This was a week of server woes.  Two servers that host many of our most important sites went offline last Friday and our IT Support people weren’t able to get them back online again until the following Tuesday.  Then just as things were getting back to normal on Tuesday an issue was spotted with another of our servers that meant it needed to be taken offline, resulting in at least 30-40 websites going down, including this one.  As I’m typing this the following Monday the server is still offline and I’ve not heard anything about a timescale for getting it back up again.

Thankfully I’m currently working for the Books and Borrowing project, which is based at servers hosted at Stirling University so my ability to work was not affected by the outages, but it’s really bad news for active research projects that require the online resources to function and it reflects really badly on both the College of Arts and the University of Glasgow as a whole.

For the Books and Borrowing project I dealt with some data correction issues for Haddington library that one of the researchers had spotted, including swapping page images around and moving borrowing records to different pages.  I also corrected the occupation errors that we’d spotted with some borrowers from Innerpeffray library using a spreadsheet that Katie sent me.  She had noticed that there were also several duplicate borrowers in the data and had noted which records needed to be amalgamated so I dealt with these as well.

My main task for the week was to update the Solr data we use for the quick search to incorporate all of the data that we will also need to use for the advanced search.  On Monday and Tuesday I spent some time reworking the Solr instance running on my laptop so as to get it ready to handle the advanced search.  This involved adding new fields for all of the types of data the advanced search needs to query.

I also figured out how to get around the stemming issue for fields like occupation.  For text fields Solr creates stemmed versions of all recognisable words in the fields, so for example ‘searching’, ‘searched’, ‘searches’ etc all have the stemmed form ‘search’.  This then allows free-text searches to find data that’s of relevance, which can be really useful.  Unfortunately when displaying search filters it’s the stemmed forms that get returned and displayed beside the checkboxes and these can be a bit confusing.  I figured out that you can create copy fields for these text fields in Solr where the text is stored as strings rather than text, and strings do not get stemmed.  The search can then use the text field and the search filters can then use the string field.  Pressing on a search filter then searches the string field, which is case sensitive, but this isn’t an issue as what’s being searched is the full text of the checkbox label (e.g. ‘Religion and Clergy’) which will always match the string form Solr stores.  This means that the search filters now say something like ‘Education’ rather than ‘educat’ and full author names now get displayed, which is great.

I also added in borrower title and ESTC as search filters as I thought these might be useful.  Plus I’ve fixed the issue of fields that hold multiple values not being sortable.  For example, a borrowing record may have multiple occupations associated with it as there may be multiple borrowers and each borrower may have several occupations.  Because of this it was not possible to sort the search results by borrower occupation.  The fix for this was to generate a further field for each that stores all of the multiple values as a single string.  For borrower occupation for sorting purposes the occupation at the bottom of the hierarchy appears first, so if a borrowing record features a borrower with occupation ‘Law -> Advocate’ the record will be sorted under ‘Advocate’ then ‘Law’.  For borrower names and author names the ordering is surname then forename.

With all of these changes in place I took a copy of the live database (also taking the opportunity to deactivate all of the test libraries in the system), regenerated the JSON files that Solr indexes and then ingested them into my updated Solr instance on my laptop.  After that I ran some tests to check all was working fine.  After that I sent the data to the IT people at Stirling (I need to get them to import the data into the Solr instance on the server) and on Wednesday morning they imported it all and thankfully everything went smoothly.

With the new data in place I updated the API and the search results page to add in the new filters (Borrower title and ESTC) and to switch the filters over to using the string versions for display so we now have full occupation and author names displayed.  I also updated the ‘Order by’ facility to allow all sorting options to work.  Unfortunately whilst doing so I spotted that I’d forgotten to add in the code to populate the book edition title single field so I’m afraid sorting by this field doesn’t work yet, but other options such as borrower occupation and author and borrower name are now working.  I updated my Solr data generation script to add in the book edition title now so next time I regenerate the data this will work.

I then started to work on implementing the advanced search.  I decided to change the way the API is referenced for the search.  Previously there was going to be one endpoint for the quick search, which would accept one search parameter, and another for the advanced search, which would accept multiple parameters.  I decided instead to amalgamate the two into one single search endpoint as in reality both search facilities will need to do the same things:  format the search options for Solr, work out the pagination, deal with ordering options and work out which filters need to be applied.

In order to amalgamate the endpoints I needed to rework the quick search facility that I had already created, and this meant breaking the quick search for a while.  Thankfully I managed to put it all back together again with the quick search working once more, but with slightly different URLs and a differently structured API call.  With this in place I began to add the advanced search data types to the API so as to construct the query that will be passed to Solr to return the advanced search results.  This basically allows specific fields in the Solr data (e.g. author names, library names, dates) to be queried rather than querying all fields, which the quick search does.  As I left things off on Friday I was in the middle of adding in the option of searching author birth and death years, but I’d run into a little difficulty when processing negative years (i.e. BC years) that I’m going to have to investigate further next week.

Also this week I made some changes to an old interactive map I’d made back in 2015 showing important places relating to Edinburgh’s enlightenment.  This is hosted on the University’s T4 system and the T4 people were keen for alt tags to be added to the image map tiles.  Thankfully I found an answer for this on Stack Overflow (https://stackoverflow.com/a/27606381) whereby attributes can be set each time a tile is loaded.  The alt tag text is an empty string so I’m uncertain whether this will actually help anyone, but it pleases the validators, anyway.  There were some other issues with the site that had been caused by the University website changing its styles since the map was published, and I fixed these too.  As of yet the changes have not been approved, even after several days, so I’m not sure what’s going on there.

My other task this week was to create an initial interface for the VARICS project website, using the logo, fonts and colour scheme that the designer had created for the project.  I spent a bit of time customising the theme to incorporate these and have emailed the PI to let her know that things are ready to add content to.  It’s possible I’ll need to make further changes to the interface, but it’s a good starting point at least.