I spent a lot of time this week continuing to implement the new versioning system for the Books and Borrowing project.  This week I continued to tackle updating the Solr index, other cached data, and the scripts that generate this data.  This involved making updates to the structure of the underlying database as where (for example) originally one cached JSON file was stored we now need to store any number of such files, with one per version.

I managed to update the code the generates the Solr index and executed it, generating the necessary JSON files, which I was then able to successfully import into a version of the Solr index that was running on my laptop.  After running a few queries through the Solr admin interface everything seemed to have worked fine, which was a relief.  I then had to work through all of the scripts that generate other cached data, such as generating the library top ten data and lists of each author that is associated with a borrowing at a library.  There are more than ten such scripts and each needed updated (sometimes, as mentioned, with updates to the database structure too), tested, executed and then the output needed to be tested.

I ran into a bit of difficulty when generating the data for the ‘Facts and Figures’ pages, as this needs to query the Solr index, sending hundreds of thousands of queries to Solr in order to generate a static cache containing figures relating to the data.  The issue was that the connection to Solr was taking a long time to complete.  Previously passing a query to Solr via HTTP has been pretty much instantaneous, but this time each request was taking at least a second to complete, and when you’re dealing with hundreds of thousands of requests this isn’t feasible.  I wasted quite a lot of time try to figure out what was going on, and it was especially perplexing as the function I used to connect to the Solr instance hadn’t been changed since I last ran it successfully and the Solr instance’s software hadn’t been updated either.  I tried installing a newer version of Solr but it had the same problem.  I tried running a test script on the actual server and it executed instantaneously, so the issue must have been something to do with my laptop.  Eventually I asked Luca if he had any ideas and we had a useful troubleshooting session over Teams chat.

During this I decided to try changing the connection string from ‘localhost’ to the localhost IP address of ‘127.0.0.1’.   As soon as I did this the connections were instantaneous again, which was a real relief, but also rather baffling, as I didn’t know why this should make a difference.  It turns out that localhost connections now default to IPv6, but Solr and my laptop were not set up to run IPv6.  This meant that every connection to ‘localhost’ was attempting to connect to Solr via IPv6, which failed, and then was falling back to IPv4 (127.0.0.1) and working.  This was taking time and was causing the connections to run so slowly.  I will have to remember this in future.  Thankfully after replacing ‘localhost’ for the IPv4 IP address of 127.0.0.1 my scripts were much quicker and what would have taken days (or possibly weeks) to execute completed in about 15 minutes.

With all of the cache generation scripts updated, the cached data regenerated with a new structure and the database structure updated I am now ready to begin the process of updating the API and front-end to incorporate versions into everything.  This is a task I’m hoping to make a start on next week.

Also this week I reviewed a shortlist of applicants for Ophira Gamliel’s project and gave feedback on their suitability.  I also fixed an error with the Metaphor Map and the Old English Metaphor Map that was preventing counts of categories from displaying in the textual browse lists.  I also liaised with our IT people to get the OHOS domain migrated to The National Archives and responded to Craig Lamont about a new project he is putting together.

The remainder of my week (about a day) was spent on the Place-names of Iona project, for which I implemented some updates that we’d discussed at a project meeting last Friday.  This included updating the site banner to cut off the top part and make it take up less of the screen and adding in the legend descriptions from the Berwickshire Place-names project.  I also updated the glossary to ensure that all listed parts of speech have a tooltip that displays the full form.

The big task was to completely overhaul the display of the glossary.  There is now a bar of letter links that feature the letters that elements start with as buttons.  A count of the number of elements starting with each letter is also included, as I thought this might be useful.  Any letters that don’t have any elements appear but are greyed out.  The list of elements is now split by letter, each of which appears as a heading with a dividing line to make it easier to tell where elements beginning with one letter end and the next set begins.  A ‘Top’ button is also present to make it easier to get back to the top and I also replaced the placeholder text in the element glossary side-menu with descriptive text that Carole came up with for Berwickshire.  These updates make the element glossary much more usable now, as the following screenshot showing personal name elements demonstrates:

Next week I’ll apply this update to the other place-names resources, and I’ll also hopefully find some time to work on the new ‘thematic maps’ that we’re hoping to implement before the resource goes live in June.