Week Beginning 21st February 2022

I participated in the UCU strike action for all of last week and on Monday and Tuesday this week.  I divided the remaining three days between three projects:  the Anglo-Norman Dictionary, the Dictionaries of the Scots Language and Books and Borrowing.

For AND I continued to work on the publication of a major update of the letter S.  I had deleted all of the existing S entries and had imported all of the new data into our test instance the week before the strike, giving the editors time to check through it all and work on the new data via the content management system of the test instance.  They had noticed that some of the older entries hadn’t been deleted, and this was causing some new entries to not get displayed (as both old and new entries had the same ‘slug’ and therefore the older entry was still getting picked up when the entry’s page was loaded).  It turned out that I had forgotten that not all S entries actually have a headword beginning with ‘s’ – there are some that have brackets and square brackets.  There were 119 of these entries still left in the system and I updated my deletion scripts to remove these additional entries, ensuring that only the older versions and not the new ones were removed.  This fixed the issues with new entries not appearing.  With this task completed and the data approved by the editors we replaced the live data with the data from the test instance.

The update has involved 2,480 ‘main’ entries, containing 4,109 main senses, 1,295 subsenses, 2,627 locutions, 1,753 locution senses, 204 locution subsenses and 16,450 citations.  In addition, 4,623 ‘xref’ entries have been created or updated.  I also created a link checker which goes through every entry, pulls out all cross references from anywhere in the entry’s XML and checks to see whether each cross-referenced entry actually exists in the system.  The vast majority of links were all working fine but there were still a substantial number that were broken (around 800).  I’ve passed a list of these over to the editors who will need to manually fix the entries over time.

For the DSL I had a meeting on Thursday morning with Rhona, Ann and Pauline to discuss the major update to the DSL’s data that is going to go live soon.  This involves a new batch of data exported from their new editing system that will have a variety of significant structural changes, such as a redesigned ‘head’ section, and an overhauled method of recording dates.  We will also be migrating the live site to a new server, a new API and a new Solr instance so it’s a pretty major change.  We had been planning to have all of this completed by the end of March, but due to the strike we now think it’s best to push this back to the end of April, although we may launch earlier if I manage to get all of the updates sorted before then.  Following the meeting I made a few updates to our test instance of the system (e.g. reinstating some superscript numbers from SND that we’d previously hidden) and had a further email conversation with Ann about some ancillary pages.

For the Books and Borrowing project I downloaded a new batch of images for five more registers that had been digitised for us by the NLS.  I then processed these, uploaded them to our server and generated register and page records for each page image.  I also processed the data from the Royal High School of Edinburgh that had been sent to me in a spreadsheet.  There were records from five different registers and it took quite some time to write a script that would process all of the data, including splitting up borrower and book data, generating book items where required and linking everything together so that a borrower and a book only exist once in the system even if they are associated with many borrowing records.  Thankfully I’d done this all before for previous external datasets, but the process is always different for each dataset so there was still much in the way of reworking to be done.

I completed my scripts and ran them on a test instance of the database running on my local PC to start with.  When all was checked and looking good I ran the scripts on the live server to incorporate the new register data with the main project dataset.  After completing the task there were 19,994 borrowing records across 1,438 register pages, involving 1,932 books and 2,397 borrowers.  Some tweaking of the data may be required (e.g. I noticed there are two ‘Alexander Adam’ borrowers, which seems to have occurred because there was a space character before the forename sometimes) but on the whole it’s all looking good to me.

Next week I’ll be on strike again on Monday to Wednesday.