My big task for this week was to handle a major update to the data for the Dictionaries of the Scots Language.  Although I’ve run some test updates on my laptop in recent months, this is the first update of the online data since 2022, so it was a bit of an important step.  For now we’re only updating our test version of the website with the new data, with a further update planned for March before an actual rollout to the live site.  On Monday I received the exported data from the DSL’s editing system and ran through all of the steps required to process the data for use on the website, running everything on my laptop.  This included importing all of the entries into our online system, extracting citations and dates, generating search data and outputting the data that will be ingested into the Solr search cache.  In previous test imports I’d spotted issues with some citations not having usable dates – there were 56 the last time I processed the data.  This time, however, there were only four such issues which is not bad going at all, considering there are a total of 748,960 citations in the system.

I managed to complete the data import and generation processes on Monday, but I don’t have direct access to the DSL’s online Solr indexes and instead I need to ask the University’s IT people to handle the updates on my behalf.  This was done on Wednesday afternoon, but unfortunately I ran into some issues.  I needed to update the online database tables that we use, but the administrative interface I use was limited to files of less than 100MB.  As the DSL’s database was almost 400MB, with the largest table taking up more than 300MB, this presented a problem.  I asked our IT people to raise the limit, which they did, but I still encountered problems uploading my file.  I was working on campus when attempting this and the file upload was timing out with an Apache error.  My helpful contact in IT Services suggested I uploaded the file to the server so he could run the import command, which I did.  However, there appears to be a major problem with uploading / downloading data from the server via sFTP as I was getting upload speeds of around 22KB/s, which is unworkably slow.  It took almost 3 hours to upload my 400MB file, but after that the IT guy was able to run the command and import the updated database.

When testing everything out I realised there was another issue, namely that the online versions of the Solr indexes contained more files than the version I had running on my laptop.  I realised that while I’d created new indexes on my laptop, to make it easier for our IT people to process the updates I’d asked them to reindex the existing cores rather than creating new ones.  The script to import the updated data only edits existing files and adds new ones – it doesn’t delete existing files that are not included in the new import file and therefore any deleted entries or citations were still in the index.  Thankfully I found a way to delete all files from a Solr index and asked our IT people to run this command before importing a fresh version of the data from the import files and this solved the problem.

There were another couple of updates that needed to be made to the system following the data import, and by this time I was working on the update from my desktop PC at home.  As always, I download a fresh copy of the site before I work on the files, but I was still encountering a speed issue when downloading files.  I had thought the problem I encountered when uploading the data was caused by some issue with the WiFi connection in my office on campus, but using a wired connection on an entirely different network at home I was also encountering download speeds of around 22KB/s, which is unworkably slow – like going back to the 90s and a dial-up connection.  I put in another IT job about this and hopefully our IT people will be able to figure out what’s clogging things up, as when I uploaded files to the server on Monday I wasn’t experiencing such issues.

The first task I still had to complete was to add links from SND entries to the DSL’s phonetic description text (https://dsl.ac.uk/about-scots/the-scots-language/), which I’d handily added IDs to the paragraphs of a few years back.  The SND’s XML contained numerical references to these paragraphs in new tags so it was simply a matter of updating the XSLT to find these tags, extract the ID’s and generate links to the relevant paragraph.

The second task was to update the way the ‘about’ text for each entry is generated, specifically the statement about the entry not having had major revisions yet.  Unfortunately I ran into some issues with this task.  This task required processing a new attribute in the entry XML files and I only started looking into this after I’d completed all other processing.  I would need to update my data processing script to extract the new ‘typeC’ attribute now contained in the entry XML and update the database to store it.  This is not a major job, but I would then have to re-run all of the scripts so it might be best if I leave this until the next update in March.  Also, it looks like every entry currently has the value ‘28’ for the new ‘typeC’ attribute but for now I’m unclear when this refers to.  The ‘About’ table in the database is used to match up the ‘typeA’ and ‘typeB’ attributes to their corresponding ‘About’ statements.  For example SND ‘dreich’ (https://dsl.ac.uk/entry/snd/dreich) has typeA=”13” and typeB=”27” which in combination gives the statement ‘First published 1952 (SND Vol. III). Includes material from the 1976 and 2005 supplements.’.  However, the ‘About’ table row with id 28 is ‘First published 1965 (SND Vol. VI).’ and presumably this is not what typeC=”28” refers to.  I suspect that ‘28’ might not be the correct ID and I’ve contacted the DSL people to find out more.

Also this week I worked on the design for a new ‘Word of the week’ feature for the DSL.  I also spent some time on the Anglo-Norman Dictionary.  The editor Delphine had accidentally deleted an entry and I reinstated it for her, but in doing so I realised that this entry did not have any search data associated with it.  I’m not sure why this should be the case as deleting an entry only deactivates it in the database, it doesn’t actually delete anything.  I re-uploaded the entry via the management system and this regenerated all of the search data (e.g. forms, citations, usage tags), but the lack of search data was a little concerning as it meant that other than via the browse and ‘jump to’ options the entry would be otherwise unfindable.  I then searched the database for other live entries that are lacking search data.  I searched for entries that have no citation data and entries that have no form data and there are 103 of the former and 8 of the latter.  Delphine checked these out and thankfully the entries without citation data are valid, but the ones without forms are errors.  These could be fixed by reuploading new XML files for them using the management system, but it’s something we’ll have to keep an eye on as I’m still not sure how the situation arose.

Other tasks this week included setting up the new subdomain for the place-names of Nairnshire project, which involved setting up the content management system, the parish boundaries, importing the GB1900 data (2520 names across the five parishes), creating historical forms associated with the ‘OS 6 inch 2nd edn’ source (used for GB1900), running a script to generate altitude for all place-names via Google Maps and setting up the map interface in the front-end, including LiDAR data.

I also met with the VARICS project team this week to discuss the development of a series of new features for their website.  This was a hugely useful meeting as it helped me to understand exactly how one of the features in particular will function, which I was somewhat confused about prior to the meeting.  It all became a lot clearer and it was great to meet with the rest of the team, even if it was just on Zoom.  I also had a chat with Craig Lamont about demonstrating the Burns Correspondence map, spoke to Sofia about the changes I’d made to the Iona map last week and had a chat with fellow developer Faidon about a WordPress-based crowdsourcing project he’s beginning to plan.