Week Beginning 29th September 2025

I spent a lot of my time this week working with the new entry XML data structure for the Dictionaries of the Scots Language.  I received an export of all of the XML data from the DSL’s editing system on Monday and set to work updating my data processing scripts, the database and the Solr index structure, all on my laptop at this stage.  I have updated the citation extraction script so that it now extracts senses at all levels throughout the entry document.  I’ve also updated the script to deal with <cit> elements that contain multiple <cref> elements, as previously only the first was being extracted.  The number of entries in the newly processed data has gone down from 78,165 to 78,141.  The number of extracted citations has gone up from 748,947 to 752,250.

I also created a new ‘citations_geo’ table that holds information about the <geo> tags found within each citation.  This is a separate table as a citation may have more than one <geo> tag.  After extracting the data this table now contains 172,018 rows relating to 171,290 citations from 22,847 entries (all SND).  I then updated the Solr indexes (both ‘entries’ and ‘quotations’) to include new fields for the <geo> data.  These allow multiple <geo> information to be stored.  I then needed to update the scripts that process the data for ingest into Solr to incorporate these fields, using the new table mentioned above.  These new fields in Solr will be used for region searching and filtering.  If we’re only wanting to add region searches to the quotations search the updates to the ‘entries’ index are not strictly necessary, but I thought I’d add this in just in case we want to use it in future.  For example “show me all of the entries that have at least one quotation from Edinburgh and at least one quotation from Shetland”.

I’ve also begun to update the XSLT to format the new elements in the entry page.  So far all I’ve done is update the <geo> tag to add in the mini-map popups, as you can see in the following screenshot:

These use the same tooltips as we use for the sparklines etc.  I’m not going to do much more with the styling of them for now as these tooltips will be replaced by a different library when I work on the new site design anyway.  Next week I’ll continue to make the necessary updates to the XSLT and I’ll then get everything transferred from my laptop to our test server.  I may also begin implementing the region search / filter option, but I need to get some feedback from the DSL team before I can so this.

The DSL ream had also spotted an issue with the dictionary browse feature missing off the first few entries from the dictionaries.  This was a strange one as I was unable to replicate the issue, and it didn’t seem to be related to browser or operating system.  After a lot of testing I finally managed to replicate it on my iPad, and after that I was able to figure out how it was being triggered.  The issue was happening on screens that are not tall enough to display the full-size browse pane.  The reason this wasn’t being triggered on my phone is because the screen was narrow enough to make the browse appear underneath the entry, meaning there was enough space to display the full browse.  On iPads and other displays that are wide but not so tall a compact version of the browse that contains less entries is displayed.  The code that returned the limited list was chopping five items off the start and end of the full list, meaning at the start of the dictionary the first five items were never getting displayed.  This was only affecting the start of the dictionary and not the end – I’ve just checked with ‘zulu’ – and I’ve now fixed it.

Also this week I created all of the necessary pages and uploaded all necessary files for a new edition of Johann Nepomuk Hummel’s Scottish Songs for George Thomson that will be added to the ‘Editing Burns’ website.  These pages are not yet live, but I completed work on them, and met with the researcher, Marjory Rycroft, who compiled the data and undertook the research.

I also had an email conversation with Henry Ivry about a new website he wants to set up.  I’m meeting with him to discuss this further next week.  I also had to spend some time with Luca and Andrew trying to fix the problem of the Emblems websites being swamped by non-legitimate page requests.  By the end of the week we’d managed to address a lot of these, but there is still more that needs to be done to improve the site’s stability.