Week Beginning 6th July 2026
I returned to work on Monday this week after a lovely holiday. Most of my time this week was spent working for the Dictionaries of the Scots Language, processing a new dataset exported from their editing system and integrating it into the website. The reason this process took considerably longer than usual is that the data had a major structural difference: Entry parts of speech had been moved, rationalised and restructured. Previously parts of speech appeared as elements within <meta> but also appeared embedded in the main entry where they were only tagged with HTML italic tags. There were more than 500 different combinations of parts of speech and things like homonym numbers were mixed in with them too.
The DSL editors have spent a huge amount of time working on the parts of speech to separate them out, standardise them and ensure other data such as homonym numbers are stored in a separate but related manner. The new structure also ensures that entry parts of speech are only stored once in the entry XML using a structure that makes sense semantically rather than for display only.
As the new entry XML in the data export now differed markedly from the earlier structure I then had to rewrite my data processing scripts, update the database and Solr cores, and the API and front-end to deal with the new structure. This was a lot of work. My first step was to create a new table to hold the individual POS data, consisting of a part pf speech and associated ‘hom’, ‘syn’ and ‘infl’ values. I then updated my extraction script so that the existing pos field in the entry table now gets its content from the ‘origpos’ attribute (so we continue to have a record of the original part of speech) and then to populate the new POS table to store each individual pos, including the value, hom, syn and infl data (where applicable) for each entry.
With this update in place I could then (after running a few smaller-scale tests) process every entry in the SND and DOST export files, which resulted in 35,655 parts of speech records being generated for SND entries and 49,828 being generated for DOST.
My next step was to update the entry browse order, which is used to decide in which order the entries appear in the browse pane when viewing entries. This previously used the old POS system to decide in which order entries with the same headword appeared (e.g. so that nouns appeared first). I had to update this to use the new POS system, and also ensure that the POS labels, which are used in the browse pane, the search results and the entry header.
With this update in place I then ran my scripts to process the citations and bibliographies. After that I worked on the Solr cores we use for search purposes. These consist of an ‘entry’ core used for headword and fulltext searching, and a ‘quotation’ core used for searching the individual quotations. Both of these needed to be updated to include new POS fields, which will eventually allow me to add parts of speech filters to the search facilities. I created new Solr cores for each, both featuring new POS fields, after which I would update my script that generates the data to populate the cores to ensure the POS data was included.
With the new cores successfully populated with the new data I then needed to update the API to work with the new POS fields (e.g. so that the labels returned for use in the search results and browse pane use the new POS data). After that I then needed to update the XSLT scripts that process the entry XML files for display to ensure that the new POS structure is displayed when viewing an entry.
At this point all of the updates were still running on my laptop, and with everything in place and tested the final step was to migrate everything to our test server. I don’t have direct access to the server so needed the help of someone with this access in order to complete tasks such as creating and populating the new Solr cores, and thankfully Luca agreed to help out. The process went remarkably smoothly and by the end of Thursday the update was complete, with our test server displaying the new POS fields. I also made a number of other minor updates to the display of entries via the XSLT files that had been requested and it’s now over to the DSL editors to test everything out and make sure all is working as it should.
I spent the remainder of the week dealing with emails I’d received whilst I was away. I also had a meeting with Joanna about the DOST Auld Laws project, which is nearing completion now, and had a meeting with Renu regarding the display of polygons in the map for the HiMuJe Malabar project.