After spending a pretty intensive few weeks working on the new interface for the Dictionaries of the Scots Language ahead of last week’s in-person meeting, this week I was able to return to other projects that I’d had to put to one side recently.  I am still unfortunately suffering from a rather bad bout of sciatica, which is now in its third week and is making it hard to work, especially in the mornings.  It has also unfortunately prevented me from travelling to the University, which meant I had to rearrange a couple of in-person meetings this week.  Despite all of this I’ve still managed to get quite a lot done this week.

I spent a lot of my time this week working on the migration of the place-names of Fife data, which I’d started to look into again last Friday.  I hadn’t had any time to work on this since September and it was good to get back into it.  I’m migrating the Fife data, which I originally extracted from a Word file into a relational database structure way back in 2016 to the same structure that I created for other place-names projects such as Berwickshire and Iona.  As the Fife data is very messy there’s much work to be done to get it ready.

This week I managed to complete work on the historical forms.  There are more than 23,000 historical forms in the data, and for Fife the sources of these forms are stored as a field in the historical form table, meaning there are more than 23,000 sources.  For the other projects an individual source is stored in a separate ‘sources’ table only once, and then is connected to all relevant historical forms via a joining table that also stores the specific reference for the form (e.g. a page number).

What I needed to do for the Fife data was to extract the unique sources, separate out the reference data (which was stored as part of the same field), insert new sources once into the ‘sources’ table and add in references to this source for each historical form where the source appears.  Thankfully there were some patterns to the source data, plus the same sources kept cropping up for many of the historical forms.  For example, the source name was italicised in more than 11,500 historical form records, which made it easy to split up the source name from the reference and process these sources.

Of the remainder there were a number of major sources, such as ‘OS 6 inch 1st edn’, found in  almost 2,200 historical form records.  I was able to process sweeps of the data that ticked off hundreds or thousands of historical forms at a time, which then left me with the awkward records that needed more manual intervention.  Even in such cases I was able to automate the process to a certain extent by first finding where the source name ended and the reference began, and then for each historical form that featured the source splitting the field at this point and storing the source and the reference.  I manage to sort out all 23,000+ historical forms this week using various methods, resulting in less than 500 unique sources being stored.  This is of course still just a first draft and there will almost certainly be some duplicates due to different spellings and such things, but the source data is now in the right format and is clean enough to be managed through the place-names CMS system, once I get round to setting it up for this project.

I also began work on the final data migration I’ll need to tackle for the Fife data:  the place-name elements.  As with the source data, these are not stored in an especially relational way.  In the more modern place-names projects each element is stored once has an associated language that is stored once.  When elements appear in a place-name they are then associated by means of a joining table that stores a reference to the element and the place-name, and information about how the element appears within the place-name, such as its position and how it is connected to a subsequent element.  In Fife this is all just stored in one table, meaning there are almost 6,500 elements.  I began cleaning this data up a bit this week, as I’d spotted some issues, such as elements being stored without a language due to the data from the original Word file not being processed successfully.  I’ll continue with this next week, if I have the time.

Also this week I met with Ophira and Renu to discuss the structure of the spreadsheet that I created to store details about the places that will appear on the map for Ophira’s HiMuJe Malaber project.  This should have been an in-person meeting but as I am still unable to leave the house much we had the meeting online instead.  We went through the spreadsheet in detail and several structural changes were proposed.  After the meeting I then spent some time making the updates and ensuring these were logged in the accompanying data dictionary.

Also this week I had a lengthy email conversation with Andrew McHugh, Luca Guariento and others about hosting the online resource for one of Garrick Allen’s projects.  This is currently hosted elsewhere but Garrick would like it to come to Glasgow.  There was a lot to discuss about this request as the resource uses technologies we don’t otherwise support at Glasgow, but by the end of the week we’d reached a decision about how and where the resource should be hosted, and Luca has agreed to oversee the process.

I also found a bit of time this week to finally swap the live Historical Thesaurus website with the new version I created earlier this year that uses a new, unified API, unlike the mess of scripts that were cobbled together over a decade or so of development that previously powered the site.  This is a major update to the resource’s back-end but includes no changes whatsoever to the front-end, so all of the work that went into it should be invisible.  But it will make it much easier to manage the resource in future as all data access now passes through one script.

I had an in-person meeting planned with Deven Parker this week to discuss her playbills project, but I was unable to travel into the University and instead we had an email conversation.  Deven is just about at the stage for me to begin creating an online resource for the data, and she shared with me the YAML files that had been generated by the AI ‘reading’ the playbill images and extracting and formatting the data.  My first task was to identify which of the playbills had been classified as ‘melodrama’ as it is these that Deven wants to initially focus on.  Deven needed some help in creating a Python script that could export a list of matching YAML files, and this was a good opportunity for me to learn a bit of Python, as this is the kind of activity I’d normally just write a PHP script for.  I managed instead to write a Python script that did was Deven needed, and we had a lengthy email conversation about the data, the project and the next steps.  I’ll hopefully be well enough to meet her in person soon to define some requirements for the resource she’d like me to build soon.