After spending a pretty intensive few weeks working on the new interface for the Dictionaries of the Scots Language ahead of last week’s in-person meeting, this week I was able to return to other projects that I’d had to put to one side recently. I am still unfortunately suffering from a rather bad bout of sciatica, which is now in its third week and is making it hard to work, especially in the mornings. It has also unfortunately prevented me from travelling to the University, which meant I had to rearrange a couple of in-person meetings this week. Despite all of this I’ve still managed to get quite a lot done this week.
I spent a lot of my time this week working on the migration of the place-names of Fife data, which I’d started to look into again last Friday. I hadn’t had any time to work on this since September and it was good to get back into it. I’m migrating the Fife data, which I originally extracted from a Word file into a relational database structure way back in 2016 to the same structure that I created for other place-names projects such as Berwickshire and Iona. As the Fife data is very messy there’s much work to be done to get it ready.
This week I managed to complete work on the historical forms. There are more than 23,000 historical forms in the data, and for Fife the sources of these forms are stored as a field in the historical form table, meaning there are more than 23,000 sources. For the other projects an individual source is stored in a separate ‘sources’ table only once, and then is connected to all relevant historical forms via a joining table that also stores the specific reference for the form (e.g. a page number).
What I needed to do for the Fife data was to extract the unique sources, separate out the reference data (which was stored as part of the same field), insert new sources once into the ‘sources’ table and add in references to this source for each historical form where the source appears. Thankfully there were some patterns to the source data, plus the same sources kept cropping up for many of the historical forms. For example, the source name was italicised in more than 11,500 historical form records, which made it easy to split up the source name from the reference and process these sources.
Of the remainder there were a number of major sources, such as ‘OS 6 inch 1st edn’, found in almost 2,200 historical form records. I was able to process sweeps of the data that ticked off hundreds or thousands of historical forms at a time, which then left me with the awkward records that needed more manual intervention. Even in such cases I was able to automate the process to a certain extent by first finding where the source name ended and the reference began, and then for each historical form that featured the source splitting the field at this point and storing the source and the reference. I manage to sort out all 23,000+ historical forms this week using various methods, resulting in less than 500 unique sources being stored. This is of course still just a first draft and there will almost certainly be some duplicates due to different spellings and such things, but the source data is now in the right format and is clean enough to be managed through the place-names CMS system, once I get round to setting it up for this project.
I also began work on the final data migration I’ll need to tackle for the Fife data: the place-name elements. As with the source data, these are not stored in an especially relational way. In the more modern place-names projects each element is stored once has an associated language that is stored once. When elements appear in a place-name they are then associated by means of a joining table that stores a reference to the element and the place-name, and information about how the element appears within the place-name, such as its position and how it is connected to a subsequent element. In Fife this is all just stored in one table, meaning there are almost 6,500 elements. I began cleaning this data up a bit this week, as I’d spotted some issues, such as elements being stored without a language due to the data from the original Word file not being processed successfully. I’ll continue with this next week, if I have the time.
Also this week I met with Ophira and Renu to discuss the structure of the spreadsheet that I created to store details about the places that will appear on the map for Ophira’s HiMuJe Malaber project. This should have been an in-person meeting but as I am still unable to leave the house much we had the meeting online instead. We went through the spreadsheet in detail and several structural changes were proposed. After the meeting I then spent some time making the updates and ensuring these were logged in the accompanying data dictionary.
Also this week I had a lengthy email conversation with Andrew McHugh, Luca Guariento and others about hosting the online resource for one of Garrick Allen’s projects. This is currently hosted elsewhere but Garrick would like it to come to Glasgow. There was a lot to discuss about this request as the resource uses technologies we don’t otherwise support at Glasgow, but by the end of the week we’d reached a decision about how and where the resource should be hosted, and Luca has agreed to oversee the process.
I also found a bit of time this week to finally swap the live Historical Thesaurus website with the new version I created earlier this year that uses a new, unified API, unlike the mess of scripts that were cobbled together over a decade or so of development that previously powered the site. This is a major update to the resource’s back-end but includes no changes whatsoever to the front-end, so all of the work that went into it should be invisible. But it will make it much easier to manage the resource in future as all data access now passes through one script.
I had an in-person meeting planned with Deven Parker this week to discuss her playbills project, but I was unable to travel into the University and instead we had an email conversation. Deven is just about at the stage for me to begin creating an online resource for the data, and she shared with me the YAML files that had been generated by the AI ‘reading’ the playbill images and extracting and formatting the data. My first task was to identify which of the playbills had been classified as ‘melodrama’ as it is these that Deven wants to initially focus on. Deven needed some help in creating a Python script that could export a list of matching YAML files, and this was a good opportunity for me to learn a bit of Python, as this is the kind of activity I’d normally just write a PHP script for. I managed instead to write a Python script that did was Deven needed, and we had a lengthy email conversation about the data, the project and the next steps. I’ll hopefully be well enough to meet her in person soon to define some requirements for the resource she’d like me to build soon.
I continued to work on the new interface for the Dictionaries of the Scots Language this week, ahead of the DSL’s face-to-face meeting on Wednesday at which I gave a demonstration of the interface. There was a lot still to do before the meeting, and I ended up working several hours over the weekend to ensure everything was ready. By Tuesday I’d managed to complete just about everything, although there are still a few formatting issues with some of the ‘history’ pages that I’ll need to tidy up at some point. With the completed interface still only running locally on my laptop the next step was to deploy it to our online test instance of the DSL site, which thankfully only took a couple of hours to complete.
I then spent most of the rest of Tuesday writing and preparing the text of a walkthrough of the new interface for Wednesday’s meeting. Of course, whilst creating this walkthrough I spotted several minor issues with the site and spent some further time addressing those too. I didn’t need to be at the meeting in Edinburgh until the afternoon so in addition to running through my demo a few times on Wednesday morning, I also added a feature to the site’s bibliography page that I’d been meaning to implement for a while. The bibliography page as it currently stands is a bit of a dead end on the site: you search for an author or title and you reach a bibliographical record for a specific work or works. There is nowhere to go from there – no pathway to take you from the record to the entries that cite the work or works. We store this information in the database already, so I figured it would be good to add the links in. I therefore updated the API to ensure a call for a bibliographical record also returned a list of all entries that cite the record, and updated the bibliography page to add in a count of these entries and links through to them. I think this works really well – for example if you search for Irvine Welsh you can now find a list of all SND entries that feature a quote from one of his books. The only downside is that some bibliographical records (especially for DOST) appear in thousands of dictionary entries, meaning thousands of links appear on the bibliography page. I’ll need to think about how to handle this, as currently all links to entries just appear as in-line buttons. We actually store the quoted text in the database so we could also display the specific quotes in the bibliography page if we want. I also still need to update things so that the links through to the entries lead directly to the first matching quote.
My demonstration of the new interface on Wednesday went very well. Everyone was very pleased with how the new site looks and functions and were also very pleased with the new bibliography page. We had a good discussion session about further possible updates, and it’s now over to the DSL team to use the new interface over the next few weeks (or possibly months) and then send on any required updates to me. I still can’t really share any screenshots of the new site at this stage, and it’s likely to be several months before it goes live, assuming the DSL team want to go live with it.
I’ve been suffering from what is most likely sciatica over the past couple of weeks, which has made it quite difficult to work. I’ve generally been unable to leave the house much, although the pain eases off as the day goes on and I’m able to move more freely in the afternoons. I’ve been unable to come into the University during this time, which has meant I’ve had to join some meetings online instead of in-person, and I also had to miss the Arts and Humanities developers coffee and catch-up that we’d scheduled for Tuesday this week. I was able to make it across to Edinburgh on Wednesday as I didn’t have to leave the house until lunchtime and the pain wasn’t as bad by then. Unfortunately the journey through exacerbated things and I was in quite a lot of pain the next day. It was impossible for me to sit at my desk for more than a few minutes at a time and I’m afraid I had to take the day off sick.
I was still in a lot of pain on Friday but I managed to work again, and I spent the day catching up with emails and investigating the Hansard data that I last worked with some seven years ago, as Marc and Fraser have a student who wants to work with it. I also spent a bit of time fixing an old Robert Burns resource that had stopped working. The Jame Currie site (https://jamescurrie.gla.ac.uk/) is not one of mine, but had apparently not worked properly since a server update. I fixed a number of issues with the code and got it working again. I’ll need to see about fully overhauling the site at some point as it contains some very useful research data but has a rather ancient interface. I also found some time to return to the migration of the Place-names of Fife data to the new place-names system, working through the historical form sources. There’s still a lot of work that needs to be done on the data before it can be fully migrated, but I’m slowly making progress.
I spent pretty much the entirety of this week continuing to develop the new interface for the Dictionaries of the Scots Language website, applying the Bootstrap-based mock-up I’d created many months ago to an instance of the actual DSL website running on my laptop. I can’t really go into too much detail about the new interface or provide any screenshots at this stage, but it’s been a pretty intensive process as every aspect of the old interface needs to be changed and various parts of it need to be integrated with WordPress, for example making widgets and ensuring the new layout works with different templates.
I managed to complete the bulk of the work this week (although this did include working several hours over the weekend too), in preparation for next week’s face-to-face DSL team meeting. This included the search results pages, the advanced search page, the dictionary entry page and the bibliography page. This may not seem like a very long list, but there was a huge amount of work to do on each of these pages, such as implementing the site panel for the entry page that features the dictionary browser, the search results browser and a new ‘entry log’ that keeps a record of entries the user has looked at during their session. I reckon the new interface is looks really good, and is a massive improvement on the live site, although there will inevitably be many further changes to be made before anything goes live.
I still need to complete the new top-level ‘About’ page, which acts as a large menu page, plus ensure that all regular WordPress pages work with the new interface and include the quick search. I’m hoping to finish these things off and then apply the interface to our online test instance of the site ahead of Wednesday’s meeting next week.
Also this week I spent a bit more time preparing for a talk about Speak For Yersel and Jennifer Smith and I were scheduled to give at the University of Edinburgh the week after next. However, later in the week we heard from the organisers that the University will be on strike when our talk is scheduled and we therefore reached a decision to cancel. It’s possible that we’ll be able to reschedule, as we are not directly involved in the strike action, but we’ll just need to see.
Also this week I created an initial version of a website for Henry Ivry’s project and contacted researcher Jenny Buckley with some further information about the processing of historical newspapers that might be of use for her project. I also made a small update to the Speech Star resource and had an email conversation with Eleanor Lawson about access restrictions for the resources data.
I participated in an online meeting regarding sharing the SCOSYA data with the Mozilla Foundation this week, and I also had a meeting with Pauline Mackay and Cleo O’Callaghan Yeoman to discuss a new phase for the Interactive Map of Burns Suppers. I subsequently spent a bit of time reviewing some materials for the site. Finally, I exported some data from the Historical Thesaurus that we’re going to share with another project.
I spent a lot of my time this week creating a new WordPress theme and complete website interface for the Dictionaries of the Scots Language, based on the Bootstrap mock-up I worked on several months ago. I had demonstrated this mock-up at the DSL’s last face-to-face meeting in February and the plan was for me to develop this into a fully functioning interface before the next meeting, which is taking place in a couple of weeks. However, since the meeting I was tasked with to major updates to the DSL: implementing the new DSL entry XML structure and developing the interactive maps of regions and dialect areas. The time I had intended to spend on the new interface was instead devoted to these two new tasks, meaning it is now unlikely that I will have the new interface completed before the next team meeting.
However, having now completed work on the to unanticipated major tasks, I am now at last able to focus on the new interface, which is what I did this week. I can’t really talk too much about the new interface or show any screenshots at this stage, but this week I set up a version of the site running on my laptop and created a new WordPress theme, initially replicating the current version. Creating the new interface from my mock-up was slightly tricky to begin with as so much needed to be done before anything could be tested. I needed to integrate Bootstrap and the JavaScript and CSS I’d created for the mock-up, replace the WordPress-powered site menu with a new one that uses Bootstrap, which requires incorporating a new class into WordPress to fully implement this (https://github.com/wp-bootstrap/wp-bootstrap-navwalker). I needed to completely replace the header and footer templates, plus the overall page template, I needed to replace the WordPress widgets I’d created for the old theme with new ones for the homepage info boxes, the quick search bar and the introductory text area, and I needed to create a second homepage for ‘Older Scots’ that could also use WordPress widgets. This, as you can maybe imagine, took a lot of time, but I managed to complete it all by the end of the week. There is still a huge amount to do, though, such as getting the new theme to actually display any dictionary content, such as the entries and the search results. I’ll continue with this next week.
Also this week I spent some time preparing some materials for the Speak For Yersel talk I’m giving with Jennifer later in November. We had intended to meet to go over things this week, but Jennifer needed to push this back to next week. I also met with Ophira Gamliel to discuss some ideas for a series of online TEI/XML/Oxygen workshops she’s running with colleagues in India, and to discuss support for digital editions in general across the College. I had another meeting with Jenny Buckley, a researcher in English Literature who is hoping to use AI tools to extract and analyse text from late 18th century newspapers, and I met with Alan Riach in Scottish Literature to help him extract some sound files that he needs for a conference.
In addition, I had an email conversation with Pauline Mackay and Cleo O’Callaghan Yeoman about the interactive map of Burns Suppers, which Cleo is going to be reworking. I managed to extract the data for the map and its collection of images and send this on to them, and I’ll be meeting with them next week to discuss the redevelopment, which I will be involved with. I also wrote a section about data management and ethics for a proposal that Garrick Allen is putting together, spoke to Mícheál Ó Mainnín at Queens University Belfast about the place-names of Armagh website I’m putting together for him and had an email conversation with Henry Ivry about the website I’m working on for him.
My final task of the week was to create a spreadsheet template and accompanying data dictionary for the data for the interactive map for Ophira’s HiMuJe Malabar project. Project RA Renu Abraham will be using this template to record the data for all of the places relating to the project’s digital editions that we want to visualise on a map, and I will then create the map. It took the best part of a morning to define the structure and describe all of the columns, but hopefully this will be of use. It’s been a rather busy week, with lots more to continue with next week too.