I continued to work on the new interface for the Dictionaries of the Scots Language this week, ahead of the DSL’s face-to-face meeting on Wednesday at which I gave a demonstration of the interface. There was a lot still to do before the meeting, and I ended up working several hours over the weekend to ensure everything was ready. By Tuesday I’d managed to complete just about everything, although there are still a few formatting issues with some of the ‘history’ pages that I’ll need to tidy up at some point. With the completed interface still only running locally on my laptop the next step was to deploy it to our online test instance of the DSL site, which thankfully only took a couple of hours to complete.
I then spent most of the rest of Tuesday writing and preparing the text of a walkthrough of the new interface for Wednesday’s meeting. Of course, whilst creating this walkthrough I spotted several minor issues with the site and spent some further time addressing those too. I didn’t need to be at the meeting in Edinburgh until the afternoon so in addition to running through my demo a few times on Wednesday morning, I also added a feature to the site’s bibliography page that I’d been meaning to implement for a while. The bibliography page as it currently stands is a bit of a dead end on the site: you search for an author or title and you reach a bibliographical record for a specific work or works. There is nowhere to go from there – no pathway to take you from the record to the entries that cite the work or works. We store this information in the database already, so I figured it would be good to add the links in. I therefore updated the API to ensure a call for a bibliographical record also returned a list of all entries that cite the record, and updated the bibliography page to add in a count of these entries and links through to them. I think this works really well – for example if you search for Irvine Welsh you can now find a list of all SND entries that feature a quote from one of his books. The only downside is that some bibliographical records (especially for DOST) appear in thousands of dictionary entries, meaning thousands of links appear on the bibliography page. I’ll need to think about how to handle this, as currently all links to entries just appear as in-line buttons. We actually store the quoted text in the database so we could also display the specific quotes in the bibliography page if we want. I also still need to update things so that the links through to the entries lead directly to the first matching quote.
My demonstration of the new interface on Wednesday went very well. Everyone was very pleased with how the new site looks and functions and were also very pleased with the new bibliography page. We had a good discussion session about further possible updates, and it’s now over to the DSL team to use the new interface over the next few weeks (or possibly months) and then send on any required updates to me. I still can’t really share any screenshots of the new site at this stage, and it’s likely to be several months before it goes live, assuming the DSL team want to go live with it.
I’ve been suffering from what is most likely sciatica over the past couple of weeks, which has made it quite difficult to work. I’ve generally been unable to leave the house much, although the pain eases off as the day goes on and I’m able to move more freely in the afternoons. I’ve been unable to come into the University during this time, which has meant I’ve had to join some meetings online instead of in-person, and I also had to miss the Arts and Humanities developers coffee and catch-up that we’d scheduled for Tuesday this week. I was able to make it across to Edinburgh on Wednesday as I didn’t have to leave the house until lunchtime and the pain wasn’t as bad by then. Unfortunately the journey through exacerbated things and I was in quite a lot of pain the next day. It was impossible for me to sit at my desk for more than a few minutes at a time and I’m afraid I had to take the day off sick.
I was still in a lot of pain on Friday but I managed to work again, and I spent the day catching up with emails and investigating the Hansard data that I last worked with some seven years ago, as Marc and Fraser have a student who wants to work with it. I also spent a bit of time fixing an old Robert Burns resource that had stopped working. The Jame Currie site (https://jamescurrie.gla.ac.uk/) is not one of mine, but had apparently not worked properly since a server update. I fixed a number of issues with the code and got it working again. I’ll need to see about fully overhauling the site at some point as it contains some very useful research data but has a rather ancient interface. I also found some time to return to the migration of the Place-names of Fife data to the new place-names system, working through the historical form sources. There’s still a lot of work that needs to be done on the data before it can be fully migrated, but I’m slowly making progress.
I spent pretty much the entirety of this week continuing to develop the new interface for the Dictionaries of the Scots Language website, applying the Bootstrap-based mock-up I’d created many months ago to an instance of the actual DSL website running on my laptop. I can’t really go into too much detail about the new interface or provide any screenshots at this stage, but it’s been a pretty intensive process as every aspect of the old interface needs to be changed and various parts of it need to be integrated with WordPress, for example making widgets and ensuring the new layout works with different templates.
I managed to complete the bulk of the work this week (although this did include working several hours over the weekend too), in preparation for next week’s face-to-face DSL team meeting. This included the search results pages, the advanced search page, the dictionary entry page and the bibliography page. This may not seem like a very long list, but there was a huge amount of work to do on each of these pages, such as implementing the site panel for the entry page that features the dictionary browser, the search results browser and a new ‘entry log’ that keeps a record of entries the user has looked at during their session. I reckon the new interface is looks really good, and is a massive improvement on the live site, although there will inevitably be many further changes to be made before anything goes live.
I still need to complete the new top-level ‘About’ page, which acts as a large menu page, plus ensure that all regular WordPress pages work with the new interface and include the quick search. I’m hoping to finish these things off and then apply the interface to our online test instance of the site ahead of Wednesday’s meeting next week.
Also this week I spent a bit more time preparing for a talk about Speak For Yersel and Jennifer Smith and I were scheduled to give at the University of Edinburgh the week after next. However, later in the week we heard from the organisers that the University will be on strike when our talk is scheduled and we therefore reached a decision to cancel. It’s possible that we’ll be able to reschedule, as we are not directly involved in the strike action, but we’ll just need to see.
Also this week I created an initial version of a website for Henry Ivry’s project and contacted researcher Jenny Buckley with some further information about the processing of historical newspapers that might be of use for her project. I also made a small update to the Speech Star resource and had an email conversation with Eleanor Lawson about access restrictions for the resources data.
I participated in an online meeting regarding sharing the SCOSYA data with the Mozilla Foundation this week, and I also had a meeting with Pauline Mackay and Cleo O’Callaghan Yeoman to discuss a new phase for the Interactive Map of Burns Suppers. I subsequently spent a bit of time reviewing some materials for the site. Finally, I exported some data from the Historical Thesaurus that we’re going to share with another project.
I spent a lot of my time this week creating a new WordPress theme and complete website interface for the Dictionaries of the Scots Language, based on the Bootstrap mock-up I worked on several months ago. I had demonstrated this mock-up at the DSL’s last face-to-face meeting in February and the plan was for me to develop this into a fully functioning interface before the next meeting, which is taking place in a couple of weeks. However, since the meeting I was tasked with to major updates to the DSL: implementing the new DSL entry XML structure and developing the interactive maps of regions and dialect areas. The time I had intended to spend on the new interface was instead devoted to these two new tasks, meaning it is now unlikely that I will have the new interface completed before the next team meeting.
However, having now completed work on the to unanticipated major tasks, I am now at last able to focus on the new interface, which is what I did this week. I can’t really talk too much about the new interface or show any screenshots at this stage, but this week I set up a version of the site running on my laptop and created a new WordPress theme, initially replicating the current version. Creating the new interface from my mock-up was slightly tricky to begin with as so much needed to be done before anything could be tested. I needed to integrate Bootstrap and the JavaScript and CSS I’d created for the mock-up, replace the WordPress-powered site menu with a new one that uses Bootstrap, which requires incorporating a new class into WordPress to fully implement this (https://github.com/wp-bootstrap/wp-bootstrap-navwalker). I needed to completely replace the header and footer templates, plus the overall page template, I needed to replace the WordPress widgets I’d created for the old theme with new ones for the homepage info boxes, the quick search bar and the introductory text area, and I needed to create a second homepage for ‘Older Scots’ that could also use WordPress widgets. This, as you can maybe imagine, took a lot of time, but I managed to complete it all by the end of the week. There is still a huge amount to do, though, such as getting the new theme to actually display any dictionary content, such as the entries and the search results. I’ll continue with this next week.
Also this week I spent some time preparing some materials for the Speak For Yersel talk I’m giving with Jennifer later in November. We had intended to meet to go over things this week, but Jennifer needed to push this back to next week. I also met with Ophira Gamliel to discuss some ideas for a series of online TEI/XML/Oxygen workshops she’s running with colleagues in India, and to discuss support for digital editions in general across the College. I had another meeting with Jenny Buckley, a researcher in English Literature who is hoping to use AI tools to extract and analyse text from late 18th century newspapers, and I met with Alan Riach in Scottish Literature to help him extract some sound files that he needs for a conference.
In addition, I had an email conversation with Pauline Mackay and Cleo O’Callaghan Yeoman about the interactive map of Burns Suppers, which Cleo is going to be reworking. I managed to extract the data for the map and its collection of images and send this on to them, and I’ll be meeting with them next week to discuss the redevelopment, which I will be involved with. I also wrote a section about data management and ethics for a proposal that Garrick Allen is putting together, spoke to Mícheál Ó Mainnín at Queens University Belfast about the place-names of Armagh website I’m putting together for him and had an email conversation with Henry Ivry about the website I’m working on for him.
My final task of the week was to create a spreadsheet template and accompanying data dictionary for the data for the interactive map for Ophira’s HiMuJe Malabar project. Project RA Renu Abraham will be using this template to record the data for all of the places relating to the project’s digital editions that we want to visualise on a map, and I will then create the map. It took the best part of a morning to define the structure and describe all of the columns, but hopefully this will be of use. It’s been a rather busy week, with lots more to continue with next week too.
I spent a lot of my time this week working with the new entry XML data structure for the Dictionaries of the Scots Language. I received an export of all of the XML data from the DSL’s editing system on Monday and set to work updating my data processing scripts, the database and the Solr index structure, all on my laptop at this stage. I have updated the citation extraction script so that it now extracts senses at all levels throughout the entry document. I’ve also updated the script to deal with <cit> elements that contain multiple <cref> elements, as previously only the first was being extracted. The number of entries in the newly processed data has gone down from 78,165 to 78,141. The number of extracted citations has gone up from 748,947 to 752,250.
I also created a new ‘citations_geo’ table that holds information about the <geo> tags found within each citation. This is a separate table as a citation may have more than one <geo> tag. After extracting the data this table now contains 172,018 rows relating to 171,290 citations from 22,847 entries (all SND). I then updated the Solr indexes (both ‘entries’ and ‘quotations’) to include new fields for the <geo> data. These allow multiple <geo> information to be stored. I then needed to update the scripts that process the data for ingest into Solr to incorporate these fields, using the new table mentioned above. These new fields in Solr will be used for region searching and filtering. If we’re only wanting to add region searches to the quotations search the updates to the ‘entries’ index are not strictly necessary, but I thought I’d add this in just in case we want to use it in future. For example “show me all of the entries that have at least one quotation from Edinburgh and at least one quotation from Shetland”.
I’ve also begun to update the XSLT to format the new elements in the entry page. So far all I’ve done is update the <geo> tag to add in the mini-map popups, as you can see in the following screenshot:
These use the same tooltips as we use for the sparklines etc. I’m not going to do much more with the styling of them for now as these tooltips will be replaced by a different library when I work on the new site design anyway. Next week I’ll continue to make the necessary updates to the XSLT and I’ll then get everything transferred from my laptop to our test server. I may also begin implementing the region search / filter option, but I need to get some feedback from the DSL team before I can so this.
The DSL ream had also spotted an issue with the dictionary browse feature missing off the first few entries from the dictionaries. This was a strange one as I was unable to replicate the issue, and it didn’t seem to be related to browser or operating system. After a lot of testing I finally managed to replicate it on my iPad, and after that I was able to figure out how it was being triggered. The issue was happening on screens that are not tall enough to display the full-size browse pane. The reason this wasn’t being triggered on my phone is because the screen was narrow enough to make the browse appear underneath the entry, meaning there was enough space to display the full browse. On iPads and other displays that are wide but not so tall a compact version of the browse that contains less entries is displayed. The code that returned the limited list was chopping five items off the start and end of the full list, meaning at the start of the dictionary the first five items were never getting displayed. This was only affecting the start of the dictionary and not the end – I’ve just checked with ‘zulu’ – and I’ve now fixed it.
Also this week I created all of the necessary pages and uploaded all necessary files for a new edition of Johann Nepomuk Hummel’s Scottish Songs for George Thomson that will be added to the ‘Editing Burns’ website. These pages are not yet live, but I completed work on them, and met with the researcher, Marjory Rycroft, who compiled the data and undertook the research.
I also had an email conversation with Henry Ivry about a new website he wants to set up. I’m meeting with him to discuss this further next week. I also had to spend some time with Luca and Andrew trying to fix the problem of the Emblems websites being swamped by non-legitimate page requests. By the end of the week we’d managed to address a lot of these, but there is still more that needs to be done to improve the site’s stability.
I had my PDR this week, which meant spending some time preparing and then attending the session, which was all very positive. Other than this I spent quite a lot of time this week working for the Anglo-Norman Dictionary, mostly working on scripts to rationalise the parts of speech used across the dictionary’s entries. The largest aspect of this was writing a script that would go through all of the live entries in the database and find all occurrences of a particular POS (e.g. ‘s._pl.’) and replace these with a standardised form (e.g. ‘s.pl.’). This involved going through the entry XML files to make changes not only to the entry-level <pos> tag, but also to look at the <senseInfo> tags for every sense and subsense, both main and locutions, to replace the ’pos’ attributes found there. The script also then needed to update the ‘pos’ table in the database (which hold a list of all parts of speech associated with each entry) to delete the old forms and add in the new forms. It was clearly important to fully test this script, and make backups of the data prior to running it, as it would need to run through every live dictionary entry, making updates to the XML where necessary. I tested everything out on an instance of the database running on my local PC, and once I was absolutely sure the script did what was required (and nothing that wasn’t required) I ran it on the live database, making changes to numerous parts of speech and updating at least a couple of thousand of entries.
I also had to update the list of parts of speech in the dicionary’s DTD, adding in a few new options that had been supplied to me. Unfortunately some of these had spaces in them and these are invalid characters in list options in a DTD. I hadn’t spotted this so making the update unfortunately stopped the dictionary entries displaying on the website, but thankfully once we did notice this it was a very quick issue to fix. I also ran a few more iterations of the POS updating script to target other parts of speech that had spaces in them that had managed to find their way into entries over the years.
I also had a lengthy email conversation with the AND editor Geert about how the new POS search facilities would function, as these steps to rationalise the parts of speech are leading up to the implementation of new search options. A POS selector will be added to the headword search, but we’ve still not fully decided which other searches should also incorporate such a selector. I had thought possibly ‘citations’ and ‘English translations’ might benefit from a POS option, but Geert didn’t seem to think so. He did suggest the ‘label’ and ‘language’ searches should possibly be updated, though.
These searches will not be able to just use the ‘entry’ level POS list that we already have – instead we’d only want to target specific citations where the <senseInfo> POS matches, otherwise entries that have multiple POS elements may bring back results that don’t actually match. For example
This will require updating the database to store POS for each citation and translation, regenerating all of the citation and translation data for every live entry to populate the new fields and updating the DMS and batch ‘upload entries’ scripts to ensure this additional data is extracted and stored. For example, a label search for ‘alchem.’ brings back https://anglo-norman.net/entry/arder where the label appears in senses that have ‘p.pr. as a.’ and ‘p.p. as a.’ as their POS. If we add a POS filter that only works at entry level then someone selecting ‘v.trans.’ and ‘alchem.’ Would still find this entry, even though the label isn’t found in the ‘v.trans.’ section. This is going to need some further work. For now I’m waiting for Geert to send me a list of the parts of speech he would like users to be able to select through the search before I take this further.
Also this week I responded to a few requests from the DSL and also Eleanor Lawson regarding a new proposal she’s putting together. I also worked with Andrew McHugh of RCaaS to migrate the old Emblems sites to a new server, although we’re encountering what looks to be a DDoS attack on this site. I can’t really go into too much detail about it here, but I spent quite some time this week with both Andrew and Luca investigating the situation and looking for solutions. As of yet the attack is still ongoing and we’re still trying to find a solution.
In addition to the above I also spent a bit of time processing some data for a new edition of Burns music that will be added to the Robert Burns website. I haven’t finished working with this data yet, as I had some questions for the researchers, but hopefully the information will be available in the next week.
Finally this week I spent a few more hours processing some of the messy data for the Place-names of Fife project. This mostly involved fixing errors in the historical forms relating to sources, and I wrote a few scripts to fix several patterns of error that I encountered. The Fife data was originally in a Word file and several years ago I managed to convert this into structured data, but some edge cases in the almost 30,000 historical forms were not successfully processed and it’s these I’m currently trying to tidy up. I think I’m just about there now, and on Friday I migrated the historical forms to a new table that will eventually be used for a new interface.
This week I completed work on a first version of the regions and dialect areas map for the Dictionaries of the Scots Language. Last week I’d begun work on the final aspect of the map: ensuring that whatever the user is viewing on the map is reflected in the hash to enable specific views of the map to be bookmarked, shared and cited. I’d begun updating the scripts to process details held in the hash and I completed this by adding the areas selected in from the info pane this week. This was slightly more complicated than I’d imagined as I needed to consider how to handle areas that are selected by default (e.g. all regions selected when the user clicks on the map) in addition to areas specifically turned on or off by the user from the checkboxes in the info pane. I decided to ensure that the default selected areas were also present in the hash to keep things simple.
With the hash processing in place I then needed to update the code to actually add the necessary information to the hash as the user interacts with the map. This was a fairly big job, involving updating much of the code, but I got there in the end. After a fair amount of testing and tweaking I was then able to send the URL for the map to the DSL people for them to test out and give feedback on – I’ll just need to see what changes may be required and also when we might go live with the map – probably not for a while yet I’d imagine.
Also for the DSL this week I had an online meeting to discuss the new entry XML structure. We went through the list of questions I’d come up with whilst working through the new XML examples a few weeks ago and had a good discussion about what would be included in the update and when I’ll start work on this. I should receive a test export of the dataset in the next week or so and once I have this I’ll be able to start implementing the necessary changes to the systems. I also investigated an issue with email lists for the DSL that Rhona contacted me about and fixed a minor layout issue with the live site.
Another sizeable task that I was able to tick off this week was finally launching the new Books and Borrowing versioning system. I’d encountered a strange issue with the online database last week that had prevented me from continuing with the rollout, but thankfully my contact in Stirling’s IT services was back at work on Tuesday and managed to identify the issue. The situation is that the online database’s data is replicated across three nodes. But this only works with database tables created in the ‘InnoDB’ format. I’d created the new tables on my laptop, which had defaulted to the ‘MyISAM’ format, something I’d not spotted. When I updated the online database these new tables were then stored in the ‘MyISAM’ format and were therefore not getting replicated. I wasn’t aware of this distinction between database tables and the implications this had for how the online data was stored – these formats refer to how the data is stored on disk and there is no difference when you view or query the tables and data through the database.
Thankfully once the issue had been identified I was then able to re-create the tables in the ‘InnoDB’ format and regenerate the data, after which all data magically appeared in both the PHPMyAdmin interface and through my scripts. With this in place I then completed the rollout of the new versioning system and the new ‘v2’ version of the API, which as far as I can tell has all been successful. As of yet there’s not really much different as we still only have version 1.0 of the data available. What you will see in the footer of every page is some text about the database version and a link to the ‘versions and changes’ page (https://borrowing.stir.ac.uk/versions-and-changes/). Also, when you perform a search and choose to cite the page the data version now appears in the citation and also the URL (so that when we do publish new versions the citation remains accurate). E.g.
“Quick search for ‘rome’. 2025. In Books and Borrowing: An Analysis of Scottish Borrowers’ Registers, 1750-1830, data version 1.0. University of Stirling. Retrieved 16 September 2025, from https://borrowing.stir.ac.uk/search/version-1.0/p-1/0/0/quick/rome”.
The URL with ‘version-1.0’ in it works in exactly the same way as the general URL for now. When we release ‘version-1.1’ the URL above will load the version 1 data with a warning that the user isn’t looking at the most up to date version. The new API is now available here: https://borrowing.stir.ac.uk/api/v2/ and I’ve updated the ‘About the database’ and ‘FAQs’ pages to point to this. I also sorted the encoded apostrophe issue and book works now appear in the filter options and ESTC has been removed. I spotted an issue with book works that had ampersands in them breaking the site when selected as a filter but I’ve managed to fix this, for example:
So after many months of work that’s the new versioning system complete. Hopefully when we come to launch version 1.1 the process will be a little more straightforward!
Also this week I had a discussion with Geert of the Anglo-Norman Dictionary about processing the data in preparation for adding in a new part of speech option to the searches, and I also fixed an issue with the management system that was preventing certain cross-reference entries from being uploaded.
I also gave some feedback to Matthew Creasy about a proposal he’s submitting soon and had a discussion with Jennifer Smith about sharing the SCOSYA data with the Mozilla Foundation. I also made a bit of progress with the arrangements for the new Place-names of Armagh project.
On Friday I had a bit of spare time that I devoted to the migration of the Place-names of Fife project to our standardised system. I realised that many of the historical forms had not been successfully processed by the script that had extracted data from the original Word files, resulting in several hundred forms that had no data in the historical form name field and everything in the ‘source’ field. I sorted these by a combination of batch processing and manual updating, and also sorted an issue with notes not appearing in the correct field either. I think I’m just about ready to migrate the historical forms now, although much work will still need to be done to extract sources.
I was intending to focus on setting up the Books and Borrowing versioning system on the server this week. I’d spent quite a long time implementing the versioning system earlier in the summer, and this required massive changes to every aspect of the B&B system: the database, the Solr index, the API, the CMS and the front-end. I’d completed work on this on my local PC just before my summer holidays and as the team were using the website quite heavily at this time we decided to hold off with the rollout until after the summer. This of course meant that I had to spend some time to familiarise myself with all of the updates, as it was mid-June when I last dealt with all of this. Unfortunately, when I came to log into Stirling University’s VPN I was notified that my account had been blocked. The B&B server is hosted at Stirling and I can’t access it without logging into the VPN so I was unable to proceed with the rollout. I remembered from the last time this happened that I can’t simply contact Stirling’s IT people and ask for my account to be unblocked – this has to be approved by a head of subject or something like that. Project PI Katie began this process on Monday but unfortunately I had still not been given access by the end of the week.
However, I did continue to work on some of the other outstanding B&B issues on my local PC. I’d noticed that the version number in the footer of the updated system wasn’t getting pulled in when the site is loaded into a browser for the first time and the first page to be loaded is a standard WordPress page (e.g. the site’s homepage). This was because the version number wasn’t getting stored as a session variable until after the page had loaded and I therefore updated the code to add a check for this.
I also investigated the issue of authors that have the same name not being differentiated in the author filter option in the search results. For example, there are two John Lelands – John Leland (Male, born 1691, died 1766) and John Leland (Male, born c.1506, died 1552) but the filter option only lists one John Leland so it’s not possible to focus on one or the other. This is because the filter uses the Solr field ‘afullnames’ to construct the list, and this simply joins an author’s forename and surname together to make the field. I’ll need to update the script that generates the Solr data to deal with this. The simplest way would be to just add dates, or possibly just birth dates for every author, but this would clutter the filter up unnecessarily. An alternative would be for the script to only add such dates where it finds multiple authors with the same name, and in such cases it would then add dates. So for example ‘John Leland, b.1691’ and ‘John Leland, b.c.1506’. I’ll need to do a full check to be sure this will cover all authors – I guess there may be some that don’t have birth years.
Whilst investigating this I ran a query to find all active authors in the system that have the same name as one (or more) other authors and there are 252, which seemed quite high to me and suggests there may be some unnecessary duplicates in the system. I exported these authors to a spreadsheet and one of the project team is going to look into this. I can’t imagine there are two different authors called ‘Cornelius a Lapide’, for example!
Project Co-I Matt had also spotted an issue with apostrophes in the filter options. The apostrophe in the author ‘Pierre Mathurin de L’Ecluse des Loges’ was causing the system to break when the name was selected in the filter options on the search results page. However, other apostrophes work perfectly. If you look at the search results for all authors with the forename ‘Pierre’ (https://borrowing.stir.ac.uk/search/0/0/0/simple/afname|pierre) the “de l’Ecluse” one is (at the time of writing) broken, but lower down we have “Pierre d’ Hancarville” and this works. Both names appear to have ‘straight’ rather than ‘curly’ apostrophes, but looking at the data, our broken Pierre’s apostrophe has actually been encoded and is stored in the system as “'” and it is this that’s breaking things. I’m not sure how this encoded form got into the system – possibly by pasting into the CMS from another website . I checked the database and there are four author surnames that have this encoding, plus one forename. I’ll fix this when I regenerate the data as part of the new version update. I also checked the other data and there are no borrowers with this encoding. However, there are 36 book works and 63 book editions that have this encoding so I’ll fix them too.
This issue got me thinking about apostrophes in the site. ‘Straight’ and ‘curly’ apostrophes are different characters and both appear in the data (see for example “Pierre Joseph d’ Orléans” in the list of Pierres, which has the ‘curly’ form). This doesn’t matter for browsing and filtering, but if the user enters an apostrophe into a search box they will only find results for the apostrophe form they entered, which will mostly be ‘straight’, but I believe Apple can make apostrophes users enter into boxes ‘curly’ on their devices. And of course there are two different ‘curly’ apostrophes – opening and closing. This is not a major issue as I can’t imagine many people will perform a search that includes an apostrophe, but I raised it with the team and we agreed that all apostrophes (and double quotes) in the data should be changed to their ‘straight’ forms, which I’ll also do once I regenerate the data.
Without having access to the server I didn’t want to continue with any further B&B work as I didn’t want to risk making updates locally and then losing track of them before I could replicate them online. I also need the current data from the server and I can’t access this yet either. I therefore moved onto other projects instead.
On Tuesday I had a meeting with Garrick Allen to discuss a new AHRC proposal he’s putting together. I can’t say much about it here, but it was a great meeting and I’ll be involved with the project if it receives funding. I spent a bit of time after the meeting writing up my notes and researching some related matters. I also had an email chat with Craig Lamont about a proposal he’s putting together that I will also be involved with.
Also this week I spent some time making further updates to the Robert Fergusson online exhibition, adding in all missing images, captions and text. The online exhibition is now publicly available and can be viewed here: https://robert-fergusson.glasgow.ac.uk/online-exhibition/.
I also provided some costings and wrote a detailed description of the place-names database tables and columns for Mícheál Ó Mainnín at QUB, who I’m hoping to work on the place-names of Armagh project with. For the DSL I had an email conversation about the new part of speech search results filtering option that I will be developing, I responded to a query about online map development, and I fixed the old Emblems websites that had stopped working due to a server upgrade.
I spent the remainder of the week returning to work on the Anglo-Norman Dictionary. The editor Geert informed me that a search for multiple words was giving a ‘403 Forbidden’ error. The AND’s various search systems don’t actually work with phrases and are all based on individual words, so a search with spaces is never going to yield meaningful results, but such a scary message is not idea. I managed to fix the issue by updating the .htaccess file to process spaces. I also spent some time investigating a strange issue Geert had experienced with cross-references, but despite my best efforts I was unable to replicate it, and neither was Geert. We’re just going to have to keep a look out for this and figure out what circumstance let to it happening.
I also updated the ‘view entry’ page in the content management system to add in buttons to download the cross-referenced entries as a ZIP file. This works for the ‘from’, ‘to’ and ‘view entries that include a cross reference in their XML’ lists, finding the matching entries, accessing their XML files, adding them to a ZIP archive and then presenting it as a download to the user. It all works very well.
I spent a lot of this week working for the Dictionaries of the Scots Language, planning and preparing for some major upcoming changes. Last week I began writing a specification document for a new region / dialect areas interactive map and I completed this on Monday, sending it out to members of the team for feedback and then replying to comments throughout the rest of the week. I then moved on to researching the new XML structure the DSL are using for entries. There have been some major updates to the structure that have been implemented in the DSL’s editing system and will require significant changes to the front-end, the search facilities and the scripts used to process entry XML files after their export from the editing system.
Over the course of the week I wrote a document that detailed all of the changes that will need to be made to the various scripts, databases, indexes and systems to incorporate the changes to the XML structure. I can’t go into too much detail about the new structure or the changes here, other than to say that the structural changes are significant. By Friday I had finished going through all of my systems and the information provided to me regarding the new structure, and had documented all of the updates that would be required. However, during the process a number of questions arose that will need further clarification from the team, and we’ve arranged to meet to discuss things before I begin implementing the updates. Unfortunately due to members of the team being on holiday this likely won’t happen until later in September, but I have plenty of other things to focus on, both for DSL and for other projects, in the meantime.
Also this week I attended online meetings with two researchers who are in the planning stages for new projects. I’d had email conversations with both of them last week, and had arrange online meetings with them this week. The first was with Emanuele Scieri, who works in Theology, and we had a good discussion about his project and the technical aspects it might involve. The second was with Mícheál Ó Mainnín, a place-names scholar at Queen’s University Belfast who is hoping to adapt the place-names system I’ve developed for a new area, and again this was a great meeting and we had some good discussions. I can’t say much more about the projects at this stage, but will need to see how things go.
Other than the above, this week I had an email conversation with Eleanor Lawson about a new project, I made some further tweaks to the VARICS lookup tool, I gave some advice to Robert Davies in the College of Social Sciences regarding interactive maps, and I had an email conversation with Simon Taylor about the updates I’m making to the Place-names of Fife, and I devotes some time on Friday to continuing with the migration of the Fife data to the more standardised structure used by the other place-names resources. I’ve now migrated the main place records, the information about map sheets, classification codes and parishes, although the latter still need parish boundaries to be added. I also began looking at historical forms, but the sources are going to need some significant work, as the source titles and references (e.g. page numbers) are stored in a single field and will need to be split up, which is going to take some processing.
I was on holiday for most of last week and a little of this week, so I’m joining the weeks together in this update. During this time I worked on several different projects and communicated with a number of people about new research projects.
Whilst at the DH2025 conference in Lisbon Jennifer Smith alerted me to a problem with the sending of emails on the Scots Syntax Atlas site. We have a couple of contact forms (including one for requesting the audio files generated by the project) and neither were working. I’d also spotted that the two-factor authentication plugin used when logging into the WordPress admin interface was failing to send the emails containing the log-in codes. This took an awfully long time to sort out as it was unclear what was causing the problem or whose responsibility it was to fix it, compounded by the fact that the site is hosted by an external company. I’d noticed that several other of our externally hosted sites had the same issue with sending the 2FA emails so it appeared to be more widespread than just one site.
I spoke to Andrew McHugh, the head of RCaaS in IT Services and he suggested contacting the third party hosting company’s support. I had a long and useful chat with their support people, who suggested that we’d need to update the DNS setting relating to mail for the scotssyntaxatlas.ac.uk domain (which Glasgow controls) to point to the hosting company’s email systems. This would then enable me to set up SMTP for email for the domain and hopefully get the emails sending again. I therefore submitted a helpdesk request to Glasgow’s IT people asking them to update the record.
At the same time, I wasn’t convinced that this would solve the problem, and it was very strange that the 2FA issue was only affecting certain sites and not others. I realised that it was only the sites that had their own domain rather than being a subdomain of Glasgow that were being affected. I spoke to Luca who confirmed that this was also the case for one of his sites that had its own domain. It was also strange that the WordFence summary emails for these sites were getting sent, which suggested that emails were somehow working. I eventually realised that the 2FA emails were being sent from the email address ‘wp2fa@domain’ while the WordFence emails were being sent from ‘wordpress@domain’. There was an optin in the 2FA plugin to change the email address its messages were sent from so I set this to ‘wordpress@domain’ and… the messages came through.
It turns out that the University must have blacklisted the ‘wp2fa@domain’ emails, meaning all messages from these addresses were being blocked. So there was never an issue with the sending of emails for the domains – the messages were being sent but were being blocked and changing the email address to ‘wordpress@domain’ allowed them to get through.
This meant that what I thought was one problem was actually two, as the contact forms on the Scots Syntax Atlas site were still broken. I’d noticed previously that there appeared to be an issue with the contact forms’ use of Google ReCaptcha, but when I’d disabled this the forms still failed to send. Now I knew the problem was definitely related to the contact form plugin I could investigate this further. I made a copy of the contact form details and then completely deleted the contact form plugin, then reinstalled it and set up the forms again without any involvement with ReCaptcha. Thankfully this fixed the issue and the contact forms successfully sent their emails. It does mean that there may be a lot more spam sent to the email address that receives the messages, and we’ll just need to keep an eye on this.
Also during this period I continued to work with the Fife Place-names data. Previously I’d written a script to add latitude and longitude to all of the place-name data (more than 3000 records) that previously only had grid references. I’d run into some issues with some of the grid references, as discussed previously, and this meant researching the correct grid references to use, which took a bit of time. I managed to complete this process during these two weeks, successfully adding latitude and longitude to all but 50 of the records. These 50 records are mostly for obsolete place-names that don’t have a specific location, and I’ve emailed the project PI Simon Saylor to ask about these.
During this time I also set up a new project website for Gavin Miller and migrated the emblems websites over to a new server in an attempt to ensure the sites don’t use up all of the server’s resources, which was occasionally happening with the previous hosting arrangement. I also made some further updates to the Robert Fegusson online exhibition, based on feedback and further data from Amy Wilcockson.
Most of the remainder of my time was spent working for the Dictionaries of the Scots Language. I had an email conversation with Ann regarding the new part of speech tagging system and a further email conversation with Becca, William and Vasilis about the new structure of the DSL entry XML files. I now have some sample data and some helpful explanatory notes about the new structure so an upcoming task will be to ensure the DSL’s online systems can work with this new structure.
I also had an online meeting with Vasilis, Becca and William on Friday to discuss the new region and dialect area maps that will be added to the DSL website. This was a very useful meeting and we went through Vasilis’ prototype and discussed how this could be turned into a resource that would work on mobile devices as well as larger monitors and how we could make access to the large amount of overlapping spatial data a little less confusing. I then began writing a specification document for the new feature and will continue with this next week.
Also this week I had email conversations with Emanuele Scieri of Theology and Mícheál Ó Mainnín, a place-names scholar at Queen’s University Belfast about new projects that I may be involved with. I can’t say much more about these for now, but I’ll be having online meetings to discuss their proposals in the next week.
I continued to work on the Online Exhibition for the Robert Fergusson project this week, spending much of Monday and Tuesday on the task and completing an initial version that I sent to the team for feedback on Tuesday afternoon. As I worked with the materials and got to know them a bit better I ended up creating a more fully-featured first draft than I was originally intending to make. It is still an initial version, and any aspects can be changed as required. I took inspiration from the Books and Borrowing online exhibition to a certain extent. As with this site, the pages all have an ‘Explore’ section with image-based buttons leading to the various sections, and each page has links to the next and previous sections at the bottom.
I can’t share screenshots or the URL just yet, but the interface is designed to work on all screen dimensions from large monitors to mobile phones and as you scroll down the pages sections animate into place. There are currently two alternative animations. Pages such as ‘Publications’ have a ‘zoom’ animation whereby the section zooms and slides up into place as you scroll. The alternative can be viewed on pages such as ‘Depictions’ where sections are already visible but the content slides in from the left and right as you scroll (except the final video section of ‘Depictions’ which zooms up). Certain pages have the first section fixed (e.g. ‘Publications’) while other pages animate the first section too (e.g. ‘Scots poems’). We might not want to mix and match the approaches so much – I included alternatives so the team can see the options.
The pages currently consist of sections with three different background colours that are alternated (a greeny-grey, a dark grey and a sort of grey-teal colour). The header font is the same one as the main site and the paragraph text is fairly large. Where a section has one image this appears to the left or right (this alternates) in a box that also features the caption. Where there are multiple images (e.g. ‘poems on various subjects’ on the ‘publications’ page) a carousel is used and you can press the arrow buttons to scroll between images. In such cases there is still currently one single caption underneath the carousel.
On the ‘depictions’ page I’ve embedded the YouTube video in the final section. On the ‘Scots poems’ page I’ve included the three audio recordings in a final section with some placeholder text. Where images appear, I’ve applied some post-processing on many of them to remove the yellowing and make them brighter. I think this makes the images look nicer as part of an online exhibition, but the team may prefer to use the originals for the sake of authenticity. I’m pretty happy with how the initial version of the exhibition has turned out. During the rest of the week I had an email conversation with the RA Amy Wilcockson about adding in new content or rearranging the content that was already there, and I spent some of my time making such updates, some of which where rather time-consuming to implement but needed to be done.
On Tuesday afternoon I met with Moira and Leo from Archives and Special Collections to discuss some of the old resources I’d created for the archives around 15-20 years ago. It was great to catch up with them both and to hear their plans for the resources. It turns out that the University Story site has been completely lost – no copy of the site and its database remains on any University servers or other hardware. This was quite a shock as a huge amount of effort went into writing the content of this site – thousands of biographies as well as structured data about (for example) holders of professorships throughout history. When I stopped working for the Archives some 13 years ago a lot of work continued to be done to the site, both in terms of the content and reworking by subsequent developers, but I wondered whether I might still have a backup of the site as I left it somewhere at home.
On Wednesday I hunted about for the data. I found my old backup external hard drive in my attic, but unfortunately this did not include anything I hadn’t already discovered on my more recent external hard drive. I then managed to locate the old laptop that I used when working at the Archives back in the day, thankfully complete with power cable. This took ages to boot up, and I couldn’t remember the password I used to use for it. And the laptop was taking more than half an hour between entering a password and coming up with the ‘incorrect password’ warning. I then decided to set up a ‘boot from USB’ installation of Ubuntu to run on the laptop instead of booting the ancient version of Windows and this worked perfectly. I was able to access the laptop’s hard drive and copy all of the files relating to my Archives work onto my external hard drive. This included one single dump of the UGS database from June 2011, plus all of the code, images, and a bunch of other stuff relating to other projects. I now have this on my desktop PC as well as on the external hard drive. I loaded the SQL dump into a MySQL instance on my new laptop and it looks like the data is complete (albeit from 2011). It consists of 93 tables, incorporating the main UGS site, the international story, the ‘world changing’ site and the WW1 roll of honour. There are records for 19,228 people and of these 1,799 have biographies. There are 1,111 image records and these link to the image files contained in the ‘images’ directory in the code. There are three derivatives per image, which is why there are so many image files compared to image records.
The code contains the code for the site and the CMS, as it was in 2011. I did try getting the site to run on my laptop, but it would take a lot of work to get it operational due to changes in technology and also because it references stylesheets from the main University website from 2011. An awful lot was added to the site after June 2011, but hopefully having the database structure and the data up to this point will be a useful starting point. Stats on the homepage from the Internet Archive snapshot from April 2018 state that there are 2390 images and 3678 biographies on the site, which is a lot more than we had in 2011.
If it’s possible to request a copy of the UGS data from the Internet Archive the Archives people could then write a script to compare the people IDs in the 2011 database with IDs embedded in the URLs of pages from the Internet Archive (e.g. ‘WH10083’ from https://web.archive.org/web/20150908140047/http://www.universitystory.gla.ac.uk/biography/?id=WH10083&type=P&o=&start=0&max=20&l=a) to then make a list of all of the people that are not found in the 2011 database. The same could be done with images (e.g. ‘UGSP01400’ in the URL https://web.archive.org/web/20130704220509/http://www.universitystory.gla.ac.uk/image/?id=UGSP01400&o=&start=0&max=20&l=A&biog=WH10083&type=P&p=2). Of course there’s no guarantee that the Internet Archive captured all of the data for every person and image, but this would at least enable the archives to create a more limited list of people and images that could then be added to the 2011 data, rather than having to process all of the Internet Archive data from scratch.
The archives people could then write a script that could extract data only from the people / image pages in the list and insert this into the database. The pages are pretty well structured so it might not be too tricky to write such a script. Of course any edits made to the existing data since 2011 wouldn’t be spotted by this approach, as only new records would be targeted.
I wouldn’t recommend using the old code for the site and CMS, which is all very outdated now. The archives would be better off building a new system around the database structure, referencing the Internet Archive version of the old site to help figure out how things used to work. They might also want to rationalise the database structure as it’s possible not all tables are actually needed, or could be structured more efficiently.
Also this week I had an informal meeting with Marc Alexander to discuss my work, and I read through an edited version of the upcoming Books and Borrowing article to make sure the edits all made sense.
For the rest of the week I looked into a task that has been on my long-term ‘to do’ list for ages: Redeveloping the Fife Place-names website (https://fife-placenames.glasgow.ac.uk/). I made this as a proof of concept many years ago, having written scripts to extract and structure the data from Word documents. What I want to do is migrate the data to the system I created for the other place-names resources, which would then mean I could create a map interface for Fife that would be comparable to the Berwickshire one, for example (see https://berwickshire-placenames.glasgow.ac.uk/map/).
The first task was to attempt to generate latitude, longitude and altitude data for Fife, as currently the data only has grid references. I wrote a script that would do this, thinking I’d just be able to leave it running. Unfortunately when I ran it I realised that many of the grid references have problems. Many records have malformed grid references, such as ones with an additional number, or a zero character instead of an ‘O’, while for some others the data extraction script did not successfully grab the grid reference. In many such cases I had to do research to discover the actual grid reference using a combination of Google Maps, the historical maps on the Berwickshire website, the UK Grid Reference Finder (https://gridreferencefinder.com/) and the NLS GB1900 website (https://maps.nls.uk/projects/os1900). As you can imagine this took quite a lot of time, and I hadn’t managed to complete the georeferencing by the end of the week.
I’m on holiday for the next week and a bit, and I’ll return to this task once I’m back.