I attended the UK-Ireland Digital Humanities Association’s conference, which was held at the University of Glasgow this week. See the programme here: https://digitalhumanities-uk-ie.org/2025-annual-event/2025-annual-event-programme/. It was a great opportunity to hear about new and ongoing Digital Humanities related research. As expected, there was a lot of discussion of AI, with a wide range of viewpoints represented, from enthusiastic adopters to a speaker giving an impassioned account of the dangers of big tech and the risks of reliance on AI. I learnt a new term – ‘retrieval-augmented generation’, a concept that allows large language models to consult and incorporate a specified set of documents in addition to training data before generating answers, which helps reduce AI hallucinations and allows users to access cited sources. Another speaker discussed ‘digital necromancy’ and ‘AI afterlives’ – how AI is used to create representations of dead people based on the data they have left behind. Another speaker discussed how we are now in the ‘post-API era’ when companies that used to make their APIs publicly available for researchers to access have now closed these off to prevent access to them by web scrapers for AI systems.
Another speaker used the term ‘digital dinosaurs’ and although this wasn’t greatly expanded upon in the talk it got me thinking that maybe I am such a digital dinosaur – one of the first generation of born digital people who still very much lives a digital life but has been somewhat left behind by the pace of change – who doesn’t engage with social media and doesn’t use AI and instead uses ‘traditional’ digital systems such as web sites and emails.
Other speakers discussed specific tools and approaches they use. A library discussed their use of the Omeka platform, a PhD student discussed using OpenRefine (https://openrefine.org/) for analysing data and Flourish (https://flourish.studio/) for generating visualisations.
There was an interesting presentation by a team who are using Transkribus to transcribe the Darmstädter Tagblatt newspaper, which has multiple layouts over its more than 200 years in print. They discussed how they trained Transkribus to identify the regions and stated that their training is available for other projects, so I’ll have to see whether I could maybe reuse this for the Edinburgh Gazetteer. They also discussed their workflow system, which they developed using the Prefect Python library (https://www.prefect.io/). They also used a local installation of Transkribus, which was interesting to hear as I didn’t realise this was a possibility.
Another project discussed qualitative spatio-temporal reasoning, which uses LLMs to extract things relating to space and time from texts, looking for terms such as ‘next to’ or ‘alongside’ in order to extract events and relationships. The project generated a nice visualisation of spatial clusters by paragraph over the course of a text, showing the number of times certain places were mentioned throughout a text.
Another speaker discussed the detailed digitisation of settlements and used the ESRI Experience Builder (https://www.esri.com/en-us/arcgis/products/arcgis-experience-builder/overview) to integrate and visualise the data. Another presentation discussed using gaming systems to encourage reading in younger people. They discussed the British Library’s ‘off the map’ game design competition, and ‘Litcraft’ (https://www.lancaster.ac.uk/litcraft/) which generates places from works of literature (e.g. Treasure Island) in Minecraft. It was really interesting to hear all of the talks and I’m glad I had the opportunity to attend the event.
At the event I bumped into Moira Rankin of the University’s Archives and Special Collections, who I used to work with before starting my current job in 2012. We discussed a couple of the old resources I’d been involved with developing that are no longer publicly available, and whether I might be available to help get them back online again. We’re going to meet at a later date to discuss this further.
Also this week I added a further updated video to the Seeing Speech IPA charts for Eleanor Lawson and I spent a bit of time working on the Anglo-Norman Dictionary. For this I created a list of all parts of speech referenced in every entry in the dictionary’s management system. There are more than 100 different parts of speech listed across the entries and these need to be rationalised before I can develop a part of speech limit option for the advanced search. This list allows the editors to see how many entries feature each part of speech and can then edit the entries as required. I will also create a script to batch edit entries once the editors have worked through the list.
A couple of weeks ago the Iona people noticed that Google wasn’t doing a very good job of indexing the Iona place-names site and I spent a bit of time investigating what was going wrong towards the end of the week. The site is hosted on a server within the university network and a while back we encountered an issue whereby web crawlers were swamping the server with requests, making all sites run slowly and in some cases crashing the server completely. My colleague Luca therefore blocked web crawlers from accessing the sites on the server and the speed increase was remarkable. However, it turns out that blocking all web crawlers also blocked Google from indexing the sites, an unintended and pretty serious consequence.
Luca and I worked together this week to test out some updates to the server software and these should hopefully block many web crawlers but will still enable legitimate indexing bots (not just Google’s) to access the sites. Using Google’s search console (a tool available to verified owners of websites) I checked that their systems can now access the Iona site (and all others on the server). It will probably take a few days (possibly longer) for Google’s indexes to be updated, but eventually we should begin to see the Iona pages appearing as they should in the Google search results again.
I also helped Rob Maslen with an issue he was experiencing with his City of Lost Books blog (https://thecityoflostbooks.glasgow.ac.uk/). WordPress had automatically classified this as a commercial website and was blocking access to the stats unless Rob paid for a commercial license. It would appear that a link to another person’s Patreon site from the blog had led WordPress to assume that the site was asking for donations. Rob has since removed the link and has requested that WordPress reassess the site, and hopefully this will fix the issue.
I spent pretty much all of this week working on the new versioning system for the Books and Borrowing project, which I have now managed to complete. Last week I made some good progress on the redevelopment of the API to enable data queries to be limited by version, and this week I was able to complete the work. With a fully working new API running on my laptop, I considered updating the live system as the next task, but decided against this. Updating the live system would have meant regenerating all of the data, and I realised that this would mean the new ‘version 1.1’ data that is already being worked on in the CMS would therefore be fully integrated into the front-end. This is something we don’t want to happen yet as Matt and Katie are still working with the data and need it to reflect the state it was in at the official end of the project.
Instead, I began the process of creating a new version of the front-end that would connect to the new API and would be able to target and display specific versions of the data. I decided to track the selected version throughout the front-end by means of both session variables and URLs. Session variables on their own could be used to track the chosen version, but we also needed to include the version number in the URL in order to enable users to cite / share / bookmark specific versions. This meant that every single URL that is generated throughout the site needed to incorporate the version number, including WordPress navigation menu items and other WordPress pages.
Adding in session tracking throughout WordPress brought some difficulties. My session data was somehow getting overwritten as I moved between pages, with WordPress session data getting added in and replacing some of the variables I had set. This took some time to get to the bottom of, and it turned out that WordPress’s emoji system was causing the problems. I found a guide to disabling this system here: https://smartwp.com/disable-emojis-wordpress/ and once I’d followed these steps my session variables worked as I’d intended.
For most users the front-end will not appear to be any different. It is now set up to default to the most recently published version of the data (including all earlier versions too) and none of the page URLs have changed. The only thing that is different is the ‘cite’ popup, which now references the current version. It also includes a link to this version in the URLs that appear, as the following screenshot demonstrates:
This is to ensure that the citation will always reference the version of the data the person cited. If we didn’t include the current live version in the citation URL then the link may lead to different results if/when we publish a new version of the data.
I also updated the footer of every page to include the current live version of the data, as you can see in the following screenshot:
I also included a link in the footer to the new ‘Versions and changes page, which you can see below:
This still has some placeholder text and then lists all of the live versions of the data, including the version number, a description and the date of publication. One version will always be selected (in blue and with a tick). If you press on ‘Version 1.0’ this then sets the entire site to use this version of the data – everything including lists of libraries, facts and figures, registers, pages, borrowing records, search results and on this day items will be limited to data created as part of this version (or earlier, if applicable).
When an earlier version is selected, every page of the site has the version number in the URL, for example ‘/libraries/version-1.0’ and the selected version is tracked throughout the site. To make it obvious to users that they are not looking at the most recent version of the database a warning is displayed on the screen, as shown below:
As with the ‘top’ link this box stays fixed in the bottom corner of the screen. Pressing the ‘return to the most recent version’ button takes you to the homepage with the most recent version reinstated.
Independently of all of this, I’ve also updated the search results to add in a version filter, as shown in the following screenshot:
So for example, if you’re viewing the current version of the data (1.1) you can see how many borrowing records were created as part of this version as opposed to the earlier version, and filter the results if you wish. Note that if you’ve set the site to use version 1.0 of the data then the 1.1 search results wouldn’t display and the 1.1 filter option won’t show up.
It’s taken a huge amount of effort to get all of this in place, updating pretty much every part of the Books and Borrowing system, but it’s now thankfully all in place. The only thing left to do is to actually add all of this to the live site, which I was intending to do next week. However, Matt and Katie are still doing research using the live site at the moment and I therefore need to hold off until they have completed this work. This is because whilst I work on the updates the live site will temporarily include data from all versions, which will obviously affect the data they extract. I’m probably going to have to hold off going live with the updates until August, as I’ll be on holiday and then at a conference between now and then.
Also this week I attended the launch of the Iona place-names resource on Monday evening, and it was great to see the resource (see https://iona-placenames.glasgow.ac.uk/map) go live after working on it over the past few years. The Berwickshire place-names team also launched their published volume at the event, which was also great to be a part of.
Also this week we finalised the archiving of the old Cognitive Toponymy website, and the archived site is now available here: https://conifer.rhizome.org/UoGLibrary/cogtop/20250430094955/https://digital-humanities.glasgow.ac.uk/cogtop/
I made the Iona place-names map resource live this week, in preparation for the official launch on Monday next week. After working on the project on and off for several years it’s great to get the resource out there. It can be accessed here: https://iona-placenames.glasgow.ac.uk/map/.
I spent a lot of my time this week continuing with the development of the versioning system for the Books and Borrowing project. Due to other commitments I didn’t manage to work on this task last week, having completed the updates to the scripts for generating the cached data the week before, so it was good to be able to return to this task and make a bit of progress. This week I focussed on updates to the API, which is a pretty major task as every single database call in more than 5000 lines of code needs to be updated. Before I did so I also had to consider how the user’s selected data version would be passed to the API. Appending it to the end of the existing set of parameters would be messy, as each endpoint would then potentially have the version passed through a different parameter. Instead I decided to that the version should appear at the very start of each endpoint call, before the file type selection. So for example, ‘/api/v2/1/json/libraries’ will retrieve all ‘version 1’ data for the ‘libraries’ endpoint as a JSON file while ‘/api/v2/2/json/libraries’ will retrieve all ‘version 2’ (and version 1) data. In order to ensure older calls that don’t feature the version number continue to work, I set the API to use the ID of the most recent live version of the data if the version number is omitted. Also, the API checks to ensure that any version number passed does actually correspond to a version of the data that is actually live in the system, so the API will never publish data that is still in development.
With this update to the API in place I then needed to work through every database query in every endpoint in order to add in the version limit. In some endpoints this was pretty straightforward to do, but other ones did get rather complicated, involving cached data in new database tables or specific versions of cached JSON files. I didn’t manage to complete the update to the API by the end of the week, but I’m hopeful I’ll be able to do so next week, at which point I aim to add a new version of the API to the live site, in preparation for redeveloping the front-end to work with the versioning system.
On Thursday this week we had the interviews for Ophira Gamliel’s project, and this took up more than half of the day. The interviews all went smoothly and we had a good discussion about potential candidates afterwards.
Also this week I had a chat with Geert, the editor of the Anglo-Norman Dictionary, about further updates to the site. We’re intending to add in a part of speech option to the advanced search and we spent some time discussing how this might function. I also gave him so advice on long-term digital preservation, helped him set up a new ‘phase’ and fixed an issue when adding lead editor initials to entries through the dictionary management system.
I also gave some advice to Tony Harris, who is developing a thesaurus system based on my code for the Bilingual Thesaurus, and fixed a couple of issues that Marc had spotted with the unified API version of the Historical Thesaurus that I worked on in April. Marc had spotted that some information was missing from the ‘recommended’ category section in the search results. The blank recommended category information was caused by the API correctly returning an empty subcat array for categories that don’t have subcats. Unfortunately the front-end code was checking for the presence of this array when deciding whether the recommended category was a sub or main category, and as the array was present (but empty) for maincats it was treating them as subcats and not displaying the correct details. It took a while to identify the cause of the problem but once I’d done so it was very easy to fix. The second issue that Marc spotted was a URL encoding issue, with both slashes and spaces causing problems. This was something I managed to sort relatively quickly.
Monday this week was a public holiday, and I divided my four working days across several projects. The Iona place-names project is having its official launch on the 9th of June and a fairly last minute request for the map interface was to incorporate ‘thematic maps’ – maps that show a set of markers that share a common theme, such as bird names. Last Friday Sofia sent me the data for two example thematic maps (which I think we’re going to be calling ‘Virtual Trails’ in the public interface) and this week I set about creating the new feature.
When creating a new feature I’d usually create the sections of the CMS for managing the data before updating the front-end but as time is rather tight I thought it best to create the necessary structures in the database, the required updates to the API and work directly on the front-end with the sample data. If there’s time before the launch I’ll create the CMS pages, but if not and we want to add more maps I’ll just have to get the team to send me spreadsheets and I’ll add the data directly to the database. Below is a screenshot showing the new map:
I’m still working on this and there is still quite a bit to do, as adding in the new section and new map type has meant a lot of reworking of existing structures. But as the above screenshot demonstrates, the interface now features a new ‘Virtual Trails’ menu section that when expanded has a placeholder description and a ‘Choose a virtual trail’ button. Pressing on this opens a popup that lists the thematic maps, showing the titles and descriptions. Pressing on a title closes the popup, adds the selected map’s title and description to the left-hand menu and loads the relevant map markers into the map. You can then change the display options (e.g. turning labels to always on, changing the classification or base map) and open the records as you would with any other map.
So that’s the basics in place, but there are still many updates I’ll need to make to the front-end to fully integrate the new feature, including ensuring that the ‘Reset map’ resets the virtual trail menu contents, ensuring citations and bookmarks / sharing URLs work when a trail is selected, ensuring the table view works properly and cancelling out any already active search or browse options when a trail is selected.
Also this week I applied the new updates I’d made last week to the element glossary to the other place-name maps, such as Berwickshire. I also added ‘deselect’ to the legend, as apparently users were unaware that the ‘select all’ option could be used to ‘deselect all’ when unchecked., and I spotted that the Iona full map ‘cite’ option was referencing the Ayr site and fixed that too.
For Speak For Yersel, I spent some time this week creating the slides for my presentation at the Lisbon DH conference, and I have now completed a first version of the slides and script. I’m probably going to have to trim it down a little, though, as when running through it I was taking longer than my allotted ten minutes. I’ll probably have to take out the parts about how the maps were generated, which is a shame as it’s the most DH section, but it’s also not something we discussed in the abstract so if anything needs cut it’s the obvious choice.
On Thursday this week I met with Ophira Gamliel to discuss the interviews for her project, which are taking place next week, and in the afternoon I attended a Modernities Research Cluster event which featured two fascinating speakers. I also fixed an issue with the Thesaurus of Old English for Jane Roberts and fixed a problem with the batch update of citations in the Anglo-Norman Dictionary. This feature of the content management system allows citations across all entries (or selected entries) to be updated simply by editing the citation once. However, there was a problem when adding dates to citations that did not already have them. The script was running into problems when adding a date for an attestation that didn’t already have a <dateInfo> element. I’d included a check for this, but the check was causing a fatal error rather than executing the code that I’d written to deal with such attestations. Thankfully once identified it was relatively easy to fix the problem, and now the batch update system is working perfectly.
I spent a lot of time this week continuing to implement the new versioning system for the Books and Borrowing project. This week I continued to tackle updating the Solr index, other cached data, and the scripts that generate this data. This involved making updates to the structure of the underlying database as where (for example) originally one cached JSON file was stored we now need to store any number of such files, with one per version.
I managed to update the code the generates the Solr index and executed it, generating the necessary JSON files, which I was then able to successfully import into a version of the Solr index that was running on my laptop. After running a few queries through the Solr admin interface everything seemed to have worked fine, which was a relief. I then had to work through all of the scripts that generate other cached data, such as generating the library top ten data and lists of each author that is associated with a borrowing at a library. There are more than ten such scripts and each needed updated (sometimes, as mentioned, with updates to the database structure too), tested, executed and then the output needed to be tested.
I ran into a bit of difficulty when generating the data for the ‘Facts and Figures’ pages, as this needs to query the Solr index, sending hundreds of thousands of queries to Solr in order to generate a static cache containing figures relating to the data. The issue was that the connection to Solr was taking a long time to complete. Previously passing a query to Solr via HTTP has been pretty much instantaneous, but this time each request was taking at least a second to complete, and when you’re dealing with hundreds of thousands of requests this isn’t feasible. I wasted quite a lot of time try to figure out what was going on, and it was especially perplexing as the function I used to connect to the Solr instance hadn’t been changed since I last ran it successfully and the Solr instance’s software hadn’t been updated either. I tried installing a newer version of Solr but it had the same problem. I tried running a test script on the actual server and it executed instantaneously, so the issue must have been something to do with my laptop. Eventually I asked Luca if he had any ideas and we had a useful troubleshooting session over Teams chat.
During this I decided to try changing the connection string from ‘localhost’ to the localhost IP address of ‘127.0.0.1’. As soon as I did this the connections were instantaneous again, which was a real relief, but also rather baffling, as I didn’t know why this should make a difference. It turns out that localhost connections now default to IPv6, but Solr and my laptop were not set up to run IPv6. This meant that every connection to ‘localhost’ was attempting to connect to Solr via IPv6, which failed, and then was falling back to IPv4 (127.0.0.1) and working. This was taking time and was causing the connections to run so slowly. I will have to remember this in future. Thankfully after replacing ‘localhost’ for the IPv4 IP address of 127.0.0.1 my scripts were much quicker and what would have taken days (or possibly weeks) to execute completed in about 15 minutes.
With all of the cache generation scripts updated, the cached data regenerated with a new structure and the database structure updated I am now ready to begin the process of updating the API and front-end to incorporate versions into everything. This is a task I’m hoping to make a start on next week.
Also this week I reviewed a shortlist of applicants for Ophira Gamliel’s project and gave feedback on their suitability. I also fixed an error with the Metaphor Map and the Old English Metaphor Map that was preventing counts of categories from displaying in the textual browse lists. I also liaised with our IT people to get the OHOS domain migrated to The National Archives and responded to Craig Lamont about a new project he is putting together.
The remainder of my week (about a day) was spent on the Place-names of Iona project, for which I implemented some updates that we’d discussed at a project meeting last Friday. This included updating the site banner to cut off the top part and make it take up less of the screen and adding in the legend descriptions from the Berwickshire Place-names project. I also updated the glossary to ensure that all listed parts of speech have a tooltip that displays the full form.
The big task was to completely overhaul the display of the glossary. There is now a bar of letter links that feature the letters that elements start with as buttons. A count of the number of elements starting with each letter is also included, as I thought this might be useful. Any letters that don’t have any elements appear but are greyed out. The list of elements is now split by letter, each of which appears as a heading with a dividing line to make it easier to tell where elements beginning with one letter end and the next set begins. A ‘Top’ button is also present to make it easier to get back to the top and I also replaced the placeholder text in the element glossary side-menu with descriptive text that Carole came up with for Berwickshire. These updates make the element glossary much more usable now, as the following screenshot showing personal name elements demonstrates:
Next week I’ll apply this update to the other place-names resources, and I’ll also hopefully find some time to work on the new ‘thematic maps’ that we’re hoping to implement before the resource goes live in June.
This week finally saw the launch of one of the new ‘map first’ interfaces for the various place-names projects I’ve been working on. I initially created the interface for the Iona place-name project and then applied it to Ayrshire, Kirkcudbrightshire, Berwickshire and Nairnshire, but none of the projects were quite ready to publicly launch the new interface. After having a char with Carole Hough last week I got the go-ahead to go live with the new Berwickshire map, and it can now be found here: https://berwickshire-placenames.glasgow.ac.uk/map with the textual list of Berwickshire place-names here: https://berwickshire-placenames.glasgow.ac.uk/list-of-berwickshire-placenames/.
It’s great to have the new interface publicly available at last, although I did have a number of last-minute tweaks to make as I worked on it. I’ve updated the legend slightly, adding in some explanatory text and ensuring that the text is hidden when the ‘hide legend’ button is pressed – previously the text remained visible, which was a bit pointless. I also spotted that the legend was not being set to hidden be default on narrow screens, even though it should have been, so I fixed this. Another issue I spotted was that the tooltips that display when the legend categorisation is by language would stop working whenever the ‘select all’ option was pressed, and I needed to update the code to ensure the tooltips were reinitialised. With these changes in place I also needed to apply them to the other place-names maps to keep things consistent.
Launching the new map wasn’t the end of the work, however. Berwickshire already had a live map resource, plus search and textual browse facilities. I needed to ensure that the quick search on the main resource website connected through to the new map’s search results and that all links / bookmarks / citations to the old interface redirected through to the appropriate section of the new interface, where possible. So for example, the old element glossary page redirects to the new element glossary popup in the new map interface: https://berwickshire-placenames.glasgow.ac.uk/place-names/?p=element-glossary. I think I’ve caught all of the old links and have ensured the necessary redirects work.
Also this week I had a meeting with Thomas and Sofia regarding the map interface for Iona, which will apparently be publicly launched on the 9th of June. We discussed some of the outstanding tasks they would like to see completed before the launch and I’m going to have a fair amount to do before then. For example, they would like the element glossary to have a ‘jump to letter’ feature, allowing the user to immediately scroll to elements beginning with a selected letter. We also discussed new ‘thematic maps’ based on static maps Sofia had created for a recent event on Iona. These maps showed place-names grouped in more fine-grained ways that are not covered by the classification options we currently have available – for example place-names featuring animal names. I suggested that I could make an interface where such thematic maps could be created in the content management system and displayed in a new menu section in the front end. This would consist of a title and description for the map, and a list of markers that should appear on the map.
I also demonstrated the ‘story map’ interface I’d created for the Scots Syntax Atlas (go here and select ‘Stories behind the examples’: https://scotssyntaxatlas.ac.uk/atlas/). This feature consists of a series slides a user can navigate through, each of which can have a different view of the map, which may feature different data and different zoom level, thus guiding the user through a particular story the map tells. I gave an example of how this could be used for iona to have a story of place-names featuring animals, with different slides showing markers featuring elements from different languages. Thomas and Sofia really liked this idea and would like to implement it, but while it might be possible for me to implement the simpler thematic maps before the 9th of June, it’s unlikely that I’d be able to get the ‘story’ approach in place. It’s something we’ll probably consider after the launch.
Continuing with place-names, I had an email conversation with Alasdair Whyte regarding his Mull / Ulva place-name data. He’s currently working towards a published volume and wanted me to add a facility to the content management system to enable him to categorise place-names by volume. I therefore updated the CMS to add in the volume field, and this now appears in the ‘add’ and ‘edit’ place forms. I also updated the ‘Browse’ page to add in volume to the search options and as a table heading for the listed data and updated the ‘Export place-name data for publication’ page in ‘Tools’ to add in a volume selection option. Thomas also contacted me this week to ask for an update to the ‘Export’ facility for Iona (adding in the ‘translation’ field) so I did this too. I also had a chat with Alasdair about the front-end for his Mull / Ulva data, as this will need to be a bit different from the other resources.
Also this week I finally managed to get my ticket sorted for the Digital Humanities event in Glasgow in June, with the help of Emma McCluskey. I also made some updates to my presentation for the DH event in Lisbon based on some very helpful feedback from Jennifer, and also added in some new publications to the Scots Syntax Atlas resource.
I also spent a bit of time working on the Dictionaries of the Scots Language. I’d been contacted last week by editor Vasilis Karaiskos about the regions data that he has been working on, and the new maps he has been developing to display these regions. I spent some time going through the interface he had developed as a proof of concept, figuring out how it all works and giving some feedback. It will be really great to get more information about the geographical regions into the website and the maps are an excellent starting point. We’re hopefully going to have an online call in the next few weeks to discuss things further.
The rest of my week was devoted to working on the new versioning system for the Books and Borrowing project. I need to update the structure of the Solr index and the other cache files, and also the scripts that generate this data, and this is my next step. As I feared, it’s proving to be an awfully tricky and time-consuming update to implement, and while I did make progress there is still a huger amount to do. I’ll be continuing with this monumental task next week.
On Monday this week I attended a ‘getting started with Transkribus’ workshop held at the University of Edinburgh. This was a really great opportunity to learn more about the Transkribus tool and to be guided through the initial stages of using it. I had somewhat naively thought that it would be possible to just upload a digitised image to the tool and it would then be able to extract and format the text with a high degree of accuracy, but in reality much more manual intervention is required before Transkribus can work effectively. Training the tool on your data is absolutely vital, and this means uploading sample images, identifying lines of text (which can be done automatically but may need manual tweaking) and then transcribing the text found on each line using Transkribus’s web-based text editor. Text can also be tagged using this facility, such as noting headings, or identifying proper names. The leader of the session suggested that around 15,000 words of text should be manually transcribed in order for the tool to be well trained, which would take a considerable amount of time and effort.
I wanted to see whether Transkribus could extract the text from the Edinburgh Gazetteer (https://edinburghgazetteer.glasgow.ac.uk/the-gazetteer/) and the first stage of this would be for the tool to recognised that each page is divided into four columns, many with a full-width heading at the top. A ‘layout’ model would need to be run on the pages in order to automatically identify the columns. There are a couple of existing newspaper layout models available but unfortunately when I ran my test image through them neither managed to divide the page, and the entire page was simply identified as ‘Region 1’, meaning all identified lines ran across all four columns. Any human looking at the image would be able to ascertain immediately that it’s four columns of text in so I was a bit disappointed that Transkribus wasn’t able to do the same. I spoke to the leader of the workshop, who has successfully used Transkribus on another historical newspaper and he suggested that I’d have to train Transkribus on this specific newspaper and that in order to do so I’d need to use a ‘Field model’. It’s not entirely clear to me how a ‘Field’ differs from a ‘Layout’ model, but unfortunately it’s not possible to create a field model with the free Transkribus account – you instead need to subscribe to a ‘Scholar’ account at €20 a month, something I didn’t want to do as I was just experimenting at the workshop, so I reached something of a dead end. Having said that it was still very useful to experiment with Transkribus in a workshop setting with an expert on hand and I will continue these experiments when I have more time available. However, I do wonder how more general AI tools might cope with the same materials and whether they might be quicker to use. It’s definitely something that would be worth exploring further.
For the rest of the week I worked on several different projects. I spent some time refamiliarising myself with the structure of the Bilingual Thesaurus of Everyday Life in Medieval England (https://thesaurus.ac.uk/bth/) as a developer who will be working with the structure for a new project got in touch with questions about it. I wrote a data description document that describes every table and column in the database, and also contains the highest ID in each table used for BTH. There are several columns that are only present as references to the original spreadsheet and are not used by the online resource, and I’ve made a note of these. Whilst working on this I also spotted an issue with the display of languages in the advanced search. These were not displaying any content and it turns out that this was because one of the referenced column names was ‘grouping’, which is a reserved word in MySQL. While the queries worked on our old server, the new server is more strict about such things and the queries failed to run on it. Having spotted this it was thankfully quick to fix – surrounding the column name with backticks (`grouping`) in the queries got things working again.
I also returned to working on the Books and Borrowing project for a few hours this week in order to update the links we have in our system to books held at St Andrews University library. The library has recently changed their online system, which has broken all existing links to records in the system. Thankfully they have a huge spreadsheet featuring the old identifiers and the corresponding new ones and I was able to write a little script that would identify the St Andrews book holding records in our system and update their links. Of the 7560 active St Andrews book holding records in our system 1666 have URLs and my script successfully updated 1662 of these. The remaining four have issues that will need to be looked at separately, and I’ve let Katie and Matt know about these.
I met briefly with Alan Riach this week to discuss the website I created for him last week. He gave me a further CD featuring a recording he wanted me to host and I was able to extract the recording and update the website to add this in. I also joined my fellow College of Arts and Humanities developers for a coffee and a catch-up this week, and it was really great to chat with them all again. Hopefully we’ll meet up again soon.
On Thursday this week I also had a meeting with Rhona Brown, Amy Wilcockson and Clare Thompson from the Mitchell Library to discuss an online exhibition for the Robert Fergusson website (https://robert-fergusson.glasgow.ac.uk/). There was a physical exhibition of materials at the Mitchell last year and we’re going to be bringing these together online in the next few months. It was good to meet everyone in person and hopefully I’ll be able to help them create an exciting online resource based on the materials.
Also this week I managed to fix the issue with capital letters not working in the ‘jump to’ option in the ‘browse’ facility of the Anglo-Norman Dictionary. You can now enter text and it will find matching entries whether they have capitals or not. E.g. enter ‘abc’ (or ‘ABC’) into the ‘jump to’ box. Brackets are also not an issue, e.g. type in ‘xilo’ and you’ll jump to ‘[xiloaloe]’. The only thing to bear in mind is that the browse is accent sensitive. It has to be as we have entries with headwords that are identical other than for accents. So for example entering ‘acel’ will jump to ‘acelément’ but typing in ‘acele’ will find nothing. However, ‘acelé’ will work. Updating this would be a major undertaking so hopefully it’s not too big a deal – it’s been like this since the new site launched, after all. I also applied the updated I’d made last week to the Iona map interface to all of the other place-names projects.
The remainder of my week was spent going through the feedback I’d been sent from testing of the new data for the Dictionaries of the Scots Language. This included fixing an issue when conducting a search featuring an apostrophe in iOS using the Safari browser. This was adding an a curly apostrophe which then didn’t work with our data. I updated the code to ensure such apostrophes are switched for regular ones when submitted, but when working on this I noted that there are some occurrences of curly apostrophes in the dictionary’s XML files. These characters are treated as entirely different from regular apostrophes and will not be found in a search that contains a regular apostrophe. This led to some discussions with the team about the use of such characters in the data. I also updated the error messages that are displayed when invalid dates are entered in the quotation date filter and spent a long time working through the sparkline data generation code to ensure certain edge cases are resolved.
It’s been a long time since I wrote the sparkline code (I began in June 2023!) and it took some time to refamiliarize myself with the it, but I managed to sort all of the issues you had been reported. Unfortunately these fixes needed to be made to the code that generates the sparkline data that then forms part of the Solr index so no updates were initially available on our test server. Instead I ran many tests on my laptop, both before regenerating the Solr data by targeting specific entries and also through a test instance of Solr and the DSL site running on my laptop after regenerating the entire dataset and as far as I can tell all is working without any unintended consequences being introduced. I then asked our IT people to update the online Solr cores and this was completed by the end of the week.
I also tweaked the display of the sparkline dates in the search results on narrow screens. I’ve updated the layout so that the sparkline text and the sparkline are treated as one single block element. If the page is too narrow for both this and the entry title to all fit on one line then the sparkline and its text drops down to a new line rather than different sections being split over different lines. I also updated the CSV output to remove the search result snippets from them and had a chat about the tagging of certain bibliography entries in the data.
For the Anglo-Norman Dictionary this week I made a couple of major updates to the data. Firstly, a researcher had gone through the whole of ‘D’ to compile links from each entry to the DEAF dictionary. This took the form of a spreadsheet and I adapted the script I’d previously written to deal with DEAF links in order to work with this new data. The script updated the XML of each entry so I made sure I backed up the data first, and ran everything on a test version prior to updating the live site, but thankfully all went smoothly and entries such as https://anglo-norman.net/entry/dubbeour now have links out to the DEAF.
The second major update was a new batch of entries and major revisions of existing entries that the team have been working on over the past few months. This covers all of ‘W’ plus may other entries across the entire dictionary. As with previous updates, I ran and tested everything locally before updating the live site, as the first step in such an update is to delete the entirety of the letter that’s being updated. After running everything on my laptop and testing things out I spotted a couple of issues with the entries and in discussion with the editor Geert we managed to sort them out. In total the update added or edited 1918 entries, of which 1161 were ‘main’ (rather than cross-reference) entries. These included 1245 main senses, 34 subsenses, 77 locutions, 41 locution senses, 1 locution subsense and 2460 citations.
Also during this process we spotted that cross references were not being listed properly in the dictionary’s content management system. The code that regenerates the cross references after changes are made to the data wasn’t specifically identifying and targeting the ‘active’ versions of entries and sometimes the links that were stored were to inactive versions, which then weren’t displaying, as the code for displaying them was specifying active records only. However, I updated the code and regenerated the cross references so they all now appear.
Also this week the editor Karen spotted that links to the FEW dictionary from commentaries weren’t working and I updated the code to fix this. I also spotted that the dictionary browse feature on the entry page wasn’t working when entries featured upper case letters. I tried a few approaches to sorting this, but the ‘slug’ field in the database needs to be case sensitive as this is the only way to make the field accent sensitive too, which is needed as we have slugs for different entries that are identical other than for an accent. I did try swapping the field the browse targets from the slug to the lemma, but this introduced further issues because the lemma can include brackets, plus the ‘next’ and ‘previous’ links in the browse rely on the slug field. I’m going to have to return to this next week.
Also this week I created a mini-site for Alan Riach’s media. This involved extracting audio and video files from CDs and DVDs, which was slightly time consuming. The DVDs were rather tricky to extract as they were in multiple parts that then needed joined together. Also, one of the DVDs was corrupted and I was only able to extract half of it. I also had to migrate transcriptions of four hour-long audio files from Word to HTML and of course create the site structure and design. The site is password protected and is intended for students and educational use only but here is a screenshot:
Also this week I addressed some issues that had been uncovered whilst testing the advanced search facilities of the test version of the Dictionaries of the Scots Language that is set up with the latest version of the data. The first issue was that in bibliography searches some entries appear twice in the drop-down list. This was happening because entries can have both long and short titles and authors and the drop-down list was accommodating both. I’ve updated this so that just the author and title that are picked as the ‘display’ versions during import are now displayed, so there are no more duplicates. However, this does mean it’s not always clear why an entry appears in the list. For example, if you select ‘author’ and enter ‘ham’ you’ll see ‘Belhaven, J. H.’ and it’s not until you click through that you can see that ‘Ham’ comes from ‘Hamilton’.
The second issue, also relating to the bibliographies was that some items were not being found in the search. This was because the import script was only picking out titles and authors for bibs that had an ‘oldid’. Bibs that didn’t have these were still being processed and display titles and authors were being stored, but as no additional titles and authors were being extracted the searches therefore weren’t working. This has now been sorted – the display titles and authors are also now being searched in addition to the other titles and authors.
Whilst testing this out I also discovered that selecting items from the drop-down list that featured apostrophes and colons broke the search and led to an empty search results page. For example a title search for ‘cottager’ and then selecting the “cottager’s daughter” item. This is still broken on the live site, but is now fixed on the test site.
Also this week I liaised with our IT Services and technical people at The National Archives to arrange for the OHOS domain to be transferred to the TNA. I also responded to a query about the Bilingual Thesaurus of Everyday Life in Medieval England, had an email conversation with Matthew Creasy about his Mallarmé project and spent some time making updates to the Iona map interface following our team meeting last week.
I think I’ve covered everything that was raised at the meeting. I created a space for each categorisation at the top of the legend with some placeholder text for now (e.g. ‘Info about language classification’). For now when you hide the legend the description remains visible, as I figured it might be useful to know what the markers represent, but I can make this hide along with everything else. I’ve also updated the altitude classification to 20m increments and added in language tooltips wherever languages appear (e.g. in the small popup and the full record). Parts of speech in the full record also now have tooltips, although I’m afraid there are a few that I can’t put a name to and there are others I may have got wrong, so these may need further work. I updated the satellite views so that they now feature markers with a white border. And I’ve fixed issue with the top legend tick box ending up unticked in Chrome. In addition, all language combinations in the legend for language now feature tooltips (if they are abbreviated). I’ve applied most of these updates to the other place-names resources too. I still have to add in the tooltips (points 3 and 4) to the other sites, but I will do so next week.
I divided my time between many different projects this week. For the Anglo-Norman Dictionary I made a few further tweaks to the new cognate references section that we went live with last week, and I also received a new batch of several hundred new or reworked entries that are to be added to the dictionary, which I’ll process next week. For the Dictionaries of the Scots Language I had an email conversation with editor Ann Ferguson about the structure of URLs and also gave some consideration to the bibliography search, based on feedback I received towards the end of the week. I’ll continue to work on this next week.
I also found a bit of time this week to begin work on a new version of the Historical Thesaurus website. The current website’s backend is a bit of a mess, as it was cobbled together over many years as new features were added. What I’ve been meaning to do for a while now was to rework this in order to extract all data querying and processing code from the front-end, separate it out and develop a single, unified API for the site, an approach I’ve taken with all of the large projects I’ve been involved with for many years now. This is going to be a rather large task, but I made a start this week: I set up a new version of the website and began the process of creating the API. So far I’ve only got the ‘random category’ feature on the homepage working with the new API, but I’ll continue working away at this as time allows until it’s all done. There won’t be any change to the front-end (although I’m also separately considering a Bootstrap-based refresh of the interface), but the update will leave the data querying side of things in much better shape for the future.
For the VARICS project I continued to work on the new look-up feature. Following last week’s traffic light experiments the team requested a more simple approach for the traffic lights, so I created new images featuring a subtle gradient and a white border, as you can see below:
I also added in a new ‘Try again’ button that would reload the form, which was subsequently renamed ‘Start again’. The team prefer this simpler approach so it looks like this is what we’ll be going with. I now need to wait until further data is sent to me before I continue to develop the feature.
For the place-names projects, I spent some time investigating an issue with the advanced search in the new map interface. An element language search was not working unless a specific element was also selected. For example, a search for all place-names containing Old English elements wouldn’t run, but selecting Old English and then selecting the element ‘tūn (OE)’ worked perfectly. It took a while to track down the issue, but thankfully once identified fixing it was trivial: there was a typo in the code for the API. The system was already set up to allow a search for element language without specifying an element, but in such circumstances the code failed to run because of the typo. I’ve fixed this now on all of the place-names sites. I also fixed a strange issue with the advanced search pop-up disappearing when parishes towards the end of the list were selected. This issue was only occurring in Chrome and not in Firefox (my default browser) and an update to the stylesheet for the map pop-up thankfully fixed the issue.
I also participated in an online meeting with the Iona project team on Wednesday this week where we discussed some of the map-related issues that had been brought up at the advisory board meeting and how (or if) we should address these. I ended up with a list of updates that I’ll need to make to the interface, another thing I’m hoping to tackle next week.
Also this week I had a brief in-person meeting with Joanna Kopaczyk and her project RA Pia Lehecka to discuss Joanna’s new DOST digitisation project. It was good to meet the other members of the team and I’m looking forward to working for the project over the next few months.
I also met with Alan Riach in Scottish Literature this week. Jean Anderson had contacted me last week to put me in touch with Alan, who was wanting some help to publish some old radio broadcasts that were previously available on one of our old servers that has since been decommissioned. Alan managed to find some audio CDs containing the recordings and I met with him to collect them, together with some additional recordings and a few DVDs. I spent some time towards the end of the week extracting the audio and video data from this media and creating a simple website where the files and accompanying transcripts will be accessible to students. I haven’t quite finished this task yet, but should have an initial version completed early next week. The resource will be password protected and only available to students and staff, as there may be licensing issues relating to the recordings – for example relating to incidental music used during the recordings.
During last week’s advisory board meeting for the Iona project someone mentioned search engine optimisation and how the Iona site (https://iona-placenames.glasgow.ac.uk/) is generally well situated in search results. This got me thinking of how our new map-based interface would be indexed by web crawlers. While the map interface is great for humans, it will make all of the data much harder for web crawlers to index. With the older style of place-name interface all of the data was easily findable in text form on web pages, but with everything now appearing within the map interface services like Google are much less likely to be able to index our data as it’s buried in pop-ups and other JavaScript elements.
To mitigate this, I decided to create a new, purely text-based alphabetical browse of place-names. This links through to the full records on the map, but more importantly for search engines, it includes the full text of fields like the description and translation, which should be straightforward for web crawlers to find and index. When we go live with the map we’ll just need to add a link to this page to the site menu too. Here’s a screenshot of the new page:
With the new page in place (but not yet live) I then returned to the other place-name sites that I’ve also developed the new map interface for (Ayrshire, Kirkcudbrightshire and Nairnshire) to add the new textual browse page to them too. I also took the opportunity to apply some recent updates to the maps for these resources, such as adding in the new ‘help and about’ pop-up.
I then moved onto setting up the new map interface for the Berwickshire project. This was the first large-scale place-names project I was involved with, and it launched back in 2018. Thankfully it didn’t prove too tricky to set up the new interface for the project, as all subsequent place-names resources are based on the same architecture as this site. I did need to set up a new version of the API, and the new map-based resource currently doesn’t have the map icons of the existing Berwickshire map, as we dropped these from all subsequent projects. There is also an issue when categorising the map markers for the full dataset by element language, as there are so many combinations of elements and not enough colours to apply to these. I’ll need to further investigate this, but on the whole the new interface is working pretty well, and also incorporates the LiDAR data, as the following screenshot demonstrates:
On Thursday I had a meeting with Matthew Creasy to discuss a potential new project about the poems of Mallarmé. We had a good chat and afterwards I spent quite a bit of time writing up my notes and thinking about how the resource would work. I sent the resulting document to Matthew and I’ll just need to see whether anything comes of this.
This took up quite a bit of time on Friday, and I spent the remainder in discussions with various people to get the required space for Deven Parker’s 18th century playbills project. It’s looking like this might be possible now and I’ll be following this up next week.