I spent pretty much all of this week working on the new versioning system for the Books and Borrowing project, which I have now managed to complete. Last week I made some good progress on the redevelopment of the API to enable data queries to be limited by version, and this week I was able to complete the work. With a fully working new API running on my laptop, I considered updating the live system as the next task, but decided against this. Updating the live system would have meant regenerating all of the data, and I realised that this would mean the new ‘version 1.1’ data that is already being worked on in the CMS would therefore be fully integrated into the front-end. This is something we don’t want to happen yet as Matt and Katie are still working with the data and need it to reflect the state it was in at the official end of the project.
Instead, I began the process of creating a new version of the front-end that would connect to the new API and would be able to target and display specific versions of the data. I decided to track the selected version throughout the front-end by means of both session variables and URLs. Session variables on their own could be used to track the chosen version, but we also needed to include the version number in the URL in order to enable users to cite / share / bookmark specific versions. This meant that every single URL that is generated throughout the site needed to incorporate the version number, including WordPress navigation menu items and other WordPress pages.
Adding in session tracking throughout WordPress brought some difficulties. My session data was somehow getting overwritten as I moved between pages, with WordPress session data getting added in and replacing some of the variables I had set. This took some time to get to the bottom of, and it turned out that WordPress’s emoji system was causing the problems. I found a guide to disabling this system here: https://smartwp.com/disable-emojis-wordpress/ and once I’d followed these steps my session variables worked as I’d intended.
For most users the front-end will not appear to be any different. It is now set up to default to the most recently published version of the data (including all earlier versions too) and none of the page URLs have changed. The only thing that is different is the ‘cite’ popup, which now references the current version. It also includes a link to this version in the URLs that appear, as the following screenshot demonstrates:
This is to ensure that the citation will always reference the version of the data the person cited. If we didn’t include the current live version in the citation URL then the link may lead to different results if/when we publish a new version of the data.
I also updated the footer of every page to include the current live version of the data, as you can see in the following screenshot:
I also included a link in the footer to the new ‘Versions and changes page, which you can see below:
This still has some placeholder text and then lists all of the live versions of the data, including the version number, a description and the date of publication. One version will always be selected (in blue and with a tick). If you press on ‘Version 1.0’ this then sets the entire site to use this version of the data – everything including lists of libraries, facts and figures, registers, pages, borrowing records, search results and on this day items will be limited to data created as part of this version (or earlier, if applicable).
When an earlier version is selected, every page of the site has the version number in the URL, for example ‘/libraries/version-1.0’ and the selected version is tracked throughout the site. To make it obvious to users that they are not looking at the most recent version of the database a warning is displayed on the screen, as shown below:
As with the ‘top’ link this box stays fixed in the bottom corner of the screen. Pressing the ‘return to the most recent version’ button takes you to the homepage with the most recent version reinstated.
Independently of all of this, I’ve also updated the search results to add in a version filter, as shown in the following screenshot:
So for example, if you’re viewing the current version of the data (1.1) you can see how many borrowing records were created as part of this version as opposed to the earlier version, and filter the results if you wish. Note that if you’ve set the site to use version 1.0 of the data then the 1.1 search results wouldn’t display and the 1.1 filter option won’t show up.
It’s taken a huge amount of effort to get all of this in place, updating pretty much every part of the Books and Borrowing system, but it’s now thankfully all in place. The only thing left to do is to actually add all of this to the live site, which I was intending to do next week. However, Matt and Katie are still doing research using the live site at the moment and I therefore need to hold off until they have completed this work. This is because whilst I work on the updates the live site will temporarily include data from all versions, which will obviously affect the data they extract. I’m probably going to have to hold off going live with the updates until August, as I’ll be on holiday and then at a conference between now and then.
Also this week I attended the launch of the Iona place-names resource on Monday evening, and it was great to see the resource (see https://iona-placenames.glasgow.ac.uk/map) go live after working on it over the past few years. The Berwickshire place-names team also launched their published volume at the event, which was also great to be a part of.
Also this week we finalised the archiving of the old Cognitive Toponymy website, and the archived site is now available here: https://conifer.rhizome.org/UoGLibrary/cogtop/20250430094955/https://digital-humanities.glasgow.ac.uk/cogtop/
This week finally saw the launch of one of the new ‘map first’ interfaces for the various place-names projects I’ve been working on. I initially created the interface for the Iona place-name project and then applied it to Ayrshire, Kirkcudbrightshire, Berwickshire and Nairnshire, but none of the projects were quite ready to publicly launch the new interface. After having a char with Carole Hough last week I got the go-ahead to go live with the new Berwickshire map, and it can now be found here: https://berwickshire-placenames.glasgow.ac.uk/map with the textual list of Berwickshire place-names here: https://berwickshire-placenames.glasgow.ac.uk/list-of-berwickshire-placenames/.
It’s great to have the new interface publicly available at last, although I did have a number of last-minute tweaks to make as I worked on it. I’ve updated the legend slightly, adding in some explanatory text and ensuring that the text is hidden when the ‘hide legend’ button is pressed – previously the text remained visible, which was a bit pointless. I also spotted that the legend was not being set to hidden be default on narrow screens, even though it should have been, so I fixed this. Another issue I spotted was that the tooltips that display when the legend categorisation is by language would stop working whenever the ‘select all’ option was pressed, and I needed to update the code to ensure the tooltips were reinitialised. With these changes in place I also needed to apply them to the other place-names maps to keep things consistent.
Launching the new map wasn’t the end of the work, however. Berwickshire already had a live map resource, plus search and textual browse facilities. I needed to ensure that the quick search on the main resource website connected through to the new map’s search results and that all links / bookmarks / citations to the old interface redirected through to the appropriate section of the new interface, where possible. So for example, the old element glossary page redirects to the new element glossary popup in the new map interface: https://berwickshire-placenames.glasgow.ac.uk/place-names/?p=element-glossary. I think I’ve caught all of the old links and have ensured the necessary redirects work.
Also this week I had a meeting with Thomas and Sofia regarding the map interface for Iona, which will apparently be publicly launched on the 9th of June. We discussed some of the outstanding tasks they would like to see completed before the launch and I’m going to have a fair amount to do before then. For example, they would like the element glossary to have a ‘jump to letter’ feature, allowing the user to immediately scroll to elements beginning with a selected letter. We also discussed new ‘thematic maps’ based on static maps Sofia had created for a recent event on Iona. These maps showed place-names grouped in more fine-grained ways that are not covered by the classification options we currently have available – for example place-names featuring animal names. I suggested that I could make an interface where such thematic maps could be created in the content management system and displayed in a new menu section in the front end. This would consist of a title and description for the map, and a list of markers that should appear on the map.
I also demonstrated the ‘story map’ interface I’d created for the Scots Syntax Atlas (go here and select ‘Stories behind the examples’: https://scotssyntaxatlas.ac.uk/atlas/). This feature consists of a series slides a user can navigate through, each of which can have a different view of the map, which may feature different data and different zoom level, thus guiding the user through a particular story the map tells. I gave an example of how this could be used for iona to have a story of place-names featuring animals, with different slides showing markers featuring elements from different languages. Thomas and Sofia really liked this idea and would like to implement it, but while it might be possible for me to implement the simpler thematic maps before the 9th of June, it’s unlikely that I’d be able to get the ‘story’ approach in place. It’s something we’ll probably consider after the launch.
Continuing with place-names, I had an email conversation with Alasdair Whyte regarding his Mull / Ulva place-name data. He’s currently working towards a published volume and wanted me to add a facility to the content management system to enable him to categorise place-names by volume. I therefore updated the CMS to add in the volume field, and this now appears in the ‘add’ and ‘edit’ place forms. I also updated the ‘Browse’ page to add in volume to the search options and as a table heading for the listed data and updated the ‘Export place-name data for publication’ page in ‘Tools’ to add in a volume selection option. Thomas also contacted me this week to ask for an update to the ‘Export’ facility for Iona (adding in the ‘translation’ field) so I did this too. I also had a chat with Alasdair about the front-end for his Mull / Ulva data, as this will need to be a bit different from the other resources.
Also this week I finally managed to get my ticket sorted for the Digital Humanities event in Glasgow in June, with the help of Emma McCluskey. I also made some updates to my presentation for the DH event in Lisbon based on some very helpful feedback from Jennifer, and also added in some new publications to the Scots Syntax Atlas resource.
I also spent a bit of time working on the Dictionaries of the Scots Language. I’d been contacted last week by editor Vasilis Karaiskos about the regions data that he has been working on, and the new maps he has been developing to display these regions. I spent some time going through the interface he had developed as a proof of concept, figuring out how it all works and giving some feedback. It will be really great to get more information about the geographical regions into the website and the maps are an excellent starting point. We’re hopefully going to have an online call in the next few weeks to discuss things further.
The rest of my week was devoted to working on the new versioning system for the Books and Borrowing project. I need to update the structure of the Solr index and the other cache files, and also the scripts that generate this data, and this is my next step. As I feared, it’s proving to be an awfully tricky and time-consuming update to implement, and while I did make progress there is still a huger amount to do. I’ll be continuing with this monumental task next week.
On Monday this week I attended a ‘getting started with Transkribus’ workshop held at the University of Edinburgh. This was a really great opportunity to learn more about the Transkribus tool and to be guided through the initial stages of using it. I had somewhat naively thought that it would be possible to just upload a digitised image to the tool and it would then be able to extract and format the text with a high degree of accuracy, but in reality much more manual intervention is required before Transkribus can work effectively. Training the tool on your data is absolutely vital, and this means uploading sample images, identifying lines of text (which can be done automatically but may need manual tweaking) and then transcribing the text found on each line using Transkribus’s web-based text editor. Text can also be tagged using this facility, such as noting headings, or identifying proper names. The leader of the session suggested that around 15,000 words of text should be manually transcribed in order for the tool to be well trained, which would take a considerable amount of time and effort.
I wanted to see whether Transkribus could extract the text from the Edinburgh Gazetteer (https://edinburghgazetteer.glasgow.ac.uk/the-gazetteer/) and the first stage of this would be for the tool to recognised that each page is divided into four columns, many with a full-width heading at the top. A ‘layout’ model would need to be run on the pages in order to automatically identify the columns. There are a couple of existing newspaper layout models available but unfortunately when I ran my test image through them neither managed to divide the page, and the entire page was simply identified as ‘Region 1’, meaning all identified lines ran across all four columns. Any human looking at the image would be able to ascertain immediately that it’s four columns of text in so I was a bit disappointed that Transkribus wasn’t able to do the same. I spoke to the leader of the workshop, who has successfully used Transkribus on another historical newspaper and he suggested that I’d have to train Transkribus on this specific newspaper and that in order to do so I’d need to use a ‘Field model’. It’s not entirely clear to me how a ‘Field’ differs from a ‘Layout’ model, but unfortunately it’s not possible to create a field model with the free Transkribus account – you instead need to subscribe to a ‘Scholar’ account at €20 a month, something I didn’t want to do as I was just experimenting at the workshop, so I reached something of a dead end. Having said that it was still very useful to experiment with Transkribus in a workshop setting with an expert on hand and I will continue these experiments when I have more time available. However, I do wonder how more general AI tools might cope with the same materials and whether they might be quicker to use. It’s definitely something that would be worth exploring further.
For the rest of the week I worked on several different projects. I spent some time refamiliarising myself with the structure of the Bilingual Thesaurus of Everyday Life in Medieval England (https://thesaurus.ac.uk/bth/) as a developer who will be working with the structure for a new project got in touch with questions about it. I wrote a data description document that describes every table and column in the database, and also contains the highest ID in each table used for BTH. There are several columns that are only present as references to the original spreadsheet and are not used by the online resource, and I’ve made a note of these. Whilst working on this I also spotted an issue with the display of languages in the advanced search. These were not displaying any content and it turns out that this was because one of the referenced column names was ‘grouping’, which is a reserved word in MySQL. While the queries worked on our old server, the new server is more strict about such things and the queries failed to run on it. Having spotted this it was thankfully quick to fix – surrounding the column name with backticks (`grouping`) in the queries got things working again.
I also returned to working on the Books and Borrowing project for a few hours this week in order to update the links we have in our system to books held at St Andrews University library. The library has recently changed their online system, which has broken all existing links to records in the system. Thankfully they have a huge spreadsheet featuring the old identifiers and the corresponding new ones and I was able to write a little script that would identify the St Andrews book holding records in our system and update their links. Of the 7560 active St Andrews book holding records in our system 1666 have URLs and my script successfully updated 1662 of these. The remaining four have issues that will need to be looked at separately, and I’ve let Katie and Matt know about these.
I met briefly with Alan Riach this week to discuss the website I created for him last week. He gave me a further CD featuring a recording he wanted me to host and I was able to extract the recording and update the website to add this in. I also joined my fellow College of Arts and Humanities developers for a coffee and a catch-up this week, and it was really great to chat with them all again. Hopefully we’ll meet up again soon.
On Thursday this week I also had a meeting with Rhona Brown, Amy Wilcockson and Clare Thompson from the Mitchell Library to discuss an online exhibition for the Robert Fergusson website (https://robert-fergusson.glasgow.ac.uk/). There was a physical exhibition of materials at the Mitchell last year and we’re going to be bringing these together online in the next few months. It was good to meet everyone in person and hopefully I’ll be able to help them create an exciting online resource based on the materials.
Also this week I managed to fix the issue with capital letters not working in the ‘jump to’ option in the ‘browse’ facility of the Anglo-Norman Dictionary. You can now enter text and it will find matching entries whether they have capitals or not. E.g. enter ‘abc’ (or ‘ABC’) into the ‘jump to’ box. Brackets are also not an issue, e.g. type in ‘xilo’ and you’ll jump to ‘[xiloaloe]’. The only thing to bear in mind is that the browse is accent sensitive. It has to be as we have entries with headwords that are identical other than for accents. So for example entering ‘acel’ will jump to ‘acelément’ but typing in ‘acele’ will find nothing. However, ‘acelé’ will work. Updating this would be a major undertaking so hopefully it’s not too big a deal – it’s been like this since the new site launched, after all. I also applied the updated I’d made last week to the Iona map interface to all of the other place-names projects.
The remainder of my week was spent going through the feedback I’d been sent from testing of the new data for the Dictionaries of the Scots Language. This included fixing an issue when conducting a search featuring an apostrophe in iOS using the Safari browser. This was adding an a curly apostrophe which then didn’t work with our data. I updated the code to ensure such apostrophes are switched for regular ones when submitted, but when working on this I noted that there are some occurrences of curly apostrophes in the dictionary’s XML files. These characters are treated as entirely different from regular apostrophes and will not be found in a search that contains a regular apostrophe. This led to some discussions with the team about the use of such characters in the data. I also updated the error messages that are displayed when invalid dates are entered in the quotation date filter and spent a long time working through the sparkline data generation code to ensure certain edge cases are resolved.
It’s been a long time since I wrote the sparkline code (I began in June 2023!) and it took some time to refamiliarize myself with the it, but I managed to sort all of the issues you had been reported. Unfortunately these fixes needed to be made to the code that generates the sparkline data that then forms part of the Solr index so no updates were initially available on our test server. Instead I ran many tests on my laptop, both before regenerating the Solr data by targeting specific entries and also through a test instance of Solr and the DSL site running on my laptop after regenerating the entire dataset and as far as I can tell all is working without any unintended consequences being introduced. I then asked our IT people to update the online Solr cores and this was completed by the end of the week.
I also tweaked the display of the sparkline dates in the search results on narrow screens. I’ve updated the layout so that the sparkline text and the sparkline are treated as one single block element. If the page is too narrow for both this and the entry title to all fit on one line then the sparkline and its text drops down to a new line rather than different sections being split over different lines. I also updated the CSV output to remove the search result snippets from them and had a chat about the tagging of certain bibliography entries in the data.
I divided my time between many different projects this week. For the Anglo-Norman Dictionary I made a few further tweaks to the new cognate references section that we went live with last week, and I also received a new batch of several hundred new or reworked entries that are to be added to the dictionary, which I’ll process next week. For the Dictionaries of the Scots Language I had an email conversation with editor Ann Ferguson about the structure of URLs and also gave some consideration to the bibliography search, based on feedback I received towards the end of the week. I’ll continue to work on this next week.
I also found a bit of time this week to begin work on a new version of the Historical Thesaurus website. The current website’s backend is a bit of a mess, as it was cobbled together over many years as new features were added. What I’ve been meaning to do for a while now was to rework this in order to extract all data querying and processing code from the front-end, separate it out and develop a single, unified API for the site, an approach I’ve taken with all of the large projects I’ve been involved with for many years now. This is going to be a rather large task, but I made a start this week: I set up a new version of the website and began the process of creating the API. So far I’ve only got the ‘random category’ feature on the homepage working with the new API, but I’ll continue working away at this as time allows until it’s all done. There won’t be any change to the front-end (although I’m also separately considering a Bootstrap-based refresh of the interface), but the update will leave the data querying side of things in much better shape for the future.
For the VARICS project I continued to work on the new look-up feature. Following last week’s traffic light experiments the team requested a more simple approach for the traffic lights, so I created new images featuring a subtle gradient and a white border, as you can see below:
I also added in a new ‘Try again’ button that would reload the form, which was subsequently renamed ‘Start again’. The team prefer this simpler approach so it looks like this is what we’ll be going with. I now need to wait until further data is sent to me before I continue to develop the feature.
For the place-names projects, I spent some time investigating an issue with the advanced search in the new map interface. An element language search was not working unless a specific element was also selected. For example, a search for all place-names containing Old English elements wouldn’t run, but selecting Old English and then selecting the element ‘tūn (OE)’ worked perfectly. It took a while to track down the issue, but thankfully once identified fixing it was trivial: there was a typo in the code for the API. The system was already set up to allow a search for element language without specifying an element, but in such circumstances the code failed to run because of the typo. I’ve fixed this now on all of the place-names sites. I also fixed a strange issue with the advanced search pop-up disappearing when parishes towards the end of the list were selected. This issue was only occurring in Chrome and not in Firefox (my default browser) and an update to the stylesheet for the map pop-up thankfully fixed the issue.
I also participated in an online meeting with the Iona project team on Wednesday this week where we discussed some of the map-related issues that had been brought up at the advisory board meeting and how (or if) we should address these. I ended up with a list of updates that I’ll need to make to the interface, another thing I’m hoping to tackle next week.
Also this week I had a brief in-person meeting with Joanna Kopaczyk and her project RA Pia Lehecka to discuss Joanna’s new DOST digitisation project. It was good to meet the other members of the team and I’m looking forward to working for the project over the next few months.
I also met with Alan Riach in Scottish Literature this week. Jean Anderson had contacted me last week to put me in touch with Alan, who was wanting some help to publish some old radio broadcasts that were previously available on one of our old servers that has since been decommissioned. Alan managed to find some audio CDs containing the recordings and I met with him to collect them, together with some additional recordings and a few DVDs. I spent some time towards the end of the week extracting the audio and video data from this media and creating a simple website where the files and accompanying transcripts will be accessible to students. I haven’t quite finished this task yet, but should have an initial version completed early next week. The resource will be password protected and only available to students and staff, as there may be licensing issues relating to the recordings – for example relating to incidental music used during the recordings.
Much of Monday morning this week was spent working with Andrew McHugh and Luca Guariento to figure out why a large number of our externally hosted websites were displaying nothing but server errors. This took some time to investigate and it turned out that one node was being affected, and this was apparently being caused by over-enthusiastic web crawlers targeting one script repeatedly, resulting in CPU resources getting used up. Once identified, we limited access to the script and everything went back to normal.
I spent a lot of this week continuing with the cognate reference updates for the Anglo-Norman Dictionary. Cognate references are links from AND entries to related entries in other dictionaries and we’re increasing the range of dictionaries that we provide active links to, as opposed to providing plain text references to. As part of this I needed to write a script to update the references to DEAF (Dictionnaire étymologique de l’ancien français) that were embedded in our entries. Last week I wrote a script that identified issues with DEAF references involving sub-headwords in brackets in 295 AND entries and we needed to update the structure of these as discussed last week. The editor Geert sent me an edited version of the spreadsheet that contained the forms that should be included and I wrote a script to extract these, generate the required XML and then find and replace the relevant section of the XML in the live entries. This obviously required quite a lot of testing before I ran it on the live data, but I’d completed and fully tested the script by the end of Tuesday and I successfully ran it on Wednesday morning.
At this point the increased selection of cognate references that we link to were still only available on a test page rather than the live site. I then moved onto updating the layout of the section in the entry page to make it easier to use. Previously the section looked something like this:
It was quite difficult to understand the section: There was no explanation as to what the section was, or what acronyms like ‘FEW’ actually meant. It was also difficult to tell where one cognate reference section ended and the next began. Geert asked me to come up with some new ideas for the section and I spent a bit of time working on a few mock-ups. We finally agreed on a new version that gets rid of the square brackets around the entire section and divides the individual dictionaries into inline-block elements that have a border and a light grey background, thus making it quicker and easier to identify the different dictionaries. I also swapped the order of the headwords and the reference numbers and make all text normal weight. Finally, I added in an ’info’ icon to the top-right that when pressed on opens up a popup with some explanatory text about the section and the dictionaries that are referenced. Here’s how the new section looks:
We went live with this new layout on Friday afternoon and so far it has been very well received. Also for the AND this week I updated the cognate references that could be actively linked to from the entry commentaries to bring these into line with the new cognate reference section.
Also this week I worked on the layout for a new ‘Scots Word of the Week’ page with William Ashford of the Dictionaries of the Scots Language. After a number of experiments I created a layout that suited, and the page is now live: https://dsl.ac.uk/scots-word-of-the-week/. I also fixed an issue with embedded videos being too large for mobile devices on the site and responded to a query from Ann Ferguson about the data import workflow and how URLs are generated.
I also spent some time updating the ‘lookup’ feature for the VARICS project, based on feedback that I’d been sent last week. I added in the proper measure labels and I updated the system so that each measure has a ‘sex’ column. The option to select a sex is only displayed when this is set to ‘Yes’. For our working measures (the maximum phonation ones in the ‘Phonation’ menu) sex is now set to ‘No’, but the other measures still have it set to ‘Yes’. This can easily be updated as more real data becomes available. I also replaced the ‘age’ input with a drop-down list of options and have dropped the ‘red’ rating for now, as suggested by the team. I also removed the ‘alternative view’ that featured all three traffic lights, plus the ‘additional information’ section and updated the output text as requested and limited the display of all numbers to 2 decimal places.
I’ve also spent a bit of time trying to find alternative ‘cartoon/clipart’ traffic light images. Finding ones that are both free to use and have a suitable license has proved to be quite a challenge. Eventually I thought I’d give AI tools a try, and decided to try Microsoft’s image creator (https://designer.microsoft.com/image-creator) as I was already signed into my Microsoft account. The results were somewhat mixed and some rather odd images were created – e.g. traffic lights with four lights, traffic lights with odd symbols on them. Attempting to specify a UK style traffic light gave an image that featured the weirdest attempt at the Union Jack I’ve ever seen. It also totally ignored my request for the traffic light to be against a white background. Here’s a screenshot of the output:
However, I managed to pick out one generated image I thought I could use and I’ve split it up into individual images. I’ve now replaced the images in the lookup with these ones. This is by no means the final version and it’s likely that we’ll change the image again before the feature is completed, but here is now it currently looks:
I’d received some helpful feedback from Eila Williamson of the Berwickshire Place-names project regarding the new map interface I’ve been developing and made some tweaks to the interface based on this. Eila also spotted that there is an issue with the element language part of the advanced search facility. At the moment selecting an element language in the form simply limits the choices that then appear in the ‘Element’ field to your chosen language – e.g. select ‘Old English’ and type in ‘tu’ and only ‘tūn (OE)’ appears. It doesn’t currently let you select an element language and then view all place-names containing it. I’m afraid this is an oversight on my part, as the existing advanced search does indeed let you do this. It’s going to be a pretty major job to update the search to allow this, but it’s something that will need addressed, not just on the Berwickshire site but in all of the place-names sites. I had hoped to have the time to tackle the issue this week but I ran out of time.
Also this week I had an email conversation with Joanna Kopaczyk about her DOST digitisation project that will be commencing next week, I attended a meeting about the teaching of Digital Humanities at Glasgow, I responded to a query from The National Archives about the hosting of the OHOS project, I did a bit of research into some old files that Jean Anderson contacted me about and I attended a meeting in Special Collections for Matthew Creasy’s Mallarmé project.
I divided most of my time this week between three projects: The Anglo-Norman Dictionary, the Dictionaries of the Scots Language and the redevelopment of the Berwickshire place-names resource. For the AND I continued to work on the new links from entries to references in other online dictionaries – the ‘cognate references’. Last week I’d created a test version of the entry page that included links out to listings about the entry on many more online resources and Geert, the editor of the AND tested it out and got back to me with some suggestions.
There was an issues in some entries that link to entries in the DEAF resource, specifically where there are both headwords and multiple sub-headwords in brackets. For example ‘Escharn’ (https://anglo-norman.net/entry/escharn) has the DEAF headword ‘escharnir (eschar, escharne)’. My new linking system doesn’t render such links correctly, and in fact this reference in the entry XML should have been split into two separate DEAF links, one for ‘escharnir (eschar)’ and one for ‘escharnir (escharne)’ as follows:
<DEAF_refs siglum="DEAF" linkable="no"> <link_form>escharnir (eschar)</link_form><link_loc/> <link_form>escharnir (escharne)</link_form><link_loc/> </DEAF_refs>
I therefore wrote a little script to find all entries that feature a DEAF headword that contained a comma or a semi-colon, outputting the entry slug and DEAF headword in a spreadsheet. There are 295 such entries, and some have a lot of different versions in the brackets. We agreed that some manual checking of these will need to be done as a few I checked don’t seem to exist on the DEAF site. E.g. ‘Message#1’ links to ‘metre 2’ (https://deaf.hadw-bw.de/lemme/metre2) with subheadwords ‘message 1’ and ‘message 2’ but neither of these appear in the DEAF entry (‘message’ doesn’t appear in the text of the page at all). Similarly the entry ‘perpetuelment’ links through to ‘perpetüel’, but while the second subheadword in brackets works (https://deaf.hadw-bw.de/lemme/perpet%C3%BCel#perpet%C3%BCellement) the first doesn’t.
We agreed that Geert would use my spreadsheet as a starting point you would add the correct forms one per column from column C onwards. I would then be able to run a script to update the existing DEAF item in the XML with multiple links. Geert worked on the spreadsheet during the week and I’ll write a script to process it next week. Geert also requested a spreadsheet listing all entries that don’t feature a DEAF link as these will probably need some further work. There are 10,831 such entries.
For the DSL I spent some time liaising with the DSL’s email supplier and our IT Services to update some DSN entries. I also removed the results filtering options from the search results of our test instance of the site as the team want to delay the rollout of this feature. I was also alerted to an issue with the new data that was added to the test instance before Christmas. Items that should have been deleted from the data in the recent update were still appearing in the search results, something that shouldn’t have been happening as I’d updated the Solr index to wipe all data prior to the recent upload.
After some communication with our IT people, it turned out that the online Solr index does not match the Solr index I have on my laptop. In the run-up to Christmas I realised I needed to completely wipe the Solr index prior to uploading the new dataset in order to clear out records for any entries that had been deleted from the dataset. I ran some tests on my local instance of Solr running on my laptop to empty and then import the data and all went fine. I don’t have direct access to Solr on our server and I have to ask our IT people to run any commands for me, so I contacted them and asked them to run the delete and import commands. It looks like the delete command (which worked on my laptop) didn’t work on the server, despite the command being exactly the same as the one I successfully ran on my laptop. We re-ran it on the server again and it failed to delete anything. However, after restarting Solr the data was cleared, so I’ll just need to bear this in mind the next time. After deleting all of the data from the indexes we re-uploaded the last dataset from December and thankfully that this has fixed all of the issues. Whilst investigating this I also noticed that some entries have URLs in their XML that won’t work in the front-end due to them having characters that don’t work in URLs such as full stops, commas and spaces. Thankfully there are only a handful of these and I’ve passed them onto the team who will update the XML.
For the new Berwickshire map interface I added in some explanatory text, fixed a couple of typos and made some other minor tweaks to the interface. I received some further feedback from another member of the team on Friday and I’ll need to make some further tweaks next week, but we’re almost ready to launch the new interface now.
In addition to the above I arranged my travel and accommodation for the DH2025 conference in the summer. I also fixed an issue with the library stats pages on the Books and Borrowing site. Unfortunately a bug must have been introduced when I reworked the facts and figures page in October last year. Prior to this each library had its own ‘top ten’ list and this meant that when viewing facts for multiple libraries only items in each top ten were being included (so if Book A at Library X appeared at position 9 but it was at position 11 in Library Y then the latter wasn’t being included in the figures). To get around this I updated the individual library lists to return the full data, then when multiple libraries were selected on the site-wide ‘Facts’ page the system could do a proper count and generate an accurate the overall top tens. What I failed to spot is that where a single library is selected (either on the site-wide ‘Facts’ page or the library’s own page) the list wasn’t being limited to ten, as in these views the code expected there to only be 10 items returned in the dataset and was just displaying all items. I’ve fixed this now. I also created screenshots for our article about the development of the resource for the Victorian’s Institute journal.
Things are looking promising for the hosting of the 18th Century Playbills project, following some email discussions towards the end of last week and a Teams call I had with Andrew Nicolson from IT Services and Luca on Monday this week. The project will require a lot of storage space (hundreds of gigabytes) and a IIIF server and it now looks like these will be available to us. It’s likely that we will have access to a Cantaloupe IIIF server that will be centrally hosted and connected to storage for our images, which will be ideal for us.
Other than making these arrangements, I spent most of the beginning of the week testing the new interface for the Berwickshire place-names resource and fixing issues that arose. This included updating the way in which map markers are coloured when categorised by element language as previously there were too many combinations for the available colours. I’ve updated the colour schemes to use different shades for specific languages. For example, all SSE categories are shades of blue, all Gaelic categories are shades of purple. It can be difficult to differentiate the shades due to the large number of categories, but you can turn a layer on or off using the legend to quickly figure out which markers belong to which group. I’ve also ensured the ‘loading’ spinner gets displayed when the quick search is reset, as this loads in the full dataset, which takes a bit of time, and without displaying the spinner it looked like the map had broken. I’ve applied this to the other place-name maps too. I made many more minor tweaks and updates to all of the resources during my period of testing. I’m still not altogether happy with how long it takes to load the default view of the data (i.e. all place-names), which can take several seconds. Next week I may investigate how I can speed up this process, but other than this (and ensuring redirects from pages of the old resource are in place) I am ready to launch the new resource, and have sent the URL to project PI Carole Hough for feedback. Hopefully we’ll be able to launch the new resource in the coming weeks.
On Thursday this week I was due to lead a session at the DSL’s annual staff meeting about the new interface mockups I’d created for the site a few months ago. I therefore spent some time on Wednesday refamiliarising myself with the mockups and why I’d taken particular approaches. I then created a runthrough of the mockups and practised going through it a few times. However, on Wednesday evening I was informed that the meeting had been cancelled due to the storm that was due to hit Scotland on Friday morning (many attendees at the meeting were due to travel home on the Friday). This was a shame, but for the best and I hope we can rearrange the meeting for another time.
Instead of attending the DSL meeting on Thursday I spent part of the day in discussions to ascertain the status of Transkribus at Glasgow. I’m involved in a project with Joanna Kopaczyk that’s due to start soon and will be digitising and extracting text from the original materials compiled for the Dictionary of the Older Scots Tongue. Joanna and I had both heard (from different people) that Glasgow has an institutional license for Transkribus, but when investigating this I’ve been unable to find any details of such a license. This is concerning as the project was funded on the assumption that such an agreement was in place, and if we have to pay per page we’re potentially going to run into difficulties. This is especially true as in the past month Transkribus has revised their payment plans, and have halved the number of pages that can be processed for free each month, from 100 to 50. Joanna is going to investigate this further and hopefully some further details can be found.
I spent the rest of the week working for the Anglo-Norman Dictionary, replacing Twitter/X with BlueSky on the website and updating the entry page to ensure more of the cognate references (the references to similar entries in other resources) actually link through to online resources. I created a test version of the entry page so nothing is live yet, but I’ve managed to add in links to Gdf, GdfC, DEAF and DMLBS.
This has involved rewriting sections of the XSLT script that processes entries for display and the JavaScript that reworks the links once the entry has been loaded into the browser. The XSLT was previously set to only process cogrefs that had their ‘linkable’ attribute set to ‘yes’ in the XML, e.g.
<FEW_refs siglum="FEW" linkable="yes"> <link_form>discipulus</link_form><link_loc>3,92a</link_loc> </FEW_refs>
I’ve had to change this to ensure the previously unlinked cogrefs do now get linked, e.g.
<Gdf_refs siglum="Gdf" linkable="no"> <link_form>disciple</link_form><link_loc>2,718a</link_loc> </Gdf_refs>
The XSLT now no longer looks at the ‘linkable’ attribute at all, as otherwise we’d have to update every entry XML file to set linkable to ‘yes’ for every cogref like the one above. Instead the XSLT will now attempt to add in a link for all cogrefs other than ‘TL’ (the only one that is not found online), or will display ‘∅’ if there is no matching cogref entry. It is slightly unsatisfying to have an attribute that no longer gets used and we should perhaps consider doing something about this, but the issue doesn’t have a bearing on how the site functions.
For Gdf the links use the numerals where possible, which means removing non-numeric characters (e.g. ‘a’ from 718a) and subtracting one from the page number. E.g. ‘Gdf: 2,718a’ gives the link https://micmap.org/dicfro/next/dictionnaire-godefroy/717/2. Where it’s not possible to use the ‘location’ like this then the headword is used, stripped of any text in brackets. GdfC is handled in the same way as Gdf, but the page number doesn’t need altered.
For DEAF headword text is used in the link. If there is any text in brackets this is extracted and placed after a hash, for example https://deaf.hadw-bw.de/lemme/paele#paelete. Any spaces in the headword are removed so that numerals work (e.g. https://deaf.hadw-bw.de/lemme/nef2). If there are any DEAF headwords that legitimately have spaces we’ll need to consider how to deal with these.
For DMLBS the headword stripped of any bracketted text is used. Unfortunately it’s not possible (as far as I can tell) to link specifically to the DMLBS tab on the Logeion website as their website has no way to reference a specific tab, which is a shame.
Once the AND editor Geert gets back to me I’ll be able to go live with this update, which I’m sure will be very useful to a lot of users.
This was my first week back after the Christmas holidays and after catching up with emails I spent the best part of two days fixing the content management system of one of the resources that had been migrated during the end of last year. The Saints Places resource (https://saintsplaces.gla.ac.uk/) is not one I created but I’ve taken on responsibility for it due to my involvement with other place-names resources. The front-end was migrated by Luca and was working perfectly, but he hadn’t touched the CMS, which is understandable given that the project launched more than ten years ago. However, I was contacted during the holidays by one of the project team who said that the resource is still regularly updated and therefore I needed to get the CMS up and running again. This required updates to database query calls and session management and it took quite some time to update and test everything. I also lost an hour or so with a script that was failing to initiate a session, even though the session start code looked identical to other scripts that worked. It turned out that this was due to the character encoding of the script, which had been set to UTF-8 BOM, which meant that hidden characters were being outputted to the browser by PHP before the session was instantiated, which then made the session fail. Thankfully once I realised this it was straightforward to convert the script from UTF-8 BOM to regular UTF-8, which solved the problem.
With this unexpected task out of the way I then returned to my work on the new map interface for the Place-names of Iona project, working through the ‘to do’ list I’d created after our last project meeting just before Christmas. I updated the map legend filter list to add in a ‘select all’ option. This took some time to implement but I think it will be really useful. You can now deselect the ‘select all’ to be left with an empty map, allowing you to start adding in the data you’re interested in rather than having to manually remove all of the uninteresting categories. You can also reselect ‘select all’ to add everything back in again.
I did a bit of work on the altitude search, making it possible to search for an altitude of zero (either on its own or with a range starting at zero such as ‘0-10’). This was not previously working as zero was being treated as empty, meaning the search didn’t run. I’ve also fixed an issue with the display of place-names with a zero altitude – previously these displayed an altitude of ‘nullm’ but they now display ‘0m’. I also updated the altitude filter groups to make them more fine-grained and updated the colours to make them more varied rather than the shades of green we previously had. Now 0-24m is a sandy yellow, 25-49, is light green, 50-74m is dark green, 75-99 is brown and anything over 99 is dark grey (currently no matching data).
I also made the satellite view the default map tileset, with the previous default moved to third in the list and labelled ‘Relief’. This proved to be trickier to update than I thought it would be (e.g. pressing the ‘reset map’ button was still loading the old default even though it shouldn’t have) but I managed to get it sorted. I also updated the map popups so they have a white background and a blue header to match the look of the full record and removed all references to Landranger maps in the popup as these were not relevant. Below is a screenshot showing these changes:
I then moved onto the development of the elements glossary, which I completed this week. This can now be accessed from the ‘Element glossary’ menu item and opens in a pop-up the same as the advanced search and the record. By default elements across all languages are loaded but you can select a specific language from the drop-down list. It’s also possible to cite or bookmark a specific view of the glossary, which will load the map with the glossary open at the required place.
I’ve tried to make better use of space than similar pages on the old place-names sites by using three columns. The place-name elements are links and pressing on one performs a search for the element in question. I also updated the full record popup to link the elements listed in it to the search results. I had intended to link to the glossary rather than the search results, which is what happens in the other place-names sites, but I thought it would be more useful and less confusing to link directly to the search results instead. Below is a screenshot showing the glossary open and displaying elements in Scottish Standard English:
I also think I’ve sorted out the issue with in-record links not working as they should in Chrome and other issues involving bar characters. I’ve done quite a bit of testing with Chrome and all seems fine to me, but I’ll need to wait an see if other members of the team encounter any issues. I also added in the ‘translation’ field to the popup and full record, although there are only a few records that currently have this field populated, relabelled the historical OS maps and fixed a bug in the CMS that was resulting in multiple ampersands being generated when an ampersand was used in certain fields.
My final update for the project this week was to change the historical forms in the full record to hide the source information by default. You now need to press a ‘show sources’ checkbox above the historical forms to turn these on. I think having the sources turned off really helps to make the historical forms easier to understand.
I also spent a bit of time this week on the Books and Borrowing project, including participating in a project team Zoom call on Monday. I had thought that we’d be ready for a final cache generation and the launch of the full website this week, but the team are still making final tweaks to the data and this had therefore been pushed back to Wednesday next week. But this week I updated the ‘genre through time’ visualisation as it turned out that the query that returned the number of borrowing records per genre per year wasn’t quite right and this was giving somewhat inflated figures, which I managed to resolve. I also created records for the first volume of the Leighton Library Minute Books. There will be three such volumes in total, all of which will feature digitised images only (no transcriptions). I processed the images and generated page records for the first volume and will tackle the other two once the images are ready.
Also this week I made a few visual tweaks to the Erskine project website (https://erskine.glasgow.ac.uk/) and I fixed a misplaced map marker in the Place-names of Berwickshire resource (https://berwickshire-placenames.glasgow.ac.uk/). For some reason the longitude was incorrect for the place-name, even though the latitude was fine, which resulted in the marker displaying in Wales. I also fixed a couple of issues with the Old English Thesaurus for Jane Roberts and responded to a query from Jennifer Smith regarding the Speak For Yersel resource.
Finally, I investigated an issue with the Anglo-Norman Dictionary. An entry was displaying what appeared to be an erroneous first date so I investigated what was going on. The earliest date for the entry was being generated from this attestation:
<attestation id="C-e055cdb1"><dateInfo> <text_date post="1390" pre="1314" cert="">1390-1412</text_date> <ms_date post="1400" pre="1449" cert="">s.xv<sup>1</sup></ms_date> </dateInfo> <quotation>luy donantz aussi congié et eleccion d’estudier en divinitee ou en loy canoun a son plesir, et ce le plus favorablement a cause de nous</quotation> <reference><source siglum="Lett_and_Pet" target=""><loc>412.19</loc></source></reference> </attestation>
Specifically the text date:
<text_date post="1390" pre="1314" cert="">1390-1412</text_date>
This particular attestation was being picked as the earliest due to a typo in the ‘pre’ date which is 1314 when it should be 1412. Where there is a range of dates the code generates a single year at the midpoint that is used as a hidden first date for ordering purposes (this was agreed upon back when we were first adding in first dates of attestation). The code to do this subtracts the ‘post’ date from the ‘pre’ date, divides this in two and then adds it to the ‘post’ date, which finds the middle point. With the typo the code therefore subtracts 1390 from 1314, giving -76. This is divided in two giving -38. This is then added onto the ‘post’ date of 1390, which gives 1352. 1352 is the earliest date for any of the entry’s attestations and therefore the earliest display date is set to ‘1390-1412’. Fixing the typo in the XML and processing the file would therefore rectify the issue.
Week 19 of Lockdown, and it was a short week for me as the Monday was the Glasgow Fair holiday. I spent a couple of days this week continuing to add features to the content management system for the Books and Borrowing project. I have now implemented the ‘normalised occupations’ part of the CMS. Originally occupations were just going to be a set of keywords, allowing one or more keyword to be associated with a borrower. However, we have been liaising with another project that has already produced a list of occupations and we have agreed to share their list. This is slightly different as it is hierarchical, with a top-level ‘parent’ containing multiple main occupations. E.g. ‘Religion and Clergy’ features ‘Bishop’. However, for our project we needed a third hierarchical level do differentiate types of minister/priest, so I’ve had to add this in too. I’ve achieved this by means of a parent occupation ID in the database, which is ‘null’ for top-level occupations and contains the ID of the parent category for all other occupations.
I completed work on the page to browse occupations, arranging the hierarchical occupations in a nested structure that features a count of the number of borrowers associated with the occupation to the right of the occupation name. These are all currently zero, but once some associations are made the numbers will go up and you’ll be able to click on the count to bring up a list of all associated borrowers, with links through to each borrower. If an occupation has any child occupations a ‘+’ icon appears beside it. Press on this to view the child occupations, which also have counts. The counts for ‘parent’ occupations tally up all of the totals for the child occupations, and clicking on one of these counts will display all borrowers assigned to all child occupations. If an occupation is empty there is a ‘delete’ button beside it. As the list of occupations is going to be fairly fixed I didn’t add in an ‘edit’ facility – if an occupation needs editing I can do it directly through the database, or it can be deleted and a new version created. Here’s a screenshot showing some of the occupations in the ‘browse’ page:
I also created facilities to add new occupations. You can enter an occupation name and optionally specify a parent occupation from a drop-down list. Doing so will add the new occupation as a child of the selected category, either at the second level if a top level parent is selected (e.g. ‘Agriculture’) or at the third level if a second level parent is selected (e.g. ‘Farmer’). If you don’t include a parent the occupation will become a new top-level grouping. I used this feature to upload all of the occupations, and it worked very well.
I then updated the ‘Borrowers’ tab in the ‘Browse Libraries’ page to add ‘Normalised Occupation’ to the list of columns in the table. The ‘Add’ and ‘Edit’ borrower facilities also now feature ‘Normalised Occupation’, which replicates the nested structure from the ‘browse occupations’ page, only features checkboxes beside each main occupation. You can select any number of occupations for a borrower and when you press the ‘Upload’ or ‘Edit’ button your choice will be saved. Deselecting all ticked checkboxes will clear all occupations for the borrower. If you edit a borrower who has one or more occupations selected, in addition to the relevant checkboxes being ticked, the occupations with their full hierarchies also appear above the list of occupations, so you can easily see what is already selected. I also updated the ‘Add’ and ‘Edit’ borrowing record pages so that whenever a borrower appears in the forms the normalised occupations feature also appears.
I also added in the option to view page images. Currently the only ledgers that have page images are the three Glasgow ones, but more will be added in due course. When viewing a page in a ledger that includes a page image you will see the ‘Page Image’ button above the table of records. Press on this and a new browser tab will open. It includes a link through to the full-size image of the page if you want to open this in your browser or download it to open in a graphics package. It also features the ‘zoom and pan’ interface that allows you to look at the image in the same manner as you’d look at a Google Map. You can also view this full screen by pressing on the button in the top right of the image.
Also this week I made further tweaks to the script I’d written to update lexeme start and end dates in the Historical Thesaurus based on citation dates in the OED. I’d sent a sample output of 10,000 rows to Fraser last week and he got back to me with some suggestions and observations. I’m going to have to rerun the script I wrote to extract the more than 3 million citation dates from the OED as some of the data needs to be processed differently, but as this script will take several days to run and I’m on holiday next week this isn’t something I can do right now. However, I managed to change the way the date matching script runs to fix some bugs and make the various processes easier to track. I also generated a list of all of the distinct labels in the OED data, with counts of the number of times these appear. Labels are associated with specific citation dates, thankfully. Only a handful are actually used lots of times, and many of the others appear to be used as a ‘notes’ field rather than as a more general label.
In addition to the above I also had a further conversation with Heather Pagan about the data management plan for the AND’s new proposal, responded to a query from Kathryn Cooper about the website I set up for her at the end of last year, responded to a couple of separate requests from post-grad students in Scottish Literature, spoke to Thomas Clancy about the start date for his Place-Names of Iona project, which got funded recently, helped with some issues with Matthew Creasy’s Scottish Cosmopolitanism website and spoke to Carole Hough about making a few tweaks to the Berwickshire Place-names website for REF.
I’m going to be on holiday for the next two weeks, so there will be no further updates from me for a while.
During week 11 of Lockdown I continued to work on the Books and Borrowing project, but also spent a fair amount of time catching up with other projects that I’d had to put to one side due to the development of the Books and Borrowing content management system. This included reading through the proposal documentation for Jennifer Smith’s follow-on funding application for SCOSYA, and writing a new version of the Data Management Plan based on this updated documentation and making some changes to the ‘export data for print publication’ facility for Carole Hough’s REELS project. I also spent some time creating as new export facility to format the place-name elements and any associated place-names for print publication too.
During this week a number of SSL certificates expired for a bunch of websites, which meant browsers were displaying scary warning messages when people visited the sites. I had to spend a bit of time tracking these down and passing the details over to Arts IT Support for them to fix as it is not something I have access rights to do myself. I also liaised with Mike Black to migrate some websites over from the server that houses many project websites to a new server. This is because the old server is running out of space and is getting rather temperamental and freeing up some space should address the issue.
I also made some further tweaks to Paul Malgrati’s interactive map of Burns’ Suppers and created a new WordPress-powered project website for Matthew Creasy’s new ‘Scottish Cosmopolitanism at the Fin de Siècle’ project. This included the usual choosing a theme, colour schemes and fonts, adding in header images and footer logos and creating initial versions of the main pages of the site. I’d also received a query from Jane Stuart-Smith about the audio recordings in the SCOTS Corpus so I did a bit of investigation about that.
Fraser Dallachy had got back to me with some further tasks for me to carry out on the processing of dates for the Historical Thesaurus, and I had intended to spend some time on this towards the end of the week, but when I began to look into this I realised that the scripts I’d written to process the old HT dates (comprising 23 different fields) and to generate the new, streamlined date system that uses a related table with just 6 fields were sitting on my PC in my office at work. Usually all the scripts I work on are located on a server, meaning I can easily access them from anywhere by connecting to the server and downloading them. However, sometimes I can’t run the scripts on the server as they may need to be left running for hours (or sometimes days) if they’re processing large amounts of data or performing intensive tasks on the data. In these cases the scripts run directly on my office PC, and this was the situation with the dates script. I realised I would need to get into my office at work on retrieve the scripts, so I put in a request to be allowed into work. Staff are not currently allowed to just go into work – instead you need to get approval from your Head of School and then arrange a time that suits security. Thankfully it looks like I’ll be able to go in early next week.
Other than these issues, I spent my time continuing to work for the Books and Borrowing project. On Tuesday we had a Zoom call with all six members of the core project team, during which I demonstrated the CMS as it currently stands. This gave me an opportunity to demonstrate the new Author association facilities I had created last week. The demonstration all went very smoothly and I think the team are happy with how the system works, although no doubt once they actually begin to use it there will be bugs to fix and workflows to tweak. I also spent some time before the meeting testing the system again, and fixing some issues that were not quite right with the author system.
I spent the remainder of my time on the project completing work on the facility to add, edit and view book holding records directly via the library page, as opposed to doing so whilst adding / editing a borrowing record. I also implemented a similar facility for borrowers as well. Next week I will begin to import some of the sample data from various libraries into the system and will allow the team to access the system to test it out.