I spent a lot of time this week continuing to implement the new versioning system for the Books and Borrowing project. This week I continued to tackle updating the Solr index, other cached data, and the scripts that generate this data. This involved making updates to the structure of the underlying database as where (for example) originally one cached JSON file was stored we now need to store any number of such files, with one per version.
I managed to update the code the generates the Solr index and executed it, generating the necessary JSON files, which I was then able to successfully import into a version of the Solr index that was running on my laptop. After running a few queries through the Solr admin interface everything seemed to have worked fine, which was a relief. I then had to work through all of the scripts that generate other cached data, such as generating the library top ten data and lists of each author that is associated with a borrowing at a library. There are more than ten such scripts and each needed updated (sometimes, as mentioned, with updates to the database structure too), tested, executed and then the output needed to be tested.
I ran into a bit of difficulty when generating the data for the ‘Facts and Figures’ pages, as this needs to query the Solr index, sending hundreds of thousands of queries to Solr in order to generate a static cache containing figures relating to the data. The issue was that the connection to Solr was taking a long time to complete. Previously passing a query to Solr via HTTP has been pretty much instantaneous, but this time each request was taking at least a second to complete, and when you’re dealing with hundreds of thousands of requests this isn’t feasible. I wasted quite a lot of time try to figure out what was going on, and it was especially perplexing as the function I used to connect to the Solr instance hadn’t been changed since I last ran it successfully and the Solr instance’s software hadn’t been updated either. I tried installing a newer version of Solr but it had the same problem. I tried running a test script on the actual server and it executed instantaneously, so the issue must have been something to do with my laptop. Eventually I asked Luca if he had any ideas and we had a useful troubleshooting session over Teams chat.
During this I decided to try changing the connection string from ‘localhost’ to the localhost IP address of ‘127.0.0.1’. As soon as I did this the connections were instantaneous again, which was a real relief, but also rather baffling, as I didn’t know why this should make a difference. It turns out that localhost connections now default to IPv6, but Solr and my laptop were not set up to run IPv6. This meant that every connection to ‘localhost’ was attempting to connect to Solr via IPv6, which failed, and then was falling back to IPv4 (127.0.0.1) and working. This was taking time and was causing the connections to run so slowly. I will have to remember this in future. Thankfully after replacing ‘localhost’ for the IPv4 IP address of 127.0.0.1 my scripts were much quicker and what would have taken days (or possibly weeks) to execute completed in about 15 minutes.
With all of the cache generation scripts updated, the cached data regenerated with a new structure and the database structure updated I am now ready to begin the process of updating the API and front-end to incorporate versions into everything. This is a task I’m hoping to make a start on next week.
Also this week I reviewed a shortlist of applicants for Ophira Gamliel’s project and gave feedback on their suitability. I also fixed an error with the Metaphor Map and the Old English Metaphor Map that was preventing counts of categories from displaying in the textual browse lists. I also liaised with our IT people to get the OHOS domain migrated to The National Archives and responded to Craig Lamont about a new project he is putting together.
The remainder of my week (about a day) was spent on the Place-names of Iona project, for which I implemented some updates that we’d discussed at a project meeting last Friday. This included updating the site banner to cut off the top part and make it take up less of the screen and adding in the legend descriptions from the Berwickshire Place-names project. I also updated the glossary to ensure that all listed parts of speech have a tooltip that displays the full form.
The big task was to completely overhaul the display of the glossary. There is now a bar of letter links that feature the letters that elements start with as buttons. A count of the number of elements starting with each letter is also included, as I thought this might be useful. Any letters that don’t have any elements appear but are greyed out. The list of elements is now split by letter, each of which appears as a heading with a dividing line to make it easier to tell where elements beginning with one letter end and the next set begins. A ‘Top’ button is also present to make it easier to get back to the top and I also replaced the placeholder text in the element glossary side-menu with descriptive text that Carole came up with for Berwickshire. These updates make the element glossary much more usable now, as the following screenshot showing personal name elements demonstrates:
Next week I’ll apply this update to the other place-names resources, and I’ll also hopefully find some time to work on the new ‘thematic maps’ that we’re hoping to implement before the resource goes live in June.
This week finally saw the launch of one of the new ‘map first’ interfaces for the various place-names projects I’ve been working on. I initially created the interface for the Iona place-name project and then applied it to Ayrshire, Kirkcudbrightshire, Berwickshire and Nairnshire, but none of the projects were quite ready to publicly launch the new interface. After having a char with Carole Hough last week I got the go-ahead to go live with the new Berwickshire map, and it can now be found here: https://berwickshire-placenames.glasgow.ac.uk/map with the textual list of Berwickshire place-names here: https://berwickshire-placenames.glasgow.ac.uk/list-of-berwickshire-placenames/.
It’s great to have the new interface publicly available at last, although I did have a number of last-minute tweaks to make as I worked on it. I’ve updated the legend slightly, adding in some explanatory text and ensuring that the text is hidden when the ‘hide legend’ button is pressed – previously the text remained visible, which was a bit pointless. I also spotted that the legend was not being set to hidden be default on narrow screens, even though it should have been, so I fixed this. Another issue I spotted was that the tooltips that display when the legend categorisation is by language would stop working whenever the ‘select all’ option was pressed, and I needed to update the code to ensure the tooltips were reinitialised. With these changes in place I also needed to apply them to the other place-names maps to keep things consistent.
Launching the new map wasn’t the end of the work, however. Berwickshire already had a live map resource, plus search and textual browse facilities. I needed to ensure that the quick search on the main resource website connected through to the new map’s search results and that all links / bookmarks / citations to the old interface redirected through to the appropriate section of the new interface, where possible. So for example, the old element glossary page redirects to the new element glossary popup in the new map interface: https://berwickshire-placenames.glasgow.ac.uk/place-names/?p=element-glossary. I think I’ve caught all of the old links and have ensured the necessary redirects work.
Also this week I had a meeting with Thomas and Sofia regarding the map interface for Iona, which will apparently be publicly launched on the 9th of June. We discussed some of the outstanding tasks they would like to see completed before the launch and I’m going to have a fair amount to do before then. For example, they would like the element glossary to have a ‘jump to letter’ feature, allowing the user to immediately scroll to elements beginning with a selected letter. We also discussed new ‘thematic maps’ based on static maps Sofia had created for a recent event on Iona. These maps showed place-names grouped in more fine-grained ways that are not covered by the classification options we currently have available – for example place-names featuring animal names. I suggested that I could make an interface where such thematic maps could be created in the content management system and displayed in a new menu section in the front end. This would consist of a title and description for the map, and a list of markers that should appear on the map.
I also demonstrated the ‘story map’ interface I’d created for the Scots Syntax Atlas (go here and select ‘Stories behind the examples’: https://scotssyntaxatlas.ac.uk/atlas/). This feature consists of a series slides a user can navigate through, each of which can have a different view of the map, which may feature different data and different zoom level, thus guiding the user through a particular story the map tells. I gave an example of how this could be used for iona to have a story of place-names featuring animals, with different slides showing markers featuring elements from different languages. Thomas and Sofia really liked this idea and would like to implement it, but while it might be possible for me to implement the simpler thematic maps before the 9th of June, it’s unlikely that I’d be able to get the ‘story’ approach in place. It’s something we’ll probably consider after the launch.
Continuing with place-names, I had an email conversation with Alasdair Whyte regarding his Mull / Ulva place-name data. He’s currently working towards a published volume and wanted me to add a facility to the content management system to enable him to categorise place-names by volume. I therefore updated the CMS to add in the volume field, and this now appears in the ‘add’ and ‘edit’ place forms. I also updated the ‘Browse’ page to add in volume to the search options and as a table heading for the listed data and updated the ‘Export place-name data for publication’ page in ‘Tools’ to add in a volume selection option. Thomas also contacted me this week to ask for an update to the ‘Export’ facility for Iona (adding in the ‘translation’ field) so I did this too. I also had a chat with Alasdair about the front-end for his Mull / Ulva data, as this will need to be a bit different from the other resources.
Also this week I finally managed to get my ticket sorted for the Digital Humanities event in Glasgow in June, with the help of Emma McCluskey. I also made some updates to my presentation for the DH event in Lisbon based on some very helpful feedback from Jennifer, and also added in some new publications to the Scots Syntax Atlas resource.
I also spent a bit of time working on the Dictionaries of the Scots Language. I’d been contacted last week by editor Vasilis Karaiskos about the regions data that he has been working on, and the new maps he has been developing to display these regions. I spent some time going through the interface he had developed as a proof of concept, figuring out how it all works and giving some feedback. It will be really great to get more information about the geographical regions into the website and the maps are an excellent starting point. We’re hopefully going to have an online call in the next few weeks to discuss things further.
The rest of my week was devoted to working on the new versioning system for the Books and Borrowing project. I need to update the structure of the Solr index and the other cache files, and also the scripts that generate this data, and this is my next step. As I feared, it’s proving to be an awfully tricky and time-consuming update to implement, and while I did make progress there is still a huger amount to do. I’ll be continuing with this monumental task next week.
It was a four-day week this week due to the May Day holiday on Monday. On Tuesday I spent quite a bit of time responding to a couple of emails from Tony Harris about the Bilingual Thesaurus system, which he is repurposing for another project. This involved going through the code and database for the Bilingual Thesaurus and refamiliarising myself with everything in order to reply to Tony’s questions. Whilst doing so I also spotted that one of the lexemes in the thesaurus was classed as a noun when it should have been a verb, so after checking with the PI Louise I corrected this.
Also on Tuesday I finally managed to register for the DH2025 conference with the help of Critical Studies Admin Emma McCluskey, and it’s great to have everything sorted for the conference now. We also tried to sign me up for the DH even that is happening in Glasgow in June but unfortunately encountered some issues with the registration process and were unable to proceed. I’ve contacted the event organisers and hopefully I’ll be able to complete the process next week. I also fixed a minor issue with the pilot website for the Books and Borrowing project and replied to Kirsteen McCue about a collection of songs and accompanying documents that we’ll be adding to the Burns website soon.
On Wednesday I focussed on preparing my presentation for the DH2025 conference. I’d met with Jennifer Smith last week to discuss this and she’d prepared an initial Powerpoint presentation, and my task was to reshape this and prepare the actual talk around the slides. The structure differs slightly from Jennifer’s original, with the biggest changes being the inclusion of a brief discussion about SCOSYA at the start, to give a bit of context for the creation of SFY, and also the inclusion of a section describing how the maps are generated as I figured the audience are likely to be more technical than linguistic and might be interested in this. There’s quite a lot of content, but I timed myself reading the text and it came in at about 8 minutes, so hopefully it will be about right. Obviously I’ll read through it many more times and will tweak things if needs be. I also sent it on to Jennifer in case she would like me to make any changes or has any suggestions.
On Thursday I had an online meeting with the Dictionaries of the Scots Language to discuss our recent rollout of the new version of the DSL data and accompanying new website features such as the sparklines. This was a very useful meeting where we all had a chance to discuss the process and any aspects that could be improved in future releases. Also mentioned at the meeting was that the team wanted to reinstate the ‘filter results by dates’ option on our test site for further testing, which I did after the meeting. I also shared the scripts and instructions needed for generating a new version of the DSL data from the XML files outputted from the DSL’s editing system so the team has a copy of everything in case I become unavailable for whatever reason.
The editor Ann Ferguson also spotted an issue with the bibliographical links referenced in some entries. These were not displaying as links and were therefore not leading to the relevant bibliographical entry. It turned out that the possibility of there being authors and no titles in the citation records had been overlooked in my code and in such cases the link to the bibliography wasn’t getting added in. Thankfully once identified it was relatively quick to fix – no need to regenerate the Solr index or anything like that. I don’t think the update will have introduced any unintended consequences so I therefore updated both the live and our test site to implement the fix.
On Thursday and Friday I also did some work for the Anglo-Norman Dictionary. The editor Geert wanted me to update the new cognate reference section in the advanced search to have all cognate dictionaries unselected by default, which I did. This does now mean that the cogref search will not submit when someone types in some text and presses ‘search’ – they will always have to make a dictionary selection first. However, Geert seemed happy with this.
Geert also wanted to make the ‘Advanced Search’ option more prominent in the site’s navigation and suggested moving it from the site menu to a separate button near the ‘Quick Search’ section in the top right of every page. I pointed out that we need to retain the ‘Advanced Search’ menu item in the menu bar as all major sections of the site are represented in this bar, and one is always highlighted. If we removed ‘Advanced search’ from the menu bar we’d end up with no item highlighted whenever a user is looking at the search form or search results, which I think would be confusing. However, I agreed that the advanced search could do with more prominence and proposed that we retain the ‘Advanced Search’ menu item, but we also include a ‘Try an advanced search’ link underneath the quick search bar in the header. I favoured text rather than a button, as a button would be too prominent and would take the user’s eye away from the actual search button. I sent a mockup image of this to Geert and he agreed with me, so I implemented the update, which is now live on all AND pages.
Geert also mentioned that the bibliography (https://anglo-norman.net/bibliography/) was very slow to load and I spent much of Friday reworking this. The bibliography page was loading all 1300+ sigla and then limiting the display to those with titles matching the selected letter. This was not a very efficient way to handle things and must have been implemented back when the bibliography was much less extensive. Instead I updated the API and the bibliography page so that only those sigla that actually begin with the selected letter are returned and processed. This has massively increased the speed at which the page loads.
I also noticed that the ‘Citation Siglum’ search box in the ‘Citations’ tab of the advanced search was not functioning as it should do. As you type into this box a list of sigla that match the characters you enter should be displayed, allowing you to select the appropriate item. However, no matter what text you entered the entire list of more than 1300 sigla was being displayed, which took time to load in and was utterly useless. I also fixed this and now (for example) when you type in ‘roy’ you are presented with a list of sigla that actually feature these characters anywhere in their titles. I’m not exactly sure when these issues arose, but I’m guessing it was when we reworked the sigla to greatly expand the information included (e.g. items, links to cognate dictionaries etc). I’m glad Geert brought the issues to my attention and I was able to fix things.
This week I began to implement the new versioning system for Books and Borrowing. I’d written a specification document for this a few weeks ago and at a meeting last Friday we agreed that I should implement the more complex system I’d proposed. This week I focussed on updating the underlying database and the content management system to add in the required version systems. The database now has a ‘version’ table, and all major tables have been updated to include a reference to this. I’ve created a ‘version 1.0’ version and have associated all existing data with this, and I’ve also created a new ‘version 1.1’ version. Now when staff log into the CMS the current version (set to ‘version 1.1’) is logged and all ‘add’ forms in the CMS have been updated to store the current version when the forms are submitted. All this happens behind the scenes, so the project team shouldn’t notice any changes.
There are, however, a number of other visible changes to the CMS that I implemented. There is now a new ‘Browse versions’ menu item that when clicked on displays a table listing the versions that exist. Version 1.0 is set to ‘live’, meaning that when I’ve updated the front-end only data from this version will be used. Note, however, that I’ve not made any updates to the front-end yet, so for now any new data that is added to the CMS will still appear when browsing registers but will not be found in the stats or searches, until the caches and Solr index are updated.
Also in the CMS, all ‘browse’ tables now have a new ‘When Created’ column that displays the version that each item was created under. Note that edits do not affect the version number – we’re only tracking new additions. I had to make changes to the database queries throughout the CMS and although I’ve tested things out pretty thoroughly on an instance running on my laptop it’s possible that a bug or two might have crept in so I’ll just have to see how the project team get on. The next step will be to update the Solr index and the other cached data, plus the scripts that generate this data. After that will come the updates to the front-end. These are going to involve some pretty hefty changes and may take some time to sort out.
On Tuesday this week I had a meeting with Marc Alexander and Faye Hammill, who is the Research Convener for the School. It was a good opportunity to catch up with Faye and to hear more about the current research landscape in the School, and to consider how my role can be promoted to academic staff in the School who possibly aren’t aware of my existence and the help I can offer. Faye also mentioned a Digital Humanities conference that is taking place in the University in June (https://digitalhumanities-uk-ie.org/). Somehow I wasn’t aware that this was taking place and I’m really glad that Faye brought it to my attention.
Also this week I met with Jennifer Smith to discuss the paper we will be giving at the DH2025 conference in Lisbon in July. Jennifer has decided not to attend the conference, so it will just be me presenting the paper, and we put together an initial version of a Powerpoint presentation that I will hopefully have some time to work on next week. I also got approval from Marc this week to register for the conference. This is something that needs to be handled by the School admin people, but unfortunately when I met with one of the team on Friday to fill out the form there were some uncertainties and we’ll need to complete the process next week. Thankfully the ’early bird’ rate for the conference has been extended by a week so that shouldn’t be a problem.
A couple of weeks ago I was contacted by Carole Hough who wanted access to the contents of Cognitive Topography, an old website that disappeared about five years ago. At that time the domain expired and Carole had arranged for the contents of the site to be moved to the University’s T4 system. Unfortunately there must have been some crossed wires as only the landing page was transferred, along with links to other pages that were still hosted on the now defunct domain. I’d been away from Glasgow for the past couple of weeks but promised to look into finding a backup of the site this week, as I am now back in Glasgow.
Thankfully I managed to locate a backup of the old site and managed to get a version of it mostly up and running again. It was a bit of a struggle getting the old site working as the WordPress version it’s running on is almost 10 years old and is incompatible with modern server technology. I had to downgrade some of the server software to get it to run, which is a security risk, so this can only be a temporary solution. Unfortunately the version of WordPress is so old it’s not even possible for me to access the admin interface (even though the credentials are correct) so getting things to display has required a good deal of furtling about directly with the database and scripts and upgrading the version of WordPress to a modern version isn’t really an option.
The site header is not displaying at the correct size and I’ve had to remove the multilingual (English and Danish, I think) plugin as it was preventing the site from loading. All of the content appears to be present, but the English parts start with [:en] and the Danish parts with [:DA].. Several of the pages also had images and links to other pages that used the full defunct URL, meaning they were broken. I therefore ran a ‘find and replace’ on these so they should now work and there hopefully won’t be any links to the defunct URL.
I shared the temporary URL with Carole and we are now thankfully in a position to rescue the content of the old site. I contacted Mick Eadie in the library, who is in charge of the library’s archiving of old University websites and he has very helpfully agreed to run the library’s archiving tool over the temporary site, after which the site should be preserved and accessible via their archiving tool, at which point the T4 page about the project can be updated to link to it. Hopefully this will happen in the next week or so.
Also this week I spoke to Eleanor Lawson about adding updated animation files to the Seeing Speech site, I gave some advice to Delphine Demelas of the Anglo-Norman Dictionary about technical expertise for future research projects, I agreed to be on an interview panel for Ophira Gamliel’s project, I had a chat to Andrew McHugh about an issue with one of the SiteGround hosted sites that resulted in the issue getting resolved, and I had a chat with Luca about issues with our locally hosted server going down, which has also now hopefully been resolved.