I’d received a new batch of data outputted from the editing system used by the Dictionaries of the Scots Language on Friday, and on Monday I processed this, which involved importing the data into the database structure used by the website, generating the data necessary for the search facilities and then outputting all of the data in a format that would then be ingested into the Solr search system. All went pretty smoothly, and after testing everything out locally I uploaded everything to the server. I don’t have direct access to update the Solr cores on the server, so I had to wait until our IT people had the time to run the necessary commands, which they did on Wednesday morning. A few subsequent tests show that all is working as it should, and we’re just about ready to replace the live site with the version we’ve been working on for more than 18 months now.
I also made a couple of further tweaks to the system, such as ensuring the new ‘TypeC’ attribute connects to the correct update statement and dynamically populates the ‘About’ section of the relevant entries. Later in the week the editor Ann got in touch to say that the bibliography search (or more specifically the dropdown list of predictive search options) was running very slowly so I investigated this. I added in some new indexes to the database and this has increased the speed considerably. Rather strangely, the live site doesn’t have these indexes and the queries appear to be exactly the same so I’m not sure why our test site is much slower – the databases are running on the same server. However, the indexes will make things more efficient so it’s good that this was spotted.
Also this week I responded to some queries from the Anglo-Norman Dictionary about our links to the DEAF site from some entries. Unfortunately it looks like their lemma-based URLs are ignoring accents, which means some of our links are leading to the wrong page on their site. For example, the DEAF link on this page: https://anglo-norman.net/entry/herit%C3%A9 leads to this page: https://deaf.hadw-bw.de/lemme/erit%C3%A9 which unfortunately displays the same content as this page: https://deaf.hadw-bw.de/lemme/erite even though ‘erite’ and ‘erité’ are different words. The only way to reach the correct page is to perform a search for it and select it from the left-hand column: https://deaf.hadw-bw.de/?type=shortarticle&id=38050#erit%C3%A9 but we can’t use such a link in our XML as it would require us to know the ID of every lemma we link to. It’s not ideal for a French language website to not be able to differentiate accented and non-accented characters, but that appears to be what’s happening. Strangely, other links on the DEAF site do differentiate accented characters. For example: https://deaf.hadw-bw.de/fiches/erit%C3%A9 and https://deaf.hadw-bw.de/fiches/erite display different content. It’s just their ‘lemme’ links that don’t work properly.
In addition, I also added a new language to the AND system this week. We now have ‘Scots’ listed and selectable in the XML, and whenever this language is added to entries the search facilities on the website will be automatically updated to enable the language to be searched for. In fact we currently have two entries that have been updated to include Scots at the time of writing: https://anglo-norman.net/search/language/14/null/
On Wednesday this week we had new broadband fitted at our house, and it took a couple of hours to get everything all working with the new system. And on Friday afternoon I attended an event organised by the ‘Realities and Immersion Glasgow’ group at the ARC. This group are looking at virtual / augmented / extended reality systems and their potential use across the university and beyond. It was very interesting to try out several of their headsets and see the work that’s going on in this area, and I’m glad I went along.
I spent most of the rest of the week continuing to develop a new, unified API for the Historical Thesaurus website. I managed to complete work on the search facilities this week, so that’s the bulk of the API sorted now. There are still several visualisations that need their data queries to be plugged in, plus I need to ensure the CSV output works correctly, but I’m hoping I’ll be able to complete work on this next week, depending on what other work needs to be done.
My main task for this week was to continue with the creation of a new, unified API for the Historical Thesaurus. This has been on my ‘to do’ list for a long time, and I began work on it back in February, but was only able to spend a day or so on it. I was able to devote a few further days to the task this week and have made good progress. The Historical Thesaurus’s existing website was created over several years, with new features added as and when required, resulting in a structure that’s got a bit messy over time. For a while now I’ve wanted to bring this into line with the other resources I’ve developed and have a single API through which all data requests are processed. Back in early February I made a start on this, but only managed to get as far as creating the endpoint to generate a random category. This week I made a lot more progress, and now have all of the endpoint needed to provide data for the category browser and other parts of the site such as the quizzes. This has also included the endpoints necessary to generate the timeline visualisations. I’m also taking the opportunity to tweak the structure of the ancillary pages slightly, stripping out unnecessary function calls.
I still have a fair amount of work ahead of me – the entire search facilities still need to be tackled, as do the endpoints for other visualisations such as the heatmaps and other search options that were developed for the Linguistic DNA project. As I’m fitting this in around other commitments I’ll just need to see when I’m able to finish work on the API, but I’m pretty pleased with the progress I’ve made this week.
Also this week I spent some further time on the Anglo-Norman Dictionary. We haven’t gone live with the new cognate reference search yet, but I implemented a new feature for the dictionary’s content management system that the editor Geert suggested recently: a list of entries that were last updated in each month. Here’s a screenshot of how this looks:
By default the page lists all of the entries that were last updated in the current month, but you can also select a different month and year to view the entries that were last updated then. As the note in bold on the page states, this is just a list of when entries were last updated, it’s not a log of every time an entry has been updated. The table lists some details about each entry and includes a link to view the entry in the CMS. You can also reorder the table by pressing on the column headings (e.g. ordering by last editor). Pressing the heading a second time reverses the order. Hopefully the feature will prove useful to the team.
Also this week I had to fix a couple of issues with the Books and Borrowing website. On Monday I noticed that the website was offline, and this was caused by the database server failing to restart following an update. I liaised with Stirling’s IT people to get this working again. A user of the website had also spotted that the CSV download of the search results was failing to work and I had to spend a bit of time investigating this. Initially I thought the problem was due to the number of records, but on further investigation it turned out to be caused by the presence of one or more borrowing records that didn’t have an associated book holding record.
For example, on this page https://borrowing.stir.ac.uk/search/p-1/0/0/advanced/brids|4473 the borrowing record with ID 48238 with the transcription ‘Prices Alexero on Sab. Chur: [?]’ doesn’t have an associated book holding. The CSV output was expecting all borrowing records to have a book holding, and when it reached one that didn’t the script stopped working. Once I’d identified this I was able to fix the issue and the output now works as expected.
Also this week I liaised with my colleague Luca to get the Helsinki Corpus up and running again. We host the site, but I’d been alerted to the fact that it was offline on Monday and correctly surmised that it was because our hosting provider had silently upgraded the version of Python that was running on the server, and some aspects of the Helsinki Corpus code were incompatible with the new version. Thankfully Luca was able to install an alternative to the library that had been deprecated and we got the site back online again. If I can find the time I’d like to redevelop the Helsinki Corpus site, migrating it to eXist and building a new site around this. Something to add to my ‘to do’ list.
Also this week the Solr server that powers the Dictionaries of the Scots Language went offline and I needed to speak to our IT people about getting this back online again, which thankfully they were able to do pretty speedily. I also made a few updates to the help text on the advanced search page of our test site and replaced ‘dates of attestation’ with ‘quotation dates’. On Friday I received a new batch of data that will hopefully be the last batch that needs processed before the changes made on our test site over the past 18 months can finally go live. I’ll begin working on this next week.
I spent most of this week continuing to implement the new cognate reference search facility for the Anglo-Norman Dictionary. When I left off last week I was midway through implementing a script to generate search form variants for all of the cognate reference forms found in the dictionary entries. These are the words found in the grey box towards the top of an entry page such as https://anglo-norman.net/entry/asseger. My script splits up multiple forms and bracketed sub-headwords, where brackets are found within a form the script generates variants with and without the bracketed text, multi-word forms have each word stored separately (in addition to the full string) to enable each word to be searched for, and where a dash is used variants are generated with the dash, with a space in place of the dash and with no space at all.
After fully testing the script and its output I then needed to work on the scripts that update dictionary entries to ensure that they also generate or regenerate the cogref search forms whenever they execute. This included the batch upload process I use when handling large-scale updates to the database and the entry publication script found in the dictionary’s content management system, as used by the editorial team when updating individual entries. I worked with the editor Geert to test out these updates and with a bit of tweaking I got everything working.
I then updated the ‘View entry’ page in the dictionary’s content management system so that all of the cogref search terms are displayed, so the team can check what can be searched for. As mentioned previously, cogrefs are split into individual search terms on the space character, allowing individual words to be searched and this does mean that parts of speech and numerals also get stored as individual search terms, but I don’t think this will be a problem – no-one will see these search terms, only the listed ‘Display form’ will be shown in the results and the ‘search forms’ just show the variants that are stored behind the scenes. I did consider stripping out all numerals and parts of speech but there are just so many possible permutations that it would be very tricky to catch them all, and storing them is not a big issue. So for example for the entry ‘asseger’ linked to above, the following forms are generated:
Where there are multiple forms (e.g. Gdf) these are stored as separate search forms. Where there are subheadwords (e.g. DEAF) these are similarly split into separate search forms. Where there are forms with spaces (e.g. OED) the full string is saved as a search form, but so too are each individual word. Where there are brackets within a word (e.g. MED) then forms are generated both with and without the bracketed text. This also works where there’s only an open bracket, as happens quite often with MED headwords. There were also a handful of MED cogref forms that had multiple brackets (around 13, I think). In such cases the generated forms only include versions with all of the bracketed text and none of the bracketed text, as the generated forms for the entry ‘Oxherd’ demonstrate:
He above also demonstrates the variants that are created when a dash is used, and any forms with slashes are treated in the same way.
With all of the cogref search data in place and facilities established to ensure the data will be updated when entries are edited I then moved onto the creation of the new search facility itself. The first step was to create a new endpoint in the API through which the search requests will be processed. It took a while to implement and test this, ensuring that wildcards can be used in the search and enabling the search to be limited to one or more cognate dictionaries. The search results initially brought back rows for each cogref form, so if (for example) the search was for ‘siege’ as found in the above screenshot there would be three individual DEAF search results, each with information about the associated entry. However, what the search needed to return was entries and then within each entry a further section listing the individual search results and I therefore needed to switch things around slightly.
With the API updated and the new search endpoint working I then needed to update the front-end to add the new search form to the advanced search page, connect the form to the API and ensure that any returned results were processed in the same way as the other advanced search options (for example with pagination, links to entries and a way to return to the form with previously entered options remembered). I created all of this in a test version of the page so the editors and I could test things out without making the update publicly available.
The new Cogref search appears as a new tab in the Advanced Search after ‘Languages’. It consists of some introductory text, a search box where a cognate form can be entered (with or without wildcards), and a section where the cognate dictionaries you want to search can be selected. These appear as checkboxes which are all checked by default. There’s an ‘All’ checkbox that allows you to select / deselect all of the other checkboxes, so to limit your search to just DEAF, for example, press ‘All’ to deselect everything then press ‘DEAF’. The cognate dictionaries have tooltips that display the full names, as the following screenshot demonstrates:
As with the headword search, by default any text you enter into the ‘Cognate Form’ box matches any corresponding string in the search forms – so for example ‘noble’ is actually treated as ‘*noble*’. To perform an exact search use double quotes. As previously mentioned, search forms have been created for many different variants and where a form has multiple words or is split with dashes each individual word is treated as a separate search form. So, for example, if you perform an exact search for “herd” you will find things like ‘nout-herd’, ‘shep-herd’ etc.
The search results display details of what you’ve searched for, the entries that feature a cogref form that matches and the specific cogrefs that feature a form that matches your search criteria, as the following screenshot demonstrates:
The editors are now testing out the new search and hopefully we’ll be able to go live with it in the next week or so.
Also for the AND this week, the editor Karen spotted an issue with the cross references between entries, which are generated whenever an entry is updated. Some cross references weren’t being found and it turns out that this was because the homonym number is separated from the lemma with a hash in the XML, but with an underscore in the ‘slug’ used in the website (this is correct and how things should be). What this meant was that the cross reference generator was taking the slug (e.g. Partein_1) and looking for that in the XML files and obviously not finding any matches as it’s recorded as ‘Partein#1’ in the XML. I therefore updated the code so that the xref search turns ‘_’ into ‘#’ and the xrefs are now being found and generated.
This week I also did a bit of work for the VARICS project, adding in a new batch of short descriptions for measurements in the lookup feature I’m developing and discussing the ancillary text with members of the project team. I also had a chat with Jennifer Smith about our Speak For Yersel paper for DH2025 and gave some feedback to Matthew Creasy about his Mallarmé Chancellor’s Fund application, which he submitted on Friday.
I also responded to a Query from William Ashford of the DSL about Google Analytics, mobile versus desktop users and search engine optimisation and responded to a query from Ann Ferguson regarding a couple of issues relating to the bibliography search.
The team meeting for the Dictionaries of the Scots Language took place in Edinburgh this week, and I attended on Wednesday. At the meeting I led a session where I demonstrated the potential new interface I’d developed for the DSL website last summer. I gave a walkthrough of all of the mock-up pages I’d created, such as the homepage, the search results and the entry page, demonstrating the interface on both desktop and mobile devices. I can’t really say too much about the interface at this stage, or post any screenshots, but the demonstration was very well received by the team and on the whole they are keen to take it forwards. We spent an hour or so discussing the interface and there were a number of helpful suggestions and new ideas proposed, and I’ll definitely take these into consideration. There was some talk of launching the new interface next year, so sometime this year (possibly in the summer) I’m intending to replace the interface for our test version of the DSL website with the new interface. This will involve creating a WordPress theme from the interface, ensuring it is fully integrated with the WordPress platform and also connecting the interface to the DSL’s API in order to pull in real data rather than displaying static mock-ups. It’s a task that will likely take a few weeks to fully sort out, but once it’s done then the team can fully test out the interface and give further feedback. I spent some time before the meeting preparing for my session and I also needed to fix a couple of issues with the mock-ups that I’d spotted – for example the tooltips were not triggering on touchscreens.
Also this week I learnt that our paper about Speak For Yersel that I submitted to DH2025 had been accepted, and I had an email conversation with Jennifer Smith about conference arrangements and the presentation. I also spoke to people at The National Archives about the transferral of the OHOS domain to TNA and updated a few St Andrews records in the Books and Borrowing website whose URLs couldn’t be fixed in the batch update I executed last week. Someone at St Andrews library made some very helpful suggestions for the books and I manually updated them. I also heard this week that the Books and Borrowing resource has won the British Society for Eighteenth Century Studies’ 2025 digital prize, which is fantastic news.
I spent the rest of my week planning and developing a new search option for the Anglo-Norman Dictionary. The editor, Geert, wanted a new advanced search facility that would enable users to select one or more cognate dictionaries and then search for a headword found in these dictionaries, returning a list of all AND entries that feature cognate references that match the criteria. The cognate references are found in the grey box towards the top of every AND entry, for example on this page: https://anglo-norman.net/entry/noble. There are ten cognate dictionaries (e.g. OED, DEAF) and the forms of the headword as found in these dictionaries are listed. It is these forms that the new search will target.
I wrote an initial specification for the new feature, the data structures that would be required and the stages that would need to be followed to generate the necessary data and shared it with Geert. After several email exchanges I wrote a second version of the document and then began work implementing the feature. The biggest issue will be processing the cognate reference forms, as these will need to be split up in various different ways. For some dictionaries there are headwords and also subheadwords in brackets. Other dictionaries use brackets to denote variant forms, sometimes without supplying a closing bracket. In other cases slashes are used to indicate separate headwords and then there are the issues of parts of speech and numerals appearing at the end of forms.
My first job was to write a script that would go through all active main AND entries and extract all of the cognate reference forms. In doing so I discovered that there are 634 such entries that don’t include the cognate reference section, and I sent the list to Geert for him to investigate. I also generated a spreadsheet containing every cognate reference for every entry, which was useful for identifying all of the variety contained in the fields. I then began writing a script that would generate individual search forms for each cognate reference, generating alternative spellings based on brackets, for example. By the end of the week I was still in the middle of this and I’ll continue with it next week.
On Monday this week I attended a ‘getting started with Transkribus’ workshop held at the University of Edinburgh. This was a really great opportunity to learn more about the Transkribus tool and to be guided through the initial stages of using it. I had somewhat naively thought that it would be possible to just upload a digitised image to the tool and it would then be able to extract and format the text with a high degree of accuracy, but in reality much more manual intervention is required before Transkribus can work effectively. Training the tool on your data is absolutely vital, and this means uploading sample images, identifying lines of text (which can be done automatically but may need manual tweaking) and then transcribing the text found on each line using Transkribus’s web-based text editor. Text can also be tagged using this facility, such as noting headings, or identifying proper names. The leader of the session suggested that around 15,000 words of text should be manually transcribed in order for the tool to be well trained, which would take a considerable amount of time and effort.
I wanted to see whether Transkribus could extract the text from the Edinburgh Gazetteer (https://edinburghgazetteer.glasgow.ac.uk/the-gazetteer/) and the first stage of this would be for the tool to recognised that each page is divided into four columns, many with a full-width heading at the top. A ‘layout’ model would need to be run on the pages in order to automatically identify the columns. There are a couple of existing newspaper layout models available but unfortunately when I ran my test image through them neither managed to divide the page, and the entire page was simply identified as ‘Region 1’, meaning all identified lines ran across all four columns. Any human looking at the image would be able to ascertain immediately that it’s four columns of text in so I was a bit disappointed that Transkribus wasn’t able to do the same. I spoke to the leader of the workshop, who has successfully used Transkribus on another historical newspaper and he suggested that I’d have to train Transkribus on this specific newspaper and that in order to do so I’d need to use a ‘Field model’. It’s not entirely clear to me how a ‘Field’ differs from a ‘Layout’ model, but unfortunately it’s not possible to create a field model with the free Transkribus account – you instead need to subscribe to a ‘Scholar’ account at €20 a month, something I didn’t want to do as I was just experimenting at the workshop, so I reached something of a dead end. Having said that it was still very useful to experiment with Transkribus in a workshop setting with an expert on hand and I will continue these experiments when I have more time available. However, I do wonder how more general AI tools might cope with the same materials and whether they might be quicker to use. It’s definitely something that would be worth exploring further.
For the rest of the week I worked on several different projects. I spent some time refamiliarising myself with the structure of the Bilingual Thesaurus of Everyday Life in Medieval England (https://thesaurus.ac.uk/bth/) as a developer who will be working with the structure for a new project got in touch with questions about it. I wrote a data description document that describes every table and column in the database, and also contains the highest ID in each table used for BTH. There are several columns that are only present as references to the original spreadsheet and are not used by the online resource, and I’ve made a note of these. Whilst working on this I also spotted an issue with the display of languages in the advanced search. These were not displaying any content and it turns out that this was because one of the referenced column names was ‘grouping’, which is a reserved word in MySQL. While the queries worked on our old server, the new server is more strict about such things and the queries failed to run on it. Having spotted this it was thankfully quick to fix – surrounding the column name with backticks (`grouping`) in the queries got things working again.
I also returned to working on the Books and Borrowing project for a few hours this week in order to update the links we have in our system to books held at St Andrews University library. The library has recently changed their online system, which has broken all existing links to records in the system. Thankfully they have a huge spreadsheet featuring the old identifiers and the corresponding new ones and I was able to write a little script that would identify the St Andrews book holding records in our system and update their links. Of the 7560 active St Andrews book holding records in our system 1666 have URLs and my script successfully updated 1662 of these. The remaining four have issues that will need to be looked at separately, and I’ve let Katie and Matt know about these.
I met briefly with Alan Riach this week to discuss the website I created for him last week. He gave me a further CD featuring a recording he wanted me to host and I was able to extract the recording and update the website to add this in. I also joined my fellow College of Arts and Humanities developers for a coffee and a catch-up this week, and it was really great to chat with them all again. Hopefully we’ll meet up again soon.
On Thursday this week I also had a meeting with Rhona Brown, Amy Wilcockson and Clare Thompson from the Mitchell Library to discuss an online exhibition for the Robert Fergusson website (https://robert-fergusson.glasgow.ac.uk/). There was a physical exhibition of materials at the Mitchell last year and we’re going to be bringing these together online in the next few months. It was good to meet everyone in person and hopefully I’ll be able to help them create an exciting online resource based on the materials.
Also this week I managed to fix the issue with capital letters not working in the ‘jump to’ option in the ‘browse’ facility of the Anglo-Norman Dictionary. You can now enter text and it will find matching entries whether they have capitals or not. E.g. enter ‘abc’ (or ‘ABC’) into the ‘jump to’ box. Brackets are also not an issue, e.g. type in ‘xilo’ and you’ll jump to ‘[xiloaloe]’. The only thing to bear in mind is that the browse is accent sensitive. It has to be as we have entries with headwords that are identical other than for accents. So for example entering ‘acel’ will jump to ‘acelément’ but typing in ‘acele’ will find nothing. However, ‘acelé’ will work. Updating this would be a major undertaking so hopefully it’s not too big a deal – it’s been like this since the new site launched, after all. I also applied the updated I’d made last week to the Iona map interface to all of the other place-names projects.
The remainder of my week was spent going through the feedback I’d been sent from testing of the new data for the Dictionaries of the Scots Language. This included fixing an issue when conducting a search featuring an apostrophe in iOS using the Safari browser. This was adding an a curly apostrophe which then didn’t work with our data. I updated the code to ensure such apostrophes are switched for regular ones when submitted, but when working on this I noted that there are some occurrences of curly apostrophes in the dictionary’s XML files. These characters are treated as entirely different from regular apostrophes and will not be found in a search that contains a regular apostrophe. This led to some discussions with the team about the use of such characters in the data. I also updated the error messages that are displayed when invalid dates are entered in the quotation date filter and spent a long time working through the sparkline data generation code to ensure certain edge cases are resolved.
It’s been a long time since I wrote the sparkline code (I began in June 2023!) and it took some time to refamiliarize myself with the it, but I managed to sort all of the issues you had been reported. Unfortunately these fixes needed to be made to the code that generates the sparkline data that then forms part of the Solr index so no updates were initially available on our test server. Instead I ran many tests on my laptop, both before regenerating the Solr data by targeting specific entries and also through a test instance of Solr and the DSL site running on my laptop after regenerating the entire dataset and as far as I can tell all is working without any unintended consequences being introduced. I then asked our IT people to update the online Solr cores and this was completed by the end of the week.
I also tweaked the display of the sparkline dates in the search results on narrow screens. I’ve updated the layout so that the sparkline text and the sparkline are treated as one single block element. If the page is too narrow for both this and the entry title to all fit on one line then the sparkline and its text drops down to a new line rather than different sections being split over different lines. I also updated the CSV output to remove the search result snippets from them and had a chat about the tagging of certain bibliography entries in the data.