On Monday this week I attended a ‘getting started with Transkribus’ workshop held at the University of Edinburgh.  This was a really great opportunity to learn more about the Transkribus tool and to be guided through the initial stages of using it.  I had somewhat naively thought that it would be possible to just upload a digitised image to the tool and it would then be able to extract and format the text with a high degree of accuracy, but in reality much more manual intervention is required before Transkribus can work effectively.  Training the tool on your data is absolutely vital, and this means uploading sample images, identifying lines of text (which can be done automatically but may need manual tweaking) and then transcribing the text found on each line using Transkribus’s web-based text editor.  Text can also be tagged using this facility, such as noting headings, or identifying proper names.  The leader of the session suggested that around 15,000 words of text should be manually transcribed in order for the tool to be well trained, which would take a considerable amount of time and effort.

I wanted to see whether Transkribus could extract the text from the Edinburgh Gazetteer (https://edinburghgazetteer.glasgow.ac.uk/the-gazetteer/)  and the first stage of this would be for the tool to recognised that each page is divided into four columns, many with a full-width heading at the top.  A ‘layout’ model would need to be run on the pages in order to automatically identify the columns.  There are a couple of existing newspaper layout models available but unfortunately when I ran my test image through them neither managed to divide the page, and the entire page was simply identified as ‘Region 1’, meaning all identified lines ran across all four columns.  Any human looking at the image would be able to ascertain immediately that it’s four columns of text in so I was a bit disappointed that Transkribus wasn’t able to do the same.  I spoke to the leader of the workshop, who has successfully used Transkribus on another historical newspaper and he suggested that I’d have to train Transkribus on this specific newspaper and that in order to do so I’d need to use a ‘Field model’.  It’s not entirely clear to me how a ‘Field’ differs from a ‘Layout’ model, but unfortunately it’s not possible to create a field model with the free Transkribus account – you instead need to subscribe to a ‘Scholar’ account at €20 a month, something I didn’t want to do as I was just experimenting at the workshop, so I reached something of a dead end.  Having said that it was still very useful to experiment with Transkribus in a workshop setting with an expert on hand and I will continue these experiments when I have more time available.  However, I do wonder how more general AI tools might cope with the same materials and whether they might be quicker to use.  It’s definitely something that would be worth exploring further.

For the rest of the week I worked on several different projects.  I spent some time refamiliarising myself with the structure of the Bilingual Thesaurus of Everyday Life in Medieval England (https://thesaurus.ac.uk/bth/) as a developer who will be working with the structure for a new project got in touch with questions about it.  I wrote a data description document that describes every table and column in the database, and also contains the highest ID in each table used for BTH.  There are several columns that are only present as references to the original spreadsheet and are not used by the online resource, and I’ve made a note of these.  Whilst working on this I also spotted an issue with the display of languages in the advanced search.  These were not displaying any content and it turns out that this was because one of the referenced column names was ‘grouping’, which is a reserved word in MySQL.  While the queries worked on our old server, the new server is more strict about such things and the queries failed to run on it.  Having spotted this it was thankfully quick to fix – surrounding the column name with backticks (`grouping`) in the queries got things working again.

I also returned to working on the Books and Borrowing project for a few hours this week in order to update the links we have in our system to books held at St Andrews University library.  The library has recently changed their online system, which has broken all existing links to records in the system.  Thankfully they have a huge spreadsheet featuring the old identifiers and the corresponding new ones and I was able to write a little script that would identify the St Andrews book holding records in our system and update their links.  Of the 7560 active St Andrews book holding records in our system 1666 have URLs and my script successfully updated 1662 of these.  The remaining four have issues that will need to be looked at separately, and I’ve let Katie and Matt know about these.

I met briefly with Alan Riach this week to discuss the website I created for him last week.  He gave me a further CD featuring a recording he wanted me to host and I was able to extract the recording and update the website to add this in.  I also joined my fellow College of Arts and Humanities developers for a coffee and a catch-up this week, and it was really great to chat with them all again.  Hopefully we’ll meet up again soon.

On Thursday this week I also had a meeting with Rhona Brown, Amy Wilcockson and Clare Thompson from the Mitchell Library to discuss an online exhibition for the Robert Fergusson website (https://robert-fergusson.glasgow.ac.uk/).  There was a physical exhibition of materials at the Mitchell last year and we’re going to be bringing these together online in the next few months.  It was good to meet everyone in person and hopefully I’ll be able to help them create an exciting online resource based on the materials.

Also this week I managed to fix the issue with capital letters not working in the ‘jump to’ option in the ‘browse’ facility of the Anglo-Norman Dictionary.  You can now enter text and it will find matching entries whether they have capitals or not.  E.g. enter ‘abc’ (or ‘ABC’) into the ‘jump to’ box.  Brackets are also not an issue, e.g. type in ‘xilo’ and you’ll jump to ‘[xiloaloe]’.  The only thing to bear in mind is that the browse is accent sensitive.  It has to be as we have entries with headwords that are identical other than for accents.  So for example entering ‘acel’ will jump to ‘acelément’ but typing in ‘acele’ will find nothing.  However, ‘acelé’ will work.  Updating this would be a major undertaking so hopefully it’s not too big a deal – it’s been like this since the new site launched, after all.   I also applied the updated I’d made last week to the Iona map interface to all of the other place-names projects.

The remainder of my week was spent going through the feedback I’d been sent from testing of the new data for the Dictionaries of the Scots Language.  This included fixing an issue when conducting a search featuring an apostrophe in iOS using the Safari browser.  This was adding an a curly apostrophe which then didn’t work with our data.  I updated the code to ensure such apostrophes are switched for regular ones when submitted, but when working on this I noted that there are some occurrences of curly apostrophes in the dictionary’s XML files.  These characters are treated as entirely different from regular apostrophes and will not be found in a search that contains a regular apostrophe.  This led to some discussions with the team about the use of such characters in the data.  I also updated the error messages that are displayed when invalid dates are entered in the quotation date filter and spent a long time working through the sparkline data generation code to ensure certain edge cases are resolved.

It’s been a long time since I wrote the sparkline code (I began in June 2023!) and it took some time to refamiliarize myself with the it, but I managed to sort all of the issues you had been reported.  Unfortunately these fixes needed to be made to the code that generates the sparkline data that then forms part of the Solr index so no updates were initially available on our test server.  Instead I ran many tests on my laptop, both before regenerating the Solr data by targeting specific entries and also through a test instance of Solr and the DSL site running on my laptop after regenerating the entire dataset and as far as I can tell all is working without any unintended consequences being introduced.  I then asked our IT people to update the online Solr cores and this was completed by the end of the week.

I also tweaked the display of the sparkline dates in the search results on narrow screens.  I’ve updated the layout so that the sparkline text and the sparkline are treated as one single block element.  If the page is too narrow for both this and the entry title to all fit on one line then the sparkline and its text drops down to a new line rather than different sections being split over different lines.  I also updated the CSV output to remove the search result snippets from them and  had a chat about the tagging of certain bibliography entries in the data.