For the Anglo-Norman Dictionary this week I made a couple of major updates to the data. Firstly, a researcher had gone through the whole of ‘D’ to compile links from each entry to the DEAF dictionary. This took the form of a spreadsheet and I adapted the script I’d previously written to deal with DEAF links in order to work with this new data. The script updated the XML of each entry so I made sure I backed up the data first, and ran everything on a test version prior to updating the live site, but thankfully all went smoothly and entries such as https://anglo-norman.net/entry/dubbeour now have links out to the DEAF.
The second major update was a new batch of entries and major revisions of existing entries that the team have been working on over the past few months. This covers all of ‘W’ plus may other entries across the entire dictionary. As with previous updates, I ran and tested everything locally before updating the live site, as the first step in such an update is to delete the entirety of the letter that’s being updated. After running everything on my laptop and testing things out I spotted a couple of issues with the entries and in discussion with the editor Geert we managed to sort them out. In total the update added or edited 1918 entries, of which 1161 were ‘main’ (rather than cross-reference) entries. These included 1245 main senses, 34 subsenses, 77 locutions, 41 locution senses, 1 locution subsense and 2460 citations.
Also during this process we spotted that cross references were not being listed properly in the dictionary’s content management system. The code that regenerates the cross references after changes are made to the data wasn’t specifically identifying and targeting the ‘active’ versions of entries and sometimes the links that were stored were to inactive versions, which then weren’t displaying, as the code for displaying them was specifying active records only. However, I updated the code and regenerated the cross references so they all now appear.
Also this week the editor Karen spotted that links to the FEW dictionary from commentaries weren’t working and I updated the code to fix this. I also spotted that the dictionary browse feature on the entry page wasn’t working when entries featured upper case letters. I tried a few approaches to sorting this, but the ‘slug’ field in the database needs to be case sensitive as this is the only way to make the field accent sensitive too, which is needed as we have slugs for different entries that are identical other than for an accent. I did try swapping the field the browse targets from the slug to the lemma, but this introduced further issues because the lemma can include brackets, plus the ‘next’ and ‘previous’ links in the browse rely on the slug field. I’m going to have to return to this next week.
Also this week I created a mini-site for Alan Riach’s media. This involved extracting audio and video files from CDs and DVDs, which was slightly time consuming. The DVDs were rather tricky to extract as they were in multiple parts that then needed joined together. Also, one of the DVDs was corrupted and I was only able to extract half of it. I also had to migrate transcriptions of four hour-long audio files from Word to HTML and of course create the site structure and design. The site is password protected and is intended for students and educational use only but here is a screenshot:
Also this week I addressed some issues that had been uncovered whilst testing the advanced search facilities of the test version of the Dictionaries of the Scots Language that is set up with the latest version of the data. The first issue was that in bibliography searches some entries appear twice in the drop-down list. This was happening because entries can have both long and short titles and authors and the drop-down list was accommodating both. I’ve updated this so that just the author and title that are picked as the ‘display’ versions during import are now displayed, so there are no more duplicates. However, this does mean it’s not always clear why an entry appears in the list. For example, if you select ‘author’ and enter ‘ham’ you’ll see ‘Belhaven, J. H.’ and it’s not until you click through that you can see that ‘Ham’ comes from ‘Hamilton’.
The second issue, also relating to the bibliographies was that some items were not being found in the search. This was because the import script was only picking out titles and authors for bibs that had an ‘oldid’. Bibs that didn’t have these were still being processed and display titles and authors were being stored, but as no additional titles and authors were being extracted the searches therefore weren’t working. This has now been sorted – the display titles and authors are also now being searched in addition to the other titles and authors.
Whilst testing this out I also discovered that selecting items from the drop-down list that featured apostrophes and colons broke the search and led to an empty search results page. For example a title search for ‘cottager’ and then selecting the “cottager’s daughter” item. This is still broken on the live site, but is now fixed on the test site.
Also this week I liaised with our IT Services and technical people at The National Archives to arrange for the OHOS domain to be transferred to the TNA. I also responded to a query about the Bilingual Thesaurus of Everyday Life in Medieval England, had an email conversation with Matthew Creasy about his Mallarmé project and spent some time making updates to the Iona map interface following our team meeting last week.
I think I’ve covered everything that was raised at the meeting. I created a space for each categorisation at the top of the legend with some placeholder text for now (e.g. ‘Info about language classification’). For now when you hide the legend the description remains visible, as I figured it might be useful to know what the markers represent, but I can make this hide along with everything else. I’ve also updated the altitude classification to 20m increments and added in language tooltips wherever languages appear (e.g. in the small popup and the full record). Parts of speech in the full record also now have tooltips, although I’m afraid there are a few that I can’t put a name to and there are others I may have got wrong, so these may need further work. I updated the satellite views so that they now feature markers with a white border. And I’ve fixed issue with the top legend tick box ending up unticked in Chrome. In addition, all language combinations in the legend for language now feature tooltips (if they are abbreviated). I’ve applied most of these updates to the other place-names resources too. I still have to add in the tooltips (points 3 and 4) to the other sites, but I will do so next week.
I divided my time between many different projects this week. For the Anglo-Norman Dictionary I made a few further tweaks to the new cognate references section that we went live with last week, and I also received a new batch of several hundred new or reworked entries that are to be added to the dictionary, which I’ll process next week. For the Dictionaries of the Scots Language I had an email conversation with editor Ann Ferguson about the structure of URLs and also gave some consideration to the bibliography search, based on feedback I received towards the end of the week. I’ll continue to work on this next week.
I also found a bit of time this week to begin work on a new version of the Historical Thesaurus website. The current website’s backend is a bit of a mess, as it was cobbled together over many years as new features were added. What I’ve been meaning to do for a while now was to rework this in order to extract all data querying and processing code from the front-end, separate it out and develop a single, unified API for the site, an approach I’ve taken with all of the large projects I’ve been involved with for many years now. This is going to be a rather large task, but I made a start this week: I set up a new version of the website and began the process of creating the API. So far I’ve only got the ‘random category’ feature on the homepage working with the new API, but I’ll continue working away at this as time allows until it’s all done. There won’t be any change to the front-end (although I’m also separately considering a Bootstrap-based refresh of the interface), but the update will leave the data querying side of things in much better shape for the future.
For the VARICS project I continued to work on the new look-up feature. Following last week’s traffic light experiments the team requested a more simple approach for the traffic lights, so I created new images featuring a subtle gradient and a white border, as you can see below:
I also added in a new ‘Try again’ button that would reload the form, which was subsequently renamed ‘Start again’. The team prefer this simpler approach so it looks like this is what we’ll be going with. I now need to wait until further data is sent to me before I continue to develop the feature.
For the place-names projects, I spent some time investigating an issue with the advanced search in the new map interface. An element language search was not working unless a specific element was also selected. For example, a search for all place-names containing Old English elements wouldn’t run, but selecting Old English and then selecting the element ‘tūn (OE)’ worked perfectly. It took a while to track down the issue, but thankfully once identified fixing it was trivial: there was a typo in the code for the API. The system was already set up to allow a search for element language without specifying an element, but in such circumstances the code failed to run because of the typo. I’ve fixed this now on all of the place-names sites. I also fixed a strange issue with the advanced search pop-up disappearing when parishes towards the end of the list were selected. This issue was only occurring in Chrome and not in Firefox (my default browser) and an update to the stylesheet for the map pop-up thankfully fixed the issue.
I also participated in an online meeting with the Iona project team on Wednesday this week where we discussed some of the map-related issues that had been brought up at the advisory board meeting and how (or if) we should address these. I ended up with a list of updates that I’ll need to make to the interface, another thing I’m hoping to tackle next week.
Also this week I had a brief in-person meeting with Joanna Kopaczyk and her project RA Pia Lehecka to discuss Joanna’s new DOST digitisation project. It was good to meet the other members of the team and I’m looking forward to working for the project over the next few months.
I also met with Alan Riach in Scottish Literature this week. Jean Anderson had contacted me last week to put me in touch with Alan, who was wanting some help to publish some old radio broadcasts that were previously available on one of our old servers that has since been decommissioned. Alan managed to find some audio CDs containing the recordings and I met with him to collect them, together with some additional recordings and a few DVDs. I spent some time towards the end of the week extracting the audio and video data from this media and creating a simple website where the files and accompanying transcripts will be accessible to students. I haven’t quite finished this task yet, but should have an initial version completed early next week. The resource will be password protected and only available to students and staff, as there may be licensing issues relating to the recordings – for example relating to incidental music used during the recordings.
Much of Monday morning this week was spent working with Andrew McHugh and Luca Guariento to figure out why a large number of our externally hosted websites were displaying nothing but server errors. This took some time to investigate and it turned out that one node was being affected, and this was apparently being caused by over-enthusiastic web crawlers targeting one script repeatedly, resulting in CPU resources getting used up. Once identified, we limited access to the script and everything went back to normal.
I spent a lot of this week continuing with the cognate reference updates for the Anglo-Norman Dictionary. Cognate references are links from AND entries to related entries in other dictionaries and we’re increasing the range of dictionaries that we provide active links to, as opposed to providing plain text references to. As part of this I needed to write a script to update the references to DEAF (Dictionnaire étymologique de l’ancien français) that were embedded in our entries. Last week I wrote a script that identified issues with DEAF references involving sub-headwords in brackets in 295 AND entries and we needed to update the structure of these as discussed last week. The editor Geert sent me an edited version of the spreadsheet that contained the forms that should be included and I wrote a script to extract these, generate the required XML and then find and replace the relevant section of the XML in the live entries. This obviously required quite a lot of testing before I ran it on the live data, but I’d completed and fully tested the script by the end of Tuesday and I successfully ran it on Wednesday morning.
At this point the increased selection of cognate references that we link to were still only available on a test page rather than the live site. I then moved onto updating the layout of the section in the entry page to make it easier to use. Previously the section looked something like this:
It was quite difficult to understand the section: There was no explanation as to what the section was, or what acronyms like ‘FEW’ actually meant. It was also difficult to tell where one cognate reference section ended and the next began. Geert asked me to come up with some new ideas for the section and I spent a bit of time working on a few mock-ups. We finally agreed on a new version that gets rid of the square brackets around the entire section and divides the individual dictionaries into inline-block elements that have a border and a light grey background, thus making it quicker and easier to identify the different dictionaries. I also swapped the order of the headwords and the reference numbers and make all text normal weight. Finally, I added in an ’info’ icon to the top-right that when pressed on opens up a popup with some explanatory text about the section and the dictionaries that are referenced. Here’s how the new section looks:
We went live with this new layout on Friday afternoon and so far it has been very well received. Also for the AND this week I updated the cognate references that could be actively linked to from the entry commentaries to bring these into line with the new cognate reference section.
Also this week I worked on the layout for a new ‘Scots Word of the Week’ page with William Ashford of the Dictionaries of the Scots Language. After a number of experiments I created a layout that suited, and the page is now live: https://dsl.ac.uk/scots-word-of-the-week/. I also fixed an issue with embedded videos being too large for mobile devices on the site and responded to a query from Ann Ferguson about the data import workflow and how URLs are generated.
I also spent some time updating the ‘lookup’ feature for the VARICS project, based on feedback that I’d been sent last week. I added in the proper measure labels and I updated the system so that each measure has a ‘sex’ column. The option to select a sex is only displayed when this is set to ‘Yes’. For our working measures (the maximum phonation ones in the ‘Phonation’ menu) sex is now set to ‘No’, but the other measures still have it set to ‘Yes’. This can easily be updated as more real data becomes available. I also replaced the ‘age’ input with a drop-down list of options and have dropped the ‘red’ rating for now, as suggested by the team. I also removed the ‘alternative view’ that featured all three traffic lights, plus the ‘additional information’ section and updated the output text as requested and limited the display of all numbers to 2 decimal places.
I’ve also spent a bit of time trying to find alternative ‘cartoon/clipart’ traffic light images. Finding ones that are both free to use and have a suitable license has proved to be quite a challenge. Eventually I thought I’d give AI tools a try, and decided to try Microsoft’s image creator (https://designer.microsoft.com/image-creator) as I was already signed into my Microsoft account. The results were somewhat mixed and some rather odd images were created – e.g. traffic lights with four lights, traffic lights with odd symbols on them. Attempting to specify a UK style traffic light gave an image that featured the weirdest attempt at the Union Jack I’ve ever seen. It also totally ignored my request for the traffic light to be against a white background. Here’s a screenshot of the output:
However, I managed to pick out one generated image I thought I could use and I’ve split it up into individual images. I’ve now replaced the images in the lookup with these ones. This is by no means the final version and it’s likely that we’ll change the image again before the feature is completed, but here is now it currently looks:
I’d received some helpful feedback from Eila Williamson of the Berwickshire Place-names project regarding the new map interface I’ve been developing and made some tweaks to the interface based on this. Eila also spotted that there is an issue with the element language part of the advanced search facility. At the moment selecting an element language in the form simply limits the choices that then appear in the ‘Element’ field to your chosen language – e.g. select ‘Old English’ and type in ‘tu’ and only ‘tūn (OE)’ appears. It doesn’t currently let you select an element language and then view all place-names containing it. I’m afraid this is an oversight on my part, as the existing advanced search does indeed let you do this. It’s going to be a pretty major job to update the search to allow this, but it’s something that will need addressed, not just on the Berwickshire site but in all of the place-names sites. I had hoped to have the time to tackle the issue this week but I ran out of time.
Also this week I had an email conversation with Joanna Kopaczyk about her DOST digitisation project that will be commencing next week, I attended a meeting about the teaching of Digital Humanities at Glasgow, I responded to a query from The National Archives about the hosting of the OHOS project, I did a bit of research into some old files that Jean Anderson contacted me about and I attended a meeting in Special Collections for Matthew Creasy’s Mallarmé project.
I divided most of my time this week between three projects: The Anglo-Norman Dictionary, the Dictionaries of the Scots Language and the redevelopment of the Berwickshire place-names resource. For the AND I continued to work on the new links from entries to references in other online dictionaries – the ‘cognate references’. Last week I’d created a test version of the entry page that included links out to listings about the entry on many more online resources and Geert, the editor of the AND tested it out and got back to me with some suggestions.
There was an issues in some entries that link to entries in the DEAF resource, specifically where there are both headwords and multiple sub-headwords in brackets. For example ‘Escharn’ (https://anglo-norman.net/entry/escharn) has the DEAF headword ‘escharnir (eschar, escharne)’. My new linking system doesn’t render such links correctly, and in fact this reference in the entry XML should have been split into two separate DEAF links, one for ‘escharnir (eschar)’ and one for ‘escharnir (escharne)’ as follows:
<DEAF_refs siglum="DEAF" linkable="no"> <link_form>escharnir (eschar)</link_form><link_loc/> <link_form>escharnir (escharne)</link_form><link_loc/> </DEAF_refs>
I therefore wrote a little script to find all entries that feature a DEAF headword that contained a comma or a semi-colon, outputting the entry slug and DEAF headword in a spreadsheet. There are 295 such entries, and some have a lot of different versions in the brackets. We agreed that some manual checking of these will need to be done as a few I checked don’t seem to exist on the DEAF site. E.g. ‘Message#1’ links to ‘metre 2’ (https://deaf.hadw-bw.de/lemme/metre2) with subheadwords ‘message 1’ and ‘message 2’ but neither of these appear in the DEAF entry (‘message’ doesn’t appear in the text of the page at all). Similarly the entry ‘perpetuelment’ links through to ‘perpetüel’, but while the second subheadword in brackets works (https://deaf.hadw-bw.de/lemme/perpet%C3%BCel#perpet%C3%BCellement) the first doesn’t.
We agreed that Geert would use my spreadsheet as a starting point you would add the correct forms one per column from column C onwards. I would then be able to run a script to update the existing DEAF item in the XML with multiple links. Geert worked on the spreadsheet during the week and I’ll write a script to process it next week. Geert also requested a spreadsheet listing all entries that don’t feature a DEAF link as these will probably need some further work. There are 10,831 such entries.
For the DSL I spent some time liaising with the DSL’s email supplier and our IT Services to update some DSN entries. I also removed the results filtering options from the search results of our test instance of the site as the team want to delay the rollout of this feature. I was also alerted to an issue with the new data that was added to the test instance before Christmas. Items that should have been deleted from the data in the recent update were still appearing in the search results, something that shouldn’t have been happening as I’d updated the Solr index to wipe all data prior to the recent upload.
After some communication with our IT people, it turned out that the online Solr index does not match the Solr index I have on my laptop. In the run-up to Christmas I realised I needed to completely wipe the Solr index prior to uploading the new dataset in order to clear out records for any entries that had been deleted from the dataset. I ran some tests on my local instance of Solr running on my laptop to empty and then import the data and all went fine. I don’t have direct access to Solr on our server and I have to ask our IT people to run any commands for me, so I contacted them and asked them to run the delete and import commands. It looks like the delete command (which worked on my laptop) didn’t work on the server, despite the command being exactly the same as the one I successfully ran on my laptop. We re-ran it on the server again and it failed to delete anything. However, after restarting Solr the data was cleared, so I’ll just need to bear this in mind the next time. After deleting all of the data from the indexes we re-uploaded the last dataset from December and thankfully that this has fixed all of the issues. Whilst investigating this I also noticed that some entries have URLs in their XML that won’t work in the front-end due to them having characters that don’t work in URLs such as full stops, commas and spaces. Thankfully there are only a handful of these and I’ve passed them onto the team who will update the XML.
For the new Berwickshire map interface I added in some explanatory text, fixed a couple of typos and made some other minor tweaks to the interface. I received some further feedback from another member of the team on Friday and I’ll need to make some further tweaks next week, but we’re almost ready to launch the new interface now.
In addition to the above I arranged my travel and accommodation for the DH2025 conference in the summer. I also fixed an issue with the library stats pages on the Books and Borrowing site. Unfortunately a bug must have been introduced when I reworked the facts and figures page in October last year. Prior to this each library had its own ‘top ten’ list and this meant that when viewing facts for multiple libraries only items in each top ten were being included (so if Book A at Library X appeared at position 9 but it was at position 11 in Library Y then the latter wasn’t being included in the figures). To get around this I updated the individual library lists to return the full data, then when multiple libraries were selected on the site-wide ‘Facts’ page the system could do a proper count and generate an accurate the overall top tens. What I failed to spot is that where a single library is selected (either on the site-wide ‘Facts’ page or the library’s own page) the list wasn’t being limited to ten, as in these views the code expected there to only be 10 items returned in the dataset and was just displaying all items. I’ve fixed this now. I also created screenshots for our article about the development of the resource for the Victorian’s Institute journal.