I began the week working for the Anglo-Norman Dictionary.  I completed the task of importing 13 new XML source texts into the taxbase and I also continued to investigate the speed issues that still seem to be affecting the site.  I figured out that while direct calls to the site’s API were pretty speedy, using the AJAX scripts to connect to the API (which the site does) was resulting in some lengthy loading times.  This demonstrated that the issue was not the speed of the database or the server, but instead that there was some blockage occurring between the API and the public website.  Further investigation uncovered that the AJAX calls were all being routed through the University’s web cache, which was not strictly necessary as both the site and the API are hosted within the University network.  When I disabled this routing the speed increase was remarkable, and it’s a relief to get to the bottom of the issue.

On Tuesday I had a meeting with Deven Parker to discuss her playbills project.  Her partners in Computing Science have made pretty amazing progress with getting ChatGPT to perform OCR on and extract structured data from the playbills images, and have so far processed around 4000 images.  The AI tool has been able to divide the images into individual plays, extract titles, dates, theatres, actors and roles and also to assign genre to the plays.  It has also been able to ascertain which list of person names are the actors and which the roles in each play, which is pretty amazing.  I haven’t seen any actual data exported from ChatGPT yet, but apparently it’s all formatted as JSON so I should be able to work with it quite easily.  The next step will be to arrange hosting for the project, although as the total collection of images numbers between 100 and 150,000 and takes up more than 700GB this might be quite tricky.  I’ve submitted a helpdesk request to enquire about this but I haven’t heard anything back yet.

Also this week I had several email conversations with the Dictionaries of the Scots Language people about the upcoming rollout of the new dataset and some issues regarding citations that have multiple dates.  Next week I’ll be processing a new batch of data for the dictionary.  I also sorted out access to the web stats for the Speech Star website for Eleanor Lawson and fixed a glitch in the placenames CMS that Alasdair Whyte had spotted with his Mull site.

I also dealt with a discrepancy with the Books and Borrowing facts and figures that project PI Katie spotted.  When looking at the stats for all libraries the number of borrowers listed in the first infobox was 11,194 whereas the number of borrowers mentioned in the occupations section was 11,197.  This required a bit of investigation.  The overall number of borrowers at the top of the page is calculated by adding up the borrowers of each gender at each library (Male, Female, Unclear and Unknown) while the number of borrowers in the occupation section was calculated from the total number of borrowers overall at each library.  The discrepancy was arising because it turns out we have three borrowers in the system who don’t have a gender specified – their gender is set to ‘null’.  They are therefore not getting picked up by the first calculation.  As an initial fix I made the occupations count use the same figure as the count in the intro so at least things look consistent, but I then needed to assign a gender to the three erroneous borrowers and then regenerate all of the cached data for the site.  This was a fairly lengthy process involving the execution of several data processing scripts and requiring the regeneration of the Solr index files.  Once all this had been completed we actually ended up with 11,198 borrowers in the system as since the last cache generation a further borrower had been added.

I spent the rest of the week adding LiDAR data to the map interface I’ve developed for the place-names websites, starting with Ayrshire.  The data comes from here: https://remotesensingdata.gov.scot/data#/list and the NLS have made it available (see https://maps.nls.uk/guides/lidar/#re-use) and it’s a fascinating resource to be able to incorporate.  Adding it in required some pretty major reworking of the code for the map, but it has been more than worth it.  The ‘display options’ map menu now contains an option to turn the LiDAR layer on or off, and if it’s on you can also change the opacity of the layer, thus allowing you to view whichever base map is selected.  So for example, ‘Stair Mount’ is one of several ‘mounts’ that were created to commemorate military service in the 1740s.  It appears on the OS 1881 map:

And can be viewed on the satellite photography:

But LiDAR lets us see several concentric circles that are not otherwise visible, plus other hidden details:

It’s really fascinating to just pan around the map looking for hidden features in the landscape and it has the potential to be a hugely useful research tool.

I also fixed a bug I spotted that was causing the exact map position and zoom level to be lost when sharing URLs of maps that featured search / browse results.  In such cases the search or browse was being performed and the map would position itself to the extent of the search results rather than retaining the specific view, which was not especially helpful for people wanting to share specific views of the map like the ones above.  Additionally, I updated the ‘display options’ to make the buttons in-line rather than being one per line which means the section takes up less space, as you can see in the above screenshots, plus I also updated the ‘Attribution and copyright’ pop-up (linked to in the bottom right of the map) to include information about the LiDAR data.  I then applied this update to Iona as well.  Unfortunately there is no LiDAR data available for Iona yet, but the other updates to the map interface were at least applied to this resource.