I was unfortunately off work sick from Monday to Wednesday this week, although I still managed to keep up with my emails and respond to queries regarding the new Nairnshire place-names project, the DSL, Garrick Allen’s new proposal, Matthew Creasy’s new proposal and to update Deven Parker on the continued lack of a response from IT Services regarding the allocation of server space for her playbills project.
I returned to work on Thursday and split most of my time between VARICS and the application of my new ‘map first’ interface to the Kirkcudbrightshire place-names resource. For VARICS I began planning the new ‘lookup’ feature, that will allow a speech therapist to select a measurement, input some details about their patient (age, sex and specific details of what is being measured, such as the duration of a feature of speech). I worked with the sample data and information I’d been sent, plus my notes from last week’s meeting, to begin considering the requirements for such a resource. This involved a further email conversation with project RA Mridhula, as several questions arose as I familiarised myself with the data. I have a much clearer understanding of what is actually required now, and I have begun to set up a mockup by importing the data into our online database. I’m hoping to have the time next Monday to continue with this.
For the place-names project I adapted my ‘map first’ place-names interface that I originally developed for the Iona project, and that I’ve subsequently applied to Ayrshire and Nairnshire, to the Place-Names of Kirkcudbrightshire resource (also known as the Place-Names of the Galloway Glens project – https://kcb-placenames.glasgow.ac.uk). None of the resources have yet to be made publicly available, and this is the first resource that already has a live place-names interface, with the current version available here: https://kcb-placenames.glasgow.ac.uk/place-names/. This means that when we do launch the new interface I’ll need to ensure there are redirects from all of the old pages to the new ones, which will take some time to sort out. For example, the URL for a search results page on the currently live site will need to redirect to the corresponding search results page in the new interface. This might prove to be slightly tricky, and it’s something I will still need to tackle when we’re ready to launch.
In order to set up the new interface I needed to create a new version of the API, as the current API is not set up to process the requests the new interface requires. With this in place I could then copy the new interface code from one of the other projects and adapt it to work with the new area. This involved updating the citation details, creating a new resource logo and a few other relatively minor tweaks.
One issue that did crop up when setting up the new resource is that the map was centring on a place far from the actual area of study. It took a bit of investigation to understand why this should be, and the issue turned out to be some erroneously located data. There are seven place-names situated in the wrong place, all of them in a vertical line off the coast of Aberystwyth, as the following screenshot demonstrates:
I’m not sure what’s going on with these so I’ve asked the researchers to take a look. There is also the place-name ‘Knockmurray’ that is currently located between Gartcosh and Coatbridge to the north of Glasgow that needs repositioned. These are causing problems because the new map interface positions the map to fit in all of the visible data, but the map has a minimum zoom level. Due to these errors the visible data is spread over a huge area and fitting them all on screen would require the map to be zoomed out further than the zoom level limits allow, meaning the map ends up showing a random area somewhere between the extremities of the data (currently the Isle of Man). Once the map markers are repositioned this issue will automatically correct itself.
I managed to complete the new interface on Friday, including adding in LiDAR data that covers about 80% of the area of study, which I’m sure will be useful to some people. Also on Friday I helped Ophira Gamliel with an issue she was having with a license for the Oxygen XML editor.
I will be working on Monday next week and then I’ll be off for the Christmas holidays.
My big task for this week was to handle a major update to the data for the Dictionaries of the Scots Language. Although I’ve run some test updates on my laptop in recent months, this is the first update of the online data since 2022, so it was a bit of an important step. For now we’re only updating our test version of the website with the new data, with a further update planned for March before an actual rollout to the live site. On Monday I received the exported data from the DSL’s editing system and ran through all of the steps required to process the data for use on the website, running everything on my laptop. This included importing all of the entries into our online system, extracting citations and dates, generating search data and outputting the data that will be ingested into the Solr search cache. In previous test imports I’d spotted issues with some citations not having usable dates – there were 56 the last time I processed the data. This time, however, there were only four such issues which is not bad going at all, considering there are a total of 748,960 citations in the system.
I managed to complete the data import and generation processes on Monday, but I don’t have direct access to the DSL’s online Solr indexes and instead I need to ask the University’s IT people to handle the updates on my behalf. This was done on Wednesday afternoon, but unfortunately I ran into some issues. I needed to update the online database tables that we use, but the administrative interface I use was limited to files of less than 100MB. As the DSL’s database was almost 400MB, with the largest table taking up more than 300MB, this presented a problem. I asked our IT people to raise the limit, which they did, but I still encountered problems uploading my file. I was working on campus when attempting this and the file upload was timing out with an Apache error. My helpful contact in IT Services suggested I uploaded the file to the server so he could run the import command, which I did. However, there appears to be a major problem with uploading / downloading data from the server via sFTP as I was getting upload speeds of around 22KB/s, which is unworkably slow. It took almost 3 hours to upload my 400MB file, but after that the IT guy was able to run the command and import the updated database.
When testing everything out I realised there was another issue, namely that the online versions of the Solr indexes contained more files than the version I had running on my laptop. I realised that while I’d created new indexes on my laptop, to make it easier for our IT people to process the updates I’d asked them to reindex the existing cores rather than creating new ones. The script to import the updated data only edits existing files and adds new ones – it doesn’t delete existing files that are not included in the new import file and therefore any deleted entries or citations were still in the index. Thankfully I found a way to delete all files from a Solr index and asked our IT people to run this command before importing a fresh version of the data from the import files and this solved the problem.
There were another couple of updates that needed to be made to the system following the data import, and by this time I was working on the update from my desktop PC at home. As always, I download a fresh copy of the site before I work on the files, but I was still encountering a speed issue when downloading files. I had thought the problem I encountered when uploading the data was caused by some issue with the WiFi connection in my office on campus, but using a wired connection on an entirely different network at home I was also encountering download speeds of around 22KB/s, which is unworkably slow – like going back to the 90s and a dial-up connection. I put in another IT job about this and hopefully our IT people will be able to figure out what’s clogging things up, as when I uploaded files to the server on Monday I wasn’t experiencing such issues.
The first task I still had to complete was to add links from SND entries to the DSL’s phonetic description text (https://dsl.ac.uk/about-scots/the-scots-language/), which I’d handily added IDs to the paragraphs of a few years back. The SND’s XML contained numerical references to these paragraphs in new tags so it was simply a matter of updating the XSLT to find these tags, extract the ID’s and generate links to the relevant paragraph.
The second task was to update the way the ‘about’ text for each entry is generated, specifically the statement about the entry not having had major revisions yet. Unfortunately I ran into some issues with this task. This task required processing a new attribute in the entry XML files and I only started looking into this after I’d completed all other processing. I would need to update my data processing script to extract the new ‘typeC’ attribute now contained in the entry XML and update the database to store it. This is not a major job, but I would then have to re-run all of the scripts so it might be best if I leave this until the next update in March. Also, it looks like every entry currently has the value ‘28’ for the new ‘typeC’ attribute but for now I’m unclear when this refers to. The ‘About’ table in the database is used to match up the ‘typeA’ and ‘typeB’ attributes to their corresponding ‘About’ statements. For example SND ‘dreich’ (https://dsl.ac.uk/entry/snd/dreich) has typeA=”13” and typeB=”27” which in combination gives the statement ‘First published 1952 (SND Vol. III). Includes material from the 1976 and 2005 supplements.’. However, the ‘About’ table row with id 28 is ‘First published 1965 (SND Vol. VI).’ and presumably this is not what typeC=”28” refers to. I suspect that ‘28’ might not be the correct ID and I’ve contacted the DSL people to find out more.
Also this week I worked on the design for a new ‘Word of the week’ feature for the DSL. I also spent some time on the Anglo-Norman Dictionary. The editor Delphine had accidentally deleted an entry and I reinstated it for her, but in doing so I realised that this entry did not have any search data associated with it. I’m not sure why this should be the case as deleting an entry only deactivates it in the database, it doesn’t actually delete anything. I re-uploaded the entry via the management system and this regenerated all of the search data (e.g. forms, citations, usage tags), but the lack of search data was a little concerning as it meant that other than via the browse and ‘jump to’ options the entry would be otherwise unfindable. I then searched the database for other live entries that are lacking search data. I searched for entries that have no citation data and entries that have no form data and there are 103 of the former and 8 of the latter. Delphine checked these out and thankfully the entries without citation data are valid, but the ones without forms are errors. These could be fixed by reuploading new XML files for them using the management system, but it’s something we’ll have to keep an eye on as I’m still not sure how the situation arose.
Other tasks this week included setting up the new subdomain for the place-names of Nairnshire project, which involved setting up the content management system, the parish boundaries, importing the GB1900 data (2520 names across the five parishes), creating historical forms associated with the ‘OS 6 inch 2nd edn’ source (used for GB1900), running a script to generate altitude for all place-names via Google Maps and setting up the map interface in the front-end, including LiDAR data.
I also met with the VARICS project team this week to discuss the development of a series of new features for their website. This was a hugely useful meeting as it helped me to understand exactly how one of the features in particular will function, which I was somewhat confused about prior to the meeting. It all became a lot clearer and it was great to meet with the rest of the team, even if it was just on Zoom. I also had a chat with Craig Lamont about demonstrating the Burns Correspondence map, spoke to Sofia about the changes I’d made to the Iona map last week and had a chat with fellow developer Faidon about a WordPress-based crowdsourcing project he’s beginning to plan.
I began the week working for the Anglo-Norman Dictionary. I completed the task of importing 13 new XML source texts into the taxbase and I also continued to investigate the speed issues that still seem to be affecting the site. I figured out that while direct calls to the site’s API were pretty speedy, using the AJAX scripts to connect to the API (which the site does) was resulting in some lengthy loading times. This demonstrated that the issue was not the speed of the database or the server, but instead that there was some blockage occurring between the API and the public website. Further investigation uncovered that the AJAX calls were all being routed through the University’s web cache, which was not strictly necessary as both the site and the API are hosted within the University network. When I disabled this routing the speed increase was remarkable, and it’s a relief to get to the bottom of the issue.
On Tuesday I had a meeting with Deven Parker to discuss her playbills project. Her partners in Computing Science have made pretty amazing progress with getting ChatGPT to perform OCR on and extract structured data from the playbills images, and have so far processed around 4000 images. The AI tool has been able to divide the images into individual plays, extract titles, dates, theatres, actors and roles and also to assign genre to the plays. It has also been able to ascertain which list of person names are the actors and which the roles in each play, which is pretty amazing. I haven’t seen any actual data exported from ChatGPT yet, but apparently it’s all formatted as JSON so I should be able to work with it quite easily. The next step will be to arrange hosting for the project, although as the total collection of images numbers between 100 and 150,000 and takes up more than 700GB this might be quite tricky. I’ve submitted a helpdesk request to enquire about this but I haven’t heard anything back yet.
Also this week I had several email conversations with the Dictionaries of the Scots Language people about the upcoming rollout of the new dataset and some issues regarding citations that have multiple dates. Next week I’ll be processing a new batch of data for the dictionary. I also sorted out access to the web stats for the Speech Star website for Eleanor Lawson and fixed a glitch in the placenames CMS that Alasdair Whyte had spotted with his Mull site.
I also dealt with a discrepancy with the Books and Borrowing facts and figures that project PI Katie spotted. When looking at the stats for all libraries the number of borrowers listed in the first infobox was 11,194 whereas the number of borrowers mentioned in the occupations section was 11,197. This required a bit of investigation. The overall number of borrowers at the top of the page is calculated by adding up the borrowers of each gender at each library (Male, Female, Unclear and Unknown) while the number of borrowers in the occupation section was calculated from the total number of borrowers overall at each library. The discrepancy was arising because it turns out we have three borrowers in the system who don’t have a gender specified – their gender is set to ‘null’. They are therefore not getting picked up by the first calculation. As an initial fix I made the occupations count use the same figure as the count in the intro so at least things look consistent, but I then needed to assign a gender to the three erroneous borrowers and then regenerate all of the cached data for the site. This was a fairly lengthy process involving the execution of several data processing scripts and requiring the regeneration of the Solr index files. Once all this had been completed we actually ended up with 11,198 borrowers in the system as since the last cache generation a further borrower had been added.
I spent the rest of the week adding LiDAR data to the map interface I’ve developed for the place-names websites, starting with Ayrshire. The data comes from here: https://remotesensingdata.gov.scot/data#/list and the NLS have made it available (see https://maps.nls.uk/guides/lidar/#re-use) and it’s a fascinating resource to be able to incorporate. Adding it in required some pretty major reworking of the code for the map, but it has been more than worth it. The ‘display options’ map menu now contains an option to turn the LiDAR layer on or off, and if it’s on you can also change the opacity of the layer, thus allowing you to view whichever base map is selected. So for example, ‘Stair Mount’ is one of several ‘mounts’ that were created to commemorate military service in the 1740s. It appears on the OS 1881 map:
And can be viewed on the satellite photography:
But LiDAR lets us see several concentric circles that are not otherwise visible, plus other hidden details:
It’s really fascinating to just pan around the map looking for hidden features in the landscape and it has the potential to be a hugely useful research tool.
I also fixed a bug I spotted that was causing the exact map position and zoom level to be lost when sharing URLs of maps that featured search / browse results. In such cases the search or browse was being performed and the map would position itself to the extent of the search results rather than retaining the specific view, which was not especially helpful for people wanting to share specific views of the map like the ones above. Additionally, I updated the ‘display options’ to make the buttons in-line rather than being one per line which means the section takes up less space, as you can see in the above screenshots, plus I also updated the ‘Attribution and copyright’ pop-up (linked to in the bottom right of the map) to include information about the LiDAR data. I then applied this update to Iona as well. Unfortunately there is no LiDAR data available for Iona yet, but the other updates to the map interface were at least applied to this resource.
I completed work on and submitted the Speak For Yersel abstract for the DH2025 conference this week. The acceptance rate for this conference is pretty low so we’ll just need to see how we get on.
I spent most of the rest of the week working on place-names projects. We have a new ‘place-names of Nairnshire’ project starting up soon and I set up the systems for this project. The most recent version of the place-names CMS is one I’d developed for the Comparative Kingship project. For this version I’d migrated everything to Bootstrap so the interface has a nicer, more intuitive layout. However, since creating the CMS Bootstrap has made some significant changes to the layout of forms and in order to use the most recent version of Bootstrap I needed to go through the code and change a lot of things. For example, ‘select’ elements previously had the class ‘form-control’ but if this is used with the most recent version of Bootstrap then select boxes don’t have a visible down-arrow to make it clear there is a drop-down list. In order to make this appear the class needs to be changed to ‘form-select’.
I’d also set up the CMS to be bilingual, with fields for both English and Gaelic forms, as this was required for Iona and Mull. However, this is not necessary for Nairnshire and I therefore updated the interface to remove the Gaelic boxes, which makes things a lot more streamlined. Once I’d completed work on the CMS I then replaced the Comparative Kingship CMS with the updated code, as it too didn’t need all of the Gaelic fields.
I then imported the data for the Nairnshire parishes from the GB1900 crowdsourced data. It took some time to get this set up as I had to redownload the full GB1900 dataset and import it into a new database. With this in place I could then extract the data for the new parishes, resulting in several thousand place-names getting added to the CMS, plus links to the OS second edition source and the creation of historical forms. What I haven’t been able to do yet is set the altitude for these place-names as I’m currently setting everything up on an instance hosted on my laptop as the new domain for the project has not yet been set up. My Google Maps API (used for requesting altitude data for a given latitude and longitude point) will only work from sites hosted on ‘glasgow.ac.uk’ so connections from my laptop are refused. Once the domain is set up I’ll ensure this data is generated. I also began working on setting up the ‘map first’ interface for the project’s front-end but again I’m limited in what I can do here until we get the actual domain set up – hopefully next week.
Also for place-names this week I investigated an issue with the Fife place-names resource that Carole Hough had spotted: a browse by classification code was resulting in a blank screen. Thankfully no other part of the site was affected and once identified it was relatively easy to fix the issue – something that had been introduced during the migration of the site to the new server and had previously been overlooked. I also looked into an issue with the Ayrshire CMS that Simon Taylor had identified.
Also this week the AND people sent me 13 new XML files that needed to be added to the Textbase. This is quite a lengthy process as I need to check the texts are in the correct format, rename and upload them and then run them through multiple stages in order to assign genre, link to the lists of source texts, extract individual pages to enable the jumping to a specific page and then generate all search data for the texts for the concordance view (e.g. KWIC). I only got round to working on this on Friday afternoon and I still have more than half of the texts left to tackle, which I’ll do next week.
There was another AND related issue I investigated this week. The site has been running rather slowly recently and I asked our IT people to investigate this. The most noticeable issue was that when loading in a dictionary entry the links to source texts appear as IDs only. The full text details for these are then loaded in via an AJAX call, but this call was taking more than 20 seconds to complete, when previously it was happening instantaneously. I did some investigation and realised that the API call was including a lot of fields that were not actually needed on the entry page, such as lists of items for each sigla, bibliographies and notes. I therefore created a new version for the entry page that only outputs the two fields that are used (slug and siglum) and this now loads much more quickly. We’ll just have to keep an eye on the overall speed of the site as it would appear that the server is struggling somewhat.
In addition to the above I had an email conversation about the hosting of the OHOS project website, made a couple of tweaks to some data and investigated web statistics for Eleanor Lawson, spoke to Garric Allen about another new proposal he’s putting together (for the AHRC this time) that I’ll probably be involved in, and had a chat with William Ashford of the DSL about the ancillary pages we host on the live and development sites.