I spent a lot of this week working for the Dictionaries of the Scots Language, planning and preparing for some major upcoming changes. Last week I began writing a specification document for a new region / dialect areas interactive map and I completed this on Monday, sending it out to members of the team for feedback and then replying to comments throughout the rest of the week. I then moved on to researching the new XML structure the DSL are using for entries. There have been some major updates to the structure that have been implemented in the DSL’s editing system and will require significant changes to the front-end, the search facilities and the scripts used to process entry XML files after their export from the editing system.
Over the course of the week I wrote a document that detailed all of the changes that will need to be made to the various scripts, databases, indexes and systems to incorporate the changes to the XML structure. I can’t go into too much detail about the new structure or the changes here, other than to say that the structural changes are significant. By Friday I had finished going through all of my systems and the information provided to me regarding the new structure, and had documented all of the updates that would be required. However, during the process a number of questions arose that will need further clarification from the team, and we’ve arranged to meet to discuss things before I begin implementing the updates. Unfortunately due to members of the team being on holiday this likely won’t happen until later in September, but I have plenty of other things to focus on, both for DSL and for other projects, in the meantime.
Also this week I attended online meetings with two researchers who are in the planning stages for new projects. I’d had email conversations with both of them last week, and had arrange online meetings with them this week. The first was with Emanuele Scieri, who works in Theology, and we had a good discussion about his project and the technical aspects it might involve. The second was with Mícheál Ó Mainnín, a place-names scholar at Queen’s University Belfast who is hoping to adapt the place-names system I’ve developed for a new area, and again this was a great meeting and we had some good discussions. I can’t say much more about the projects at this stage, but will need to see how things go.
Other than the above, this week I had an email conversation with Eleanor Lawson about a new project, I made some further tweaks to the VARICS lookup tool, I gave some advice to Robert Davies in the College of Social Sciences regarding interactive maps, and I had an email conversation with Simon Taylor about the updates I’m making to the Place-names of Fife, and I devotes some time on Friday to continuing with the migration of the Fife data to the more standardised structure used by the other place-names resources. I’ve now migrated the main place records, the information about map sheets, classification codes and parishes, although the latter still need parish boundaries to be added. I also began looking at historical forms, but the sources are going to need some significant work, as the source titles and references (e.g. page numbers) are stored in a single field and will need to be split up, which is going to take some processing.
I was on holiday for most of last week and a little of this week, so I’m joining the weeks together in this update. During this time I worked on several different projects and communicated with a number of people about new research projects.
Whilst at the DH2025 conference in Lisbon Jennifer Smith alerted me to a problem with the sending of emails on the Scots Syntax Atlas site. We have a couple of contact forms (including one for requesting the audio files generated by the project) and neither were working. I’d also spotted that the two-factor authentication plugin used when logging into the WordPress admin interface was failing to send the emails containing the log-in codes. This took an awfully long time to sort out as it was unclear what was causing the problem or whose responsibility it was to fix it, compounded by the fact that the site is hosted by an external company. I’d noticed that several other of our externally hosted sites had the same issue with sending the 2FA emails so it appeared to be more widespread than just one site.
I spoke to Andrew McHugh, the head of RCaaS in IT Services and he suggested contacting the third party hosting company’s support. I had a long and useful chat with their support people, who suggested that we’d need to update the DNS setting relating to mail for the scotssyntaxatlas.ac.uk domain (which Glasgow controls) to point to the hosting company’s email systems. This would then enable me to set up SMTP for email for the domain and hopefully get the emails sending again. I therefore submitted a helpdesk request to Glasgow’s IT people asking them to update the record.
At the same time, I wasn’t convinced that this would solve the problem, and it was very strange that the 2FA issue was only affecting certain sites and not others. I realised that it was only the sites that had their own domain rather than being a subdomain of Glasgow that were being affected. I spoke to Luca who confirmed that this was also the case for one of his sites that had its own domain. It was also strange that the WordFence summary emails for these sites were getting sent, which suggested that emails were somehow working. I eventually realised that the 2FA emails were being sent from the email address ‘wp2fa@domain’ while the WordFence emails were being sent from ‘wordpress@domain’. There was an optin in the 2FA plugin to change the email address its messages were sent from so I set this to ‘wordpress@domain’ and… the messages came through.
It turns out that the University must have blacklisted the ‘wp2fa@domain’ emails, meaning all messages from these addresses were being blocked. So there was never an issue with the sending of emails for the domains – the messages were being sent but were being blocked and changing the email address to ‘wordpress@domain’ allowed them to get through.
This meant that what I thought was one problem was actually two, as the contact forms on the Scots Syntax Atlas site were still broken. I’d noticed previously that there appeared to be an issue with the contact forms’ use of Google ReCaptcha, but when I’d disabled this the forms still failed to send. Now I knew the problem was definitely related to the contact form plugin I could investigate this further. I made a copy of the contact form details and then completely deleted the contact form plugin, then reinstalled it and set up the forms again without any involvement with ReCaptcha. Thankfully this fixed the issue and the contact forms successfully sent their emails. It does mean that there may be a lot more spam sent to the email address that receives the messages, and we’ll just need to keep an eye on this.
Also during this period I continued to work with the Fife Place-names data. Previously I’d written a script to add latitude and longitude to all of the place-name data (more than 3000 records) that previously only had grid references. I’d run into some issues with some of the grid references, as discussed previously, and this meant researching the correct grid references to use, which took a bit of time. I managed to complete this process during these two weeks, successfully adding latitude and longitude to all but 50 of the records. These 50 records are mostly for obsolete place-names that don’t have a specific location, and I’ve emailed the project PI Simon Saylor to ask about these.
During this time I also set up a new project website for Gavin Miller and migrated the emblems websites over to a new server in an attempt to ensure the sites don’t use up all of the server’s resources, which was occasionally happening with the previous hosting arrangement. I also made some further updates to the Robert Fegusson online exhibition, based on feedback and further data from Amy Wilcockson.
Most of the remainder of my time was spent working for the Dictionaries of the Scots Language. I had an email conversation with Ann regarding the new part of speech tagging system and a further email conversation with Becca, William and Vasilis about the new structure of the DSL entry XML files. I now have some sample data and some helpful explanatory notes about the new structure so an upcoming task will be to ensure the DSL’s online systems can work with this new structure.
I also had an online meeting with Vasilis, Becca and William on Friday to discuss the new region and dialect area maps that will be added to the DSL website. This was a very useful meeting and we went through Vasilis’ prototype and discussed how this could be turned into a resource that would work on mobile devices as well as larger monitors and how we could make access to the large amount of overlapping spatial data a little less confusing. I then began writing a specification document for the new feature and will continue with this next week.
Also this week I had email conversations with Emanuele Scieri of Theology and Mícheál Ó Mainnín, a place-names scholar at Queen’s University Belfast about new projects that I may be involved with. I can’t say much more about these for now, but I’ll be having online meetings to discuss their proposals in the next week.
I continued to work on the Online Exhibition for the Robert Fergusson project this week, spending much of Monday and Tuesday on the task and completing an initial version that I sent to the team for feedback on Tuesday afternoon. As I worked with the materials and got to know them a bit better I ended up creating a more fully-featured first draft than I was originally intending to make. It is still an initial version, and any aspects can be changed as required. I took inspiration from the Books and Borrowing online exhibition to a certain extent. As with this site, the pages all have an ‘Explore’ section with image-based buttons leading to the various sections, and each page has links to the next and previous sections at the bottom.
I can’t share screenshots or the URL just yet, but the interface is designed to work on all screen dimensions from large monitors to mobile phones and as you scroll down the pages sections animate into place. There are currently two alternative animations. Pages such as ‘Publications’ have a ‘zoom’ animation whereby the section zooms and slides up into place as you scroll. The alternative can be viewed on pages such as ‘Depictions’ where sections are already visible but the content slides in from the left and right as you scroll (except the final video section of ‘Depictions’ which zooms up). Certain pages have the first section fixed (e.g. ‘Publications’) while other pages animate the first section too (e.g. ‘Scots poems’). We might not want to mix and match the approaches so much – I included alternatives so the team can see the options.
The pages currently consist of sections with three different background colours that are alternated (a greeny-grey, a dark grey and a sort of grey-teal colour). The header font is the same one as the main site and the paragraph text is fairly large. Where a section has one image this appears to the left or right (this alternates) in a box that also features the caption. Where there are multiple images (e.g. ‘poems on various subjects’ on the ‘publications’ page) a carousel is used and you can press the arrow buttons to scroll between images. In such cases there is still currently one single caption underneath the carousel.
On the ‘depictions’ page I’ve embedded the YouTube video in the final section. On the ‘Scots poems’ page I’ve included the three audio recordings in a final section with some placeholder text. Where images appear, I’ve applied some post-processing on many of them to remove the yellowing and make them brighter. I think this makes the images look nicer as part of an online exhibition, but the team may prefer to use the originals for the sake of authenticity. I’m pretty happy with how the initial version of the exhibition has turned out. During the rest of the week I had an email conversation with the RA Amy Wilcockson about adding in new content or rearranging the content that was already there, and I spent some of my time making such updates, some of which where rather time-consuming to implement but needed to be done.
On Tuesday afternoon I met with Moira and Leo from Archives and Special Collections to discuss some of the old resources I’d created for the archives around 15-20 years ago. It was great to catch up with them both and to hear their plans for the resources. It turns out that the University Story site has been completely lost – no copy of the site and its database remains on any University servers or other hardware. This was quite a shock as a huge amount of effort went into writing the content of this site – thousands of biographies as well as structured data about (for example) holders of professorships throughout history. When I stopped working for the Archives some 13 years ago a lot of work continued to be done to the site, both in terms of the content and reworking by subsequent developers, but I wondered whether I might still have a backup of the site as I left it somewhere at home.
On Wednesday I hunted about for the data. I found my old backup external hard drive in my attic, but unfortunately this did not include anything I hadn’t already discovered on my more recent external hard drive. I then managed to locate the old laptop that I used when working at the Archives back in the day, thankfully complete with power cable. This took ages to boot up, and I couldn’t remember the password I used to use for it. And the laptop was taking more than half an hour between entering a password and coming up with the ‘incorrect password’ warning. I then decided to set up a ‘boot from USB’ installation of Ubuntu to run on the laptop instead of booting the ancient version of Windows and this worked perfectly. I was able to access the laptop’s hard drive and copy all of the files relating to my Archives work onto my external hard drive. This included one single dump of the UGS database from June 2011, plus all of the code, images, and a bunch of other stuff relating to other projects. I now have this on my desktop PC as well as on the external hard drive. I loaded the SQL dump into a MySQL instance on my new laptop and it looks like the data is complete (albeit from 2011). It consists of 93 tables, incorporating the main UGS site, the international story, the ‘world changing’ site and the WW1 roll of honour. There are records for 19,228 people and of these 1,799 have biographies. There are 1,111 image records and these link to the image files contained in the ‘images’ directory in the code. There are three derivatives per image, which is why there are so many image files compared to image records.
The code contains the code for the site and the CMS, as it was in 2011. I did try getting the site to run on my laptop, but it would take a lot of work to get it operational due to changes in technology and also because it references stylesheets from the main University website from 2011. An awful lot was added to the site after June 2011, but hopefully having the database structure and the data up to this point will be a useful starting point. Stats on the homepage from the Internet Archive snapshot from April 2018 state that there are 2390 images and 3678 biographies on the site, which is a lot more than we had in 2011.
If it’s possible to request a copy of the UGS data from the Internet Archive the Archives people could then write a script to compare the people IDs in the 2011 database with IDs embedded in the URLs of pages from the Internet Archive (e.g. ‘WH10083’ from https://web.archive.org/web/20150908140047/http://www.universitystory.gla.ac.uk/biography/?id=WH10083&type=P&o=&start=0&max=20&l=a) to then make a list of all of the people that are not found in the 2011 database. The same could be done with images (e.g. ‘UGSP01400’ in the URL https://web.archive.org/web/20130704220509/http://www.universitystory.gla.ac.uk/image/?id=UGSP01400&o=&start=0&max=20&l=A&biog=WH10083&type=P&p=2). Of course there’s no guarantee that the Internet Archive captured all of the data for every person and image, but this would at least enable the archives to create a more limited list of people and images that could then be added to the 2011 data, rather than having to process all of the Internet Archive data from scratch.
The archives people could then write a script that could extract data only from the people / image pages in the list and insert this into the database. The pages are pretty well structured so it might not be too tricky to write such a script. Of course any edits made to the existing data since 2011 wouldn’t be spotted by this approach, as only new records would be targeted.
I wouldn’t recommend using the old code for the site and CMS, which is all very outdated now. The archives would be better off building a new system around the database structure, referencing the Internet Archive version of the old site to help figure out how things used to work. They might also want to rationalise the database structure as it’s possible not all tables are actually needed, or could be structured more efficiently.
Also this week I had an informal meeting with Marc Alexander to discuss my work, and I read through an edited version of the upcoming Books and Borrowing article to make sure the edits all made sense.
For the rest of the week I looked into a task that has been on my long-term ‘to do’ list for ages: Redeveloping the Fife Place-names website (https://fife-placenames.glasgow.ac.uk/). I made this as a proof of concept many years ago, having written scripts to extract and structure the data from Word documents. What I want to do is migrate the data to the system I created for the other place-names resources, which would then mean I could create a map interface for Fife that would be comparable to the Berwickshire one, for example (see https://berwickshire-placenames.glasgow.ac.uk/map/).
The first task was to attempt to generate latitude, longitude and altitude data for Fife, as currently the data only has grid references. I wrote a script that would do this, thinking I’d just be able to leave it running. Unfortunately when I ran it I realised that many of the grid references have problems. Many records have malformed grid references, such as ones with an additional number, or a zero character instead of an ‘O’, while for some others the data extraction script did not successfully grab the grid reference. In many such cases I had to do research to discover the actual grid reference using a combination of Google Maps, the historical maps on the Berwickshire website, the UK Grid Reference Finder (https://gridreferencefinder.com/) and the NLS GB1900 website (https://maps.nls.uk/projects/os1900). As you can imagine this took quite a lot of time, and I hadn’t managed to complete the georeferencing by the end of the week.
I’m on holiday for the next week and a bit, and I’ll return to this task once I’m back.