I spent most of Monday and Tuesday this week completing the requirements document for Deven Parker’s Playbills project, which clocks in at almost 5,500 words and covers all of the main aspects of the front-end that will be developed for the project.  I’ve sent it to Deven for feedback and we’re going to meet to discuss updates after the Christmas holidays.  There are bound to be some aspects that will need changed and I’ll write an updated version after our meeting.  After that I’ll begin to implement everything.

Also this week I completed writing the information that Garrick Allen had requested for his proposal, and I made the ‘Urtext Edition of Johann Nepomuk Hummel’s Scottish Songs for George Thomson’ live on the Burns website (see https://burnsc21.glasgow.ac.uk/hummel-edition/).

I also spent some time investigating the code for an app that had been developed for the VARICS project to see whether we might be able to get this hosted through the University of Glasgow’s app store account.  Unfortunately I didn’t have much luck.  When we’d initially discussed hosting the app I had assumed that it was already published on the App Store and it would be a case of arranging for ownership of the app and its store listing to be transferred to Glasgow, but the app was never actually officially released so this approach won’t work.  As the app isn’t available I’ve been unable to even try it out to get an idea of what it is and how it works.

I downloaded the code for the app from GitHub, but I’m not familiar with the Expo framework (https://expo.dev/) that was used to create the app.  I followed a tutorial to install the components and get a test app working using the framework, but when I tried to follow the same steps to launch the VARICS app I just get a bunch of errors.

I spent several hours trying to figure out what’s going on but had no success.  It may be the case that the code was written a few years ago now and the framework has gone through several updates since then and this is causing issues.  However, the real problem is that I have no idea what I’m doing and my attempts to resolve errors are just leading to further errors.  Without some guidance from the original app developers I’m not going to be able to make any progress, and we also need to consider whether this is all more trouble than it’s worth.  It’s not clear to me what the app will it be used for, and who will use it if we can even get it working.  I’ve not been able to get the app working, I’ve not seen any documentation about it or even a screenshot so I know very little about its purpose, but looking at the code it would appear to be an app that allows users to record their speech.  It’s unclear to me where these recordings end up and how are they used, and I would imagine that there are considerable ethical and data protection issues that would need to be considered if the app was publicly available, especially as it is aimed at children.  I put these concerns to the project team and it looks like they agree that it might not be worth pursuing this, but I’ll learn more in the new year.

Also this week I had a Teams call with the place-names of Armagh project team where I demonstrated the content management system and discussed the project’s data.  The session went well, and they raised some interesting questions about how certain aspects of the CMS function.  I’ve set up user accounts for the researchers and they’re going to start experimenting with the CMS.  I’m also going to think about the structure of their existing data and how this might be migrated into the new system.

I had hoped to spend the rest of my available time this week continuing to work on the place-names of Fife data, but I was only able to spend an hour or two on this, continuing to sort out the place-names elements.  The rest of my time was spent looking into the semantically tagged Hansard dataset.  I’d worked with this almost ten years ago, but nothing really cam of it.  Recently Marc and Fraser have wanted to access it again, and I’d managed to track down the data.  I set up a Team in Microsoft Teams where we could all access the data, but the file is not in a usable format.  It’s a 10GB tar.gz file containing Base64 encoded lines of text.  Years ago I’d managed to extract this and create a database for the data.  I had the files for this (taking up around 70GB) on an external hard drive, and it was also set up on an old desktop PC in my office.

When I was on campus on Thursday I attempted to get this PC working, but it only has VGA out and I no longer have a monitor that has a VGA connection.  I tried attaching the PC to the projector in the STELLA lab, but was unable to get it working, and the PC made some rather worrying beeps when I tried starting it up.  I therefore tried to get the database files running on my laptop, but you can’t just plug 10 year old MySQL data files into the current version of MySQL (as I found out, after trying and temporarily breaking the MySQL instance on my laptop).  Instead I downloaded and set up an old version of MySQL and thankfully this worked.  But the frequencies table in the database contains 718,424,384 rows and there are no indexes other than the primary key.  A query to display the distinct years in the dataset took 22 minutes to execute!

I began the process of indexing the data by year on Thursday evening but it ate up all available disk space on my laptop and then gave an error.  On Friday I emptied most of the data from my laptop onto an external hard drive and began the process again.  The process took 52 minutes to complete and the index takes up around 17GB, but now a query to return the distinct years is instantaneous, which is pretty promising.  The next stage will be to write a script to output the data for each year for each house (commons and lords) into separate CSV files, which will hopefully be more usable.  But that’s all for this week, and indeed for this year.  If anyone is reading this I wish you a merry Christmas and a happy New Year!