I spent most of Monday and Tuesday this week completing the requirements document for Deven Parker’s Playbills project, which clocks in at almost 5,500 words and covers all of the main aspects of the front-end that will be developed for the project. I’ve sent it to Deven for feedback and we’re going to meet to discuss updates after the Christmas holidays. There are bound to be some aspects that will need changed and I’ll write an updated version after our meeting. After that I’ll begin to implement everything.
Also this week I completed writing the information that Garrick Allen had requested for his proposal, and I made the ‘Urtext Edition of Johann Nepomuk Hummel’s Scottish Songs for George Thomson’ live on the Burns website (see https://burnsc21.glasgow.ac.uk/hummel-edition/).
I also spent some time investigating the code for an app that had been developed for the VARICS project to see whether we might be able to get this hosted through the University of Glasgow’s app store account. Unfortunately I didn’t have much luck. When we’d initially discussed hosting the app I had assumed that it was already published on the App Store and it would be a case of arranging for ownership of the app and its store listing to be transferred to Glasgow, but the app was never actually officially released so this approach won’t work. As the app isn’t available I’ve been unable to even try it out to get an idea of what it is and how it works.
I downloaded the code for the app from GitHub, but I’m not familiar with the Expo framework (https://expo.dev/) that was used to create the app. I followed a tutorial to install the components and get a test app working using the framework, but when I tried to follow the same steps to launch the VARICS app I just get a bunch of errors.
I spent several hours trying to figure out what’s going on but had no success. It may be the case that the code was written a few years ago now and the framework has gone through several updates since then and this is causing issues. However, the real problem is that I have no idea what I’m doing and my attempts to resolve errors are just leading to further errors. Without some guidance from the original app developers I’m not going to be able to make any progress, and we also need to consider whether this is all more trouble than it’s worth. It’s not clear to me what the app will it be used for, and who will use it if we can even get it working. I’ve not been able to get the app working, I’ve not seen any documentation about it or even a screenshot so I know very little about its purpose, but looking at the code it would appear to be an app that allows users to record their speech. It’s unclear to me where these recordings end up and how are they used, and I would imagine that there are considerable ethical and data protection issues that would need to be considered if the app was publicly available, especially as it is aimed at children. I put these concerns to the project team and it looks like they agree that it might not be worth pursuing this, but I’ll learn more in the new year.
Also this week I had a Teams call with the place-names of Armagh project team where I demonstrated the content management system and discussed the project’s data. The session went well, and they raised some interesting questions about how certain aspects of the CMS function. I’ve set up user accounts for the researchers and they’re going to start experimenting with the CMS. I’m also going to think about the structure of their existing data and how this might be migrated into the new system.
I had hoped to spend the rest of my available time this week continuing to work on the place-names of Fife data, but I was only able to spend an hour or two on this, continuing to sort out the place-names elements. The rest of my time was spent looking into the semantically tagged Hansard dataset. I’d worked with this almost ten years ago, but nothing really cam of it. Recently Marc and Fraser have wanted to access it again, and I’d managed to track down the data. I set up a Team in Microsoft Teams where we could all access the data, but the file is not in a usable format. It’s a 10GB tar.gz file containing Base64 encoded lines of text. Years ago I’d managed to extract this and create a database for the data. I had the files for this (taking up around 70GB) on an external hard drive, and it was also set up on an old desktop PC in my office.
When I was on campus on Thursday I attempted to get this PC working, but it only has VGA out and I no longer have a monitor that has a VGA connection. I tried attaching the PC to the projector in the STELLA lab, but was unable to get it working, and the PC made some rather worrying beeps when I tried starting it up. I therefore tried to get the database files running on my laptop, but you can’t just plug 10 year old MySQL data files into the current version of MySQL (as I found out, after trying and temporarily breaking the MySQL instance on my laptop). Instead I downloaded and set up an old version of MySQL and thankfully this worked. But the frequencies table in the database contains 718,424,384 rows and there are no indexes other than the primary key. A query to display the distinct years in the dataset took 22 minutes to execute!
I began the process of indexing the data by year on Thursday evening but it ate up all available disk space on my laptop and then gave an error. On Friday I emptied most of the data from my laptop onto an external hard drive and began the process again. The process took 52 minutes to complete and the index takes up around 17GB, but now a query to return the distinct years is instantaneous, which is pretty promising. The next stage will be to write a script to output the data for each year for each house (commons and lords) into separate CSV files, which will hopefully be more usable. But that’s all for this week, and indeed for this year. If anyone is reading this I wish you a merry Christmas and a happy New Year!
My time this week was mostly divided between two projects: The Place-names of the Coalfield Communities (Ayrshire) and the Eighteenth Century Playbills project. We launched the interactive map of Ayrshire place-names during an online event on Wednesday this week and there were a lot of last-minute updates that I needed to implement leading up to this. This included replacing a lot of placeholder text with actual text, both on the map and the project website more generally and fixing a number of issues with the data. It turned out that there were almost 250 place-names in the system that had a grid reference that was used by another place-name. This meant that the map markers were sitting on top of each other and unless the data was filtered it was impossible to tell that multiple markers existed at the point. To rectify this I wrote a little script that identified the places that shared a grid reference and all places after the first had their longitude updated to move them slightly east of the original location, ensuring the markers did not overlap. This approach has worked pretty well, although the researchers do still need to properly investigate the duplicates as there are some that are genuine duplicates and should probably be removed.
There was also a slightly strange issue with the CMS, whereby making updates to a specific place-name element was getting blocked by the WordPress plugin Wordfence that runs on the domain. What was especially strange was that the update was getting blocked in Chrome-based browsers but was completing successfully in Firefox, even though the issue was server-side and the posted data should have been identical in both browsers. It was the description field that was getting blocked, and it looked like this was due to the element having a fairly long description, as when I replaced it with a shorter text the update was fine. There was nothing in the contents of the original description that would have caused issues – just a few simple HTML tags. In the end I just had to add an exception to Wordfence to stop it blocking the upload page, and this worked. It’s not like Wordfence should really be processing data added via the non-Wordpress, password protected CMS anyway.
I also needed to fix element languages in the elements glossary, as several of these were appearing as beginning with ‘other’ in the alphabetical list, even though they clearly shouldn’t have been. This was because I created a new ‘element sort order’ field in the database to be used for the alphabetical list and generated the data for it using a script. But what I forgot to do was update the CMS pages where elements are added or edited to ensure this field subsequently gets populated. This meant any elements added after I ran the script to generate the field didn’t have data in the field and so the elements were appearing as ‘other’. I fixed this for the Ayrshire site and also ensured the fix was in place for the other place-names resources too.
Some further tweaks to the elements were required as there were seven different ‘?’ elements (used where an element is unknown) and I merged these into one record. I also ensured that element certainty in the ‘more information’ view of place-name elements was appearing. This should have been visible but wasn’t showing up in any of the place-names resources. A quick tweak of the various APIs got this working across all of the place-names resources.
With all of these updates in place I went live with the map on Wednesday morning, creating a cached version of the full map view data to ensure that the initial map loads speedily. I also spotted some issues where place-names hadn’t been assigned a classification code, resulting in a blank item getting added to the map legend, and I fixed these.
The online launch of the map (now available at https://ayr-placenames.glasgow.ac.uk/map/) went very well and everyone seemed happy with the map, its data and its potential uses. I’d say the session was a great success.
I spent most of the remainder of the week writing a requirements document for the front-end for Deven Parker’s Playbills project. I had hoped to begin work on this earlier in the week, but there was rather more to do before the project launch than I was expecting. There is a lot to consider for the playbills project in terms of how to process and store the data and present everything online, and by the end of the week I was probably only about half-way through writing the requirements document. I’m hopeful that I’ll be able to complete it next week, though.
Also this week I updated the facility to save an SVG version of the Historical Thesaurus timeline visualisation to make the process fully JavaScript based. Previously the SVG was exported for download by posting the XML data to a PHP script, which then triggered the download. The purely JavaScript way means that no data needs to be posted and echoed out by PHP, which is more secure. I also had some discussions with the Stirling IT people about bot traffic on the Books and Borrowers site, which is beginning to cause issues. I talked about our recent experiences with the Emblems site and shared the htaccess script Luca created to sort out that issue. Hopefully this will help. I also received a request from Garrick Allen to provide some text for his latest research proposal, and although I began this on Friday afternoon (when the request came in) I didn’t quite manage to finish it. I’ll get this sorted next week.
My time this week was divided between several different projects and meetings. I spent quite a bit of time working with Transkribus, ahead of Friday’s Transkribus workshop at which I was speaking about my text extraction experiments with the Edinburgh Gazetteer (https://edinburghgazetteer.glasgow.ac.uk/). Back when I worked on the project with Rhona Brown (almost ten years ago now) we’d attempted to extract the text from the images using OCR but our experiments had been pretty hopeless. I attended a Transkribus event in Edinburgh earlier this year and had done a little bit of work with the Gazetteer in Transkribus then, but hadn’t progressed very far. This week I made considerably more progress, experimenting with a three-column subsection of one page, as you can see below:
Transkribus can identify columns of text by analysing what it calls ‘Fields’ so this is what I asked it to do first, using the ‘Baroness of Blocks’ model. This is something that’s only available with a subscription, but I was able to make use of a free trial. Unfortunately the process was not very successful. It did correctly identify the columns, but split the image up into sections within each column, with some parts of the image entirely missing from the classification (e.g. the top of column 2) in the image below:
I manually corrected this using the interface, as the following image shows:
However, I’m uncertain how I would be able to train the system to automatically and correctly identify such columns for other images – this would require further investigation. The next step was to identify lines within each column, which is accomplished using a ‘Layout’ model. I chose the default ‘Mixed line orientation’ model which was pretty successful in identifying all of the lines in each column. It wasn’t perfect but it was good enough for test purposes, as you can see below:
The third step was to extract the text. For test purposes I wanted to see how the model would work without any training, and I chose the ‘Text Titan I ter’ model. This took several minutes to process, but the results were very encouraging, as the following image demonstrates:
There were some issues, however, such as the large drop-characters at the beginning of sections being omitted, and some words that are legible to humans being incorrect, such as ‘acie’ instead of ‘acre’. The ends of lines in the first column were also missing, so line identification would need to be tweaked. Despite these issues the text is broadly understandable and complete. The poor print quality and the long ‘S’ character were processed successfully and some sections that were very difficult for a human to decipher were processed successfully by the tool. There are still issues to be ironed out with regards to successfully identifying columns of text, and these would need to be addressed before any batch processing of the entire Gazetteer, but it’s looking very promising.
I spent most of Tuesday this week attending the presentations for the new Grade 8 and Grade 7 roles for the post-graduate course in Digital Humanities that is being set up in Information Studies. It was really interesting to hear the presentations and to learn more about what the candidates would bring to the roles. There were some really excellent candidates and it was very useful to hear from them.
Also this week I spent a little time working on the Place-names of Ayrshire project ahead of next week’s launch and engaged in a continuing email conversation about how the data for the interactive map will be gathered and stored for Ophira Gamliel’s Malabar project. We’ve now managed to reach an agreement on how to proceed with this, which is a relief. I also spent some time continuing to make updates to the Place-names of Armagh resource, creating a nice interface for the project website using suggested public domain images and fonts, and adding parishes and baronies to the content management system. The new project website is not yet live, but here’s how the new design currently looks:
I’ll meet with the project team next week to give a run-through of the CMS and working with WordPress, after which they should be in a position to start adding data to the resource. I also managed to spent a little more time working with the data for the Fife place-names project, continuing to rationalise the place-name elements and their connections, but there is still more to do for this.
On Thursday this week I met with Deven Parker to discuss the requirements for an online resource for her Playbills project. We discussed the kinds of search and browse facilities she would like to include and other features such as visualisations and data summaries. Next week I’m going to write up the requirements and share the document with her. Also on Thursday I met with Wendy Anderson and Carole Hough to discuss some potential future updates to the Mapping Metaphor resource. I can’t really go into any details here, but we had a good meeting and will consider the options before meeting again in the New Year to see how this might be taken forward.
This week I began setting up the infrastructure for the new Place-names of Armagh project, which I’m working on with Mícheál Ó Mainnín at Queen’s University Belfast, setting up the database, content management system and the project’s WordPress-based front-end. For now the WordPress site just has a very plain interface and two pages – a welcome page and a project news page, which is a blog page that can be used to post updates about the project. I’ll continue to work on this once I hear more from Mícheál.
A slightly biggest task was setting up the database and CMS. These are based on the system created for previous place-name projects so most of the work was already done, but as is usually the case, I needed to tailor the system to the current project, removing some fields that were not required, adding new ones in and ensuring everything works with place-names in both the NI and ROI areas. We’d decided that the system would not be fully bilingual but would have fields for specifying English and Gaelic fields for names of places (main place-name and historical forms); all other fields will just be single fields, so for example will not be separate fields for English and Gaelic descriptions.
As with the CMS I set up for the Comparative Kingship project (which covered both NI and ROI), the grid references are set up to use Irish grid references (e.g. ‘H 88144 4476’) and also ITM co-ordinates (e.g. 688077,844763). A researcher can supply either the grid reference or ITM co-ordinates and the system then automatically generates the other, plus latitude, longitude and altitude. These can all be manually overwritten if required. There’s a handy resource called https://irish.gridreferencefinder.com that can be used to view the grid reference and ITM for a point clicked on if the researchers are not gathering grid references from physical maps. I still need to check whether using Irish grid references and ITM the correct approach for the project, as it’s possible that UK OS grid references will be more appropriate, as with the Scottish projects. If so I can swap to them.
I also still need to know what parishes the project will cover so I can add these to the system, and eventually track down the GeoJSON shapefiles for them so we can plot parish boundaries. There are a few other aspects that I may need to tweak too. For example, for the Comparative Kingship project (an earlier project I was involved with which covered NI and ROI) we incorporated baronies in addition to parishes, and I need to know whether this are needed or not. For now I have included them, but I may need to remove them if they are not needed.
It was as bit of a week for place-names projects, as on Thursday I met with Thomas Clancy to discuss the official launch of the interactive map-based resource for the Coalfield Communities / Place-names of Ayrshire project. This resource features the same map interface I’ve developed for other place-names projects, such as Iona and Berwickshire but is not yet live. We’re going to make some final tweaks to the data and the interface over the next week and the place is to launch it in mid December.
I also continued to work on the Fife place-names data, making good progress with the processing of the place-name elements. As mentioned last week, there are more than 6,500 elements (i.e. distinct parts of a place-name, such as ‘-ton’) in the Fife data, and these need to be properly split up so they’re only stored once and then link to as many place-names as are relevant. Before I get to this stage, though, I need to clean the data, as the scripts that migrated the original data form Word to the relational database structure was not 100% successful. My first sweep was to fix issues with the language field, as if erroneous data appeared in this field it was likely that the entire record would need to be fixed. I managed to batch-process many such issues, and I’ve almost managed to complete this stage of the cleaning process – there are maybe around 10 or 20 records that I still need to sort next week. After that I should be able to migrate the element data to the data structure used by the other place-names projects and then think about setting up a CMS to enable researchers to take control of the data.
Also this week I participated in a lengthy call about the relationships between the interactive map and the digital edition for the HiMuJe Malabar project. This wasn’t an easy meeting, but I think we may have identified a solution that might work for everyone. I also attended the English Language and Linguistics seminar on Thursday and discussed next week’s Transkribus workshop with Joanna.
I also spent a bit of time working on Deven Parker’s Playbills project. When processing the YAML files last week my Python script had encountered a few errors so I wrote another script to investigate these. It turns out that 109 of the 110 error files are in the JSON rather than YAML format, which obviously caused problems for a script created to work with YAML data. The final file had a value that began with an apostrophe that did not have a corresponding closing apostrophe, which the YAML processor didn’t like. I’m hopefully going to meet with Deven next week to discuss the requirements for the online resource she wants to create for her data.
Finally this week, I had an email conversation with Wendy Anderson and Carole Hough about possible updates to the Mapping Metaphor resource. We are going to meet next week to discuss this further. They had also spotted an issue with the resource’s advanced search. There was a problem with searches for ‘examples of metaphor’ that caused no results to be returned. Other searches (e.g. for category name / descriptors) were working. Thankfully it was pretty quick to fix this.