I mostly divided my time between three projects this week: Playbills, Burns Supper Map and the Place-names of Armagh.  Unfortunately I was still suffering from the monstrous cold I started with last week and struggled through some of the week, but I still managed to get quite a lot done.

For the Playbills project I wrote, tested and implemented a script that extracts the data from the JSON versions of the playbill files I generated, splits this up and inserts everything into a relational database. The reason I’m doing this is to make it easier to generate canonical records for venues, plays, performers and roles, as it will be much easier to query the data and track records in a relational database.

The data I’ve extracted consists of 1902 playbill records that feature 6434 plays.  These are categorised by one or more of 188 distinct genres (with ‘melodrama’ associated with 2026 plays and ‘melo-drama’ a further 10).  I’ve extracted 185 distinct venues, 49498 performers, 49495 roles and 6083 contributors.

As of yet I haven’t done anything to generate canonical records, which will be the next major step, and I need to discuss things with project PI Deven before I proceed with this.  For example, the role ‘Macbeth’ appears 27 times, with a further three appearances in other strings (not including ‘Lady Macbeth’) e.g. ‘Macbeth’s Last appearance’.  These would need to link to one single canonical ‘Macbeth’ role.  Similarly, there are 28 plays that have ‘Macbeth’ somewhere in their title, with variants such as ‘MACBETH, KING OF SCOTLAND’, ‘Macbeth; King of Scotland’, ‘MACBETH, KING OF SCOTLAND.’ In addition to ‘MACBETH’ and ‘Macbeth’ and these would need to link to a single canonical ‘Macbeth’ play.

There’s also some data cleaning that we should perform, e.g. amalgamating data that doesn’t have the same form but should be the same thing.  For example, there are a lot of possible duplicates in the ‘Genre’ data.  There’s ‘acrobatic’, ‘acrobatic display’, ‘acrobatic performance’ and ‘acrobatics’ all as different genres when presumably these should be the same.

I also still need to work on the performer names to split them into titles, forenames and surnames, and to ascertain gender based on titles.  Venues also need some work as there are many that are the same but have slightly different text, e.g. ‘Royal Theatre, Aberdeen’, ‘Theatre Royal, Aberdeen’ and ‘Theatre Royal Aberdeen’.  There’s the same issue with printers too, although perhaps this isn’t so important.  E.g. ‘Keenes, Kingsmead-Street, Bath’, ‘Keenes, Bath, Kingsmead-Street’ and ‘Keenes, Bath’.  It’s possible that we might be able to get some sort of AI processes to help with such tasks.

We’re also going to have to give some thought about how to handle updates to the data.  I’m generating canonical records, extracting things like performer gender and generating unique identifiers for things like plays in my database, and I’ll be creating new JSON files that incorporate this new data that will then be ingested into Solr for search purposes.  Therefore the data will be quite different to the original YAML files.  When updates need to be made should these then be made to the original YAML files, which would necessitate much regeneration of data, or should the updates be made elsewhere, such as through the database?  I don’t have an answer to this yet, but it’s something we’ll need to consider.

For the Burns Supper Map project I set up the online database for the supper data and have been working on a script that imports the data from the spreadsheets into this database.  I have got everything working for the spreadsheet of the online survey, so my database currently has 308 suppers that include data for 6902 filter options.

What I haven’t been able to do yet is to import the data from the public domain spreadsheet, as this currently contains a lot of inconsistencies in how the data are recorded.  The data in the filter columns (“frequency”, “category”, “toast”, “food”, “style”, “drink”, “entertainment”, “poem”, “music”, “dance”, “dress”) must exactly match the options found in the online form for my import script to work.  This includes capitalisation / case and ensuring that a semi-colon is used to separate multiple items.  I had a meeting with the project RA Cleo on Friday to discuss this, and she’s going to work on tidying things up.

I also wrote a script that posts the address for each record to Google Maps which then returns the latitude and longitude (something we’re going to need in order to pin the records on a map).  The user inputted location data can be somewhat variable, as you might imagine, but Google Maps has generally done a very good job at identifying places from the data, and we can always tweak things once we see the locations on the map.  I’m hoping to start development of the map next week.

For the Place-names of Armagh project I uploaded a large number of place-names that I’d been sent.  We now have 2932 place-names in the system.  I also processed the existing historical forms CSV and this has found historical forms for 1056 of these new place-names.  The new place-names had additional parishes and baronies that were not already in the system and in such cases these have been created, but there are some issues, as the data appears to be somewhat messy at times and will need some cleaning.  For example, there’s a ‘Forkhill’ and a ‘Forkill’ and these may be the same, there are forms with question marks and multiple forms and descriptive text, e.g. ‘Killevy/Partly in Dundonald Parish’ and ‘Armagh?/Eglish?’.  These will all need separated out and fixed as required.

I also spent some time updating the CMS to convert the townland field from a textbox to a list, thus enabling multiple townlands to be associated with a place-name and ensuring each townland is only stored once in the system.  This involved extracting the townlands from all of the 2932 placename records, splitting forms up that have multiple townlands in ‘x or y’ or ‘x / y’ format, storing the unique townlands and then associating the corresponding ones with each placename record.  There are 997 unique townlands (although some of these may need amalgamated) and 3007 connections between townlands and placenames.

I then updated the CMS to replace the existing ‘townland’ textbox with a list of townlands as checkboxes, in the same way as parishes and baronies.  We might need to rethink this, though, as scrolling through 997 townlands to find the right ones takes time.  I also included an option to add a new townland when adding / editing a placename record as I’m guessing there will be more to come.  This should only be used when the townland isn’t already in the list, otherwise we’ll end up with duplicates.

I also made some further updates to the CMS, namely simplifying historical forms so there is just one ‘form’ field rather than separate English and Irish boxes, and adding a flag to record whether the form is a ‘previously suggested form’ or not.  I also renamed the ‘Discovery’ maps to ‘1:50,000’ as this is how the maps tend to be referred to.

Also this week I replied to a couple of emails from the DSL people about future developments and I added a new video to the Seeing Speech resource.  I also investigated an issue with the Books and Borrowing website and discussed the migration of the resource to a new server with the Stirling IT people, and I generated CSV files for all of the survey answers for Speak For Yersel and sent them on to Janine Illian in Statistics, who Jennifer and I met with last week.