I had a bit of a disrupted week this week, as I started feeling unwell on Tuesday morning and ended up off work sick for the rest of Tuesday and Wednesday.  During this time I felt absolutely wiped out and could barely do anything other than sleep, but by Wednesday evening this had developed into a monstrous cold, the likes of which I’ve not had for several years.  Thankfully once the symptoms had moved to my nose and throat my head was a bit clearer and I was able to work on Thursday and Friday, but I was still pretty far from feeling 100%.

I spent most of Monday this week preparing for, travelling to and co-presenting a talk about Speak For Yersel at the Edinburgh Futures Institute with Jennifer Smith.  The talk went pretty well and it was good to meet some of our linguistics colleagues at Edinburgh, plus others involved with the EFI.  I spent some of my other available time reading through and commenting on an AHRC proposal that will involve Glasgow and the Historical Thesaurus that had been sent by Sara Pons-Sanz at Cardiff University, and looking through some further place-name data I’d been sent for the Place-names of Armagh project.

Despite being off work sick on Wednesday I still managed to attend an online meeting for the Burns Supper Map project to discuss the specification document I’d prepared for the project.  This was all very positive and there weren’t any major issues that anyone had spotted whilst reading through it.

For the remainder of the week I spent a bit of time investigating some issues that had been encountered when publishing pure xref entries through the Anglo-Norman Dictionary’s management system.  Certain cross references were not appearing in the published entries despite being in the XML and a bit of investigation uncovered why.  The entries contained cross references to entries that don’t actually exist in the dictionary.  For example, Mars_2 references ‘march’, which is not an entry and respundre_2 references ‘repundre’ which is also not an entry (they both need homonym numbers added).  When xref entries are published the cross references are extracted and stored, and at this point the system checks that the references are valid, and only links to entries that are valid.  It is these that are displayed in the front-end, so even though invalid xrefs may exist in the XML they don’t get displayed.  The ‘preview’ generates its view directly from the XML without checking validity, which is why this view doesn’t match the front-end.  I ran a check and it turns out that there are around 500 xref entries that include a reference to an entry that doesn’t exist, and I passed these onto the editor who will get these sorted.

On Friday I met with Jennifer and Janine Illian, who is the current Head of Statistics, to discuss the Speak For Yersel data and what kind of additional statistical analysis might be possible.  Janine is particularly interested in spatial modelling and has a keen interest in linguistics and it was really great to hear her thoughts about the Speak For Yersel data.  I’m going to send her the data for all survey responses next week so she can experiment with it, and we’ve arranged to meet again later this month.

I spent the rest of my available time this week working on Deven Parker’s Playbills project, working with the YAML files, figuring out how these might be imported into Solr and how we can extract canonical records for things like venues from them.  It turns out that Solr can’t index YAML files (at least not without creating a custom data importer), which is a bit of a surprise.  This isn’t a major issue, though, as I can convert them to JSON, although this also proved to be trickier than I’d anticipated.  Normally I’d use PHP to process data, but PHP also can’t read YAML files, at least not without installing extensions and this process seemed far too convoluted to bother with.  Instead I used Python to convert the files, but this involved a bit of trial and error as I’m not used to Python and it’s bizarre insistence on whitespace being important, and the fact that if you mix up spaces and tabs to create this whitespace the scripts fall over.  I got there in the end, though.

The bigger issue I encountered was with the unit of data that gets indexed.  I’d previously said that we’d index entire playbill files and use ‘playbill’ as the smallest item that gets returned in the search results, but it turns out there are some problems with this, and I think indexing individual plays is going to work better.  I’m still experimenting with the data and Solr’s capabilities, but initial impressions are that it isn’t very good when working with subsets of data within individual files, or more complex queries.  For example, you can search the playbills for the title ‘Macbeth’ and find matching playbills.  But if you combine this with another field that exists in another play in the playbill (e.g. role ‘Jacques Strop’) the playbill record will still be returned.  So even though the role mentioned actually belongs to a different play in the playbill, because both pieces of information exist somewhere in the playbill it gets returned.

With my initial experiments Solr also flattened out the data – all performer names appear in one list per playbill, not separate lists per play, and it’s the same with roles.  Other than the order of the items in the lists, there is nothing to connect the two.  The following screenshot shows one playbill record indexed within Solr (just using Solr’s default post and without customising a schema).  You can maybe see how Solr has flattened things out, resulting in data being lost (e.g. which performer belongs to which play).

I then tried to index the data at play level, adding in a play ID and also any playbill level data (thus ensuring it’s still possible to search for date, venue etc).  You can see the results in the following screenshot, which includes 5 separate records.

Here at least it’s possible to tell which performer / role belongs to which play.  But performers / roles are still only connected by their position in the lists.  Record 5 is a duplicate I made of record 4, but I deleted the ‘role’ text for one performer to see what would happen.  And Solr indexed the record as it was, with 5 performers and 4 roles, so based on list order ‘Miss Newton’ is now ‘Landlord’ and not ‘Marie’, and ‘Mr. Watkins’ now had no role.

After further investigation I realised that it is possible to get sole to properly index nested data (see https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-nested-documents.html) although instructions on how to actually import nested data into Solr are pretty thin on the ground – you can’t just use the default ‘post’ command as this flattens all data.  I ended up following another tutorial (see https://docs.arenadata.io/en/ADH/current/how-to/solr/solr-index-nested-docs.html) and importing the data using the Solr admin interface.  This thankfully worked, as the following screenshot demonstrates.  You can see that individual performers are directly associated with roles.

There’s still a massive amount to do with the data, though.  I need to extract unique venues, plays, performers and roles and assign IDs to them to enable them to be searches for.  I decided that it would be easier to manage such processes via a relational database, so on Friday and mapped out a structure for the playbill data and bean working on an import script that would process the JSON files.  Lots more to do in the coming weeks!