After being off sick on Thursday and Friday last week I returned to work for a full five days this week, although I was still in the throes of ‘post viral fatigue’ and it wasn’t until Friday that I stopped feeling exhausted.  Despite this I still managed to get a lot done for several projects this week.  I had been planning on continuing with the redevelopment of Mapping Metaphor towards the end of last week, and as I was unable to do so I decided to focus on this on Monday.

As I looked through the partially updated site I spotted an issue when combining E and OE data:  Sometimes the metaphorical connection between two categories has the categories the other way round in the other dataset (i.e. in E we have cat1 and cat2 whereas in OE we have cat2 and cat1).  This was resulting in metaphors in the combined view sometimes being treated as different when they’re actually the same, which was affecting some counts and also sometimes making the figure for number of OE lexemes appear next to the wrong category in the card view.  I managed to sort this, and I also ensured that if the category order needs to be swapped then the direction is also swapped.

I then spent most of the day implementing the combined table view, and it should be fully operational now. It was extremely tricky to implement this as the script that generates the data for the table view can handle many different data requests and (unlike the visualisation script) also processes individual metaphor connections for display, including directionality and examples of metaphor.  There was a lot to update and check, and it was probably not the best of choices of things to work on during my first day back at work.  However, I got there in the end and was able to share the update with Wendy and Carole by the end of the day.  I still need to update the CSV download option, but that will need to wait for another day.

I also spent about half a day or so on the Playbills project this week.  I fixed the few remaining plays that didn’t have links to canonical records so that every play now includes such a link.  I then moved onto looking at performers.  I wrote a script that outputs a spreadsheet that lists all performers arranged alphabetically by surname with the corresponding play, date of performance and venue.  I’m hoping we’ll be able to do something with this to figure out which performers are actually the same person, but with almost 52,000 rows we’ll probably want to try some automated processes rather than figure it all out manually.  I’m just not sure how best to proceed with this.

For example, there are 17 ‘Mrs Ashton’ performers, with performances from 1827 to 1834.  12 of these are at the Theatre Royal, Birmingham in 1827, then there’s nothing until 1832 at Bristol (4 performances) then one final performance in 1834 in Edinburgh.  Would we consider these all to be the same performer?  Or should we treat the Bristol and Edinburgh ones as different people?  This will need some discussion with Deven.

I also began looking into generating canonical roles for plays.  I wrote a script that for a given canonical play ID returns all plays that have the ID and for each lists its name and all associated roles.  The individual play names are links through to the play page, from which it’s possible to access the playbill page and check the original image.  At the very bottom of the output is a list of all unique roles with a count of the number of times they appear.  This is only an initial version but it has highlighted many issues due to variant spellings (either in the original playbill or due to the AI text extraction).  I can further tweak this, for example making all lower case, removing punctuation, removing text in brackets, replacing “M’” with ‘Mac’ or ‘Mc’ to see if these match.  We could also use Levenshtein distance to try and find forms that differ by one character, which would match ‘Baillie Nicol Jarvie’ and ‘Balie Nicol Jarvie’ with ‘Balie Nicol Jarvie’ and ‘Captain Thorton’ with ‘Captain Thornton’, for example.  We’d have to watch out for false positives, though (e.g. ‘Jane’, ‘Jean’ and ‘Janet’), and also make sure the form that has the most hits is taken as the canonical form, rather than it just being the first form found.  But it’s also possible that asking AI to do this might be a better approach.

As I was working on this I noticed the interestingly named role ‘MacSycophant’ in the list, which is definitely an AI hallucination!  Looking at the original playbill image it should be ‘MacStewart’.  I also spotted that many of the actors are incorrectly assigned, with ‘Mr Harrold’ and ‘Mr Felton’ being swapped and Rob Roy listed as being performed by Miss A Murray instead of Mr Pritchard).  Checking the original YAML shows that the issue was with the AI extraction and thankfully it was not any subsequent processing scripts that have introduced the errors.

Following on from this, Deven has suggested that she and some other researchers may manually proofread all of the actors and roles across the entire dataset.  This would really help to ensure that the data is more accurate, but it is also a huge amount of work, as there are 1902 playbills containing tens of thousands of performers and roles.

I spent some time proposing a means of editing the data so that the correct updates are made in the database. We would need to generate lists of performers from the database so we don’t lose the work we’ve already done to split the names and assign genders, and presumably the task won’t just involve swapping roles between performers, but will involve role names being edited too (e.g. to fix things like ‘MacSycophant’).  I think it’s likely that some issues with performer names that have been extracted incorrectly too will also be spotted and need fixing too.  We’ll need to ensure any updates to performers’ names and role names can be tracked to the relevant record in the database using the corresponding ID fields.

To achieve all this I generated a spreadsheet that includes information about the playbill, play, date and venue, with a link to the page for each playbill for checking the original image.  The Role data and Performer data then follow on each row.  Using this a proofreader could then make changes to the ‘Role Name’ field where required, but also make changes to the performer fields too if anything needs corrected there, and also add in new rows if required.

I then returned to the DOST Auld Laws project, for which I created an initial version of a user interface for the site.  This is still just a work in progress and may change depending on feedback given, but below is a screenshot:

I spent most of the rest of the week working on created a system to manage travel routes for the interactive map for the HiMuJe Malabar project.  My first task was to create a test route based on data provided by project Co-I Ines.  I added this to a spreadsheet template I’d created, and which we’ll hopefully be able to use for future routes.

With this data in place I then wrote a script that converts the spreadsheet into JSON data for use on the map, so if further travel routes are created (one per spreadsheet) with the same structure I’ll be able to add them to the map.

Once I’d completed my updates to the map interface, when you press on the ‘Travel Routes’ section of the map menu each travel route is now listed (there is currently only one). Each route appears with its title and type (we can maybe split and/or limit the list by type in future) and the description given in the spreadsheet.  Pressing on the checkbox or the route name adds the route to the map.  This currently highlights the relevant locations with a red border and adds a dotted red line connecting each location in the itinerary.  We may want to use different colours when multiple routes are added at the same time to help differentiate them.

As the route can get lost in amongst all of the other data on the map I’ve added an option to show or hide unrelated places.  If you deselect the ‘Show unrelated places’ checkbox any locations that are not part of the travel route are removed from the map, making it much easier to see the route.  The following screenshot shows this:

The legend is removed from the map when the ‘Travel Routes’ map menu is active.  This is because we need the space to display the information about a specific stage in the travel route itinerary, as described below.  Also, if the legend was visible users may end up removing a map layer (e.g. ‘Temple’) that contains locations that appear in a selected route, which would cause confusion.  When viewing travel routes, all of the categorisation layers are set to on for this reason.  If you navigate from the ‘Travel Routes’ menu to another menu any selected travel routes are removed from the map, the legend is reinstated and all locations are added to the map.

If you press the ‘Explore’ button for a route this adds the route to the map (if you’ve not already added it by selecting the checkbox) and the map will reposition to display the first stage in the itinerary.  This displays an info box in the top right of the map (in the place the legend would otherwise be).  This box displays the route title and the information about the first stage in the itinerary.  This can include any information about the stage in the itinerary (including duration, if of relevance).  I’ve also included information about the number of stages in the route and the number of the current stage (e.g. ‘Stage 2 of 10’), as I figured this would be useful for people.  There is also a ‘close’ icon in the top right that closes the info box and ‘Next’ and ‘Previous’ buttons at the bottom that can be used to traverse the travel route.  Pressing on one of these buttons repositions the map to the next or previous stage in the itinerary and loads the information about this stage into the info box.  Using these options you can step through the travel itinerary and view all of the information relating to it.  The following screenshot shows the map with ‘Stage 2’ of the travel route loaded.  Note that you can also still open the popup for any location on the route or manually scroll the map between locations.

This is only a first version of the feature and there’s still a lot to do.  Displaying multiple routes at the same time is going to require further work, and I also want to add the selected route options to the page URL to enable citation / sharing / bookmarking of specific route selections and also possibly individual stage selections.  We may also want to add information about relevant routes to the location popup (in a new tab) so users can tell at a glance which routes a location appears in.  I guess adding links from this to the relevant stage in each route would also be useful.

Also, the current system is only set up to work with a purely linear route (e.g. a->b->c).  If we are to include routes that branch off (e.g. a->b then b->c and also b->d then c->e and also d->e) then we’ll need to consider how to handle this as the traversal via ‘Next’ and ‘Previous’ buttons would not work.

Also this week I made a few further tweaks to the Burns Supper Map, including updating the site title to include ‘Worldwide’, fixing some typos in the data and adding in some new videos.  I’ll be continuing with this next week.