This was a two-day week for me, as I headed off on holiday on Wednesday and will be away until Monday the 6th of July. I mostly spent the two days tidying up loose ends before I went away. This included making some updates to some of the data for the Burns Supper Map that Cleo asked me to do. This was mainly fixing typos and updating the categorisation, plus adding, deleting and replacing a selection of images.
I also wrote scripts to process the edited announcements for the Playbills project. Deven had been working on a spreadsheet version of the data that featured columns noting where announcement text had been edited, where the announcement needed to be deleted, reclassified as a ‘special attraction’ or in some cases where a new announcement needed to be added. After writing and running my script a total of 1836 announcements were updated, reclassified or deleted. I then ran my script that generates complete JSON files for every playbill in the system, featuring all data about associated plays, venues, actors, roles and such things. This took quite a while to execute as a lot of data needs to be processed, but after half an hour or so I had a collection of 1902 JSON files that I sent to Deven so she can run some offline queries for a paper she’s writing.
I spent the rest of my two days continuing to work on the redevelopment of the Mapping Metaphor resource, which will enable both English and Old English metaphors to be viewed in combination. I managed to complete the combined card view, as the following screenshot demonstrates:
The cards feature both main and OE metaphor IDs, counts of lexemes in both E and OE and examples of metaphor from both E and OE. Where a metaphor exists in both E and OE and the E date is later than OE the later date is highlighted in the card in green. I still need to work on the timeline view, the CSV download and the search facilities, plus I need to add in an option to switch between the combined and E / OE specific maps, and also update the user interface, so there is still a lot to do. But not until after my holiday.
I began feeling unwell on Monday this week, but managed to struggle through until Thursday morning, by which point I just couldn’t sit at my desk any more. I was then off work sick on Thursday and Friday.
Despite coming down with something I still managed to get quite a bit done on Monday to Wednesday this week. On Monday I mainly focussed on the Playbills project. I manually sorted all but three of the remaining plays that didn’t have canonical records and asked Deven to further investigate the remaining three. I then added the canonical play data to the output produced by the API and wrote a script that would output all of the playbill data for every playbill as JSON files. This will be used for offline analysis by Deven.
On Tuesday I continued to work on the map for the HiMuJe Malabar project, mostly tidying up some loose ends with the map. I implemented the URL shortener and set up the ‘share / cite’ options so that now if you open a place record and press on the ‘Share / Cite’ tab this loads content, with the citation text being dynamically generated based on the map options that are currently selected. It’s also now possible to press the ‘Share’ button at the bottom of the map menu to share or cite a specific map view as opposed to a specific record.
I also implemented the ‘Reset map’ option on the ‘Home’ menu and pressing on this resets the map to the default view and categorisation type and I finally sorted out the polygons on the map so they should always now appear behind the markers, even when turning layers on and off in the legend.
The last thing I implemented was the ‘Table view’. If you press on this button at the bottom of the map menu it opens a popup containing all of the data in tabular form. I haven’t included the references in this as this is too much data for such a table, but I have included a count of the number of references for each place. Similarly, I’ve only included the latitude and longitude for each place and not the full GeoJSON shapes for the polygons. You can press on a column heading to order the table by that column (press a second time to reverse the order). I think it will be quite a useful view, for example if you know the name of a place but are not sure exactly where it is located you can order the table by placename and find it. Pressing on a placename in the table closes the popup, centres the map on the location and (after an intentional brief delay so you can appreciate where on the map you’re looking at) the relevant record popup opens. I might see about highlighting the place’s marker to make it clearer which is the relevant one in areas where there are many.
I still need to implement categorisation by certainty, but as of yet we don’t have any certainty data in the JSON file so I’m leaving this for now. I’m intending to start on the travel routes soon. We don’t have any routes available yet, but I’m going to create a test route and create the interface for plotting the connections along the route so we can see how this might work.
On Wednesday I continued to work on the DOST Auld Laws project. I managed to fix the issue with multiple results in one line causing the in-page navigation to stop working. This required quite a bit of reworking as the results were identified by line and instead I needed to add in a new way of identifying that a result may be on the same line but is actually a different result. I did this by counting the number of results per line and using this figure in addition to the line ID to track the results, adding this counter to the URL that is used to reach the document page from the search results, with the document page then taking this figure and using it to ascertain which result is the current one and allowing the results traversal on the document page to function.
I then moved onto looking at results page ordering. Previously the search results were ordered by document name, document page and line, but I wanted to add further sort options via a drop-down list to the right of the ‘you searched for…’ box. In addition to the default ordering, this allows you to order by highlighted term (useful for searches like ‘*other’ where the start of terms may differ) and a concordance-like word position. The latter allows you to sort the results by up to 5 words to the left or right of the term, with the selected position highlighted in cyan. The screenshot below shows the search for ‘*other’ ordered by the second word to the left of the term:
This was a particularly large and complex update to implement as many parts of the code had to be rewritten, but I think it’s worth it. The final update I implemented was to ensure that the ‘return to search results’ button in the search bar of the document page now takes you back to the specific results page and retains the ordering you’ve selected rather than taking you to the first page of results and the default ordering.
Also this week I set up a bare-bones WordPress site for Craig Lamont’s new project, and I’ll meet with him over the summer to get this fully set up. I also tweaked the Burns Supper Map to ensure long subtitles for the charts don’t end up overlapping with the ‘hamburger’ menus for each chart and read through the findings of the user survey for the Dictionaries of the Scots Language that I’d been sent, and which was on the whole very positive.
The project I spent the most time working on this week was the DOST Auld Laws project, which I hadn’t worked on for a couple of weeks. The last time I worked on the project I’d managed to get an initial version of the quick search working, but there were some issues with it. A phrase search wasn’t working, wildcards at the start of a search term were not working, the full term was not getting highlighted in the results where an <expan> is present in the term, I hadn’t implemented the pagination of search results and I still needed to update the page view to enable results traversal directly from the page when accessed via the search results.
I was struggling somewhat with phrase searching and full term highlighting and ended up going round in circles with ChatGPT for several hours and getting nowhere. I eventually asked my colleague Luca, who has considerably more experience and knowledge of eXist-db and XML document querying, for some advice. Thankfully he was able to come up with a query script that did exactly what I needed it to do, which was a massive help as I was really struggling. This is definitely an example of where a conversing with actual person is much more effective than AI and I really must give Luca credit for the help he gave.
By the end of the week my updates meant that it was possible to use wildcards at the start of a search term, as you can see from the following screenshot that shows the results for ‘*other’:
With Luca’s help, phrase searches now work, and the following screenshot shows the results of a search for “the landis of”:
Again with Luca’s help, the search term in each result is now fully highlighted even when an <expan> is present in the word, so for example a search for ‘pebillis’ previously found two results but only highlighted ‘pebill’ in the results as the term was recorded as ‘pebill<expan>is</expan>’. But now the full term is highlighted.
I also implemented results pagination, with is currently set to display a maximum of 20 results per page. The following screenshot shows a search for ‘r?cht’ with the pagination in place:
And now when you reach a document page that is in the search results (e.g. by selecting it from the search results) a search results navigation bar appears above the document navigation bar, allowing you to navigate to the next or previous search result or return to the full search results, as the following screenshot demonstrates:
I’ll probably add in a ‘clear search results’ button here too, and I still need to fix an issue with the results navigation when there are multiple results on one line. At the moment there is no way for the code to differentiate these so the ‘next’ and ‘previous’ links get stuck. I also need to look into speed issues with the search too. But some good progress has been made this week and I feel much more confident using eXist-db now.
Also this week I spent a bit more time on the Burns Supper Map, adding in videos and images for the recently imported supper records. I also had an email conversation with the Dictionaries of the Scots Language people about parts of speech and new data imports, which will probably be taking place in the next few weeks. I also attended a ‘coffee and catch-up’ with the other developers in the College, which was really valuable as always.
The rest of my week was divided between the Playbills project and the HiMuJe Malabar project. For Playbills I processed a spreadsheet containing around 400 performers that Deven had manually processed last week. I ended up doing a bit more manual tweaking after investigating the appearance of some of the performers in the playbill images, and when I ran the spreadsheet through my import script we then had 52,178 performers in the system, and of these some 52,048 of these have a gender assigned, which I pretty amazing.
I then moved onto processing the plays based on the spreadsheet Deven had worked on before Easter that notes which performances actually feature the same play, even if it is not referred to in exactly the same way. Using this I created canonical records for plays, I set some marked plays as ‘special attractions’ and I deleted certain plays that had been marked for deletion. Before I did this I updated the database so that all tables include an ‘isactive’ field, and updated the API so that it only includes data where ‘isactive’ is set to ‘Y’. Then when it came to deleting plays I didn’t actually delete them but set them (and all associated data such as performers and roles) to ‘isactive = N’. This means if we realise something has been ‘deleted’ that needs to be reinstated I’ll just need to update the relevant fields back to ‘isactive = Y’ rather than having to find and re-insert all associated data.
For the most part the script was successful and result in 261 plays being deleted and 1098 converted to special attractions. It then created 2039 canonical records and 4134 other plays were then assigned to these. There were a few issues with some plays referencing canonical records for other plays that were set to be deleted (so canonical records weren’t created for them), and I passed these on to Deven so she could look into them.
I then created an API endpoint and front-end pages for listing the canonical plays and the details for a selected canonical play. Below is a screenshot of part of the list of canonical plays, ordered by number of plays:
It’s just an initial version that lists the titles, the record type (either play or ‘special attraction’) and a count of the number of associated plays. You can press on column headings to order the table by the column and if you press on a canonical play name you can access a list of plays that are associated with it, as you can see below:
This lists each associated play’s name, date, playbill, venue, location and genres, and you can click on each linked item to reach the relevant page (e.g. the details for a play or the associated playbill page). This is just a work in progress and we’ll probably want to update it, for example to include a genre filter on the canonical plays list, or including access to lists of associated roles and performers. I also updated the playbill page to add links through from play titles to the relevant play page, and I’ve added a ‘Type’ field for each play that either displays ‘Play’ or ‘Special Attraction’.
For the Malabar project I sorted out the legend for source texts, alphabetising the list and ensuring the pane has a maximum width. I also added in the two other categorisation options that I’d included in my specification document: Number of references and placename languages. Number of references categorises the markers and polygons based on the number of times each place is referenced in the source texts. I set up the categories to match the available data (0, 1-5, 6-10, 11-20, 21+) but these can very easily be altered as the data and number of references grow. I think it will prove quite useful to be able to quickly identify the places that are referenced the most in the texts. Below is a screenshot showing the currently available data categorised by number of references:
The other new categorisation option was ‘Placename languages’ and this categorises the places based on the languages of the placename variants included in each record. This will likely need some further work, such as adding in full language names rather than the codes, and possibly filtering out ‘en’ as I’m guessing these would not have been found in the original sources, but it’s still interesting to use the categorisation – for example finding all placenames that have a Hebrew form. Here is a screenshot showing placename languages:
The data itself is still being compiled and I still need to work on the marker colours (and icons) and also to ensure the polygons always appear behind the markers and don’t make the markers unclickable, as is sometimes the case at the moment. Also this week I added the selection of base map, menu section and categorisation type to the URL, meaning it’s now possible to bookmark / share / cite specific views of the map. You can also link directly to a specific record as well.
Monday was a holiday this week and on Tuesday I worked on the Burn Supper map. I updated the narrative summary so that it features percentages based on the number of suppers that supplied the data type rather than the overall total. This means that (for example) Whisky now has the percentage 88% rather than 57%. The new summary layout is shown below:
I then ran an import of new suppers, taking the total of suppers up to 1044, and as the facts and figures are all dynamically generated these updated to reflect the new data. I had to do some manual tweaking of the data, for example some countries had typos and needed fixing, plus I needed to process and upload all new images that were associated with the new suppers. Other than future imports of new data I think the Burns Supper map is now pretty much complete.
On Wednesday I focussed on the Playbills project. Last week I’d executed my script to process performers, and this left a few hundred that needed to be manually checked. I started to work on these myself initially, but it was pretty slow going, especially when checking data by querying the database directly. Instead I decided to create a new spreadsheet that includes all of the fields needed for checking (e.g. the role information as well as the performer details) and it should be quicker to edit this. I sent it, along with detailed instructions on how to update the data, to Deven and hopefully she’ll be able to work through the list in an hour or two.
I met with Deven to discuss our next steps for the project during the afternoon, and before this meeting I prepared a list of discussion points. As we would ideally like to know the genders of all roles in the data (so it will be possible, for example, to ascertain when female performers take on male roles) but as roles are often just names without titles assigning gender would require manual checking. Instead I wondered whether we could get AI to assign gender and then leave us with only the trickier ones that are less clear.
As a quick experiment I passed a short list of roles from one play to ChatGPT and asked it to ascertain the gender of each. The prompt I gave was “for the following list of names work out whether each is male, female or unknown: Edmund (the Blind Boy), Stanislaus, Oberto, Rodolph, Kalig, Molino, Starrow, Elvina, Lida” and the response was: “Edmund (the Blind Boy) — Male, Stanislaus — Male, Oberto — Male, Rodolph — Male, Kalig — Unknown, Molino — Unknown, Starrow — Unknown, Elvina — Female, Lida — Female”
So two thirds of roles were correctly assigned a gender and there were no mistakes, leaving one third that would need manual checking, which I think is looking fairly promising.
On Thursday I continued to develop the interactive map for the HiMuJe Malabar project. I sorted out the type / subtype categorisation in the legend to include both types and subtypes, as shown in the following screenshot:
Subtypes are indented within the type now and any places with a type but no subtype (e.g. Sri Lanka) are now appearing. Currently all types and subtypes appear in the legend, even if they have no associated places, mainly so we can see what the full list will look like. It is rather long and I may need to add in a scrollbar, although we may also want to rework the categorisation too – e.g. we have ‘Town’ as both a type and a subtype of ‘Settlement’, plus we have two occurrences of both ‘Hinterland’ and ‘Backwater’.
I was intending the counts beside the types to be a total of all places categorised by the respective subtypes, but some places only have a type and no subtype so the counts represent these instead (e.g. the ‘4’ beside ‘Region’ shows the number of places that have ‘Region’ and no subtype). There are also 5 places that have a subtype within ‘Region’ in addition to this but I’m not sure how best to represent this without confusing people. We could have something like ‘Region (4+5)’ or ‘Region (9)’ but both of these seem a bit unclear to me. I was also thinking of having the checkbox beside each main type select / deselect all subtypes, but if we did this it wouldn’t be possible to just display the main type without its subtypes. These issues need further consideration.
I also implemented the record pop-up that now appears when you press on a marker or polygon, as you can see in the following screenshot:
The popup header displays the ‘preferred name’ for the place and the ‘general information’ tab features the ID, all names and their languages, the category and subcategory (I guess I should standardise this to ‘type’ and ‘subtype’ to avoid confusion) and any supplied description. The ‘references’ tab shows a count of the number of references to the placename in the source texts and the content of the tab lists the filenames and snippets for each reference, with the actual text highlighted in yellow, as shown in the following screenshot:
We should probably have actual titles for the source texts rather than filenames, but these are not included in the output and is maybe something to add in, along with references to specific lines / pages. Another possible issue is that I am aware some of the text will be read right to left and at the moment all text is just displayed as ‘prefix+extract+suffix’. We might need a flag in the data for when the text should instead be ‘suffix+extract+prefix’ (or I guess set the direction to ‘rtl’ in the stylesheet). When we have any images I’ll add these as a further tab, but we don’t have any yet. Also, I haven’t implemented the ‘Share / Cite’ tab yet.
The final thing I’ve implemented is an alternative categorisation for the map, based on the source texts the place is referenced in. In the data I was working with there are only four places that have references, and I’ve added a further ‘No source’ category that all other places are added to, as the following screenshot demonstrates:
I renamed the ‘Place’ menu in the left-hand menu to ‘Place Categorisation’ and there is now an option to switch the categorisation from type to source text. I’ll add in ‘language’ and ‘frequency of reference’ next week, all being well. After I sent an update to the team, Christian, the technical person for the digital edition, informed me that a new output of the data was available that featured many more references and other updates. I therefore replaced the data in the map with the new version and it made a huge difference – the list of cats and subcats is now shorter, there are many more source texts (although this demonstrated that I have some further work to do with the legend for source text categorisation) and many more places with references. I’ll continue with this next week.
During the week Katie Halsey, the PI of the Books and Borrowing project, contacted me to ask for some help in creating some queries of the data for the monograph that she and Matt are writing. These involved working out which books and authors were found at ten or more libraries, and which books and authors were borrowed in every decade from the 1750s to the 1830s. I looked into this on Friday, and it took most of the day to work on it, partially because it’s been a while since I worked with the data and it took some time to remember how everything fitted together. However, I managed to produce the data required data and I sent it to Katie and Matt in four spreadsheets.
Also on Friday I did some work for the Dictionaries of the Scots Language, setting up a new user account that will be used by some interns that are starting with the project over the summer, sorting out access to the Google Search Console and removing the user survey.
This was another week of many projects, the first being the Burns Supper project. This week I finished implementing all of the data visualisations on the ‘facts and figures’ pop-up, with a mixture of bar, column and pie charts depending on how many data types there are. All charts are ordered by number of suppers, other than the pie charts. I also added in a statement above each visualisation about the number of suppers that supplied the data and a note when a supper may have more than one type of data. The screenshot below shows part of the popup with three visualisation types (mostly) visible:
I haven’t updated the narrative text to update the percentages to be based around the number of suppers that supplied the data rather than the total number of suppers and to include additional explanatory text, but I’ll do this next week.
For the HiMuJe Malabar project I have now incorporated the updated JSON file that contains cats and subcats into the map. I have also incorporated the GeoJSON polygons that project RA Renu has created so far. Where a place has a polygon this is used instead of a marker, and polygons feature a tooltip on hover-over in the same way as markers. I added the type and subtype to the tooltip as well. I also made a start on the categorisation of places based on cat and subcat, although this still needs some work. I added in a ‘legend’ box on the right of the map and have split the places into layers based on their cat and subcat. Each of these appears in the legend, and you can turn each layer on or off. You can also select or deselect all layers, which I find quite useful (e.g. deselect all then only add in the layers you’re interested in). There is also a count of the number of places in each layer as part of the layer label. Note that the cats and subcats only appear if there is at least one place that has latitude and longitude assigned to the subcat, which is why (for example) ‘Mosque’ is not currently listed. Below is a screenshot showing the categorisation options that are currently in place:
There’s still quite a lot to do and some issues that need to be addressed. At the moment only places that have a subcat appear. I still need to implement the top-level cats, so for example ‘Sri Lanka’ that has cat ‘Region’ and no subcat is not currently appearing on the map. I’m hoping to update the legend to give it a two-level hierarchy (as mentioned in my specification document) but I didn’t have time to implement this. Also, marker and layer colours are currently arbitrarily assigned. I think we will eventually have icons as well as colours, but in addition to this I will update the colours, probably to have different shades for each subcat in a cat. I also need to ensure that the polygons always appear behind the markers as this is not currently the case. E.g. if you turn the ‘Region: Kingdom’ layer off and then on again the layer is then added to the front, meaning it’s no longer possible to press on any markers located within the polygons. Also, there will be several other categorisation options in addition to cat and subcat (as discussed in the specification document) and these still need to be implemented.
Also this week, I returned to working on the Playbills project for the first time since before Easter. This included writing an executing a script to merge genre classifications, which reduced the number of genres from 188 to 69 and reassigned any plays that were assigned to a deleted genre. I also executed my scripts that process performers to extract individual names and titles from the free-text ‘full name’ column and assigned gender based on the title. My scripts process individual performers and also split multiple performers into individual records, resulting in the number of performers going up from 49498 to 52146. In these cases the newly extracted performers have also been assigned to the same role as the existing row. Of the 52146 performers we now have a Male or Female gender assigned to 51724, which I think is pretty good. I also updated the playbill and play pages so that the extracted names and genders now appear, and multiple performers for a role all now appear. I still need to deal with the ‘unprocessed’ performer spreadsheet, which is probably going to need some manual work as this contains the ‘edge cases’ that my processing scripts were unable to tackle. I hope be able to continue with this next week.
My fourth project of the week was the DOST Auld Laws project, for which I continued to work on the search facilities. This has involved setting up an Exist-db XML database and learning how to run full-text queries on the documents contained in it and how to generate KWIC (keyword in context) snippets for the results. There’s still a lot to do, but an initial version of the quick search is now operational, as the screenshot below demonstrates:
You can also use an asterisk wildcard to represent any number of characters, for example ‘wyn*g’ finds all terms beginning ‘wyn’ and ending in ‘g’. A question mark wildcard can also be used to represent a single character, e.g. ‘r?cht’ matches ‘richt’ and ‘rycht’. I haven’t added in pagination of results yet, so they currently all appear on one page. Each result features the document name and the page number where the result is found, plus the snippet with the term highlighted. Pressing on the snippet loads the page with the line where the term is found highlighted in both the image and the text.
It’s definitely something of a milestone to get the quick search working, but there is still a lot to be done. The highlighting of the term in the snippet is not currently working properly when the term includes characters in <expan>. The search finds the term, but the highlighting is only getting applied to the part of the term before the <expan> tag. Also, Exist-db uses the Lucene full-text engine and this does not support wildcards at the beginning of terms, so while a search for ‘grant*’ finds all terms beginning with ‘grant’, it’s not possible to find all words ending in ‘*ting’, for example. I’m not sure how big an issue this is, but if such functionality is required I’ll have to investigate an alternative. Also, phrase searching is not currently operational. I have a query that should be able to handle this, but I haven’t had time to add this in yet. It’s been quite tricky to implement as phase searches need to cross line and page boundaries whilst still returning the line and page IDs where the start of the phrase is found.
In addition, Boolean searches are not yet working (e.g. term1 AND term2, or term1 OR term2). However, we might want to consider how useful these would be and how they would work. Currently the search looks for occurrences of the term in entire documents and returns snippets showing the context of the term. If a Boolean search is using an entire document as its source then how useful would it be if (for example) term1 is found on page one and term2 is found on page 29? I’m just not sure how helpful this would be. I guess an OR search would be useful in finding variant spellings?
I also still need to add in results traversal to the document page, so you can navigate directly through results when looking at one specific result, and I need to add in a link back to the results page from the document page. There’s also still a lot of work to do on the user interface, which is still not finalised and I’ll work on this (e.g. colours, fonts, layout) once I’ve finished with the search facilities.
My fifth and final project of the week was the redevelopment of Mapping Metaphor. I didn’t have much time left to spend on this, but I did manage to tidy up the loose ends from last week. Where a combined metaphor has both OE data and E data and the E data does not begin in the OE period the timeline in the visualisation card popup now includes OE selected and highlights the period when the E data begins. This second highlight currently uses the green used for the visualisation background but can be changed. We should also maybe include some explanatory text so users understand why two periods are selected. The following screenshot shows the timeline with two periods highlighted:
I also updated the card popups so that both E and OE IDs appear in the header of the card (where applicable) now, as you can also see in the above screenshot. Also, where a metaphor connection exists in E but not OE (or vice-versa) counts of lexemes in each joining category in the other period are now displayed.
Also this week I had an email conversation with Craig Lamont about an online resource for his new research project and arranged for a subdomain to be set up for it. I also updated the DSL survey so that it only appears on mobile devices and responded to a request from Kirsteen McCue regarding stats for song downloads on the Editing Burns and Burns Choral websites.
This was a pretty busy week that had me dividing my time between five main projects. I spent most of Monday working on the Burns Suppers project, beginning development of the ‘facts and figures’ popup. I added a button in the ‘Home’ menu labelled ‘Facts & figures’, that features a pie chart icon. Pressing on this opens the facts and figures popup part of which you can see in the screenshot below:
The popup features the narrative summary section, with the figures in this section being dynamically generated. It took quite some time to write the code to generate them, but at least there will be no further work to do when we import more data. For now all percentages are the percentage of the total number of suppers rather than a percentage of the number of suppers that supplied the data, so for example, the ‘annual supper’ figure is 39% rather than displaying 83% (based on there being 420 suppers that actually have frequency data). This is because I realised that without a lot of additional explanatory text users will likely think our figures are wrong. If people only see the total number of suppers (886) and the number that are annual (347) then displaying 83% will be misleading. We’d have to include lots of additional data such as “of the 420 suppers that included frequency data, 347 (83%) were annual events”. Depending on feedback I may implement this later.
I’ve only implemented one graph in the pop-up so far, which is the graph of countries. This is a long list, but I think it works ok. I also included an option to switch from a graph ordered by number of suppers to an alphabetical version, and this is all operational. I’ll continue to add further graphs next week.
I spent most of Tuesday working on the interactive map for the HiMuJe Malaber project. I’ve created an initial version of the map now, which uses my map menu interface that I’ve used on several other resources, and below is a screenshot:
By default the map uses the ‘Watercolour’ base map, as we used for a static map on the main site (https://himuje-malabar.glasgow.ac.uk/about/summary/). You can also switch to a satellite map using the ‘Change the base map’ buttons. Currently markers as just displayed as red dots, and if you hover over them the ‘Preferred Name’ from the JSON file is displayed as a tooltip (although I’ve noticed that some places don’t seem to have a ‘Preferred Name’). There are no popups or filters or anything like that yet. Also note that any places in the JSON file that don’t currently have latitude and longitude values are not displayed as there is nowhere to ‘pin’ them.
You can zoom and pan the map as with Google maps, and the icon in the bottom right makes the map full screen. The map also works on mobile devices (hiding the left-hand menu using the ‘<’ button above the menu helps when using a mobile device). Other than the base map selection options, nothing works in the left-hand menu yet. ‘Places’ will eventually include the place filters. ‘Travel Routes’ will list travel routes involving people and organisations once this data is available. There’s still a lot to do, and I’ll hopefully begin work on the place popups and the categorisation options next week.
I spent most of Wednesday working on the Place-names of Armagh project. I’ve updated my ‘Irish form / translation’ script to remove ‘More details: Unverified’ and to also attempt to split the translation up based on apostrophes. This has been a bit of a nightmare, firstly because apostrophes are not only used to denote the translation but appear within the text, and secondly because what looks like an apostrophe can actually be many different characters, including straight, curly opening and closing apostrophes and several HTML codes that are rendered as apostrophes but are stored as codes. This has all made splitting the text up rather challenging. However, I’ve got something that mostly works.
The script now deals with things like Ir. <em>Coill Uí Fhloinn ‘O’Flynn’s </em>wood’ and outputs “O’Flynn’s wood” as the translation. Where there are multiple sections some unnecessary apostrophes appear, for example: “Ir. <em>Tír Garbh </em>’rough land or district’ or perhaps Ir. <em>Baile Uí Aodha </em>'<em>O’Hugh’s</em> homestead or townland’” results in: “rough land or district’ or perhaps Ir. Baile Uí Aodha ‘O’Hugh’s homestead or townland” This will probably need some manual fixing. The script also now processes rows that don’t have ‘Ir.’ and italics, for example “Críonchoill ‘withered decayed wood’” now has Irish form “<em>Críonchoill </em>” and translation “withered decayed wood”.
I sent an Excel version of the script output to the project team as this will likely require some (but hopefully not too much) manual intervention to fully sort out. For example, the text “Eng./Sc. ‘hill of the military camp’ or perhaps Ir. <em>Mullach na Críne </em>’hilltop of decay’ or Ir. <em>Mullach na Craoibhe </em>’hilltop of branch, tree’” will need some work as the translation omits the ‘Eng./Sc.’ Text as it’s before the first apostrophe.
The other thing I’ve managed to do today is to set up an initial version of the public map interface. This currently takes quite a long time to load as it’s processing a lot of data. As with the other sites, I’ll create a cached version of the data once we’re ready to launch, which will be much faster to load. The reason it’s not in place now is that a new version of the cache will need to be generated any time you want subsequent changes made to the CMS to appear on the map, and we’re still very much working on the data.
For now, three base maps are available (satellite, satellite with labels and relief). We can add more in later. There are also no parish boundaries or townland / barony boundaries as I don’t have this data yet. I also still need to work on the colours of the markers as there are no colours for a lot of the classifications so they’re defaulting to purple. There’s also nothing in the elements glossary as we don’t have this data yet. But the search, browse and categorisation options all work. For example, here are the placenames beginning with ‘A’ categorised by altitude:
There are some issues with the data. For some reason there are three place-names miles away from Armagh, around Ballybofey, and there are a couple of place-names appearing in the Irish Sea. We also have an issue of the same coordinates being used for multiple places, thus resulting in markers appearing on top of markers. There’s also an issue with the accuracy of markers too. For example, the marker for ‘Lowry’s Lough’ is found about 500m south of the actual body of water. But the good thing about having this map available (despite the speed issues for now) is that it will help when working on the data.
I spent all of Thursday and most of Friday morning learning how to use the Exist-DB XML database that I’m hoping to use for the DOST Auld Laws project and developing the query that will eventually power the quick search, and form the basis for the advanced search. I installed Exist on my laptop and managed to set up a collection for the project’s XML files, which I then uploaded into the system. I followed the documentation available on the Exist website and was able to create a full-text index for the collection, and I followed a useful tutorial here: https://dh.obdurodon.org/php-xquery.xhtml about how to query Exist using the REST interface. Setting up and querying the texts in Exist was all new to me and there was a lot to try and take in, with many configuration options that were not all that easy to follow in the Exist documentation. I ended up using ChatGPT quite a lot to help me figure out how everything should work and why some of my initial tests were not producing any results. This proved to be extremely useful and really helped increase my understanding of Xquery. I was able to get a search working that queries the full text and returns contextual snippets for each result, together with the IDs of the line, page and document, which is everything I’ll need for the quick search. On Friday I asked Luca to help set up the necessary collection on the server and hopefully I’ll be able to get an initial version of the quick search working on the website next week.
This left me with a few hours on Friday afternoon to devote to the redevelopment of the Mapping Metaphor resource, for which I’m creating a unified view of the data, joining both the English and Old English datasets together. I used this time to implement the visualisation card view when viewing connections between specific categories. The combined card view shows a bidirectional arrow if the OE and E directions differ. It also defaults to the E strength. The counts of lexemes in each category include both full and OE counts and the examples of metaphor feature both OE and E examples, with OE coming first. If the metaphor exists in the OE data then the ‘start era’ now defaults to OE (but as of yet I’ve not added in a further highlight in the timeline to show when the first non-OE occurrence was documented). Below is an example of the combined card view:
In addition to the above I also looked into an issue raised by Ann Fergusson for the Dictionaries of the Scots Language regarding accented characters, search results and entry slugs. This took some time to investigate but I think my response proved useful. I also had email discussions with Deven Parker about her Playbills project, which she now has time to look into again. I’ll probably be working on this again next week.
This was a three-day week for me because Friday was Good Friday and I’d taken the Thursday off as well. Despite this, I still managed to work on four different projects. Last week for the Playbills project I’d written a script to process performer names, including splitting this up into titles, forenames, surnames and other names, and also assigning gender. What I’d spotted and hadn’t had time to address is that there are several thousand performer names that are actually multiple performers that would need to be split into individual people. I spent most of Monday working on a script that would split these multiple performers up and process them all individually. This managed to reduce the number of unprocessed performers from 1271 to 414 (and most of the ones remaining are either blank or have text like ‘performer unknown’). I saved all performers in spreadsheets so Deven can check them – as of yet I haven’t made any updates to the database. There are 48,234 individual performers that were successfully processed by my script and a further 3,491 that were processed from multiple performers, leaving just 414 unprocessed performers, so I’m pretty happy at how successful my scripts have been.
Also for the Playbills project this week I had further discussions with Deven about downgrading certain plays to ‘special attractions’, which are things like songs and dance that are not full plays. Deven had also completed work on the spreadsheet that mapped out which performances involved the same plays (even if their titles were not exactly the same) that I will use to generate canonical records for plays. This spreadsheet also notes which plays should be downgraded and others that should be deleted entirely.
Whilst working on the scripts that will generate canonical records, downgrade plays and delete others I spotted some issues with the spreadsheet, such as plays being marked for deletion that looks like legitimate plays to me. Some other rows in the spreadsheet had been marked as both to be merged and deleted, which wasn’t right. I sent a list of possible issues to Deven and I’ll need to take this up with her once I’m back from my holiday.
For the Burns Supper Map project I had an in-person meeting the Cleo and Pauline to discuss the map, the interface, the data and our plans for the weeks ahead. It was a great meeting and really useful to meet up in person. Everyone is very happy with how the map interface is coming along and we have decided which of the test interfaces I’d developed would be used for the live site (mostly the first test interface but with the fonts from the second one). I have a 14 point ‘to do’ list for the project that I’ll work through once I’m back from my holiday.
For the HiMuJe Malabar project I wrote a non-technical specification document for the map I’ll be developing for the project, based on the discussions I had with Renu and Ophira at our meeting last week. I’ve got a pretty good idea about what needs to be developed now and I sent the first draft of my document to Renu and Ophira for feedback.
The remainder of my time was spent trying to digitise an old cassette tape of Scots and Gaelic poetry for Alan Riach in Scottish Literature. I had hoped I’d be able to plug an old tape player into my laptop to do this, but then I realised my laptop (and most modern computing equipment) doesn’t have a line-in port so this wouldn’t work. I then asked Jane Stuart-Smith whether there may be facilities in the GULP lab that I could use, and while there was a tape player we were unable to get it connected to any computer in the lab. I finally managed to digitise the tape by digging my old Hi-Fi out of my attic at home and plugging it into an old PC. Thankfully this worked and I recorded each individual poem as a separate MP3, then sent these to Alan.
I was off on Thursday and Friday, and I’ll be off for Easter for all of the following week and will return to work on Monday the 13th of April. So that’s all from me for now.
I continued to mainly work on the Playbills and Burns Supper projects this week, but also spent some time on several other projects. For the Playbills project I re-ran my venue and printer merge scripts as previously the scripts only merged records and didn’t update their names and locations. I hadn’t realised that Deven was going to edit some of these, but thankfully we spotted this and my updated scripts incorporated the edits.
I also began work on a script that will create canonical records for plays. These records will track the appearance of plays across multiple playbills, even if the actual names of the plays differ. Deven has been updating a spreadsheet to note which plays are actually the same and I’ll be using this to create the canonical records. Unfortunately Deven encountered a bit of an issue with Teams, which resulted in her losing a tranche of updates she had made, and this set us back a bit. I have the new database and script in place, so hopefully I’ll be able to run the data through this next week.
I then began to tackle the performers – splitting the names up into constituent parts, reformatting them and assigning gender. It all went pretty well, and my script successfully processed 48,227 performers (34420 male, 13807 female), leaving 1271 that will need further work. The majority of the problematical rows are due to there being multiple performers and it should be possible to refine my script to further process these and create separate performer records for each. This is something I’m going to look into next week. The remainder are either blank, have text such as ‘[Actor not named]’ or are problems with the AI output (e.g. ‘celebrated clowns’ is not a performer name).
For the 48,227 that have been processed my script extracts the first word of the full name and checks it against lists of male and female titles. If the title is in one of the lists the script processes the performer, assigning it the relevant gender and removing bracketed text and saving this in a separate field (any forms of ‘junior’ and ‘senior’ such as ‘Jun.’ are also extracted and stored in this field as standardised ‘Junior’ and ‘Senior’ text).
Some names have exclamation marks and these are removed, and names are converted to lower case with upper case letters for each word. If the name (minus title and other text) is multiple words the final word is treated as a surname and the rest as forenames, otherwise the name is set as surname. Finally, if the surname starts with Mc, M’ or O’ the following letter is capitalised. In the case of M’ this is also standardised to Mc. Note that ‘Mac’ is not changed as names like ‘Macauly’ are not written as ‘MacAuly’ (at least, not usually!).
With the names split like this we’ll be able to have alphabetical lists of performers, which will be useful, as will having gender. It will also help with identifying the same performer across multiple plays.
I also spent a bit of time researching the locations of venues, so we have a few to use for test purposes when it comes to developing a map-based interface to the data. Unfortunately this proved to be quite difficult. Google is pretty hopeless when searching for historical theatre names and just fills its results with random current ‘theatre’ links and ‘book tickets now’ adverts when trying to find actual historical information. It was all very frustrating and I’m sure Google never used to be this rubbish. I did manage to find the locations for around 20 venues, which is a good start, but there are still around 80 left to do.
For the Burns Supper project I mostly spent my time processing data. This included uploading images that had been converted from PDF files and associating them with supper records and running another import of the survey and public domain spreadsheets. This resulted in a further 141 suppers being added to the system, and I also took the opportunity to create a document that lists the steps needed to import new suppers, and this documentation should ensure that future data imports are easier to manage. As part of this process I also dealt with new images for the suppers, running scripts to convert, resize and rename image files and create image records in the database. I also manually split one supper record into three distinct records as it actually covered three different events at different locations.
I also added in links to videos that RA Cleo had uploaded to YouTube. It was a little trickier than I anticipated to get the embedded videos working properly. There were then some issues with the videos not taking up the correct amount of space in the carousel, but I rectified this. I then noticed that the carousel’s ‘next’ and ‘previous’ buttons were obscuring the YouTube player buttons, so if you tried to enter full screen or pressed the play button you’d just change slide. I managed to fix this, but needed to tweak it further when I realised Chrome-based browsers display a different player to Firefox.
There was also an issue with the YouTube player displaying thumbnails of other, entirely unrelated videos in the player when you paused the video. I was shown videos of Donald Trump as a young man and a video about Epstein, which was all pretty awful. Thankfully I found a way to disable this. I then noticed that if you didn’t manually pause a video or watch it to the end before either navigating to another slide in the carousel, selecting one of the other tabs in the record popup or closing the popup, the video (although now hidden) kept playing so you’d still hear the audio. This was clearly no good and I managed to sort it, although my first attempt at a fix didn’t work when a record had multiple videos (it does now). So eventually all was working correctly and we now have 53 videos that are accessible as part of the map.
I also added in a ‘Random supper’ feature to the ‘Home’ tab. This navigates to and (after a short delay) opens the record for a random supper, based on your currently chosen filters, so for example if you have filtered the map to only show suppers that have images or videos you can press the button to load a random supper that has media.
Also this week I had a meeting with Ophira and Renu about the HiMuJe Malabar interactive map. It’s been a while since I met with them as they’ve been focussing on adding place data to the digital edition, which is being overseen by a partner institution. However, we’re reaching the point where the map needs to be developed. We had a productive meeting, discussing the data and how it might be represented on a map. Next week I’m hoping to write a brief, non-technical specification document so we all know what will be developed.
Also this week I ran a few queries and outputted a spreadsheet of data for the Anglo-Norman Dictionary, and I made some updates to the map of dialect areas for the Dictionaries of the Scots Language based on feedback I’d been sent a while back. I think this map is nearing completion now, although I still need to develop region search and filtering for the DSL’s advanced search, which this map will then connect to.
I continued to work on the Burns Supper Map this week, implementing the ‘click through’ options in the record pop-up. The record pop-ups now feature a dotted line underneath certain information, and if you press on this the relevant filter for the selected item will display, as the following screenshot demonstrates:
Other than importing more data, integrating the videos and things like the ‘more info’ pop-up and some ‘fact and figures’ that’s pretty much all of the functionality in place now, and I therefore moved on to the user interface, a task that took a fair amount of time, but has been useful. I developed three alternative interfaces using different colour schemes and fonts, and in some cases different layouts. For the first version I took the head of Burns from the stained glass image and used this in the ‘Home’ tab and also as the supper icon and the browser tab icon. The menu sections have a black background and white text and I’ve used the font ‘Quintessential’ for the site heading and also the popup headings. You can see this in the following screenshot:
For the second version I used a traditional Scottish Blue and White colour scheme. This version doesn’t feature the disembodied Burns head and it uses the ‘whisky glass’ icon on the map and in the browser tab. The whole site uses the ‘Baskervville’ serif font throughout (the other versions use the sans-serif ‘Montserrat’ font for most text) with the ‘Baskervville SC’ small-caps variant used for headings:
The third version uses a sort of teal-ish colour scheme with a sort of whisky-orange colour for highlights. For this one I incorporated the Nasmyth painting of Burns in the ‘Home’ tab and I used the cursive font ‘Tangerine’ for the ‘Burns Suppers’ text in the site heading and in the record pop-up. Map icons use the cutlery icon with the whisky-orange background. This is also used in the browser tab, although the icon and colour scheme are rather similar to the ‘Just Eat’ logo and this might be an issue. Here’s a screenshot:
For this version I also redesigned the record pop-up to make it look sort of like a fancy menu. This does take up more vertical space and the cursive font used in the headings might be difficult for some people to read, but I quite like the way it looks:
These are just work-in-progress versions and I’ll hopefully meet with the other team members next week to discuss how we should take this forward. Also this week the project RA Cleo went through the images and reduced the number of images for each supper to a maximum of ten (some had hundreds before). I then ran a script to resize these images to a maximum of 1600px, which has greatly reduced the file size of the images – there were some previously that were 16MB each. I then replaced the images on the server with this smaller set, reducing the number of images from 1696 to 893 and reducing the total size of the images from more than 4GB to about 350MB.
I also continued with the Playbills project this week, giving advice to project PI Deven as she edited the data and processing updates to the data. This included merging and deleting venue and printer records to remove duplicates and reassign playbills to the merged records. I also began to investigate assigning location data (latitude and longitude) to venues, which is something we’re going to need of we want to create a map-based interface for the data. We have around 100 venues after merging and deleting, and it’s possible that some others will also be deleted, so it’s not a huge number of records. However, it’s likely that the location data will need to be manually researched as many of the theatres no longer exist, or at least the buildings no longe exists as theatres. For example, our first listed venue is ‘Royal Theatre Aberdeen’, which doesn’t exist anymore, but I found an address for it on this page, meaning I could find the building (now Aberdeen Elm Church) on Google Maps and get the latitude and longitude (57.146607938894356, -2.0920931547723085). I’ve created a spreadsheet on the project’s Team that we can use to add this data in, and I’ll probably research a few records myself, if I have the time.
I also spent about a day this week writing an initial version of a specification for Joanna’s DOST Auld Laws front-end. It’s a non-technical overview of what will be developed and how things will work and I created it mainly so we all know what the plan is. The document I wrote is just a first draft and I sent it on to the team for feedback. It’s likely that aspects of it will need to be changed and in fact there is a list of questions at the end that will need consideration. I’ll hopefully get some feedback from the team next week and I’m hoping that we’ll have a final version of the document decided upon before Easter, so I can start development after I get back from my Easter holidays.
I also spent some time investigating a data issue on the Books and Borrowing website. Certain records for Innerpeffray library didn’t have associated borrowers so Katie and Kit added these in, including creating new borrowers where necessary. However, the data in the front-end wasn’t updating to show the borrowers when viewing register pages and I needed to figure out why. It turns out that this is a consequence of the versioning system I implemented last year. The CMS is now set to data version 2, meaning anything newly added has this version number applied to it. The front-end is set to only display data version 1, so any new borrowers created will not get displayed as they are version 2 records. I manually updated the new borrower records to assign them to version 1 and after that they appeared in the front-end. We still need to fully update the data caches and Solr indexes before the new borrowers are fully integrated with the front-end (e.g. appearing in search results), and this is a task I’ll have to tackle in the next few months.
I divided my time between four projects this week: Playbills, the Burns Supper Map, the place-names of Armagh and the Dictionaries of the Scots Language. For the Playbills project I spent some time processing the Playbill images. These were stored in hundreds of directories in many different file formats, including HEIC that is an Apple-specific format, JPEG, PNG, GIF, TIFF and even PDF. Thankfully I discovered that the command-line tool ImageMagick can process HEIC files in Windows and I wrote a script to find all image files across the hundreds of directories, move them to a single folder and convert them (where necessary) into JPEGs. It took a while for the script to execute, but it did so during Monday evening and successfully migrated 17,406 image files.
On Tuesday I then wrote a further script that picked out just the images actually referenced in our current dataset (so 1902 images) and I uploaded these to the server. I then worked on the page through which the image can be viewed. I’d already created the playbill page last week, through which the textual representation of each playbill can be accessed, and it was now a matter of adding in an OpenLayers viewer that would load in the image and enable users to zoom and pan around it. Eventually we’re hoping to connect this to a IIIF server, which will make loading the image a lot quicker, but for now the view just loads in the full, high-resolution image when the page loads. As this can take some time I added in a loading ‘spinner’. I also made the image pane as tall as the browser window and the text pane is the same height, with text scrolling if it’s longer than the pane. This ensures you can still see the image as you scroll down the text. You can also press on the icon in the top right of the image pane to view the image full screen. The following screenshot shows how things currently look, although bear in mind that the interface is still very much a work in progress – it doesn’t work very well on mobile devices yet, for example.
I also had a chat with Deven about the process for consolidating and amalgamating records using spreadsheets, and once Deven has completed this process I’ll update the data and continue to work on the site.
The Burns Supper Map took up much of the rest of the week. My work included fixing an issue with ‘guests’ not working in the short URLs, relocating a New York supper that was appearing in Barcelona to its proper location and adding ‘Source’ to the table view. I then implemented the ‘Download CSV’ option, which includes all of the fields from the table view and outputs the CSV using a purely JavaScript approach rather than connecting to the server.
I also added ‘country’ to the advanced filters, and fixed a few typos (e.g. ‘Astria’). It took several hours to implement the update, as ‘country’ is free-text and includes spaces and characters such as the colon that don’t work well in URLs. I’ve therefore had to create a new field that stores URL-friendly versions of the name and it is this that is then used for search purposes.
I then moved on to processing the supper images. This was quite nicely timed as the processing was very similar to what was required for the Playbills project (lots of different image formats with files spread across countless directories) so I could adapt the script I’d written earlier in the week. I set the script to migrate all of the images to JPEG and to copy them all to once single location with a standardised filename.
With image processing complete, I uploaded the files to the server and began work on the ‘images’ tab in the supper record pop-up, which I managed to complete. I decided to use the Bootstrap carousel to display the images, as it works very well and has support for touch controls (e.g. swiping left and right). Addin in Bootstrap (which the interface was not previously using) caused a few layout issues as some existing styles were overwritten by Bootstrap’s styles, but I was able to sort these issues out.
Where the supper has one image it just gets displayed, but where there is more than one image the carousel navigation buttons appear on the left and right of the image, allowing you to scroll through. One issue I encountered is that we have a mixture of portrait and landscape images, and this causes some layout problems. However, I’ve managed to sort this and I’ve given any extra space a black background, which I think works quite well. Another thing to note is that all of the images are loaded into the carousel at once, so for a record that has hundreds of images, or very large image files, it can take some time before everything gets loaded in. This won’t be an issue for long, though, as we’re going to pick out just a few imager per supper (some currently have as many as 200) and I’m going to resize the images (some are currently up to 16MB). Below is a screenshot of the image tab showing the image tab and accordion for one Burns Supper:
I also realised that people are going to be pretty interested in seeing the images so I added an option to the advanced filters that allows you to limit the display to records that include (or don’t include) media.
For the DSL this week I added in a user feedback popup to our test site and also at a temporary URL on the live site so that the team can test out how the popup will function. As the live and test sites use different user interface frameworks I needed to implement the popup separately for each site, but it didn’t take too long to set up.
On Friday I returned to the Place-names of Armagh resource, which I’ve not worked on for a few weeks. The PI Mícheál had gone through the lists of parishes and baronies and had noted which needed to be merged or deleted, and I spent some time processing the data, moving place-name records to different parishes or baronies as required. I also spent some time writing a script to extract the Irish forms and translations from the ‘analysis’ field (which is actually called ‘description’ in the database). The outputted table displays fields for the PID, the name and the current contents of the Description field in these columns. The ‘Irish Form’ column extracts all italic text. As there are often multiple italicised pieces of text and in such cases each is extracted and separated with a slash, and tags are displayed rather than being rendered in this and the following two columns as I’ll need the tags when I grab the data.
The ‘Translation’ column includes all of the text from the first paragraph of the description, except where this contains ‘reference’, ‘uncertain’ and ‘details’ as these text to signify that it’s not a translation. Any other text gets added to the ‘Remaining Text’ column. Unfortunately it looks like the policy of using italics for Irish forms is not used in later records, and also it looks like the HTML is malformed in a number of these later records, so all text for these rows has just been added to the ‘Remaining Text’ column. I might be able to further process things to sort some of this. But for now I’ll need to wait until Mícheál has a chance to look through the output and decides whether he wants to use it.