This was a two-day week for me, as I headed off on holiday on Wednesday and will be away until Monday the 6th of July. I mostly spent the two days tidying up loose ends before I went away. This included making some updates to some of the data for the Burns Supper Map that Cleo asked me to do. This was mainly fixing typos and updating the categorisation, plus adding, deleting and replacing a selection of images.
I also wrote scripts to process the edited announcements for the Playbills project. Deven had been working on a spreadsheet version of the data that featured columns noting where announcement text had been edited, where the announcement needed to be deleted, reclassified as a ‘special attraction’ or in some cases where a new announcement needed to be added. After writing and running my script a total of 1836 announcements were updated, reclassified or deleted. I then ran my script that generates complete JSON files for every playbill in the system, featuring all data about associated plays, venues, actors, roles and such things. This took quite a while to execute as a lot of data needs to be processed, but after half an hour or so I had a collection of 1902 JSON files that I sent to Deven so she can run some offline queries for a paper she’s writing.
I spent the rest of my two days continuing to work on the redevelopment of the Mapping Metaphor resource, which will enable both English and Old English metaphors to be viewed in combination. I managed to complete the combined card view, as the following screenshot demonstrates:
The cards feature both main and OE metaphor IDs, counts of lexemes in both E and OE and examples of metaphor from both E and OE. Where a metaphor exists in both E and OE and the E date is later than OE the later date is highlighted in the card in green. I still need to work on the timeline view, the CSV download and the search facilities, plus I need to add in an option to switch between the combined and E / OE specific maps, and also update the user interface, so there is still a lot to do. But not until after my holiday.
After being off sick on Thursday and Friday last week I returned to work for a full five days this week, although I was still in the throes of ‘post viral fatigue’ and it wasn’t until Friday that I stopped feeling exhausted. Despite this I still managed to get a lot done for several projects this week. I had been planning on continuing with the redevelopment of Mapping Metaphor towards the end of last week, and as I was unable to do so I decided to focus on this on Monday.
As I looked through the partially updated site I spotted an issue when combining E and OE data: Sometimes the metaphorical connection between two categories has the categories the other way round in the other dataset (i.e. in E we have cat1 and cat2 whereas in OE we have cat2 and cat1). This was resulting in metaphors in the combined view sometimes being treated as different when they’re actually the same, which was affecting some counts and also sometimes making the figure for number of OE lexemes appear next to the wrong category in the card view. I managed to sort this, and I also ensured that if the category order needs to be swapped then the direction is also swapped.
I then spent most of the day implementing the combined table view, and it should be fully operational now. It was extremely tricky to implement this as the script that generates the data for the table view can handle many different data requests and (unlike the visualisation script) also processes individual metaphor connections for display, including directionality and examples of metaphor. There was a lot to update and check, and it was probably not the best of choices of things to work on during my first day back at work. However, I got there in the end and was able to share the update with Wendy and Carole by the end of the day. I still need to update the CSV download option, but that will need to wait for another day.
I also spent about half a day or so on the Playbills project this week. I fixed the few remaining plays that didn’t have links to canonical records so that every play now includes such a link. I then moved onto looking at performers. I wrote a script that outputs a spreadsheet that lists all performers arranged alphabetically by surname with the corresponding play, date of performance and venue. I’m hoping we’ll be able to do something with this to figure out which performers are actually the same person, but with almost 52,000 rows we’ll probably want to try some automated processes rather than figure it all out manually. I’m just not sure how best to proceed with this.
For example, there are 17 ‘Mrs Ashton’ performers, with performances from 1827 to 1834. 12 of these are at the Theatre Royal, Birmingham in 1827, then there’s nothing until 1832 at Bristol (4 performances) then one final performance in 1834 in Edinburgh. Would we consider these all to be the same performer? Or should we treat the Bristol and Edinburgh ones as different people? This will need some discussion with Deven.
I also began looking into generating canonical roles for plays. I wrote a script that for a given canonical play ID returns all plays that have the ID and for each lists its name and all associated roles. The individual play names are links through to the play page, from which it’s possible to access the playbill page and check the original image. At the very bottom of the output is a list of all unique roles with a count of the number of times they appear. This is only an initial version but it has highlighted many issues due to variant spellings (either in the original playbill or due to the AI text extraction). I can further tweak this, for example making all lower case, removing punctuation, removing text in brackets, replacing “M’” with ‘Mac’ or ‘Mc’ to see if these match. We could also use Levenshtein distance to try and find forms that differ by one character, which would match ‘Baillie Nicol Jarvie’ and ‘Balie Nicol Jarvie’ with ‘Balie Nicol Jarvie’ and ‘Captain Thorton’ with ‘Captain Thornton’, for example. We’d have to watch out for false positives, though (e.g. ‘Jane’, ‘Jean’ and ‘Janet’), and also make sure the form that has the most hits is taken as the canonical form, rather than it just being the first form found. But it’s also possible that asking AI to do this might be a better approach.
As I was working on this I noticed the interestingly named role ‘MacSycophant’ in the list, which is definitely an AI hallucination! Looking at the original playbill image it should be ‘MacStewart’. I also spotted that many of the actors are incorrectly assigned, with ‘Mr Harrold’ and ‘Mr Felton’ being swapped and Rob Roy listed as being performed by Miss A Murray instead of Mr Pritchard). Checking the original YAML shows that the issue was with the AI extraction and thankfully it was not any subsequent processing scripts that have introduced the errors.
Following on from this, Deven has suggested that she and some other researchers may manually proofread all of the actors and roles across the entire dataset. This would really help to ensure that the data is more accurate, but it is also a huge amount of work, as there are 1902 playbills containing tens of thousands of performers and roles.
I spent some time proposing a means of editing the data so that the correct updates are made in the database. We would need to generate lists of performers from the database so we don’t lose the work we’ve already done to split the names and assign genders, and presumably the task won’t just involve swapping roles between performers, but will involve role names being edited too (e.g. to fix things like ‘MacSycophant’). I think it’s likely that some issues with performer names that have been extracted incorrectly too will also be spotted and need fixing too. We’ll need to ensure any updates to performers’ names and role names can be tracked to the relevant record in the database using the corresponding ID fields.
To achieve all this I generated a spreadsheet that includes information about the playbill, play, date and venue, with a link to the page for each playbill for checking the original image. The Role data and Performer data then follow on each row. Using this a proofreader could then make changes to the ‘Role Name’ field where required, but also make changes to the performer fields too if anything needs corrected there, and also add in new rows if required.
I then returned to the DOST Auld Laws project, for which I created an initial version of a user interface for the site. This is still just a work in progress and may change depending on feedback given, but below is a screenshot:
I spent most of the rest of the week working on created a system to manage travel routes for the interactive map for the HiMuJe Malabar project. My first task was to create a test route based on data provided by project Co-I Ines. I added this to a spreadsheet template I’d created, and which we’ll hopefully be able to use for future routes.
With this data in place I then wrote a script that converts the spreadsheet into JSON data for use on the map, so if further travel routes are created (one per spreadsheet) with the same structure I’ll be able to add them to the map.
Once I’d completed my updates to the map interface, when you press on the ‘Travel Routes’ section of the map menu each travel route is now listed (there is currently only one). Each route appears with its title and type (we can maybe split and/or limit the list by type in future) and the description given in the spreadsheet. Pressing on the checkbox or the route name adds the route to the map. This currently highlights the relevant locations with a red border and adds a dotted red line connecting each location in the itinerary. We may want to use different colours when multiple routes are added at the same time to help differentiate them.
As the route can get lost in amongst all of the other data on the map I’ve added an option to show or hide unrelated places. If you deselect the ‘Show unrelated places’ checkbox any locations that are not part of the travel route are removed from the map, making it much easier to see the route. The following screenshot shows this:
The legend is removed from the map when the ‘Travel Routes’ map menu is active. This is because we need the space to display the information about a specific stage in the travel route itinerary, as described below. Also, if the legend was visible users may end up removing a map layer (e.g. ‘Temple’) that contains locations that appear in a selected route, which would cause confusion. When viewing travel routes, all of the categorisation layers are set to on for this reason. If you navigate from the ‘Travel Routes’ menu to another menu any selected travel routes are removed from the map, the legend is reinstated and all locations are added to the map.
If you press the ‘Explore’ button for a route this adds the route to the map (if you’ve not already added it by selecting the checkbox) and the map will reposition to display the first stage in the itinerary. This displays an info box in the top right of the map (in the place the legend would otherwise be). This box displays the route title and the information about the first stage in the itinerary. This can include any information about the stage in the itinerary (including duration, if of relevance). I’ve also included information about the number of stages in the route and the number of the current stage (e.g. ‘Stage 2 of 10’), as I figured this would be useful for people. There is also a ‘close’ icon in the top right that closes the info box and ‘Next’ and ‘Previous’ buttons at the bottom that can be used to traverse the travel route. Pressing on one of these buttons repositions the map to the next or previous stage in the itinerary and loads the information about this stage into the info box. Using these options you can step through the travel itinerary and view all of the information relating to it. The following screenshot shows the map with ‘Stage 2’ of the travel route loaded. Note that you can also still open the popup for any location on the route or manually scroll the map between locations.
This is only a first version of the feature and there’s still a lot to do. Displaying multiple routes at the same time is going to require further work, and I also want to add the selected route options to the page URL to enable citation / sharing / bookmarking of specific route selections and also possibly individual stage selections. We may also want to add information about relevant routes to the location popup (in a new tab) so users can tell at a glance which routes a location appears in. I guess adding links from this to the relevant stage in each route would also be useful.
Also, the current system is only set up to work with a purely linear route (e.g. a->b->c). If we are to include routes that branch off (e.g. a->b then b->c and also b->d then c->e and also d->e) then we’ll need to consider how to handle this as the traversal via ‘Next’ and ‘Previous’ buttons would not work.
Also this week I made a few further tweaks to the Burns Supper Map, including updating the site title to include ‘Worldwide’, fixing some typos in the data and adding in some new videos. I’ll be continuing with this next week.
This was another week of many projects, the first being the Burns Supper project. This week I finished implementing all of the data visualisations on the ‘facts and figures’ pop-up, with a mixture of bar, column and pie charts depending on how many data types there are. All charts are ordered by number of suppers, other than the pie charts. I also added in a statement above each visualisation about the number of suppers that supplied the data and a note when a supper may have more than one type of data. The screenshot below shows part of the popup with three visualisation types (mostly) visible:
I haven’t updated the narrative text to update the percentages to be based around the number of suppers that supplied the data rather than the total number of suppers and to include additional explanatory text, but I’ll do this next week.
For the HiMuJe Malabar project I have now incorporated the updated JSON file that contains cats and subcats into the map. I have also incorporated the GeoJSON polygons that project RA Renu has created so far. Where a place has a polygon this is used instead of a marker, and polygons feature a tooltip on hover-over in the same way as markers. I added the type and subtype to the tooltip as well. I also made a start on the categorisation of places based on cat and subcat, although this still needs some work. I added in a ‘legend’ box on the right of the map and have split the places into layers based on their cat and subcat. Each of these appears in the legend, and you can turn each layer on or off. You can also select or deselect all layers, which I find quite useful (e.g. deselect all then only add in the layers you’re interested in). There is also a count of the number of places in each layer as part of the layer label. Note that the cats and subcats only appear if there is at least one place that has latitude and longitude assigned to the subcat, which is why (for example) ‘Mosque’ is not currently listed. Below is a screenshot showing the categorisation options that are currently in place:
There’s still quite a lot to do and some issues that need to be addressed. At the moment only places that have a subcat appear. I still need to implement the top-level cats, so for example ‘Sri Lanka’ that has cat ‘Region’ and no subcat is not currently appearing on the map. I’m hoping to update the legend to give it a two-level hierarchy (as mentioned in my specification document) but I didn’t have time to implement this. Also, marker and layer colours are currently arbitrarily assigned. I think we will eventually have icons as well as colours, but in addition to this I will update the colours, probably to have different shades for each subcat in a cat. I also need to ensure that the polygons always appear behind the markers as this is not currently the case. E.g. if you turn the ‘Region: Kingdom’ layer off and then on again the layer is then added to the front, meaning it’s no longer possible to press on any markers located within the polygons. Also, there will be several other categorisation options in addition to cat and subcat (as discussed in the specification document) and these still need to be implemented.
Also this week, I returned to working on the Playbills project for the first time since before Easter. This included writing an executing a script to merge genre classifications, which reduced the number of genres from 188 to 69 and reassigned any plays that were assigned to a deleted genre. I also executed my scripts that process performers to extract individual names and titles from the free-text ‘full name’ column and assigned gender based on the title. My scripts process individual performers and also split multiple performers into individual records, resulting in the number of performers going up from 49498 to 52146. In these cases the newly extracted performers have also been assigned to the same role as the existing row. Of the 52146 performers we now have a Male or Female gender assigned to 51724, which I think is pretty good. I also updated the playbill and play pages so that the extracted names and genders now appear, and multiple performers for a role all now appear. I still need to deal with the ‘unprocessed’ performer spreadsheet, which is probably going to need some manual work as this contains the ‘edge cases’ that my processing scripts were unable to tackle. I hope be able to continue with this next week.
My fourth project of the week was the DOST Auld Laws project, for which I continued to work on the search facilities. This has involved setting up an Exist-db XML database and learning how to run full-text queries on the documents contained in it and how to generate KWIC (keyword in context) snippets for the results. There’s still a lot to do, but an initial version of the quick search is now operational, as the screenshot below demonstrates:
You can also use an asterisk wildcard to represent any number of characters, for example ‘wyn*g’ finds all terms beginning ‘wyn’ and ending in ‘g’. A question mark wildcard can also be used to represent a single character, e.g. ‘r?cht’ matches ‘richt’ and ‘rycht’. I haven’t added in pagination of results yet, so they currently all appear on one page. Each result features the document name and the page number where the result is found, plus the snippet with the term highlighted. Pressing on the snippet loads the page with the line where the term is found highlighted in both the image and the text.
It’s definitely something of a milestone to get the quick search working, but there is still a lot to be done. The highlighting of the term in the snippet is not currently working properly when the term includes characters in <expan>. The search finds the term, but the highlighting is only getting applied to the part of the term before the <expan> tag. Also, Exist-db uses the Lucene full-text engine and this does not support wildcards at the beginning of terms, so while a search for ‘grant*’ finds all terms beginning with ‘grant’, it’s not possible to find all words ending in ‘*ting’, for example. I’m not sure how big an issue this is, but if such functionality is required I’ll have to investigate an alternative. Also, phrase searching is not currently operational. I have a query that should be able to handle this, but I haven’t had time to add this in yet. It’s been quite tricky to implement as phase searches need to cross line and page boundaries whilst still returning the line and page IDs where the start of the phrase is found.
In addition, Boolean searches are not yet working (e.g. term1 AND term2, or term1 OR term2). However, we might want to consider how useful these would be and how they would work. Currently the search looks for occurrences of the term in entire documents and returns snippets showing the context of the term. If a Boolean search is using an entire document as its source then how useful would it be if (for example) term1 is found on page one and term2 is found on page 29? I’m just not sure how helpful this would be. I guess an OR search would be useful in finding variant spellings?
I also still need to add in results traversal to the document page, so you can navigate directly through results when looking at one specific result, and I need to add in a link back to the results page from the document page. There’s also still a lot of work to do on the user interface, which is still not finalised and I’ll work on this (e.g. colours, fonts, layout) once I’ve finished with the search facilities.
My fifth and final project of the week was the redevelopment of Mapping Metaphor. I didn’t have much time left to spend on this, but I did manage to tidy up the loose ends from last week. Where a combined metaphor has both OE data and E data and the E data does not begin in the OE period the timeline in the visualisation card popup now includes OE selected and highlights the period when the E data begins. This second highlight currently uses the green used for the visualisation background but can be changed. We should also maybe include some explanatory text so users understand why two periods are selected. The following screenshot shows the timeline with two periods highlighted:
I also updated the card popups so that both E and OE IDs appear in the header of the card (where applicable) now, as you can also see in the above screenshot. Also, where a metaphor connection exists in E but not OE (or vice-versa) counts of lexemes in each joining category in the other period are now displayed.
Also this week I had an email conversation with Craig Lamont about an online resource for his new research project and arranged for a subdomain to be set up for it. I also updated the DSL survey so that it only appears on mobile devices and responded to a request from Kirsteen McCue regarding stats for song downloads on the Editing Burns and Burns Choral websites.
This was a pretty busy week that had me dividing my time between five main projects. I spent most of Monday working on the Burns Suppers project, beginning development of the ‘facts and figures’ popup. I added a button in the ‘Home’ menu labelled ‘Facts & figures’, that features a pie chart icon. Pressing on this opens the facts and figures popup part of which you can see in the screenshot below:
The popup features the narrative summary section, with the figures in this section being dynamically generated. It took quite some time to write the code to generate them, but at least there will be no further work to do when we import more data. For now all percentages are the percentage of the total number of suppers rather than a percentage of the number of suppers that supplied the data, so for example, the ‘annual supper’ figure is 39% rather than displaying 83% (based on there being 420 suppers that actually have frequency data). This is because I realised that without a lot of additional explanatory text users will likely think our figures are wrong. If people only see the total number of suppers (886) and the number that are annual (347) then displaying 83% will be misleading. We’d have to include lots of additional data such as “of the 420 suppers that included frequency data, 347 (83%) were annual events”. Depending on feedback I may implement this later.
I’ve only implemented one graph in the pop-up so far, which is the graph of countries. This is a long list, but I think it works ok. I also included an option to switch from a graph ordered by number of suppers to an alphabetical version, and this is all operational. I’ll continue to add further graphs next week.
I spent most of Tuesday working on the interactive map for the HiMuJe Malaber project. I’ve created an initial version of the map now, which uses my map menu interface that I’ve used on several other resources, and below is a screenshot:
By default the map uses the ‘Watercolour’ base map, as we used for a static map on the main site (https://himuje-malabar.glasgow.ac.uk/about/summary/). You can also switch to a satellite map using the ‘Change the base map’ buttons. Currently markers as just displayed as red dots, and if you hover over them the ‘Preferred Name’ from the JSON file is displayed as a tooltip (although I’ve noticed that some places don’t seem to have a ‘Preferred Name’). There are no popups or filters or anything like that yet. Also note that any places in the JSON file that don’t currently have latitude and longitude values are not displayed as there is nowhere to ‘pin’ them.
You can zoom and pan the map as with Google maps, and the icon in the bottom right makes the map full screen. The map also works on mobile devices (hiding the left-hand menu using the ‘<’ button above the menu helps when using a mobile device). Other than the base map selection options, nothing works in the left-hand menu yet. ‘Places’ will eventually include the place filters. ‘Travel Routes’ will list travel routes involving people and organisations once this data is available. There’s still a lot to do, and I’ll hopefully begin work on the place popups and the categorisation options next week.
I spent most of Wednesday working on the Place-names of Armagh project. I’ve updated my ‘Irish form / translation’ script to remove ‘More details: Unverified’ and to also attempt to split the translation up based on apostrophes. This has been a bit of a nightmare, firstly because apostrophes are not only used to denote the translation but appear within the text, and secondly because what looks like an apostrophe can actually be many different characters, including straight, curly opening and closing apostrophes and several HTML codes that are rendered as apostrophes but are stored as codes. This has all made splitting the text up rather challenging. However, I’ve got something that mostly works.
The script now deals with things like Ir. <em>Coill Uí Fhloinn ‘O’Flynn’s </em>wood’ and outputs “O’Flynn’s wood” as the translation. Where there are multiple sections some unnecessary apostrophes appear, for example: “Ir. <em>Tír Garbh </em>’rough land or district’ or perhaps Ir. <em>Baile Uí Aodha </em>'<em>O’Hugh’s</em> homestead or townland’” results in: “rough land or district’ or perhaps Ir. Baile Uí Aodha ‘O’Hugh’s homestead or townland” This will probably need some manual fixing. The script also now processes rows that don’t have ‘Ir.’ and italics, for example “Críonchoill ‘withered decayed wood’” now has Irish form “<em>Críonchoill </em>” and translation “withered decayed wood”.
I sent an Excel version of the script output to the project team as this will likely require some (but hopefully not too much) manual intervention to fully sort out. For example, the text “Eng./Sc. ‘hill of the military camp’ or perhaps Ir. <em>Mullach na Críne </em>’hilltop of decay’ or Ir. <em>Mullach na Craoibhe </em>’hilltop of branch, tree’” will need some work as the translation omits the ‘Eng./Sc.’ Text as it’s before the first apostrophe.
The other thing I’ve managed to do today is to set up an initial version of the public map interface. This currently takes quite a long time to load as it’s processing a lot of data. As with the other sites, I’ll create a cached version of the data once we’re ready to launch, which will be much faster to load. The reason it’s not in place now is that a new version of the cache will need to be generated any time you want subsequent changes made to the CMS to appear on the map, and we’re still very much working on the data.
For now, three base maps are available (satellite, satellite with labels and relief). We can add more in later. There are also no parish boundaries or townland / barony boundaries as I don’t have this data yet. I also still need to work on the colours of the markers as there are no colours for a lot of the classifications so they’re defaulting to purple. There’s also nothing in the elements glossary as we don’t have this data yet. But the search, browse and categorisation options all work. For example, here are the placenames beginning with ‘A’ categorised by altitude:
There are some issues with the data. For some reason there are three place-names miles away from Armagh, around Ballybofey, and there are a couple of place-names appearing in the Irish Sea. We also have an issue of the same coordinates being used for multiple places, thus resulting in markers appearing on top of markers. There’s also an issue with the accuracy of markers too. For example, the marker for ‘Lowry’s Lough’ is found about 500m south of the actual body of water. But the good thing about having this map available (despite the speed issues for now) is that it will help when working on the data.
I spent all of Thursday and most of Friday morning learning how to use the Exist-DB XML database that I’m hoping to use for the DOST Auld Laws project and developing the query that will eventually power the quick search, and form the basis for the advanced search. I installed Exist on my laptop and managed to set up a collection for the project’s XML files, which I then uploaded into the system. I followed the documentation available on the Exist website and was able to create a full-text index for the collection, and I followed a useful tutorial here: https://dh.obdurodon.org/php-xquery.xhtml about how to query Exist using the REST interface. Setting up and querying the texts in Exist was all new to me and there was a lot to try and take in, with many configuration options that were not all that easy to follow in the Exist documentation. I ended up using ChatGPT quite a lot to help me figure out how everything should work and why some of my initial tests were not producing any results. This proved to be extremely useful and really helped increase my understanding of Xquery. I was able to get a search working that queries the full text and returns contextual snippets for each result, together with the IDs of the line, page and document, which is everything I’ll need for the quick search. On Friday I asked Luca to help set up the necessary collection on the server and hopefully I’ll be able to get an initial version of the quick search working on the website next week.
This left me with a few hours on Friday afternoon to devote to the redevelopment of the Mapping Metaphor resource, for which I’m creating a unified view of the data, joining both the English and Old English datasets together. I used this time to implement the visualisation card view when viewing connections between specific categories. The combined card view shows a bidirectional arrow if the OE and E directions differ. It also defaults to the E strength. The counts of lexemes in each category include both full and OE counts and the examples of metaphor feature both OE and E examples, with OE coming first. If the metaphor exists in the OE data then the ‘start era’ now defaults to OE (but as of yet I’ve not added in a further highlight in the timeline to show when the first non-OE occurrence was documented). Below is an example of the combined card view:
In addition to the above I also looked into an issue raised by Ann Fergusson for the Dictionaries of the Scots Language regarding accented characters, search results and entry slugs. This took some time to investigate but I think my response proved useful. I also had email discussions with Deven Parker about her Playbills project, which she now has time to look into again. I’ll probably be working on this again next week.
This was a four-day week as Monday was a bank holiday. I spent most of Tuesday and Wednesday continuing to work on the DOST Auld Laws project, working with the XML files. The XML files generated by Transkribus and exported by the tool as TEI XML contained many elements that were not valid TEI elements, such as the custom <Aitken> element that had been applied to notes added by A J Aitken. My first task of the week was to write and apply transformations to the XML files to convert them into fully valid TEI. I achieved this using XSLT, which is a language I find very unintuitive and frustrating to work with, no doubt exacerbated by the fact that I don’t work with it very often. I struggled to get any transformations to run initially, and ended up turning to AI to figure out why my scripts were not working. In this instance AI proved to be extremely useful as it identified what the problems were (e.g. I hadn’t declared the correct namespace or used it when writing the rules) and really helped me to understand how everything fitted together. I still wrote the bulk of the code myself, but AI was very helpful in identifying errors or issues. By the end of Tuesday I had generated (and checked) a collection of TEI files that successfully validated in Oxygen, which was a good milestone to reach.
On Wednesday I then worked on the front-end for the project, figuring out how to transform the valid TEI XML into HTML for display in the ‘text and image’ and ‘text only’ views of document pages. As this was once more using XSLT I enlisted the help of AI to figure out specific issues that I was unfamiliar with. The biggest of these was how to pick out and process one single page from a document’s XML file based on the <pb/> element. I had no idea how to achieve this, and despite this being a fairly fundamental issue when processing TEI documents I didn’t manage to find any useful information online. However, AI came up with a solution (and equally importantly an explanation) in a few seconds and I was then able to incorporate this into my code, transforming the contents of one page, whose ID was passed to the XSLT script as a parameter, to HTML with a variety of styles applied to the various elements.
I also updated the ‘click on a line in the image’ feature, and it’s now possible to deselect the line if it’s already selected – previously once you’d highlighted a line you couldn’t get rid of the highlighting, only move it to a different line but now if you press on the highlight it’s removed. The lines are also now connected to the text view – pressing on a line in the image also highlights the corresponding line in the text. You can also press on a line in the text to highlight it and the corresponding line in the image.
I also updated the height of the text pane so that it matches the height of the image pane and if the text is longer the pane scrolls. This ensures that if the text is very long it’s still possible to see the image, rather than having the entire page scrolling, which may result in the image not being visible when you’re at the end of the text. It is how we did things in Books and Borrowing, but having a scrollbar in a section of the page in addition to the browser’s scrollbar can be annoying for some people so I might revert to the previous layout depending on feedback. Here’s a screenshot showing the transformed text, a highlighted line and some of the formatting that’s been added:
Also this week I continued to work on the Place-names of Armagh project. I wrote and executed a script that generated Irish grid references and ITM values for all places based on their latitude and longitude, a task that I completed using the ‘GridRefUtils’ scripts as detailed here: https://www.howtocreate.co.uk/php/gridrefapi.php. I’ve used these scripts before on previous place-name projects and they’ve been hugely useful. I then ran a further script to generated altitude for the place-names by connecting to the Google Maps API, so we now have complete geospatial data for all of the places (other than the 386 that didn’t include Easting and Northing data in the original spreadsheet). I also added in a new parish and barony in a different county that one place-name requires, and had discussions with the team about the splitting of analysis data into Irish forms and translations. Removing the Irish forms from the translation field is going to be rather tricky to automate as the Irish forms often form part of the translation. It’s looking like I’ll need to automate the transformation of some of these, with the rest then requiring manual intervention.
I also continued working on the redevelopment of the Mapping Metaphor resource to create a combined English and Old English map, something I began last week. This week I managed to get the combined view of the drilldown of the visualisation working. The counts represented by the yellow circles also use the combined data, and these are also displayed in the pop-up card, for example, if you select 1K02 Creation and press on the yellow line or circle for 1B Life. In the combined map there are connections to 6 categories in Life whereas there are 5 in the E map and 2 in the OE map (one of the OE ones is also present in the E map, which is why the combined total is 6). The combined pop-up card also now lists the number of OE lexemes in addition to the E lexemes, for example “1K02 Creation 1088 lexemes / 127 OE lexemes”.
I still need to implement the combined visualisation view of the search results, and also the card pop-up between individual categories, which is going to take quite some reworking (possible direction changes, combined example lexemes, timeline updates etc). I’ll hopefully find some time to continue with this next week.
Also this week I processed and added some more images to a few Burns Suppers, had a chat with Garrick Allen about a research project he’s wanting me to be involved with later in the year, and contacted Lindsay Balfour about a proposal she’s writing that will have some technical requirements. I also made a couple of tweaks to the Thesaurus of Old English website after Fraser go in touch with some suggestions, made some tweaks to the survey popup on the Dictionaries of the Scots Language website, gave some mapping advice to Renu of the HiMuJe Malabar project, and created an alternative version of one of the Anglo-Norman Dictionary’s textbase documents that strips out all Latin text.
I had several meetings this week, the first being to discuss the development of the map for Ophira Gamliel’s HiMuJe Malabar project. In the run-up to the meeting it became clear that no-one had read the specification document I’d written regarding the development of the map, which I’d sent out several weeks ago, but thankfully after I raised this it was distributed before the meeting and we had a very useful discussion about the map and how to prepare the data for the map. It feels like some real progress has been made in our joint understanding of the data and what needs to be presented, and following the meeting I sent out a proposed structure for the JSON data that will be generated from the digital edition files and a proposal for how we would document travel routes on the map. I also met with project RA Renu a couple of days later to discuss geoJSON data and the creation of polygons for certain locations in the data.
My second meeting was with the Place-names of Armagh people, and this was also very useful. We discussed some additional explanatory text found in the original data that I hadn’t spotted before, the splitting up of the existing ‘analysis’ data into Irish forms and translations and the creation of the front-end. Now all I need to do is actually find the time to work on all of this. My third meeting was with the Burns Supper team, at which we went over the ‘facts and figures’ popup that I will be developing. We also discussed the launch of the resource and a demo that was given on Monday and was very well received.
I spent a fair amount of time this week working on the DOST Auld Laws project, beginning with an analysis of the XML files in order to work out what non-standard elements generated by Transkribus needed to be converted into which TEI elements. It took some time to go through everything but it was very useful to get it all documented and subsequent email discussions with Joanna and Pia were really helpful.
I then ran a script to update the XML files to change the filenames, add in IDs and replace the referenced image filenames (both in the XML files and the image files themselves) with a more rational naming structure as I’d previously discussed with the team. For example, the file with the title ‘115 – Peebles B. Rec.’ was given the ID ‘peebles-b-rec’ and image files and references were renamed from (for example) ‘0001_115 001.jpg’ to ‘peebles-b-rec_001.jpg’. With this in place I then arranged with Luca for the images to be uploaded to the IIIF server, after which I could begin working on the front-end on the server, as opposed to on my laptop as I’d been doing previously.
As part of this work I also decided to migrate the document data I’d extracted from the XML from JSON to a MySQL database. The reason for this was that some of the document JSON files were rather large (a couple more than 6MB) and it seemed rather inefficient to load all of this data in just to work out counts of the number of pages and such things. With all of the data in a relational database it’s much easier to query and return just the data that is needed.
During the week I completed work on an initial version of the image parts of the site. The user interface is by no means complete and is fairly rudimentary – we will decide thinks like fonts, colour schemes, illustrative images and ancillary content later on. For now there is just placeholder text on all but the ‘Documents’ pages and the search option doesn’t work yet. However, it is possible to browse the documents and all facsimile images contained in each. Pressing on the ‘Documents’ menu item brings up a sub-menu listing the documents, plus an ‘overview’ page which is currently empty. If you select a document you’re presented with an overview page. This can feature a description of the document (there’s just placeholder text for now) a link to open the document at the first page and a randomly selected image displayed to the right of the description for illustrative purposes.
Beneath this is a section through which you can see a count of the number of pages and access a list of thumbnails of every page. For some of the longer documents it can take some time to load in all of these thumbnails. We can maybe have this section hidden by default, or possibly paginated. Pressing on the ‘open document at first page’, the random image or any of the image thumbnails opens the page in question, a screenshot of which is shown below:
I’ve borrowed much of the layout of this page from the work I did on the Books and Borrowing project. The page features a navigation bar at the top and bottom, through which you can navigate to the next or previous pages, or use the ‘jump to’ feature to select a specific page. There are three views of the page: an image and text view which displays both side by side and then text or image only views. When you select a view it is remembered as you use the navigation options.
As of yet there is no content in the text view (processing the XML will be my next task) so for now it’s all about the images. You can zoom and pan the image or open it full screen. You can also press on a line to highlight it. Eventually this will allow highlighting of the text from the image and vice-versa, but for now all that happens is the section of the image is highlighted. As mentioned in an earlier post, I adapted an existing ‘simplify coordinates’ script (https://github.com/dariok/page2tei/blob/master/simplify-coordinates.xsl) to generate simple rectangular sections for each Transkribus polygon, but we may need to have more complex shapes as in some pages (particularly the handwritten ones) the boxes are not very accurate. I’ve also spotted an issue on touchscreens (on my Android phone using Firefox, at least) whereby the boxes for the lines stop the ‘pinch to zoom’ feature working, meaning zoom will only work using the icons in the top left. I’ll need to investigate this further.
My next task will be to work with the XML, firstly replacing the Transkribus tags with valid TEI ones and then generating the text view for the website, which will link each line to the image. Once this is in place I’ll begin to think about the search facilities.
On Friday I spent a bit more time working on Sara Ponz-Sanz’s AHRC application and spent the rest of the day beginning work on the redevelopment of the Mapping Metaphor website. It took a while to refamiliarize myself with the code and to work out how things function, but once I made some progress with this I decided to start work on a new ‘combined’ version of the map. This will display the combined E and OE data but will not initially allow you to switch the display to just E or OE. There is a LOT of work that needs to be done to get this working fully, and I only made a start on things this week. For now only the visualisation works, and only the top level view of this works.
Working out the totals for the visualisation is not as simple as just getting the E and OE data and adding them together, as this would duplicate data for metaphors that appear in both sets. Instead, the new version of the visualisation grabs all of the E data and then only adds OE data for connections that are not already present. So for example, in the new visualisation pressing on ‘1K’ displays a connection to ‘3E’, which is not present in the E map but is present in the OE map.
In the combined map with strength set to ‘Strong’ and with ‘1K’ selected, pressing on the yellow line or circle for ‘1B’ the card shows 18 connections. If you do the same for the E map you’ll also see 18 connections, while the OE map shows 3 connections. The reason the combined map shows 18 rather than 21 connections is because the three OE connections already exist in the E data. Setting the strength to ‘Both’ in the combined map shows 38 connections between these categories, whereas there are 35 in the E map and 10 in the OE map, and this is because there are three additional weak connections between categories in ‘1K’ and ‘1B’ in OE that are not found in E. There is going to be a lot of figuring out how exactly to amalgamate the data across all views and all levels, which is going to be a pretty large undertaking. But I feel that I’ve begun to figure it out, at least.
I spent much of Monday and Tuesday this week working on the DOST Auld Laws project, which I’d begun work on last week. I set up a test instance of the Cantaloupe IIIF image server on my laptop and began experimenting with the OpenSeadragon IIIF image viewer. It was pretty straightforward to set this up and to create a test page where the viewer connected to an image stored in the IIIF server and enables the user to zoom and pan around the image. The trickier issue was setting up the infrastructure so as to allow lines of text in the image to be clicked on and highlighted, and for IDs associated with each of these line regions to be accessible by JavaScript code (to eventually enable the corresponding line in the text view to also be highlighted).
Luca is currently working on a digital edition that allows regions of an image to be highlighted when buttons are pressed on and he very helpfully gave me access to the test site he’s working on so I could see how things work. I had been expecting that the region data would need to be somehow stored in the IIIF manifests for each image and that the IIIF server would be a lot of the processing to enable the display of and interaction with regions in the image, but both Luca’s site and another one I was referencing stored the region data as coordinates pulled into the front-end from a separate source, such as a JSON file. I was very happy to go with this approach, as firstly it meant I didn’t need to spend ages investigating IIIF manifests and how to update them and secondly it means the region data is not tied into a specific server technology, meaning it will be easier to repurpose it in future if that technology changes.
The line data exported from Transkribus consisted of detailed coordinates such as:
1309,2018 1469,2068 1716,2037 1876,2123 2092,2037 2234,2111 2389,2062 2555,2123 2734,2062 2796,2111 2975,2068 3555,2130 3629,2080 3808,2111 4062,2037 4457,2043 4524,2080 4599,2018 5055,2099 5142,2049 5321,2099 5438,2043 5586,2086 5845,2043 5901,2080 6271,2062 6271,1901 6129,1944 5802,1839 5710,1920 5284,1889 5234,1938 5166,1883 4771,1907 4481,1796 4333,1907 4222,1870 4043,1957 3950,1907 3808,1920 3753,1864 3574,1913 3506,1852 3426,1907 3006,1839 2895,1895 2438,1870 2265,1913 2055,1815 1975,1852 1759,1839 1660,1920 1506,1920 1309,184
But for the most part these refer to simple rectangles and with our data this level of detail isn’t really needed. Thankfully there is a script available to simplify the coordinates (https://github.com/dariok/page2tei/blob/master/simplify-coordinates.xsl) and I was able to adapt this to export the pixel coordinates of each corner of the line rectangle, such as (for the above polygon):
1309,1796 6271, 1796 6271, 2130 1309, 2130
OpenSeadragon doesn’t work directly with pixel dimensions, but has a function to convert these into its required format (see https://openseadragon.github.io/docs/OpenSeadragon.Viewport.html#imageToViewportRectangle) and I could therefore plug in the rectangle data and make the viewer display a region overlay on the image. I made a test that displayed all lines at once, as you can see below:
There is quite a lot of overlap between the lines here, but this isn’t a big issue. These areas will not be displayed all at once – only one will be displayed when the user clicks on the image. If the highlighted line isn’t the one the user wants due to there being an overlap it’s very easy for the user to just click again in a slightly different location to highlight the required line.
Having managed to get the regions displayed on the image, the next step was to process click events. Despite the overlays appearing as HTML elements, each sharing the same class, it was not possible to simply use jQuery to process clicks on this class. Instead, OpenSeadragon’s clickHandler needed to be used. With this in place it was then possible to process the click, to add a new highlighting class and to grab the ID of the element, which will eventually be used to find and highlight the line in the text view.
With all of this in place I then wrote a script to export line data (IDs, coordinates) from the TEI XML and store all of this in JSON files (one per document) that can then be loaded in whenever a page is displayed. I then moved on to setting up an initial version of the website for the digital edition, setting up an initial interface, menus, pages and such things. It’s all still running on my laptop for now, but I’ve made really good progress this week and will continue with it next week.
For the remainder of the week I worked on a variety of other projects. I made some further changes to the user survey popup for the Dictionaries of the Scots Language and I added some text to Sara Pons-Sanz’s AHRC proposal document. I set up the mapping metaphor site at the new URL we will be using for the site, meaning everything is ready for the major updates that I’ll hopefully be working on in the coming weeks.
I also wrote a document outlining the ‘facts and figures’ popup for the Burns Supper map. This took some time to research, but gives a handy overview of what the popup will contain in terms of summary data and visualisations.
Finally, I returned to the Place-names of Armagh project. The existing data for this project specified location data as Eastings and Northings and I need this as latitude and longitude. When I’d previously imported the data into QGIS the markers were all in the wrong place and I didn’t know why. Thankfully Frances Kane, who is working on the project suggested that this might be because I hadn’t set the coordinate reference system to Irish Grid, and this proved to be the answer.
After making the update the data all loaded at the correct locations and with this in place I was then able to follow this answer: https://gis.stackexchange.com/a/64700 to generate latitude and longitude fields for the data and then export this from QGIS as a CSV. After that I imported the data into the CMS and all records now have latitude and longitude. There’s still more I need to do, though. At the moment the map in the ‘edit place’ page in the CMS is generated based on the supplied grid reference or ITM coordinates, which the records still don’t have, so no map displays. I think I have a script that generates this data from latitude and longitude and I’ll look into this next week. I’ll also run the data through my script that grabs the altitude for places from Google Maps by sending latitude and longitude to the service. The front-end map uses latitude and longitude directly (no messing about with grid references or ITM) so I should be all set to start deploying the front-end map soon.
This was my first week back after my Easter holidays and I returned to discover that one of my oldest sites had been flagged with security issues. I created the Emblems sites (https://emblems.arts.gla.ac.uk/french/ and https://emblems.arts.gla.ac.uk/alciato/) more than twenty years ago and the way they connected to their database sources was obsolete and could be exploited. This wasn’t a huge issue as no personal or private data was stored in the database anyway, but it still wasn’t ideal and needed sorting. It would be good to completely overhaul these resources as they are still actively used as research tools but their interfaces are somewhat creaky now. I don’t have time to do this at the moment, unfortunately, so instead I focussed on replacing all database calls with a more modern solution and ensuring all user submitted content is fully sanitised. This in itself took two full and pretty intense days to sort out. I tackled the French site first and when that was sorted I’d hope I’d be able to replicate these updates on the Alciato site, which I could do to a certain extent. However, the latter site has several additions to its data and functionality so it was not possible to simply replace the entire code, and each script needed to be checked and manually updated. Thankfully it’s all done now and both sites should be useable and safe for the foreseeable future, even if they still look a bit ancient.
I then spent a couple of days continuing to work on the Burns Supper map, working through my list of ‘to do’ items for the project. I started off with processing a spreadsheet to update supper venue titles and delete certain suppers that had been flagged as no longer needed (usually duplicates). I then wrote a further script to reassign images from one supper to another, which involved renaming images and updating database records to process the reassignment. I then updated the advanced filters in the front-end to enable suppers with images and videos to be filtered independently. Previously there was one filter for displaying suppers that featured (or didn’t feature) media files but I reckoned it would be useful for people to find and display just those suppers that feature videos. Now there are options for selecting suppers with images, suppers with videos or suppers with no media, and it’s also possible to combine these (e.g. showing all suppers that have either images or videos).
I also removed the ‘Download CSV’ option from the map as it was decided that we would not offer such a facility, and I added in a copyright statement that appears underneath the media carousel. I updated the ‘Share’ options to add in LinkedIn and BlueSky and remove X, and added in the welcome message on the ‘Home’ tab.
With these updates in place I then focussed on updating the user interface. At our last meeting we’d agreed that we’d mostly use elements from the first test interface I created (as shown in my post from a couple of weeks ago) but that we’d use the font I’d selected for the second test interface. It took several hours of implementing and testing to get the new interface working, but it’s now in place, as you can see in the following screenshot:
I then created the image slideshow, which appears as a button on the ‘Home’ menu. The slideshow picks a random selection of up to 20 images from the suppers that are currently displayed on the map (i.e. it takes any filters into consideration) and displays them in a slideshow that automatically scrolls (but pauses when your cursor is over the image). An image caption consisting of the venue and country of the supper plus a link through to the full record for the supper also appears, as the following screenshot demonstrates:
Each time you load the slideshow a different selection of images is pulled in, and I think it’s quite a nice feature to use. Note that there are often black bars visible as the images are all different dimensions and have to fit in the same overall space. I think it looks ok, though.
Other than adding help and about text and attribution text the only thing left to tackle now is the facts and figures popup, which I’m hoping to look into next week.
Also this week I spent a little bit of time working for the Dictionaries of the Scots language, updating the user survey popup based on requests for updates I’d been sent. I also set up a new subdomain for the Mapping Metaphor project, which I’ll use for the redevelopment of the site, which I’m hoping to start work on next week.
I spent more of Friday beginning to work on Joanna Kopacyk-McPherson’s DOST Auld Laws project. This project has around 500 pages of handwritten and printed text that has been transcribed using the Transkribus tool, and I need to create a front-end to allow this to be browsed, searched and viewed. My first task was to export the Transkribus text as TEI XML, something that I thankfully managed to achieve. I then spent some time familiarising myself with the texts and thinking about what technical solutions should be used to create the front-end. We are going to need to display images and text and highlight sections of text in the images (and vice-versa), which is not something I’ve ever tackled before, so it’s going to take some time to get my head around.
I spent a lot of this week continuing to work on the Playbills project. Last week I began working on a very rough first draft of a front-end through which the playbills data can be browse, and this week I completed this but creating the API call and corresponding page that displays the full data about a specific playbill. As of yet I’ve not included the actual playbill images, so for now there’s just a placeholder section where the image will fit. There is still a huge amount to do for the project and this is just a first draft of the ‘browse and view’ functionality only with practically no time spent on the user interface, but here’s an example of the work in progress:
We still need to do a lot of work with the data, such as amalgamating records for things like genres and venues that have similar but not identical text, plus making canonical records for things like plays. Until this work is completed I can’t really do much more with the front-end, as we need this data before I can generate the Solr search index, and there’s the possibility that all of the existing data will need to be wiped and regenerated as proofreading progresses.
Towards the end of the week I began to download all of the playbill images from OneDrive, a process that took a long time as the downloads kept quitting with an error whenever the ZIP file became larger than around 10Gb – instead I had to download the images in smaller batches. I also realised that a lot of the images are in the HEIC format, which is an Apple-specific format that can’t easily be opened in Windows and is unsuitable for use in websites. At the moment I’m unable to even view the images and it’s going to take some time to figure out firstly how to open them and secondly how to batch convert them to a more widely supported format.
On Tuesday this week I met with Wendy Anderson and Carole Hough to discuss updates to the Mapping Metaphor resource. Currently the resource is divided into separate sections for Old English and the rest of English and we’re hoping to amalgamate these (as well as continuing to give options to focus on one or the other). As well as attending the meeting I spent most of the day preparing a document that discusses the work that will need to be carried out and some of the issues that will need to be addressed. We discussed a lot of these issues at the meeting and the document will prove useful when I come to begin the work, which I’m hoping to do after Easter.
I also spent some time this week working on a few items for the Anglo-Norman Dictionary. A few weeks ago I updated the entry publication workflow so that entries that are cross references are checked to ensure that their references actually refer to entries that exist in the system and the editor Geet had spotted a couple of instances where cross references were continuing to be flagged even though they were main entries rather than cross references. I checked the data and this was happening because both xref and main entries with the slugs are active in the database. I think this may have been caused by the issue of uploaded entries not having the existing ID in the filename, something we discussed a few weeks ago, and I manually deactivated the xref entries, which got rid of the warnings.
I also ran a query to identify duplicate active entries in the system and there were 23 of them. 22 of them were cross references to single entries so it was pretty easy to deactivate the duplicates. One of them had one active version that is a cross reference to four entries and another that only has a single cross reference to a single, further entry, and Geert created a new amalgamated version after I flagged it. The remaining duplicate had one live entry that’s a main and another live entry that’s an xref, but the xref version was malformed so I deleted it.
I also spent a bit of time creating a new cross-reference checker, this time rather than focussing on purely xref entries the new checker instead finds all xref elements in main entries and checks whether they actually link through to an active entry in the system. This script takes a while to run as it needs to check the XML of several tens of thousands of entries, but it’s going to be a useful tool. Currently it has identifies 880 broken xrefs in main entries and Geert can use the output to decide what the do about these. It is something that will more than likely require manual editing rather than any batch processing, unfortunately.
The rest of my available time this week was spent on the Burns Supper map, which is coming along pretty nicely. I’ve updated the records to include information about the source of the data. If it’s a survey response the text is ‘Information about this supper was submitted as a survey response.’. If it’s a PD supper the text is ‘Information about this supper was gathered from publicly available sources.’ I also added a new filter to the advanced filters that allows you to select the source. I figured this would be sufficient, rather than adding another option to the standard filter options. I also updated the map to move the side panel to the right, both to differentiate the site a little from the place-names maps and also because when zoomed out the panel only now obscures parts of Russia (and Japan, unfortunately) rather than North America.
I then moved onto a larger task: getting the map to remember filters and open records so people can bookmark / share / cite specific views of the data. This has been a pretty major update that required a lot of changes to be made throughout the code, but it’s going to be a very useful addition.
Whilst working on this I realised that the advanced filter popup wasn’t working properly in Chrome-based browsers – the popup was unexpectedly closing in Chrome when filter options lower down the form were clicked on. It would appear that there is some kind of conflict with the UI framework I was using for the filter buttons and the map popup, because links that didn’t use this framework worked fine. I’ve therefore replaced the UI framework checkbox buttons with my own and the form now works perfectly in Chrome. The buttons mostly look the same, but the ‘ticks’ are now the browser’s default ticks and can’t be styled. This isn’t a big issue, though.
I also fixed an issue whereby opening records from the table view wasn’t adding the record ID to the URL, and I also implemented the short URLs and the ‘share’ options. If you press on the ‘Share’ button in the side menu this will display a popup containing some buttons linking to sharing platforms, and the citation styles are also displayed and feature a description of the contents of the map based on filters applied. I also updated the record popup to add the ‘details’ to a separate tab labelled ‘Further Details’, with the other textual data in a tab labelled ‘General Information’, and added in a ‘Share this record’ tab, which works in the same way as the map share option. There is still a lot to do for the resource, but I feel like I’m making good progress.
My time this week was divided between several different projects and meetings. I spent quite a bit of time working with Transkribus, ahead of Friday’s Transkribus workshop at which I was speaking about my text extraction experiments with the Edinburgh Gazetteer (https://edinburghgazetteer.glasgow.ac.uk/). Back when I worked on the project with Rhona Brown (almost ten years ago now) we’d attempted to extract the text from the images using OCR but our experiments had been pretty hopeless. I attended a Transkribus event in Edinburgh earlier this year and had done a little bit of work with the Gazetteer in Transkribus then, but hadn’t progressed very far. This week I made considerably more progress, experimenting with a three-column subsection of one page, as you can see below:
Transkribus can identify columns of text by analysing what it calls ‘Fields’ so this is what I asked it to do first, using the ‘Baroness of Blocks’ model. This is something that’s only available with a subscription, but I was able to make use of a free trial. Unfortunately the process was not very successful. It did correctly identify the columns, but split the image up into sections within each column, with some parts of the image entirely missing from the classification (e.g. the top of column 2) in the image below:
I manually corrected this using the interface, as the following image shows:
However, I’m uncertain how I would be able to train the system to automatically and correctly identify such columns for other images – this would require further investigation. The next step was to identify lines within each column, which is accomplished using a ‘Layout’ model. I chose the default ‘Mixed line orientation’ model which was pretty successful in identifying all of the lines in each column. It wasn’t perfect but it was good enough for test purposes, as you can see below:
The third step was to extract the text. For test purposes I wanted to see how the model would work without any training, and I chose the ‘Text Titan I ter’ model. This took several minutes to process, but the results were very encouraging, as the following image demonstrates:
There were some issues, however, such as the large drop-characters at the beginning of sections being omitted, and some words that are legible to humans being incorrect, such as ‘acie’ instead of ‘acre’. The ends of lines in the first column were also missing, so line identification would need to be tweaked. Despite these issues the text is broadly understandable and complete. The poor print quality and the long ‘S’ character were processed successfully and some sections that were very difficult for a human to decipher were processed successfully by the tool. There are still issues to be ironed out with regards to successfully identifying columns of text, and these would need to be addressed before any batch processing of the entire Gazetteer, but it’s looking very promising.
I spent most of Tuesday this week attending the presentations for the new Grade 8 and Grade 7 roles for the post-graduate course in Digital Humanities that is being set up in Information Studies. It was really interesting to hear the presentations and to learn more about what the candidates would bring to the roles. There were some really excellent candidates and it was very useful to hear from them.
Also this week I spent a little time working on the Place-names of Ayrshire project ahead of next week’s launch and engaged in a continuing email conversation about how the data for the interactive map will be gathered and stored for Ophira Gamliel’s Malabar project. We’ve now managed to reach an agreement on how to proceed with this, which is a relief. I also spent some time continuing to make updates to the Place-names of Armagh resource, creating a nice interface for the project website using suggested public domain images and fonts, and adding parishes and baronies to the content management system. The new project website is not yet live, but here’s how the new design currently looks:
I’ll meet with the project team next week to give a run-through of the CMS and working with WordPress, after which they should be in a position to start adding data to the resource. I also managed to spent a little more time working with the data for the Fife place-names project, continuing to rationalise the place-name elements and their connections, but there is still more to do for this.
On Thursday this week I met with Deven Parker to discuss the requirements for an online resource for her Playbills project. We discussed the kinds of search and browse facilities she would like to include and other features such as visualisations and data summaries. Next week I’m going to write up the requirements and share the document with her. Also on Thursday I met with Wendy Anderson and Carole Hough to discuss some potential future updates to the Mapping Metaphor resource. I can’t really go into any details here, but we had a good meeting and will consider the options before meeting again in the New Year to see how this might be taken forward.