This week I continued with the interactive map for the HiMuJe Malabar project and made some good progress with its development. All subtypes now have an associated icon that is used on the map when the ‘type/subtype’ categorisation is selected, and I also reworked the colours of these, as you can see in the following screenshot:
After last week’s frustrations with the map legend, I reworked it to divide it a little better into sections by type, which you can also see in the above screenshot. When a location popup is opened the corresponding subtype icon also now appears in the popup header. I also added an option to the ‘Home’ menu to make all marker labels permanently visible on the map (until you turn them off again). With all locations on this does clutter up the map, but I think it’s a useful option for people who are looking for a place but aren’t entirely sure where it is. With labels on you can see the names of all places without having to hover over each marker in turn. This works better when the map is filtered in the legend. I also intentionally set the map so that I you set the labels to on and open the travel route menu the labels are removed as the travel routes would be very difficult to see with the labels on.
I also updated the colours for the other categorisation types to hopefully make them work a little better. When the ‘number of references’ categorisation is selected the marker colours are now a gradient. It’s also now possible to cite a specific stage in a travel route. This appears in the citation text, e.g. “HiMuJe Malabar Digital Map: Watercolor map, travel routes menu, showing the travel route ‘Brahmins’, with unrelated places hidden, viewing stage 2 of the route, placenames categorised by language. 2026. In HiMuJe Malabar. Glasgow: University of Glasgow. Retrieved 28 August 2026 and following URL opens the route at Stage 2. This option will mean I’ll be able to update the location popup to add a new tab that lists all of the stages in routes that the place appears in, and give links to each of these. I haven’t done this yet, though.
Also this week I spent quite some time working for the Dictionaries of the Scots Language. This included attending an online meeting to discuss the structure of the new site, at which some fairly major revisions to the new design were proposed. I also engaged in a couple of email conversations about the updates that still need to be undertaken before the hopefully launch the new site later this year. In preparation for all of this work I also went back through all of my emails from the DSL team and my notes to create a bit ‘to do’ list containing everything that needs done. It’s going to be a busy couple of months.
Also this week I met with Craig Lamont and his RA Emily Hay to discuss the Life Writing project. They’re going to be compiling a bibliography of Scottish life writing and I’m going to develop the systems for this. I did look into using existing bibliography software but nothing I looked at did exactly what they are after so I’m going to create a simple CMS for them to use myself, with a front-end consisting of search and browse options and a map interface sometime next year. I spent a bit of time after the meeting going through the sample spreadsheet they had prepared and sending a list of questions about the data.
I also met with Deven Parker this week to discuss an AHRC proposal she’s involved with. I can’t say too much about this, but it’s related to the work we’re doing on the playbills and I would be involved in some technical capacity. I also had a chat with Garrick Allen about a proposal he’s trying to get funding for after an earlier AHRC submission was unsuccessful. I read through the documentation and gave feedback, and I’ve also put Garrick in touch with my colleague Luca Guariento who is probably going to provide the technical assistance.
Other tasks I carried out this week included investigating an issue with the Anglo-Norman Dictionary (which turned out to be an issue with the data and not the system), making some final tweaks to the Burns Supper Map before the project RA finished working on the project, and generating some Books and Borrowing data for a publication Katie Halsey is working on.
Monday was a holiday this week and on Tuesday I worked on the Burn Supper map. I updated the narrative summary so that it features percentages based on the number of suppers that supplied the data type rather than the overall total. This means that (for example) Whisky now has the percentage 88% rather than 57%. The new summary layout is shown below:
I then ran an import of new suppers, taking the total of suppers up to 1044, and as the facts and figures are all dynamically generated these updated to reflect the new data. I had to do some manual tweaking of the data, for example some countries had typos and needed fixing, plus I needed to process and upload all new images that were associated with the new suppers. Other than future imports of new data I think the Burns Supper map is now pretty much complete.
On Wednesday I focussed on the Playbills project. Last week I’d executed my script to process performers, and this left a few hundred that needed to be manually checked. I started to work on these myself initially, but it was pretty slow going, especially when checking data by querying the database directly. Instead I decided to create a new spreadsheet that includes all of the fields needed for checking (e.g. the role information as well as the performer details) and it should be quicker to edit this. I sent it, along with detailed instructions on how to update the data, to Deven and hopefully she’ll be able to work through the list in an hour or two.
I met with Deven to discuss our next steps for the project during the afternoon, and before this meeting I prepared a list of discussion points. As we would ideally like to know the genders of all roles in the data (so it will be possible, for example, to ascertain when female performers take on male roles) but as roles are often just names without titles assigning gender would require manual checking. Instead I wondered whether we could get AI to assign gender and then leave us with only the trickier ones that are less clear.
As a quick experiment I passed a short list of roles from one play to ChatGPT and asked it to ascertain the gender of each. The prompt I gave was “for the following list of names work out whether each is male, female or unknown: Edmund (the Blind Boy), Stanislaus, Oberto, Rodolph, Kalig, Molino, Starrow, Elvina, Lida” and the response was: “Edmund (the Blind Boy) — Male, Stanislaus — Male, Oberto — Male, Rodolph — Male, Kalig — Unknown, Molino — Unknown, Starrow — Unknown, Elvina — Female, Lida — Female”
So two thirds of roles were correctly assigned a gender and there were no mistakes, leaving one third that would need manual checking, which I think is looking fairly promising.
On Thursday I continued to develop the interactive map for the HiMuJe Malabar project. I sorted out the type / subtype categorisation in the legend to include both types and subtypes, as shown in the following screenshot:
Subtypes are indented within the type now and any places with a type but no subtype (e.g. Sri Lanka) are now appearing. Currently all types and subtypes appear in the legend, even if they have no associated places, mainly so we can see what the full list will look like. It is rather long and I may need to add in a scrollbar, although we may also want to rework the categorisation too – e.g. we have ‘Town’ as both a type and a subtype of ‘Settlement’, plus we have two occurrences of both ‘Hinterland’ and ‘Backwater’.
I was intending the counts beside the types to be a total of all places categorised by the respective subtypes, but some places only have a type and no subtype so the counts represent these instead (e.g. the ‘4’ beside ‘Region’ shows the number of places that have ‘Region’ and no subtype). There are also 5 places that have a subtype within ‘Region’ in addition to this but I’m not sure how best to represent this without confusing people. We could have something like ‘Region (4+5)’ or ‘Region (9)’ but both of these seem a bit unclear to me. I was also thinking of having the checkbox beside each main type select / deselect all subtypes, but if we did this it wouldn’t be possible to just display the main type without its subtypes. These issues need further consideration.
I also implemented the record pop-up that now appears when you press on a marker or polygon, as you can see in the following screenshot:
The popup header displays the ‘preferred name’ for the place and the ‘general information’ tab features the ID, all names and their languages, the category and subcategory (I guess I should standardise this to ‘type’ and ‘subtype’ to avoid confusion) and any supplied description. The ‘references’ tab shows a count of the number of references to the placename in the source texts and the content of the tab lists the filenames and snippets for each reference, with the actual text highlighted in yellow, as shown in the following screenshot:
We should probably have actual titles for the source texts rather than filenames, but these are not included in the output and is maybe something to add in, along with references to specific lines / pages. Another possible issue is that I am aware some of the text will be read right to left and at the moment all text is just displayed as ‘prefix+extract+suffix’. We might need a flag in the data for when the text should instead be ‘suffix+extract+prefix’ (or I guess set the direction to ‘rtl’ in the stylesheet). When we have any images I’ll add these as a further tab, but we don’t have any yet. Also, I haven’t implemented the ‘Share / Cite’ tab yet.
The final thing I’ve implemented is an alternative categorisation for the map, based on the source texts the place is referenced in. In the data I was working with there are only four places that have references, and I’ve added a further ‘No source’ category that all other places are added to, as the following screenshot demonstrates:
I renamed the ‘Place’ menu in the left-hand menu to ‘Place Categorisation’ and there is now an option to switch the categorisation from type to source text. I’ll add in ‘language’ and ‘frequency of reference’ next week, all being well. After I sent an update to the team, Christian, the technical person for the digital edition, informed me that a new output of the data was available that featured many more references and other updates. I therefore replaced the data in the map with the new version and it made a huge difference – the list of cats and subcats is now shorter, there are many more source texts (although this demonstrated that I have some further work to do with the legend for source text categorisation) and many more places with references. I’ll continue with this next week.
During the week Katie Halsey, the PI of the Books and Borrowing project, contacted me to ask for some help in creating some queries of the data for the monograph that she and Matt are writing. These involved working out which books and authors were found at ten or more libraries, and which books and authors were borrowed in every decade from the 1750s to the 1830s. I looked into this on Friday, and it took most of the day to work on it, partially because it’s been a while since I worked with the data and it took some time to remember how everything fitted together. However, I managed to produce the data required data and I sent it to Katie and Matt in four spreadsheets.
Also on Friday I did some work for the Dictionaries of the Scots Language, setting up a new user account that will be used by some interns that are starting with the project over the summer, sorting out access to the Google Search Console and removing the user survey.
I continued to work on the Burns Supper Map this week, implementing the ‘click through’ options in the record pop-up. The record pop-ups now feature a dotted line underneath certain information, and if you press on this the relevant filter for the selected item will display, as the following screenshot demonstrates:
Other than importing more data, integrating the videos and things like the ‘more info’ pop-up and some ‘fact and figures’ that’s pretty much all of the functionality in place now, and I therefore moved on to the user interface, a task that took a fair amount of time, but has been useful. I developed three alternative interfaces using different colour schemes and fonts, and in some cases different layouts. For the first version I took the head of Burns from the stained glass image and used this in the ‘Home’ tab and also as the supper icon and the browser tab icon. The menu sections have a black background and white text and I’ve used the font ‘Quintessential’ for the site heading and also the popup headings. You can see this in the following screenshot:
For the second version I used a traditional Scottish Blue and White colour scheme. This version doesn’t feature the disembodied Burns head and it uses the ‘whisky glass’ icon on the map and in the browser tab. The whole site uses the ‘Baskervville’ serif font throughout (the other versions use the sans-serif ‘Montserrat’ font for most text) with the ‘Baskervville SC’ small-caps variant used for headings:
The third version uses a sort of teal-ish colour scheme with a sort of whisky-orange colour for highlights. For this one I incorporated the Nasmyth painting of Burns in the ‘Home’ tab and I used the cursive font ‘Tangerine’ for the ‘Burns Suppers’ text in the site heading and in the record pop-up. Map icons use the cutlery icon with the whisky-orange background. This is also used in the browser tab, although the icon and colour scheme are rather similar to the ‘Just Eat’ logo and this might be an issue. Here’s a screenshot:
For this version I also redesigned the record pop-up to make it look sort of like a fancy menu. This does take up more vertical space and the cursive font used in the headings might be difficult for some people to read, but I quite like the way it looks:
These are just work-in-progress versions and I’ll hopefully meet with the other team members next week to discuss how we should take this forward. Also this week the project RA Cleo went through the images and reduced the number of images for each supper to a maximum of ten (some had hundreds before). I then ran a script to resize these images to a maximum of 1600px, which has greatly reduced the file size of the images – there were some previously that were 16MB each. I then replaced the images on the server with this smaller set, reducing the number of images from 1696 to 893 and reducing the total size of the images from more than 4GB to about 350MB.
I also continued with the Playbills project this week, giving advice to project PI Deven as she edited the data and processing updates to the data. This included merging and deleting venue and printer records to remove duplicates and reassign playbills to the merged records. I also began to investigate assigning location data (latitude and longitude) to venues, which is something we’re going to need of we want to create a map-based interface for the data. We have around 100 venues after merging and deleting, and it’s possible that some others will also be deleted, so it’s not a huge number of records. However, it’s likely that the location data will need to be manually researched as many of the theatres no longer exist, or at least the buildings no longe exists as theatres. For example, our first listed venue is ‘Royal Theatre Aberdeen’, which doesn’t exist anymore, but I found an address for it on this page, meaning I could find the building (now Aberdeen Elm Church) on Google Maps and get the latitude and longitude (57.146607938894356, -2.0920931547723085). I’ve created a spreadsheet on the project’s Team that we can use to add this data in, and I’ll probably research a few records myself, if I have the time.
I also spent about a day this week writing an initial version of a specification for Joanna’s DOST Auld Laws front-end. It’s a non-technical overview of what will be developed and how things will work and I created it mainly so we all know what the plan is. The document I wrote is just a first draft and I sent it on to the team for feedback. It’s likely that aspects of it will need to be changed and in fact there is a list of questions at the end that will need consideration. I’ll hopefully get some feedback from the team next week and I’m hoping that we’ll have a final version of the document decided upon before Easter, so I can start development after I get back from my Easter holidays.
I also spent some time investigating a data issue on the Books and Borrowing website. Certain records for Innerpeffray library didn’t have associated borrowers so Katie and Kit added these in, including creating new borrowers where necessary. However, the data in the front-end wasn’t updating to show the borrowers when viewing register pages and I needed to figure out why. It turns out that this is a consequence of the versioning system I implemented last year. The CMS is now set to data version 2, meaning anything newly added has this version number applied to it. The front-end is set to only display data version 1, so any new borrowers created will not get displayed as they are version 2 records. I manually updated the new borrower records to assign them to version 1 and after that they appeared in the front-end. We still need to fully update the data caches and Solr indexes before the new borrowers are fully integrated with the front-end (e.g. appearing in search results), and this is a task I’ll have to tackle in the next few months.
I mostly divided my time between three projects this week: Playbills, Burns Supper Map and the Place-names of Armagh. Unfortunately I was still suffering from the monstrous cold I started with last week and struggled through some of the week, but I still managed to get quite a lot done.
For the Playbills project I wrote, tested and implemented a script that extracts the data from the JSON versions of the playbill files I generated, splits this up and inserts everything into a relational database. The reason I’m doing this is to make it easier to generate canonical records for venues, plays, performers and roles, as it will be much easier to query the data and track records in a relational database.
The data I’ve extracted consists of 1902 playbill records that feature 6434 plays. These are categorised by one or more of 188 distinct genres (with ‘melodrama’ associated with 2026 plays and ‘melo-drama’ a further 10). I’ve extracted 185 distinct venues, 49498 performers, 49495 roles and 6083 contributors.
As of yet I haven’t done anything to generate canonical records, which will be the next major step, and I need to discuss things with project PI Deven before I proceed with this. For example, the role ‘Macbeth’ appears 27 times, with a further three appearances in other strings (not including ‘Lady Macbeth’) e.g. ‘Macbeth’s Last appearance’. These would need to link to one single canonical ‘Macbeth’ role. Similarly, there are 28 plays that have ‘Macbeth’ somewhere in their title, with variants such as ‘MACBETH, KING OF SCOTLAND’, ‘Macbeth; King of Scotland’, ‘MACBETH, KING OF SCOTLAND.’ In addition to ‘MACBETH’ and ‘Macbeth’ and these would need to link to a single canonical ‘Macbeth’ play.
There’s also some data cleaning that we should perform, e.g. amalgamating data that doesn’t have the same form but should be the same thing. For example, there are a lot of possible duplicates in the ‘Genre’ data. There’s ‘acrobatic’, ‘acrobatic display’, ‘acrobatic performance’ and ‘acrobatics’ all as different genres when presumably these should be the same.
I also still need to work on the performer names to split them into titles, forenames and surnames, and to ascertain gender based on titles. Venues also need some work as there are many that are the same but have slightly different text, e.g. ‘Royal Theatre, Aberdeen’, ‘Theatre Royal, Aberdeen’ and ‘Theatre Royal Aberdeen’. There’s the same issue with printers too, although perhaps this isn’t so important. E.g. ‘Keenes, Kingsmead-Street, Bath’, ‘Keenes, Bath, Kingsmead-Street’ and ‘Keenes, Bath’. It’s possible that we might be able to get some sort of AI processes to help with such tasks.
We’re also going to have to give some thought about how to handle updates to the data. I’m generating canonical records, extracting things like performer gender and generating unique identifiers for things like plays in my database, and I’ll be creating new JSON files that incorporate this new data that will then be ingested into Solr for search purposes. Therefore the data will be quite different to the original YAML files. When updates need to be made should these then be made to the original YAML files, which would necessitate much regeneration of data, or should the updates be made elsewhere, such as through the database? I don’t have an answer to this yet, but it’s something we’ll need to consider.
For the Burns Supper Map project I set up the online database for the supper data and have been working on a script that imports the data from the spreadsheets into this database. I have got everything working for the spreadsheet of the online survey, so my database currently has 308 suppers that include data for 6902 filter options.
What I haven’t been able to do yet is to import the data from the public domain spreadsheet, as this currently contains a lot of inconsistencies in how the data are recorded. The data in the filter columns (“frequency”, “category”, “toast”, “food”, “style”, “drink”, “entertainment”, “poem”, “music”, “dance”, “dress”) must exactly match the options found in the online form for my import script to work. This includes capitalisation / case and ensuring that a semi-colon is used to separate multiple items. I had a meeting with the project RA Cleo on Friday to discuss this, and she’s going to work on tidying things up.
I also wrote a script that posts the address for each record to Google Maps which then returns the latitude and longitude (something we’re going to need in order to pin the records on a map). The user inputted location data can be somewhat variable, as you might imagine, but Google Maps has generally done a very good job at identifying places from the data, and we can always tweak things once we see the locations on the map. I’m hoping to start development of the map next week.
For the Place-names of Armagh project I uploaded a large number of place-names that I’d been sent. We now have 2932 place-names in the system. I also processed the existing historical forms CSV and this has found historical forms for 1056 of these new place-names. The new place-names had additional parishes and baronies that were not already in the system and in such cases these have been created, but there are some issues, as the data appears to be somewhat messy at times and will need some cleaning. For example, there’s a ‘Forkhill’ and a ‘Forkill’ and these may be the same, there are forms with question marks and multiple forms and descriptive text, e.g. ‘Killevy/Partly in Dundonald Parish’ and ‘Armagh?/Eglish?’. These will all need separated out and fixed as required.
I also spent some time updating the CMS to convert the townland field from a textbox to a list, thus enabling multiple townlands to be associated with a place-name and ensuring each townland is only stored once in the system. This involved extracting the townlands from all of the 2932 placename records, splitting forms up that have multiple townlands in ‘x or y’ or ‘x / y’ format, storing the unique townlands and then associating the corresponding ones with each placename record. There are 997 unique townlands (although some of these may need amalgamated) and 3007 connections between townlands and placenames.
I then updated the CMS to replace the existing ‘townland’ textbox with a list of townlands as checkboxes, in the same way as parishes and baronies. We might need to rethink this, though, as scrolling through 997 townlands to find the right ones takes time. I also included an option to add a new townland when adding / editing a placename record as I’m guessing there will be more to come. This should only be used when the townland isn’t already in the list, otherwise we’ll end up with duplicates.
I also made some further updates to the CMS, namely simplifying historical forms so there is just one ‘form’ field rather than separate English and Irish boxes, and adding a flag to record whether the form is a ‘previously suggested form’ or not. I also renamed the ‘Discovery’ maps to ‘1:50,000’ as this is how the maps tend to be referred to.
Also this week I replied to a couple of emails from the DSL people about future developments and I added a new video to the Seeing Speech resource. I also investigated an issue with the Books and Borrowing website and discussed the migration of the resource to a new server with the Stirling IT people, and I generated CSV files for all of the survey answers for Speak For Yersel and sent them on to Janine Illian in Statistics, who Jennifer and I met with last week.
My time this week was mostly divided between two projects: The Place-names of the Coalfield Communities (Ayrshire) and the Eighteenth Century Playbills project. We launched the interactive map of Ayrshire place-names during an online event on Wednesday this week and there were a lot of last-minute updates that I needed to implement leading up to this. This included replacing a lot of placeholder text with actual text, both on the map and the project website more generally and fixing a number of issues with the data. It turned out that there were almost 250 place-names in the system that had a grid reference that was used by another place-name. This meant that the map markers were sitting on top of each other and unless the data was filtered it was impossible to tell that multiple markers existed at the point. To rectify this I wrote a little script that identified the places that shared a grid reference and all places after the first had their longitude updated to move them slightly east of the original location, ensuring the markers did not overlap. This approach has worked pretty well, although the researchers do still need to properly investigate the duplicates as there are some that are genuine duplicates and should probably be removed.
There was also a slightly strange issue with the CMS, whereby making updates to a specific place-name element was getting blocked by the WordPress plugin Wordfence that runs on the domain. What was especially strange was that the update was getting blocked in Chrome-based browsers but was completing successfully in Firefox, even though the issue was server-side and the posted data should have been identical in both browsers. It was the description field that was getting blocked, and it looked like this was due to the element having a fairly long description, as when I replaced it with a shorter text the update was fine. There was nothing in the contents of the original description that would have caused issues – just a few simple HTML tags. In the end I just had to add an exception to Wordfence to stop it blocking the upload page, and this worked. It’s not like Wordfence should really be processing data added via the non-Wordpress, password protected CMS anyway.
I also needed to fix element languages in the elements glossary, as several of these were appearing as beginning with ‘other’ in the alphabetical list, even though they clearly shouldn’t have been. This was because I created a new ‘element sort order’ field in the database to be used for the alphabetical list and generated the data for it using a script. But what I forgot to do was update the CMS pages where elements are added or edited to ensure this field subsequently gets populated. This meant any elements added after I ran the script to generate the field didn’t have data in the field and so the elements were appearing as ‘other’. I fixed this for the Ayrshire site and also ensured the fix was in place for the other place-names resources too.
Some further tweaks to the elements were required as there were seven different ‘?’ elements (used where an element is unknown) and I merged these into one record. I also ensured that element certainty in the ‘more information’ view of place-name elements was appearing. This should have been visible but wasn’t showing up in any of the place-names resources. A quick tweak of the various APIs got this working across all of the place-names resources.
With all of these updates in place I went live with the map on Wednesday morning, creating a cached version of the full map view data to ensure that the initial map loads speedily. I also spotted some issues where place-names hadn’t been assigned a classification code, resulting in a blank item getting added to the map legend, and I fixed these.
The online launch of the map (now available at https://ayr-placenames.glasgow.ac.uk/map/) went very well and everyone seemed happy with the map, its data and its potential uses. I’d say the session was a great success.
I spent most of the remainder of the week writing a requirements document for the front-end for Deven Parker’s Playbills project. I had hoped to begin work on this earlier in the week, but there was rather more to do before the project launch than I was expecting. There is a lot to consider for the playbills project in terms of how to process and store the data and present everything online, and by the end of the week I was probably only about half-way through writing the requirements document. I’m hopeful that I’ll be able to complete it next week, though.
Also this week I updated the facility to save an SVG version of the Historical Thesaurus timeline visualisation to make the process fully JavaScript based. Previously the SVG was exported for download by posting the XML data to a PHP script, which then triggered the download. The purely JavaScript way means that no data needs to be posted and echoed out by PHP, which is more secure. I also had some discussions with the Stirling IT people about bot traffic on the Books and Borrowers site, which is beginning to cause issues. I talked about our recent experiences with the Emblems site and shared the htaccess script Luca created to sort out that issue. Hopefully this will help. I also received a request from Garrick Allen to provide some text for his latest research proposal, and although I began this on Friday afternoon (when the request came in) I didn’t quite manage to finish it. I’ll get this sorted next week.
This week I completed work on a first version of the regions and dialect areas map for the Dictionaries of the Scots Language. Last week I’d begun work on the final aspect of the map: ensuring that whatever the user is viewing on the map is reflected in the hash to enable specific views of the map to be bookmarked, shared and cited. I’d begun updating the scripts to process details held in the hash and I completed this by adding the areas selected in from the info pane this week. This was slightly more complicated than I’d imagined as I needed to consider how to handle areas that are selected by default (e.g. all regions selected when the user clicks on the map) in addition to areas specifically turned on or off by the user from the checkboxes in the info pane. I decided to ensure that the default selected areas were also present in the hash to keep things simple.
With the hash processing in place I then needed to update the code to actually add the necessary information to the hash as the user interacts with the map. This was a fairly big job, involving updating much of the code, but I got there in the end. After a fair amount of testing and tweaking I was then able to send the URL for the map to the DSL people for them to test out and give feedback on – I’ll just need to see what changes may be required and also when we might go live with the map – probably not for a while yet I’d imagine.
Also for the DSL this week I had an online meeting to discuss the new entry XML structure. We went through the list of questions I’d come up with whilst working through the new XML examples a few weeks ago and had a good discussion about what would be included in the update and when I’ll start work on this. I should receive a test export of the dataset in the next week or so and once I have this I’ll be able to start implementing the necessary changes to the systems. I also investigated an issue with email lists for the DSL that Rhona contacted me about and fixed a minor layout issue with the live site.
Another sizeable task that I was able to tick off this week was finally launching the new Books and Borrowing versioning system. I’d encountered a strange issue with the online database last week that had prevented me from continuing with the rollout, but thankfully my contact in Stirling’s IT services was back at work on Tuesday and managed to identify the issue. The situation is that the online database’s data is replicated across three nodes. But this only works with database tables created in the ‘InnoDB’ format. I’d created the new tables on my laptop, which had defaulted to the ‘MyISAM’ format, something I’d not spotted. When I updated the online database these new tables were then stored in the ‘MyISAM’ format and were therefore not getting replicated. I wasn’t aware of this distinction between database tables and the implications this had for how the online data was stored – these formats refer to how the data is stored on disk and there is no difference when you view or query the tables and data through the database.
Thankfully once the issue had been identified I was then able to re-create the tables in the ‘InnoDB’ format and regenerate the data, after which all data magically appeared in both the PHPMyAdmin interface and through my scripts. With this in place I then completed the rollout of the new versioning system and the new ‘v2’ version of the API, which as far as I can tell has all been successful. As of yet there’s not really much different as we still only have version 1.0 of the data available. What you will see in the footer of every page is some text about the database version and a link to the ‘versions and changes’ page (https://borrowing.stir.ac.uk/versions-and-changes/). Also, when you perform a search and choose to cite the page the data version now appears in the citation and also the URL (so that when we do publish new versions the citation remains accurate). E.g.
“Quick search for ‘rome’. 2025. In Books and Borrowing: An Analysis of Scottish Borrowers’ Registers, 1750-1830, data version 1.0. University of Stirling. Retrieved 16 September 2025, from https://borrowing.stir.ac.uk/search/version-1.0/p-1/0/0/quick/rome”.
The URL with ‘version-1.0’ in it works in exactly the same way as the general URL for now. When we release ‘version-1.1’ the URL above will load the version 1 data with a warning that the user isn’t looking at the most up to date version. The new API is now available here: https://borrowing.stir.ac.uk/api/v2/ and I’ve updated the ‘About the database’ and ‘FAQs’ pages to point to this. I also sorted the encoded apostrophe issue and book works now appear in the filter options and ESTC has been removed. I spotted an issue with book works that had ampersands in them breaking the site when selected as a filter but I’ve managed to fix this, for example:
So after many months of work that’s the new versioning system complete. Hopefully when we come to launch version 1.1 the process will be a little more straightforward!
Also this week I had a discussion with Geert of the Anglo-Norman Dictionary about processing the data in preparation for adding in a new part of speech option to the searches, and I also fixed an issue with the management system that was preventing certain cross-reference entries from being uploaded.
I also gave some feedback to Matthew Creasy about a proposal he’s submitting soon and had a discussion with Jennifer Smith about sharing the SCOSYA data with the Mozilla Foundation. I also made a bit of progress with the arrangements for the new Place-names of Armagh project.
On Friday I had a bit of spare time that I devoted to the migration of the Place-names of Fife project to our standardised system. I realised that many of the historical forms had not been successfully processed by the script that had extracted data from the original Word files, resulting in several hundred forms that had no data in the historical form name field and everything in the ‘source’ field. I sorted these by a combination of batch processing and manual updating, and also sorted an issue with notes not appearing in the correct field either. I think I’m just about ready to migrate the historical forms now, although much work will still need to be done to extract sources.
I continued to work on the new region / dialect area map for the Dictionaries of the Scots Language for around half of this week. I’d made really good progress with the map last week and by the end of the week I’d figured out a possible way to grab all areas that intersected a given GeoJSON area using the Turf.js library (version 6 as for some reason version 7 didn’t work). This week I wrote a script to go through all of the regions, identify all intersecting regions using Turf.js and store these in a JSON file.
All went well to begin with, but midway through processing my script threw an error, with Turf unable to process a geoJSON aera. It turned out that some of the coordinates in our region data use a lot of decimal places – more than Turf can handle, in fact. Thankfully Turf has a helpful ‘truncate’ function that truncates the precision of a region, and once I applied this the script executed successfully.
Once I had my JSON file containing all of the intersects for every region and dialect area I could then work on what to do when the user selects a region / area from the drop-down lists. When this happens my original intention was that the corresponding area would be highlighted in yellow, with all intersecting regions given the usual red colour, while intersecting dialect areas would be ‘off’ by default but the user could turn these on from the info pane. I also wanted the map to zoom and pan to focus on the area that had been selected.
However, all of this brought some difficulties. It turns out that there are a lot of intersections with our data. Not all adjacent areas neatly line up and there are frequently very small (or not so small) overlaps. These are of course considered intersects, meaning that many adjacent areas get added in when really they shouldn’t. This meant that lots of areas were getting highlighted in red, and the yellow highlighting of the area the user actually selected was getting somewhat lost in the mix, due to the transparent nature of the area fills. I therefore decided that I would only highlight the area the user actually selected (in the usual red, not yellow), and all other intersecting areas would be switched off by default, but would be selectable in the info pane, as the following screenshot demonstrates:
I think this works pretty well and is not too overwhelming for users. The next issue I needed to address was zooming and panning to the selected area. In order to do this I needed to use Leaflet’s ‘fitBounds’ method, which expects latitude and longitude values for the North East and South West corners of a rectangle. Unfortunately all I had was very complex geoJSON polygons so I needed to investigate how to generate the ‘bounds’ from this. Initially I thought I’d have to do this on the fly (i.e. when the user selected a region) and figuring out how to get Leaflet to grab a specific area within the geoJSON file and calculate this proved very tricky. Eventually I realised that I didn’t actually need to do this: I could simply store the ‘bounds’ for each area along with the intersects. Turf has a plugin called ‘bbox’ (https://turfjs.org/docs/api/bbox) that can take a geoJSON shape and create a ‘bounding box’ from it. After a bit of experimentation I realised that this solution would work perfectly, so I updated my script that generated the intersects to also generate and store the ‘bounds’ for each area. I was then able to update my map code to grab these bounds when the user selects an area and pass them to Leaflet’s ‘fitBounds’ method, making the map show the desired location. I updated this to add in a bit of padding and was very pleased with the result.
With this in place I then moved onto the region search. In addition to selecting a region from a pretty sizeable drop-down list, we also wanted users to be able to search the areas, using an autocomplete textbox. Vasilis’ prototype had used a neat little jQuery plugin called Select2 (https://select2.org/). This included an autocomplete search box within the drop-down menu, which was nice and compact. However, it isn’t immediately apparent to users that this search box is an option – both I and another member of the DSL team who tried out the prototype didn’t even notice this was an option until Vasilis pointed it out. For this reason I decided to separate out the list and the search option, to hopefully make it clearer to users that a search option is even available.
I used the usual jQuery UI autocomplete that I’ve used many times before on other projects, connecting via AJAX to a PHP script that worked with the ‘intersects’ JSON file to find matching regions. This appeared to work well, and on selection of a matching item in the autocomplete list the same code that handles the list executes to select the area on the map, as the screenshot below demonstrates. However, I spotted that the autocomplete was not appearing when the map was in full-screen mode, and remembered I’d encountered a similar issue with the place-names projects. For these projects the autocomplete was in a modal pop-up and using the autocomplete’s ‘appendTo’ option (https://api.jqueryui.com/autocomplete/#option-appendTo) to attach it specifically to this modal in full-screen mode did the trick. For the current map, however, the autocomplete is in the map menu, which is a limited height based on the menu’s contents. When the autocomplete was added it was too tall for the menu and caused awkward scrollbars to appear. I tried several approaches to get around this issue, but in the end decided to just make the menu tall enough to hold the autocomplete when in full-screen mode, as the following screenshot demonstrates:
With this in place the ‘regions’ map was pretty much complete, and I moved onto the dialect area map. I updated the code so that when the ‘Dialect Areas’ menu item is selected the map resets and displays the dialect areas rather than the regions. Pressing on the ‘Regions’ menu swaps things back again, as does pressing ‘reset map’. With the dialect areas visible, I then needed to update the code for clicking on the map so that it would display dialect areas rather than regions, and update the info pane so that dialect areas appear first, as you can see in the following screenshot, which shows one dialect area and several regions selected:
I now had the core functionality of the map in place and I then spent some time adding in more of the necessary features, including the help and about modal window, the attribution modal window and a ‘loading’ spinner that gets displayed until the geoJSON data has loaded. A also added in the option to select a satellite base map. I’m not sure how useful this will be, but it’s good to have it as an option that can be selected at any time from the ‘Home’ menu. A satellite view of the previous map can be seen here:
I then moved on to what will hopefully be the final big issue relating to the development of the map: adding in the option to bookmark / cite / share a specific view of the map. As with the place-names resources, this will involve not just tracking the position and zoom level, but also exactly which data is visible on the map as the user zooms, pans, clicks and selects things in the resource. It’s hugely useful to be able to save or share a specific view of the map rather than only being able to link to the default view of the map, and even though it will take some time to implement such a feature I think it’s worth it. The first thing I did was add in the leaflet hash plugin (https://github.com/mlevans/leaflet-hash) which tracks and processes position and zoom. As with the place-name projects, I adapted this plugin to also track and process further data held in the hash (the part of the URL after that ‘#’ sign). I decided on a structure for this data and began implementing the code that will process the various elements, ensuring that (for example) when the user has selected the satellite map this is picked up by the code and the relevant base map is shown. By the end of the week I’d added in the code for processing the base map, the active map type (region or dialect area), the browse option (region search, region browse, dialect area browse, map click) and the browse data. I still need to add in the selected items in the story pane, which I’ll do next week. I also still need to update the code to add all of this information to the hash based on the user actions – for now all the code does is process information in the hash rather than add data to it. Hopefully I’ll finish this and complete an initial version of the map next week.
My other big task for the week was to roll-out the new versioning system for Books and Borrowing, replacing all of the code and the data on the online server. I’d got everything sorted on my laptop last week and had sent the data structure and files to Stirling’s IT people so they could set up the new Solr core that was required. This was set up the day I asked for it, which was great, and I therefore began regenerating all of the database data on the live server.
Unfortunately I began to encounter some strange behaviour with the online version of the database, and as my contact in Stirling’s IT department is on holiday until Tuesday next week I had to abandon my attempts at rolling out the new system.
I first encountered the issue when running my script to populate the book holding totals cached data. These are now stored in a separate table as there may be different totals for different versions of the data. The script ran fine on my laptop, and appeared to run perfectly on the server, but when I checked the online admin interface to the database (PHPMyAdmin) the book holding totals table was empty.
I spent ages trying to work out why the ‘insert’ queries were not running, but also not giving any errors. The exact query when entered directly into PHPMyAdmin worked and displayed the inserted row, but when executed through my script the row wasn’t shown in PHPMyAdmin.
Eventually I figured out that the queries WERE executing successfully in my script and WERE being interested into the database, as a test script I wrote connected to the database and listed the rows. But still PHPMyAdmin was showing an empty table. It was almost as if my scripts were connecting to a different version of the database to the one PHPMyAdmin was accessing. However, this wasn’t the case as if I ran the ‘empty’ command in PHPMyAdmin to clear the data from the table my test script also then showed no data. It was all very odd, but having identified the issue I could proceed to run my cache generation scripts (which included several more similar tables) and test that the data had been inserted using scripts rather than PHPMyAdmin.
However, the library cached data is pretty sizeable and instead of generating it on the server I decided to export the tables from the version of the database running on my laptop and import them into the server’s database using PHPMyAdmin. This appeared to have worked and I could see the data in PHPMyAdmin. I completed the data import scripts and then started on upgrading the online scripts to use my new versioning code. I started by setting up the new ‘V2’ version of the API. However, API calls in this that worked perfectly on my laptop were giving errors when run on the server. After much more investigation it would appear that the library cache tables that I imported via PHPMyAdmin are coming back as empty when accessed via the API scripts, even though they contain data when viewed in PHPMyAdmin.
There is clearly something very wrong with the online database and I need someone at Stirling to investigate. It almost looks like there are two separate versions of the database, but for the fact that running an ‘empty’ query in PHPMyAdmin definitely affected the data my scripts access. I can’t really proceed with the rollout of the new system until this is resolved, which will hopefully be next week. It’s all very frustrating as I just want to get the new system finished.
Also this week I spent some time fixing an old resource with Luca. I’d received an email saying that the Working Class Marriage website (https://workingclassmarriage.gla.ac.uk/) was not working. This wasn’t a website I had worked on, but it contains a bunch of interactive maps and a lot of data and Luca and I wanted to get it back online again, which we managed to do after some tweaking of the code.
I also spent a bit of time working for the Anglo-Norman Dictionary, fixing the ‘add source’ feature of the management system, which was unfortunately broken due to a database update not being reflected in the code.
I spent the majority of this week working on the new, interactive region and dialect area map for the Dictionaries of the Scots Language. DSL editor Vasilis had created a prototype interface and geoJSON data before the summer and a few weeks ago I had a meeting with a few members of the DSL team about creating an new version of this that would be more suited to the DSL website and would be optimised for mobile use. I then created a specification document which I shared with the group and after some feedback we reached agreement on what needed to be developed.
I was able to base the new interface on my previous map-based projects such as the place-name resources and the Scots Syntax Atlas, and as with these I decided to use the Leaflet.js mapping library. My first task was to incorporate all of the geoJSON data into an interactive map to get an idea of how it might work and whether there would be too many areas for people to make sense of. The first issue I encountered was that the geoJSON data had been created using a projection that Leaflet did not support. I’m not exactly a GIS expert, but apparently Leaflet expects all coordinates to be latitude and longitude using the Coordinate Reference System (CRS) WGS84. The DSL regions geoJSON data stored its coordinates as EPSG 3857, so the coordinates were things like ‘[ -203142.297936542046955, 7877431.976650067605078 ]’ rather than ‘[-1.804316665436796,57.55628903006679]’. This meant that no data was getting displayed on the map. Thankfully I’d encountered this issue before whilst working on the Speak For Yersel project, and I was able to use QGIS to convert the data into the alternative CRS, after which it appeared on the map correctly.
The next issue was that the geoJSON was around 11MB in size, which is far too big for downloading into a webpage. Again, we’d dealt with this issue with Speak For Yersel, where we’d used an algorithm to simplify the polygons and massively reduce the file size. Unfortunately this was a task that another member of the team had performed and going through my blogs and emails I was unable to find details of exactly how this had been accomplished. I tried a couple of approaches in QGIS but they converted the polygons to really basic shapes that were totally unsuitable. Thankfully I found a really amazing resource called mapshaper (https://mapshaper.org/) that allows you to paste in a geoJSON file, visualise it and adapt it, including simplifying it. After some experimentation I exported a file that was just over 1MB in size, which was much more suitable whilst retaining sufficient detail.
I was then able to plug this data into a Leaflet map and visualise it. As agreed with the team, we would have separate maps for regions and dialect areas, so I updated my code to only display regions, stripping out the dialect areas. Initially I created an interface based on the Speak For Yersel maps, with polygons appearing with a faint grey dotted border and the ability to hover over a region, which would highlight it with a yellow border and display the region’s name in a box in the top-right of the screen. I realised that this approach was not going to work very well, firstly because hover-over doesn’t work very well on touchscreens and secondly because there are generally multiple layers present at any one point of the map and highlighting these and displaying all of their names as the cursor moves around the map would be messy.
Instead I decided to remove the hover-over and focus on the click event. We’d decided that when the user clicks a point on the map all regions represented at this point would be highlighted on the map, and information about them, plus the dialect areas would appear in a pane in the map, with options to turns on and off each layer.
In order to grab details about each layer that was represented at the point the user clicked on I needed to use a Leaflet plugin called ‘Leaflet point in polygon’ (https://github.com/mapbox/leaflet-pip). Using this gave my code access to as handy array of all of the layers found at the point where the user clicks, thus allowing me to highlight them on the map and access their details to present them in the information pane.
While Vasilis’ prototype had used solid blocks on the map to represent the regions, we’d decided instead to make these transparent so people could still make out the placenames and features of the underlying map. I also decided to replace the dotted lines with solid ones, as these looked a little neater where multiple polygons converged. I had been worried that there would be too much data and that the map would be too cluttered to be understandable, but I think the map works with all regions visible, and it helps give people a sense of which area might be interesting to click on.
I worked on the information pane, creating lists of regions and dialect areas found at the point clicked on and making it possible to turn these on or off, using a red border and transparent background for regions and a purple border and background for dialect areas, with dialect areas switched off by default. I also decided to add a map marker to where the user clicked to make it clearer how the highlighting has been calculated. I also added in the option to close the information pane (which would also remove all highlighting). This can also be achieved by pressing anywhere on the map that isn’t a region (e.g. in the sea). The screenshot below shows a section of the map with all regions located at the point clicked on highlighted in red and one dialect area (Northeastern Scots (a)) selected and shown in purple:
This is quite a lot of information, but note that the purple area is not visible by default and has to be manually selected. Also, using the options in the information pane you can very easily filter out any layers you’re not interested in. The following screenshot only has one layer (Moray Firth Fishing Villages) highlighted, for example:
There’s still work to be done with the information pane – for example the listed regions and dialect areas should ideally be hierarchically ordered, but I’m pretty happy with how things are developing so far.
I then decided to switch my focus to the map menu, which I wanted to add to the left of the map using the code I’d developed for other resources. This menu would provide lists of regions and dialect areas, allowing the user to select one to view it on the map rather than just clicking round the map. Vasilis had sent me lists of dialect areas and an XML file containing the hierarchical structure of regions and I spent quite a lot of time working with this data. I needed to ensure that the data presented in these files tied into the geoJSON data, and there were a few inconsistencies that took some time to sort out. For example, the ‘abbr’ element in the XML file did not always match up with the ‘abbr’ attribute in the geoJSON data and there were a few typos. Also, the regions XML file also contained dialect areas and I needed to strip these out. It took somewhat longer and was rather more tricky than anticipated to extract the data in a format that I could use, but I got there in the end. I think there’s still work to be done on both the hierarchical structure and the presentation, but it’s a start at least. The screenshot below shows the hierarchical layout of the dialect areas using a drop-down list:
I still need to add in a search option and I might replace the HTML select box with an alternative interface that can better display the hierarchy. As of yet it’s not possible to select an item and actually visualise it on the map, and this led onto another issue:
Selecting an entire region needs to behave differently to clicking on the map; when the user clicks on the map the regions that need to be highlighted are simply all of those regions that are present at this exact point on the map. But when an entire region is selected this may include several intersecting regions at various points within the region so (for example) ascertaining the centre of the selected region and working out which regions are present at this point would not give an accurate picture. I therefore needed a method to ascertain all regions that intersect with the selected region anywhere within the region.
I cam across someone else who had this same issue (https://gis.stackexchange.com/questions/170919/how-to-tell-if-a-geojson-path-intersects-with-another-feature-in-leaflet) and the answers pointed me towards the turf.js library, which I had also previously used for Speak For Yersel. I downloaded the most recent version of Turf (version 7) and wrote a test script to see whether it would work. However, I just couldn’t seem to get Turf to work with the geoJSON data – it just gave errors. I also tried plugging in the example given in the answer on the page linked to above and it also gave errors, which was very frustrating. Eventually I decided to try an older version of Turf (version 6) and both the example code and my geoJSON areas worked perfectly, so there much be something in Turf 7 that has changed. However, this means I can process all of the intersects using Turf 6 and store these as a cached list to be used whenever a region is selected from the lists. This will be my task of the start of next week.
Also this week I worked on the Books and Borrowing project. I now have access to the Stirling servers again and was therefore able to work through some of the outstanding tasks, although I won’t be able to update the online systems until the duplicate author work that another member of the team is working on has been completed (well, I could, but then I’d have to do it all again afterwards) so for now most of the updates are limited to the version of the site running on my laptop.
What I did manage to do was fix the weird apostrophe encoding issues across all data types, replacing the encoded apostrophes with regular straight ones. I’ve made this change on the live database, but until I regenerate the search indexes the online searches won’t be changed. I also replaced all curly apostrophes and double-quotation marks found in authors, borrowers, book editions, works, holdings and borrowing records with their straight equivalents. This took some time to implement and test as it involved updating several thousand records. All seems to have worked, though. Again, this change has been made in the online database and you can see the change when viewing records (see for example: https://borrowing.stir.ac.uk/search/0/0/0/advanced/brids|32203 where the text ‘Peveril of the Peak : by the author of “Waverley, Kenilworth”’ once had curly double-quotes).
I also removed ESTC from the search results filters (again, only on my laptop for now) and added in Book Work title to the search results filters (also only oh my laptop for now) as the following screenshot shows:
The book work titles are rather long, but I think it works ok and I’m sure will be a really useful addition. In order to implement this update I needed to update the Solr index structure to add a new field, reindex the data, update the API and the script that processes search results, so a fair amount of work, but worth it. I also still need to update the script that generates the Solr data to ensure it differentiates authors that have the same name, but I’ll wait to do this until the duplicate author work is complete. After that I’ll then grab a fresh copy of the online data, run it through the various stages needed to generate indexes and caches, test everything out on my laptop again, and after that I’ll need to get Stirling’s IT people to set up a new Solr index and import the new files. Once this has been completed I’ll then be able to update the online API and front-end.
Also this week I made some updates to the structure of the SpeechStar resource for Eleanor Lawson. These included some major changes of page URLs, which introduced some complexity as we don’t want people who may have cited a view of the old pages, or a specific video on one of those pages to end up with a ‘not found’. To avoid this I put in redirects from the old URLs, but this wasn’t enough as I also needed to ensure any filters were also passed over to the new pages, but this is all in place now. So for example, someone who has bookmarked or cited the ‘Child speech error database’, listing the videos by word, filtered by sound: k and sex: female with the video for ‘school’ open would have this link:
And rather than giving a ‘not found’ or redirecting to the default view of the page the above link continues to open the correct video with the correct filters.
I was intending to focus on setting up the Books and Borrowing versioning system on the server this week. I’d spent quite a long time implementing the versioning system earlier in the summer, and this required massive changes to every aspect of the B&B system: the database, the Solr index, the API, the CMS and the front-end. I’d completed work on this on my local PC just before my summer holidays and as the team were using the website quite heavily at this time we decided to hold off with the rollout until after the summer. This of course meant that I had to spend some time to familiarise myself with all of the updates, as it was mid-June when I last dealt with all of this. Unfortunately, when I came to log into Stirling University’s VPN I was notified that my account had been blocked. The B&B server is hosted at Stirling and I can’t access it without logging into the VPN so I was unable to proceed with the rollout. I remembered from the last time this happened that I can’t simply contact Stirling’s IT people and ask for my account to be unblocked – this has to be approved by a head of subject or something like that. Project PI Katie began this process on Monday but unfortunately I had still not been given access by the end of the week.
However, I did continue to work on some of the other outstanding B&B issues on my local PC. I’d noticed that the version number in the footer of the updated system wasn’t getting pulled in when the site is loaded into a browser for the first time and the first page to be loaded is a standard WordPress page (e.g. the site’s homepage). This was because the version number wasn’t getting stored as a session variable until after the page had loaded and I therefore updated the code to add a check for this.
I also investigated the issue of authors that have the same name not being differentiated in the author filter option in the search results. For example, there are two John Lelands – John Leland (Male, born 1691, died 1766) and John Leland (Male, born c.1506, died 1552) but the filter option only lists one John Leland so it’s not possible to focus on one or the other. This is because the filter uses the Solr field ‘afullnames’ to construct the list, and this simply joins an author’s forename and surname together to make the field. I’ll need to update the script that generates the Solr data to deal with this. The simplest way would be to just add dates, or possibly just birth dates for every author, but this would clutter the filter up unnecessarily. An alternative would be for the script to only add such dates where it finds multiple authors with the same name, and in such cases it would then add dates. So for example ‘John Leland, b.1691’ and ‘John Leland, b.c.1506’. I’ll need to do a full check to be sure this will cover all authors – I guess there may be some that don’t have birth years.
Whilst investigating this I ran a query to find all active authors in the system that have the same name as one (or more) other authors and there are 252, which seemed quite high to me and suggests there may be some unnecessary duplicates in the system. I exported these authors to a spreadsheet and one of the project team is going to look into this. I can’t imagine there are two different authors called ‘Cornelius a Lapide’, for example!
Project Co-I Matt had also spotted an issue with apostrophes in the filter options. The apostrophe in the author ‘Pierre Mathurin de L’Ecluse des Loges’ was causing the system to break when the name was selected in the filter options on the search results page. However, other apostrophes work perfectly. If you look at the search results for all authors with the forename ‘Pierre’ (https://borrowing.stir.ac.uk/search/0/0/0/simple/afname|pierre) the “de l’Ecluse” one is (at the time of writing) broken, but lower down we have “Pierre d’ Hancarville” and this works. Both names appear to have ‘straight’ rather than ‘curly’ apostrophes, but looking at the data, our broken Pierre’s apostrophe has actually been encoded and is stored in the system as “'” and it is this that’s breaking things. I’m not sure how this encoded form got into the system – possibly by pasting into the CMS from another website . I checked the database and there are four author surnames that have this encoding, plus one forename. I’ll fix this when I regenerate the data as part of the new version update. I also checked the other data and there are no borrowers with this encoding. However, there are 36 book works and 63 book editions that have this encoding so I’ll fix them too.
This issue got me thinking about apostrophes in the site. ‘Straight’ and ‘curly’ apostrophes are different characters and both appear in the data (see for example “Pierre Joseph d’ Orléans” in the list of Pierres, which has the ‘curly’ form). This doesn’t matter for browsing and filtering, but if the user enters an apostrophe into a search box they will only find results for the apostrophe form they entered, which will mostly be ‘straight’, but I believe Apple can make apostrophes users enter into boxes ‘curly’ on their devices. And of course there are two different ‘curly’ apostrophes – opening and closing. This is not a major issue as I can’t imagine many people will perform a search that includes an apostrophe, but I raised it with the team and we agreed that all apostrophes (and double quotes) in the data should be changed to their ‘straight’ forms, which I’ll also do once I regenerate the data.
Without having access to the server I didn’t want to continue with any further B&B work as I didn’t want to risk making updates locally and then losing track of them before I could replicate them online. I also need the current data from the server and I can’t access this yet either. I therefore moved onto other projects instead.
On Tuesday I had a meeting with Garrick Allen to discuss a new AHRC proposal he’s putting together. I can’t say much about it here, but it was a great meeting and I’ll be involved with the project if it receives funding. I spent a bit of time after the meeting writing up my notes and researching some related matters. I also had an email chat with Craig Lamont about a proposal he’s putting together that I will also be involved with.
Also this week I spent some time making further updates to the Robert Fergusson online exhibition, adding in all missing images, captions and text. The online exhibition is now publicly available and can be viewed here: https://robert-fergusson.glasgow.ac.uk/online-exhibition/.
I also provided some costings and wrote a detailed description of the place-names database tables and columns for Mícheál Ó Mainnín at QUB, who I’m hoping to work on the place-names of Armagh project with. For the DSL I had an email conversation about the new part of speech search results filtering option that I will be developing, I responded to a query about online map development, and I fixed the old Emblems websites that had stopped working due to a server upgrade.
I spent the remainder of the week returning to work on the Anglo-Norman Dictionary. The editor Geert informed me that a search for multiple words was giving a ‘403 Forbidden’ error. The AND’s various search systems don’t actually work with phrases and are all based on individual words, so a search with spaces is never going to yield meaningful results, but such a scary message is not idea. I managed to fix the issue by updating the .htaccess file to process spaces. I also spent some time investigating a strange issue Geert had experienced with cross-references, but despite my best efforts I was unable to replicate it, and neither was Geert. We’re just going to have to keep a look out for this and figure out what circumstance let to it happening.
I also updated the ‘view entry’ page in the content management system to add in buttons to download the cross-referenced entries as a ZIP file. This works for the ‘from’, ‘to’ and ‘view entries that include a cross reference in their XML’ lists, finding the matching entries, accessing their XML files, adding them to a ZIP archive and then presenting it as a download to the user. It all works very well.
I continued to work on the Online Exhibition for the Robert Fergusson project this week, spending much of Monday and Tuesday on the task and completing an initial version that I sent to the team for feedback on Tuesday afternoon. As I worked with the materials and got to know them a bit better I ended up creating a more fully-featured first draft than I was originally intending to make. It is still an initial version, and any aspects can be changed as required. I took inspiration from the Books and Borrowing online exhibition to a certain extent. As with this site, the pages all have an ‘Explore’ section with image-based buttons leading to the various sections, and each page has links to the next and previous sections at the bottom.
I can’t share screenshots or the URL just yet, but the interface is designed to work on all screen dimensions from large monitors to mobile phones and as you scroll down the pages sections animate into place. There are currently two alternative animations. Pages such as ‘Publications’ have a ‘zoom’ animation whereby the section zooms and slides up into place as you scroll. The alternative can be viewed on pages such as ‘Depictions’ where sections are already visible but the content slides in from the left and right as you scroll (except the final video section of ‘Depictions’ which zooms up). Certain pages have the first section fixed (e.g. ‘Publications’) while other pages animate the first section too (e.g. ‘Scots poems’). We might not want to mix and match the approaches so much – I included alternatives so the team can see the options.
The pages currently consist of sections with three different background colours that are alternated (a greeny-grey, a dark grey and a sort of grey-teal colour). The header font is the same one as the main site and the paragraph text is fairly large. Where a section has one image this appears to the left or right (this alternates) in a box that also features the caption. Where there are multiple images (e.g. ‘poems on various subjects’ on the ‘publications’ page) a carousel is used and you can press the arrow buttons to scroll between images. In such cases there is still currently one single caption underneath the carousel.
On the ‘depictions’ page I’ve embedded the YouTube video in the final section. On the ‘Scots poems’ page I’ve included the three audio recordings in a final section with some placeholder text. Where images appear, I’ve applied some post-processing on many of them to remove the yellowing and make them brighter. I think this makes the images look nicer as part of an online exhibition, but the team may prefer to use the originals for the sake of authenticity. I’m pretty happy with how the initial version of the exhibition has turned out. During the rest of the week I had an email conversation with the RA Amy Wilcockson about adding in new content or rearranging the content that was already there, and I spent some of my time making such updates, some of which where rather time-consuming to implement but needed to be done.
On Tuesday afternoon I met with Moira and Leo from Archives and Special Collections to discuss some of the old resources I’d created for the archives around 15-20 years ago. It was great to catch up with them both and to hear their plans for the resources. It turns out that the University Story site has been completely lost – no copy of the site and its database remains on any University servers or other hardware. This was quite a shock as a huge amount of effort went into writing the content of this site – thousands of biographies as well as structured data about (for example) holders of professorships throughout history. When I stopped working for the Archives some 13 years ago a lot of work continued to be done to the site, both in terms of the content and reworking by subsequent developers, but I wondered whether I might still have a backup of the site as I left it somewhere at home.
On Wednesday I hunted about for the data. I found my old backup external hard drive in my attic, but unfortunately this did not include anything I hadn’t already discovered on my more recent external hard drive. I then managed to locate the old laptop that I used when working at the Archives back in the day, thankfully complete with power cable. This took ages to boot up, and I couldn’t remember the password I used to use for it. And the laptop was taking more than half an hour between entering a password and coming up with the ‘incorrect password’ warning. I then decided to set up a ‘boot from USB’ installation of Ubuntu to run on the laptop instead of booting the ancient version of Windows and this worked perfectly. I was able to access the laptop’s hard drive and copy all of the files relating to my Archives work onto my external hard drive. This included one single dump of the UGS database from June 2011, plus all of the code, images, and a bunch of other stuff relating to other projects. I now have this on my desktop PC as well as on the external hard drive. I loaded the SQL dump into a MySQL instance on my new laptop and it looks like the data is complete (albeit from 2011). It consists of 93 tables, incorporating the main UGS site, the international story, the ‘world changing’ site and the WW1 roll of honour. There are records for 19,228 people and of these 1,799 have biographies. There are 1,111 image records and these link to the image files contained in the ‘images’ directory in the code. There are three derivatives per image, which is why there are so many image files compared to image records.
The code contains the code for the site and the CMS, as it was in 2011. I did try getting the site to run on my laptop, but it would take a lot of work to get it operational due to changes in technology and also because it references stylesheets from the main University website from 2011. An awful lot was added to the site after June 2011, but hopefully having the database structure and the data up to this point will be a useful starting point. Stats on the homepage from the Internet Archive snapshot from April 2018 state that there are 2390 images and 3678 biographies on the site, which is a lot more than we had in 2011.
If it’s possible to request a copy of the UGS data from the Internet Archive the Archives people could then write a script to compare the people IDs in the 2011 database with IDs embedded in the URLs of pages from the Internet Archive (e.g. ‘WH10083’ from https://web.archive.org/web/20150908140047/http://www.universitystory.gla.ac.uk/biography/?id=WH10083&type=P&o=&start=0&max=20&l=a) to then make a list of all of the people that are not found in the 2011 database. The same could be done with images (e.g. ‘UGSP01400’ in the URL https://web.archive.org/web/20130704220509/http://www.universitystory.gla.ac.uk/image/?id=UGSP01400&o=&start=0&max=20&l=A&biog=WH10083&type=P&p=2). Of course there’s no guarantee that the Internet Archive captured all of the data for every person and image, but this would at least enable the archives to create a more limited list of people and images that could then be added to the 2011 data, rather than having to process all of the Internet Archive data from scratch.
The archives people could then write a script that could extract data only from the people / image pages in the list and insert this into the database. The pages are pretty well structured so it might not be too tricky to write such a script. Of course any edits made to the existing data since 2011 wouldn’t be spotted by this approach, as only new records would be targeted.
I wouldn’t recommend using the old code for the site and CMS, which is all very outdated now. The archives would be better off building a new system around the database structure, referencing the Internet Archive version of the old site to help figure out how things used to work. They might also want to rationalise the database structure as it’s possible not all tables are actually needed, or could be structured more efficiently.
Also this week I had an informal meeting with Marc Alexander to discuss my work, and I read through an edited version of the upcoming Books and Borrowing article to make sure the edits all made sense.
For the rest of the week I looked into a task that has been on my long-term ‘to do’ list for ages: Redeveloping the Fife Place-names website (https://fife-placenames.glasgow.ac.uk/). I made this as a proof of concept many years ago, having written scripts to extract and structure the data from Word documents. What I want to do is migrate the data to the system I created for the other place-names resources, which would then mean I could create a map interface for Fife that would be comparable to the Berwickshire one, for example (see https://berwickshire-placenames.glasgow.ac.uk/map/).
The first task was to attempt to generate latitude, longitude and altitude data for Fife, as currently the data only has grid references. I wrote a script that would do this, thinking I’d just be able to leave it running. Unfortunately when I ran it I realised that many of the grid references have problems. Many records have malformed grid references, such as ones with an additional number, or a zero character instead of an ‘O’, while for some others the data extraction script did not successfully grab the grid reference. In many such cases I had to do research to discover the actual grid reference using a combination of Google Maps, the historical maps on the Berwickshire website, the UK Grid Reference Finder (https://gridreferencefinder.com/) and the NLS GB1900 website (https://maps.nls.uk/projects/os1900). As you can imagine this took quite a lot of time, and I hadn’t managed to complete the georeferencing by the end of the week.
I’m on holiday for the next week and a bit, and I’ll return to this task once I’m back.