Week Beginning 22nd January 2024

I’d hoped that we’d be able to go live with the full Books and Borrowing site this week, but unfortunately there are still some final tweaks to be made so we’re not live yet.  I did manage to get everything ready to go, but then some final checking by the team uncovered some additional issues that needed sorting.

Migrating all of the pages in the development site to their final URLs proved to be quite challenging as it was not simply a matter of copying the pages.  Instead I needed to amalgamate three different sets of scripts and stylesheets (the WordPress site, the dev site and the Chambers map), which involved some rewriting and quite a lot of checking.  However, by the end of Monday I had completed the process and I sent the URLs to the team for a final round of checking.

The team uncovered a few issues that I managed to address, including some problems with the encoding of ampersands and books and borrowers without associated dates not getting returned in the site-wide browse options.  I needed to regenerate the cache files after sorting some of these, which also took a bit of time.

I also realised that the default view of the ‘Browse books’ page was taking fat too long to load, with load times of more than 20 seconds.  I therefore decided to create a cache for the standard view of books, with separate cache files for books by first letter of their title and their author.  These caches would then be used whenever such lists were requested rather than querying everything each time.  When filters are applied (e.g. date ranges, genres) the cache would be ignored, but such filtered views generally bring back less books and are returned quicker anyway.  It took a while to write a script to generate the cache, to update the API to use the cache and then add the cache generation script to the documentation, but once all was in place the page was much quicker to load.  It does still take a few seconds to load in as there are almost 3000 books beginning with the letter ‘A’ in our system and even processing a static file containing this much data takes time.  But it’s a marked improvement.

On Friday afternoon the project PI Katie Halsey got back to me with a list of further updates that I need to make before the launch, one of which will require the Solr index structure to be updated and all of the Solr data to be regenerated, which I’ll tackle next week.  Hopefully after that we’ll finally be able to launch the site.

Also this week I fixed a small bug that had been introduced to the DSL sites when we migrated to a new server before Christmas.  This was causing an error when the ‘clear results’ option was selected when viewing the entry page and I’ve sorted it now.  I also investigated an issue then including URLs in the description of place-names on the Ayrshire place-names site.  It turned out that relative URLs were getting blocked as a security risk, but a switch to absolute URLs sorted this.

I also made some fixes to some of the data stored in the Speech Star databases that had been uploaded with certain features missing and provided some CV-like information about the projects I’ve worked on for Clara Cohen in English Language to include in a proposal I’m involved with.

I then investigated a couple of issues with the Anglo-Norman Dictionary.  The first was that the Editor Geert needed a new ‘usage label’ to the XML for the site, which would then need to be searchable in the front-end.  After spending some time refamiliarising myself with how things worked I realised it should be very easy to add new labels.  All I needed to do was to add the label to the database and then it was automatically picked up by the DTD and given as an option in Oxygen.  Then when an entry is uploaded to our dictionary management system it will be processed and should (hopefully) automatically appear in the advanced search label options (once it has at least one active association to an entry).  Geert has begun using the new label and we’ll see if the process works as I think it will.

The second issue was one that Geert had noticed in the Dictionary Management System when manually editing the citations.  The ‘Edit citations’ feature allows you to select a source and then view every time this source is cited in all of the entries.  The list then lets you manually edit each of these.  The issue was cropping up when an entry featured more than one citation for a source and more than one of these needed to be edited.  In such cases some of the edits were being lost.  The problem was that when multiple citations were being edited at the same time the code was extracting the XML from the database and editing it afresh for each one, meaning only the changes for the last were being stored.  I’ve sorted this now.

I also spent some time continuing to develop the new place-names map for the Iona project.  I updated the elements glossary to add a new hidden column to the data that stores the plain alphanumeric value of the name (i.e. all weird characters removed).  This is now used for ordering purposes, meaning any elements beginning with a character such as an asterisk or a bracket now appear in the correct place.

I also began working on the ‘Table view’ and for now I’ve set this to appear in a pop-up, as with the place-name record.  Pressing on the ‘Table view’ button brings up the table, which contains all of the data currently displayed on the map.  You can reorder the table by any of the headings by pressing on them (press a second time to reverse the order).  Below is a screenshot demonstrating how this currently looks:

You can also press on a place-name to close the table and open the place-name’s full record.  The map also pans and zooms to the relevant position.  What I haven’t done yet is managed to get the marker pop-up to also appear, so it’s currently still rather difficult to tell which marker is the one you’re interested in.  I’m working on this, but it’s surprisingly difficult to achieve due to the leaflet mapping library not allowing you to assign IDs to markers.

Using the pop-up for the table is good in some ways as it keeps everything within the map interface, but it does have some issues.  I’ve had to make the pop-up wider to accommodate the table, so it’s not going to work great on mobile phones.  Also when the table is very long and you try to use the browser’s scrollbar the pop-up closes, which is pretty annoying.  I think I might have found a solution to this (adding a scrollbar to the pop-up content itself), but I haven’t had time to implement it yet.  An alternative may be to just have the table opening in a new page, but it seems a shame to not have everything contained in the map.

Finally this week I spent some time working through the poems on the old ‘Anthology of 16th and Early 17th Century Scots Poetry’ site.  I created a new interface for this last week (which is not yet live) and decided it would make sense to migrate the poems from ancient HTML to TEI XML.  This week I began this process, manually converting the firs 13 poems (out of about 130) to XML.  With these in place I then worked on an XSLT stylesheet that would transform this XML into the required HTML.  It was here that I ran into an issue.  I absolutely hate working with XSLT – it’s really old fashioned and brittle and cumbersome to use.  Things I can literally achieve in seconds for HTML using jQuery can take half a day of struggling with XSLT.  This time I couldn’t even get the XSLT script to output anything.  I was using similar code to that which I’d created for the DSL and AND, but nothing was happening.  I spent hours trying to get the XSLT to match any element in the XML but nothing I tried worked.  Eventually I asked Luca, who has had more experience with XML than I have had, for advice and thankfully he managed to identify the issue.  I hadn’t included a namespace for TEI in the XSLT file.  I needed to include the following reference:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:tei="http://www.tei-c.org/ns/1.0">

And then each time I wanted to match an XML element such as <text> I couldn’t just use ‘text’ but instead needed to add a tei prefix ‘tei:text’.  Without this the XSLT wouldn’t do anything.  Yep, XSLT is brittle and cumbersome and an absolute pain to use.  But with Luca’s help I managed to get it working and other than transcribing the remaining texts (a big job I’ll tackle bit by bit over many weeks) all is pretty much in place.