For the Books and Borrowing project this week I completed an initial version of the site-wide facts and figures page.  The page functions in a very similar way to the library-specific facts and figures page as discussed in previous weeks, but here you can view the data for any or all libraries rather than just a single one.  Libraries are selectable from a series of checkboxes as in the advanced search, allowing you to group together and compare library data, for example the data for the three University libraries or data from libraries in southern Scotland.  However, the page can take about 30 seconds to load as it’s processing an awful lot of data for the default view (which is data for every library).  I’m going to create a cached version of the data for this particular view and possible others, but I haven’t got round to it yet.

The ‘Summary’ section provides an overview of the amalgamated data for your chosen libraries and the ‘top ten lists’ are also amalgamated.  The book holding and prolific borrowers lists include the relevant library in square brackets after the number of associated borrowing records as these lists contain items that are related to specific libraries (borrowers and book holdings ‘belong’ to a specific library).  I’ve also added in the top ten borrowed book editions, as these are site-wide, as are authors.  The links through to the search results for these lists include your selected libraries to ensure the search results match up.  For example, Sir Walter Scott is the most borrowed author when viewing data for Dumfries, Selkirk and Wigtown and when you press on the author’s name you search is limited to these libraries.  The occupations and borrowings through time visualisations contain data for all selected libraries and rather than a ‘book holding frequency of borrowing’ chart at the bottom of the page there is a ‘book edition’ version as editions are site-wide.  As with other data, the editions and borrowings over time returned here are limited to your chosen libraries and links through to the search results also incorporate these.

I still need to address a couple of things, though.  Firstly, book edition titles have not currently been cut off after 50 characters as I realised doing so for titles with non-Latin characters (e.g. Greek) broke the page.  This is because characters such as Greek take up multiple bytes while Latin characters take up one byte each.  The default substring method for chopping up strings is not ‘multibyte safe’, meaning non-Latin characters can get split in the middle of the data, which results in an error.  There is a handy multibyte version of the substring method but unfortunately multibyte functions have not been installed on the server so I can’t use it.  Once I’ve managed to get this installed I’ll add in the limit.  Secondly, I’ve noticed that the visualisations don’t work very well on touchscreens so I’m going to have to rework them to try and improve matters.  For example, the donut charts allow you to click on a section to perform a search for the section’s occupation, but as touchscreens have no hover state this means on a touchscreen it’s not possible to view the tooltip that includes the name of the occupation and the exact percentage.  Also, the click through to a year on the ‘borrowings through time’ chart don’t seem to work on my iPad and the book edition autocomplete doesn’t seem to be firing either.

Also for the project this week I assigned some borrowing records for a borrower in Wigtown to another borrower, as it turned out these were the same person.  This also involved updating the borrower’s total number of borrowings and active borrowing period.  I then began to look into migrating the website’s theme to Bootstrap.  I’m going to be working on this on a local version of the site running on my laptop and I began getting things set up.

Also this week I uploaded a new set of videos and an updated version of the database for the Child Speech Error Database that as part of the Speech Star project and was involved in the migration of Seeing Speech and Dynamic Dialects to a new server.  I was also involved in the migration of two further WordPress sites to our external provider and sorted out some issues with a couple of other WordPress sites that had been moved to our new in-house server.

I also spent a bit of time working for the DSL, engaging in an email conversation about how the new part of speech search will work and replacing the ‘Share’ buttons on the entry page with an alternative as the ‘AddThis’ service that we previously used has been shut down by parent company Oracle (See https://www.addtoany.com/blog/replace-addthis-with-addtoany-the-addthis-alternative/).  This involved investigating an issue with one member of staff who was just seeing a blank space rather than the new options.  It turned out the Chrome plugin ‘DuckDuckGo privacy essentials’ was blocking the service so we’ll have to watch out for this.

Also this week I spent a bit of time working for the Anglo-Norman Dictionary.  I had an email discussion about collaborative XML tools with one of the project team and we will have a Zoom call next week to go over this in more detail.  I also wrote a little script to export a set of XML files from the live database, in this case all entries beginning with the letter ’U’.  After doing this the editor happened to notice that the ‘language’ tags seem to have disappeared from entries.

This then led onto some detailed investigation into this very worrying situation.  It would appear that some process that modified the XML has at some point removed the language tag.  This appears to have happened up to and including the import of R, but whatever was causing the issue had been rectified by the time we imported S.  Looking at specific examples I noticed that the language tag was present in the XML for ‘rother’ prior to upload, but is not present in the XML in the online database.  On a version of the data on my local PC I deleted ‘rother’ and ran it through the upload process that was used for ‘S’ and the language tag remained intact.  I can’t be certain but I presume the error was caused by the script that assigns IDs and numbers to <sense> and <senseInfo> tags as it is this script that edits the XML.  We currently have 493 entries that feature a language tag.  Heather asked me to export counts of languages found in entries in March last year and there were a similar number then.  We still have a column holding the initial data that was first imported into the new system and in the current data there are 1759 ‘active’ entries that feature a language tag.  This does not include any R data as it was added after this point.  Looking back through old versions of the database, the version of the data I have from 26/11/20 does not have the ‘old’ column and features 1867 entries with language tags.  The import of R must have happened sometime shortly after I created this version.  The version of the data I have from 03/26/21 features the R data and also is the first version I have with the ‘old’ column.  In this version the live data now only has 47 entries with language tags while ‘old’ has 1868 entries with the language tag.

In order to fix this issue I’m going to have to somehow write a script to look at ‘old’ for language tags then attempt to extract these.  I’ll also need to do this for the R data too, directly from the XML files that I was sent prior to upload.  Thankfully after further discussion with Geert it turns out that he thinks the language tag should be an ‘entry’ level rather than ‘sense’ / ‘subsense’ level.  Moving the language tag out of senses and placing it at entry level should actually make the data retrieval much more straightforward.  The biggest issue would have been figuring out which sense each language tag needs to be added to, as the old data does not have sense IDs or any other easy way of identifying which sense is which.  If instead a script just needs to pull out all language tags in the entry’s senses / subsenses and then add them to a new section directly in the entry that should be much simpler.

I tested out the DMS (downloading and then uploading the entry ‘slepewrt’ which features a language tag) and it is retaining the tag, so it is thankfully safe to continue to use the DMS.  Next week I’ll begin working on a script for reinstating the missing tags.