Week Beginning 8th May 2023

This was a three-day week for me as Monday was the coronation holiday and Tuesday was (thankfully) my final day of jury duty.  I spent some more time this week working on the migration of websites to external hosting and fixing a number of issues that have arisen during this process.  I also set up a website for Ophira Gamliel’s new project including engaging in an email discussion with the two project Co-Is about the interface and design.  I also made some further updates to the teaching version of the SpeechStar website, which has now launched (https://www.seeingspeech.ac.uk/speechstar/).  There are still some things to do for the website, such as adding in citation options for videos and adding more descriptive metadata to some of the videos, but the majority of content is now available.

I also investigated an issue the editor of the Anglo-Norman Dictionary was encountering with the proof-reader feature of the dictionary management system I’ve created.  This feature allows a ZIP file containing any number of dictionary entry XML files to be uploaded and these are then formatted in the same was a ‘live’ dictionary entries and displayed on one long page that can then be downloaded or copied into Word.  When Geert, the editor, was uploading a ZIP file the page was encountering an error and displaying a blank screen.

After some investigation I discovered this was an issue with the data.  The proof-reader is not an XML validator and expects valid XML files.  When it encounters a malformed XML file the proof-reader breaks.  It turned out there was one problematic XML file which includes a malformed tag:

<lemma>tresaiele/lemma>

This was causing a fatal error in the script, resulting in a blank page.  What I have done now is to update the proof-reader so it can handle problems with an XML file.  In such cases the file is now skipped over and a warning is displayed at the top of the page, such as  “WARNING: Malformed XML file skipped: test/tresaile.xml” while all other files get processed correctly.  This is a much better approach and will be very helpful in future.

I spent the rest of my week working on the Books and Borrowing project.  I sorted out the page numbers for one of the new registers I added to the system last week (the other one didn’t need sorting).  Page numbering was a real mess in this one and it’s taken quite some time to get our page numbers aligned with what is written in the images.

I also spent some time investigating a couple of issues that Matt had reported.  The first one was empty, unlabelled book items sometimes appearing in book holding records.  I think there is a bug in the CMS that can result in a blank book item getting created in certain circumstances.  As we’re sort of nearing the end of the project and debugging the CMS would likely be a long and painful process I pointed out that I would rather run a script to strip out all unlabelled book items that have no associated borrowings as a quick fix instead.  We could then run this script multiple times as required.  I ran a query that identified all unlabelled book items that have no associated borrowings, and there are about 2,400 of them.  I could then run a script to delete all of these items, which would remove the various ‘Volume (0 borrowings)’ from the book holding records.  What it won’t do is fix the unlabelled volumes that do have borrowings (e.g. ‘Volume (1 borrowing)’ items).  However, I could run another script to output a list of all book items that have no label but have at least one borrowing if that might help to fix these.

The second issue Matt pointed out is that the number of borrowing records listed for book holdings in the overall list of book editions does not match up with the number of search results found when you perform a search for the holding.  This is because the counts on the list of books page are the total number of times each book item for each book holding have been borrowed (so in Matt’s example adding up all of the times each book item was borrowed gives 159).  The search results display a count of borrowing records, and a single borrowing record can include multiple book items.  For example one borrowing record may involve three book items but only counts as one borrowing record.  In Matt’s example the 108 search results are borrowing records, not book items.

This is a bit confusing and I wonder what we could do to clarify this.   I did wonder whether the search results could also tally up the number of book items in the results and display this alongside the ‘your search matched x borrowing records’, but unfortunately we only have this data for a subset of results (up to 100 on each page of the results).  Perhaps we need to add some explanatory text somewhere, or perhaps the number of borrowings on the ‘Books’ page should also count the number of associated borrowing records rather than the total number of times each item was borrowed.  Or I could update the Solr index to cache the total number of book items for each borrowing record.

I also updated the overall list of borrowers in the front-end, adding in a ‘limit by library’ option.  I had hoped that this would only take a couple of hours to implement but it ended up being rather complicated to get working as I needed to update several API endpoints to incorporate the limit by library, update the way the URLs work on the borrowers page and update the JavaScript that processes the changing of the view and the selection of filter items, in addition to incorporating the list of libraries into the page.  However, I manage to get the feature completed by the end of the week.

One thing to note is that there is often a discrepancy between the number of borrowers listed in the occupations section and the number in the tabs.  For example, when limiting the list to female borrowers with an ‘Arts and Letters’ occupation the ‘Arts and Letters’ occupation shows 7 borrowers but if you count the borrowers in the tabs there are only 5.  The reason is a borrower can have multiple occupations.  For example, Anne Grant is both poet and author so counts as one borrower in the ‘G’ tab but one each for ‘Poet’ and ‘Author’ in the occupation counts.  I also wonder whether I should add in counts of borrowers at each library to the ‘limit by library’ page.  Something to consider, anyway.

I also encountered some weirdness with the site today – specifically any searches for things with spaces in them (e.g. occupation ‘Arts and Letters’) just displayed a 403 Forbidden page.  This was before I’d made any updates to the code so it wasn’t that I’d inadvertently broken something.  It turned out that a recent update to the Apache server software was breaking any URL that had a space in it.  If you really want to find out more you can read here:

https://stackoverflow.com/questions/75684314/ah10411-error-managing-spaces-and-20-in-apache-mod-rewrite

I had to spend some time figuring out how this change has affected the site and making updates to avoid the 403 error pages.  I think I’ve sorted most things out when using the front-end, although accessing the API directly is still causing problems.  I’ll need to sort this out next week.  I’ll also deal with the Hunterian images, which it turns out had not yet been added to the system, and will hopefully move on to the ‘Facts and Figures’ pages and their visualisations next week too.