I spent most of this week working towards adding genre to the Books and Borrowing front-end, working on a version running on my laptop. My initial task was to update the Solr index to add in additional fields for genre. With the new fields added I then had to update my script that generates the data for Solr to incorporate the fields. The Solr index is of borrowing records so as with authors, I needed to extract all genre associations at all book levels (work, edition, holding, item) for each book that was associated with a borrowing record, ensuring lower level associations replaced any higher level associations and removing any duplicates. This is all academic for now as all genre associations are at Work level, but this may not always be the case. It took a few attempts to get the data just right (e.g. after one export I realised it would be good to have genre IDs in the index as well as their names) and each run-through took about an hour or so to process, but all is looking good now. I’ll need to ask Stirling IT to create a new Solr core and ingest the new data on the server at Stirling as this is not something I have the access to do myself, and I’ll do this next week. The screenshot below shows one of the records in Solr with the new genre fields present.
With Solr updated I then began updating the front-end, in a version of the site running on my laptop. This required making significant updates to the API that generates all of the data for the front-end by connecting to both Solr and the database as well as updating the actual output to ensure genre is displayed. I updated the Advanced Search forms (simple and advanced) to add in a list of genres from which you can select any you’re interested in (see the following two screenshots) and updated the search facilities to enable the selected genres to be searched, either on their own or in combination with the other search options.
On the search results page any genres associated with a matching record are displayed, with associations at higher book levels cascading down to lower book levels (unless the lower book level has its own genre records). Genres appear in the records as clickable items, allowing you to perform a search for a genre you’re interested in by clicking on it. I’ve also added in genre as a filter option down the left of the results page. Any genres present in the results are listed, together with a count of the number of associated records, and you can filter the results by pressing on a genre, as you can see in the following screenshot, which shows the results of a quick search for ‘Egypt’, displaying the genre filter options and showing the appearance of genre in the records.
Genre is displayed in a similar way wherever book records appear elsewhere in the site, for example the lists of books for a library, the top-level ‘book editions’ page and when viewing a specific page in a library register.
There is still more to be done with genre, which I’ll continue with next week. This includes adding in new visualisations for genre, adding in new ‘facts and figures’ relating to genre and adding in facilities to limit the ‘browse books’ pages to specific genres. I’ll keep you posted next week.
I also spent some time going through the API and front-end fixing any notifications and warnings given by the PHP scripting language. These are not errors as such, just messages that PHP logs when it thinks there might be an issue, for example if a variable is referenced without it being explicitly instantiated first. These messages get added to a log file and are never publicly displayed (unless the server is set to display them) but it’s better to address them to avoid cluttering up the log files so I’ve (hopefully) sorted them all now. Also for the project this week I generated a list of all book editions that currently have no associated book work. There are currently 2474 of these and they will need to be investigated by the team.
I also met with Luca Guariento and Stevie Barret to have a catch-up and also to compile a list of key responsibilities for a server administrator who would manage the Arts servers. We discovered this week that Arts IT Support is no longer continuing, with all support being moved to central IT Services. We still have our own servers and require someone to manage them so hopefully our list will be taken into consideration and we will be kept informed of any future developments.
Also this week I created a new blog for a project Gavin Miller is setting up, fixed an issue that took down every dictionary entry in the Anglo-Norman Dictionary (caused by one of the project staff adding an invalid ID to the system) and completed the migration of the old Arts server to our third-party supplier.
I also investigated an issue with the Place-names of Mull and Ulva CMS that was causing source details to be wiped. The script that populates the source fields when an existing source is selected from the autocomplete list was failing to load in data. This meant that all other fields for the source were left blank, so when the ‘Add’ button was pressed the script assumed the user wanted all of the other fields to be blank and therefore wiped them. This situation was only happening very infrequently and what I reckon happened is that the data for the source that failed included a character that is not permitted in JSON data (maybe a double quote or a tab), meaning when the script tried to grab the data it failed to parse it and silently failed to populate the required fields. I therefore updated the script that returns the source fields so that double quotes and tab characters are stripped out of the fields before the data is returned. I also created a script based on this that outputs all sources as JSON data to check for errors and thankfully the output is valid JSON.
I also made a couple of minor tweaks to the Dictionaries of the Scots Language site, fixing an issue with the display of the advanced search results that had been introduced when I updated the code prior to the site’s recent migration to a new server and updating the wording of the ‘About this entry’ box. I also had an email conversation with Craig Lamont about a potential new project and spoke to Clara Cohen about a project she’s putting together.
This week I began the major task of integrating book genre with the Books and Borrowing dataset. The team had been working on a spreadsheet that enabled them to assign top-level Book Work records to more than 13,000 Book Edition records and also assign up to three genres to each Work. I had to write a script to parse this data, which involved extracting and storing the distinct genres, creating Book Work records, assigning Book Work authors, adding in associations to Book Edition records, deleting any author associations at Edition level and creating associations between Works and genres. It took the best part of two days to create and test the script, running it on a local version of the data stored on my laptop. After final testing the number of active Book Works increased from 75 to 9808 and the number of active Book Editions that have a Work association grew from 72 to 13,099. The number of genre connections for Works stood at 11,536 and the number of active Book Works that have at least one author association stood at 9,808, up from 70, while the number of active Book Editions with at least one direct author association decreased to 2,191 from 14,384, due to the author association being shifted up to Work (and it will cascade from there).
With the data import sorted I then moved onto updating the project’s content management system to incorporate all facilities to add, edit, browse and delete genres. This included creating facilities for associating genres with book records at any level (from Work down to Item) wherever books can be edited in the CMS. The ‘Browse Genres’ page works in a similar way to ‘Browse Authors’, giving you a list of genres and a count of the number of each book at each level that has an association, as the following screenshot shows:
Pressing on a number opens a pop-up containing a list of the associated books and you can connect through to each book record from this. As with authors, genre will cascade down from whichever level of book it is associated with to all lower levels. You only need to make an association at a lower level if it differs from the genre at a higher level. The counts in the ‘browse’ page show only the direct associations, so for now there are no editions or lower with any numbers listed. Wherever a book at any level can be edited in the CMS a new ‘Genre’ section has been added to the edit form. This consists of a list of genres with checkboxes beside them, as the following screenshot demonstrates:
You can tick as many checkboxes as are required and when updating the record the changes will be made. I tested out the new genre features in the CMS and all seem to be working well. I also imported all of the genre data so hopefully everything is now in place. Next week I will move onto the front-end, where there is much to do – not only making genre visible wherever books are viewed but updating the search facilities and adding in a number of new visualisations for genre as well. I also fixed a few issues with images of Registers from the Royal High School – a few that were missing I added in and the order of others needed to be updated.
Also this week I finalised the new project website for Rhona Brown’s new project. It’s not live yet, but my work on it is now complete. I was also involved in the migration of a number of my sites to a new server. As always seems to be the case, the DSL website migration did not go very smoothly, with the DNS update taking a many hours to propagate and in the meantime the domain was serving up the Anglo-Norman Dictionary, which was not good at all. This wasn’t something I had any direct control over, unfortunately, but thankfully the situation rectified itself the following day.
I also had to make a number of tweaks to the data in the Child speech error database for Speech Star, after many transcriptions were revised. I also updated the Mull / Ulva place-names CMS to add in a facility to export place-names for publication limited by one or more selected islands. In addition I began creating a new website for a project Gavin Miller is running and I created some new flat spreadsheet exports of the Historical Thesaurus for Fraser Dallachy and Marc Alexander to work with.
I spent the first two days of this week finishing off checking my locally hosted sites were compatible with PHP8, upgrading the version of jQuery they use and dealing with any warnings and notifications thrown by PHP. The dictionary sites (Anglo-Norman and the Dictionaries of the Scots Language) were especially time consuming to deal with, and the DSL sites needed to be upgraded from a very early version of jQuery and jQuery UI which was a bit of a nightmare. However, I’d completed all of the required work on Tuesday, which was quite a relief as it’s been a long and tedious task. I discovered later in the week that the server these sites is hosted on, which we purchased less than two years ago is now going to be decommissioned, and all of the sites will be moved to an entirely different server. This was a bit of a shock, and will require yet another round of migration and dealing with issues. I’m not directly involved with most of this, but I did have to help out with setting up Solr cores and important data into Solr on the new server on Friday this week, which took some time to sort out when I was really needing to focus on other things, which was rather frustrating.
I spent most of the rest of the week continuing to develop the new map interface for the Iona place-names project, working on the Advanced Search, which is accessed by pressing the ‘Advanced search’ button in the ‘Search’ section of the map’s left-hand menu. This opens up a pop-up that contains all of the fields as found in previous place-names projects (e.g. https://berwickshire-placenames.glasgow.ac.uk/place-names/?p=search), but in a more compact layout. This pop-up is an ‘in map’ pop-up meaning that it still works when the map is in full-screen mode. It has been trickier than you might think to implement – it wasn’t just a case of taking the existing form and sticking it in the pop-up as fields such as parishes, codes and element languages are dynamically generated based on the available data and the existing form was generated on the server-side whereas the new form is generated on the client-side (i.e. by code running in the user’s browser). I therefore had to write a new series of scripts to generate the data on the server and make it available to be pulled into the form in the browser. This also applies to the ‘autocomplete’ fields (‘Source’ and ‘Element’), where you can type a few characters in and view a list of matching items.
Having the form in a pop-up in a map also made it tricky to get the tooltips (when you hover-over one of the ‘(?)’ icons) and the autocomplete selection lists working. The webpage is comprised of different elements, some of which sit on top of each other, and the order of these is controlled by something called the ‘z-index’. This tells the browser (for example) that when the pop-up opens, the map layer should be covered by a transparent grey layer and on top of this layer the popup should sit. But the tooltips and autocomplete use a different library that doesn’t expect there to be a map and then a popup sitting over the webpage, meaning these elements were appearing underneath the popup. It took a while to figure out these elements were actually working, but were being hidden.
However, I managed to deal with these issues and the upshot is that the advanced search form works. The following screenshot is an example of the form, with ‘Scottish Gaelic’ chosen as ‘Element Language’ and the autocomplete list visible for ‘Element’ after typing in the characters ‘du’.
The form can be used to search for any combination of the search boxes, but we may want to trim these down (‘Parish’ may not be necessary and I’m not sure ‘Linear’ is relevant, for example). When you press on the ‘Search’ button the pop-up closes and the script generates a search query based on the options you have chosen. All of this code had to be migrated over to the client-side too (converting it from PHP to JavaScript) so this wasn’t a straightforward matter either. The query is then passed to the API and any relevant data is returned and displayed. The ‘Search’ section of the menu is also updated to display a list of the things you searched for and a count of the number of place-names found. The screenshot below shows the results of a search for place-names that have an earliest recorded date of before 1700. This is a full-screen view of the map, set to satellite view with place-name labels visible and one place-name pop-up open.
There are still a number of tweaks I need to make to the search. Most importantly, your search form choices are not yet ‘remembered’ by the system and when you return to the search form any previously entered options are lost. I will sort this, along with the other things previously mentioned such as updating the URL to reflect the map contents, enabling bookmarks and citations, and also ensuring the map ‘remembers’ your display option choices too. I also implemented the ‘Attribution and Copyright’ pop-up this week (press on the link in the very bottom right to view this).
Also this week I sorted an issue that was preventing the bulk download of texts as a ZIP from the SCOTS corpus from working. Thankfully this was a simple permissions issue and was quick to resolve. I also received the data about Genre from Matt Sangster for the Books and Borrowing project and have arranged to focus on implementing this over the coming weeks. This will involve setting up the data structures to store genre data, writing and testing the scripts to import the data then running the scripts, updating the CMS to enable data to be managed, updating the Solr index and the scripts for generating Solr data for the front-end search to incorporate genre and updating the front-end to incorporate genre including the display, search and browse of genre. This is going to be a pretty major job.
I also spent some time adding new videos and metadata for American speakers to the Speech Star database (see https://www.seeingspeech.ac.uk/speechstar/speech-database/), spoke to Gavin Miller about the new project website I’m setting up for him, helped Matthew Creasy with an issue with his James Joyce conference website and created an interface for Rhona Brown’s new project website.
I continued with the new ‘map-first’ front-end for the Iona project this week, adding in an option to reset the map to the default view on the ‘Home’ section adding the scale of the map to the bottom left. I then spent quite a bit of time working on a first version of the ‘Browse’ option. Pressing on the ‘Browse’ accordion header now displays a drop-down list through which you can select what type of browse you want to do. This defaults to ‘Current place-name’ and displays the letters that place-names begin with together with a count of the number of places beginning with each letter. The browse accordion replicates all that is found on the browse pages of the place-names resources I’ve previously developed (e.g. https://berwickshire-placenames.glasgow.ac.uk/place-names/?p=browse), but is much more efficient with the use of space. Pressing on a letter updates the map markers to only show those of places beginning with the letter, as the following screenshot demonstrates (with a pop-up open):
I haven’t implemented the automatic zooming to show all markers yet, and selected display options such as categorisation are not yet remembered, but I will be adding these features in. The other browse options are also all fully functioning, although ‘Parish’ might not be much use as currently every place-name is found in the same parish. ‘Source’ has a lot of data, but the area scrolls so I think it works ok. The screenshot below shows a browse for sources with ‘Admiralty Chart no. 2617’ selected:
I also implemented the quick search this week, which as with the other place-name resources searches current names, historical forms and elements. The screenshot below shows a search for ‘cnoc’, zoomed in on the north of the island with labels turned on:
Note that in addition to pressing the ‘reset map’ button on the ‘Home’ section if you’ve performed a quick search and want to reset things you can just delete the text from the searchbox and press the search button. I hope to be able to continue with this next week. The big things remaining are the advanced search, the full place-name record popup and also what to do about the elements glossary. There are lots of smaller things still to do too, such as allowing specific views to be bookmarked, shared and cited.
I spent most of the remainder of the week preparing the sites on our internally hosted server for an upgrade to PHP8. This involved getting PHP8 working on my local PC and then setting up each site locally with errors turned on in order to test each script. I’ve also been taking the opportunity to deal with any warnings and notices, plus upgrading jQuery where applicable. The only fatal issue I’ve encountered so far has been the use of the ‘usort’ function with a Boolen return value, which is deprecated in PHP 8 (see https://stackoverflow.com/questions/65382799/what-happened-in-php8-0-0-to-break-usort-intstrlenastrlenb). This has affected a few sites. Other than that I’ve not found anything that breaks in PHP8. I haven’t finished checking all sites yet, but will hopefully finish this tedious but necessary task next week.
Also this week I helped Petra Poncarova with an issue with her project website. She was wanting to add accordions to pages and I found and installed a plugin that works with the Classic WordPress editor (https://en-gb.wordpress.org/plugins/easy-accordion-free/). It’s very easy to use and I added an accordion section to one of the pages on her site to show how it works: https://erskine.glasgow.ac.uk/people/donald-sinclair/.
I also gave some advice to Craig Lamont as he works on the banner for Rhona Brown’s new website, spoke to Thomas Clancy about another new place-names project and arranged to have a new WordPress site set up for a project Gavin Miller is setting up.