The Books and Borrowing project (https://borrowing.stir.ac.uk/) had its official launch on Friday this week, and it was great to celebrate the completion of a project that has been such a major part of my working life for these past four years. I spent a lot of this week preparing for the launch, at which I gave a talk about the creation of the resource. This covered the definition of the data structures, the creation of the database, the planning and development of the content management system and then the front-end and API for the project. There was a lot more I could have said about our use of technologies such as the IIIF server for images and Apache Solr for search facilities, but I had to keep the talk relatively brief and couldn’t include everything. I think my talk went pretty well, and it was also really great to hear from the other members of the project team, many of whom also gave talks about the research they had undertaken using the resource.
As part of my preparations I also get the site running on my laptop in case there were any issues with the server or general internet connection during the launch. I also spotted and fixed a small bug with the search results filter options. The filter by place of publication was working, but when a place was selected it was not getting ‘ticked’ in the left-hand filter options. This meant it was rather difficult to unselect the filter. The issue must have been introduced when I updated how publication places were stored in the Solr index a few months ago and thankfully I was able to fix it without having to regenerate the data.
Also this week I made a small tweak to the website for the International James Joyce Symposium (https://ijjf2024.glasgow.ac.uk/), adding a further logo and link to the footer. I also made a few more minor updates to the Speech Star website.
I also continued to work on the new language search for the Anglo-Norman Dictionary. I updated my data import scripts so that language data would be extracted during batch import of data and ensured existing language data was deleted during the process of deleting older data. I then updated the dictionary’s content management system to ensure that language data was properly dealt with when entries were added or edited through the system and updated the ‘view entry’ page in the system so that language data for each entry is now visible on the page, in the same way as parts of speech, labels and other data.
I spent the remainder of my week working on the Speak For Yersel follow-on projects, updating several of the questions for the Republic of Ireland, adding in introductory text for this survey area and updating all three survey areas to add some explanatory text to the start of each survey question.
The Books and Borrowing project has its official launch next week, and I spent some time this week preparing for it. This included fixing a speed issue with the site-wide fact page (https://borrowing.stir.ac.uk/facts/) that was taking far too long to load its default view, something that has somehow got worse since I originally created the feature. After undertaking some investigation into what was causing the slow loading time it turned out that the main sticking point was the loading of data for the two ‘Borrowings through time’ visualisations. This was taking quite a long time to calculate for all libraries. I therefore created cached files for the data used for all libraries and have set the API to call on these rather than querying the database whenever specific libraries are not requested. This has greatly speeded up the loading of the page, which (for me at least) is now practically instantaneous, while before it was taking up to 30 seconds.
I also went through all of my blog posts to extract text relating to the project going back to create a single document. It’s 130 pages long and contains more than 56,000 words, covering all of the work I did for the project over the course of four years. I then spent some time preparing a talk I’ll give at the launch about the development of the resource. I haven’t finished working on this yet and will continue with it next week. I also met with Matt to discuss the British Association for Romantic Studies journal (https://www.bars.ac.uk/review/index.php/barsreview) that needs and overhaul. I’m going to work on this for Matt and hopefully get an updated version in place before the end of May.
Also this week I continued to work for the Speech Star project, adding another batch of videos to the website. This took a fair amount of time to implement as the videos needed to get added to a variety of different pages.
I then returned to working on the Speak For Yersel follow-on project, focussing on Wales this time. I needed to generate a top-level ‘regions’ geojson file that amalgamated the area polygons into 22 larger regions. I did this in QGIS, manually selecting and merging the areas. With the regions file in place I could then set up the Wales survey using my survey setup script. This all went pretty smoothly, although there were some inconsistencies between the area geojson file and the settlements spreadsheet that I needed to fix before I could get the survey to setup successfully.
I’d also received feedback about the Republic of Ireland and Northern Ireland surveys, including many changes that needed to be made to the survey questions. I decided that the easiest way to handle this would be to delete the surveys (which were still only test versions with no real data) and start again with updated question / answer spreadsheets. There were some other updates that had been suggested that would apply to all surveys and I updated all three sites to incorporate these. I then spent a bit of time investigating QGIS and whether it could be used to create simpler versions of the region polygons in order to generate a logo for each of our new sites that would be analogous to the Scottish survey. After discussing this with Jennifer and Mary we agreed that Mary would take this forward, so we’ll hopefully see the outcome next week.
I’d taken a day off this week, so only worked four. I began the week continuing to work on the Anglo-Norman Dictionary, making a tweak to the publications scripts I was working on last week and then planning a new search of language tags that the editor wanted to be added. Language tags are at entry level (i.e. they apply to the whole entry) and are used to denote loanwords.
There are only 2660 entries that currently feature the language tag (and 24,762 that don’t) so the search is going to be fairly limited. I explored two possible developments. Firstly, we could have a separate tab on the ‘Advanced Search’ page for ‘Language’, as we do for ‘Semantic & Usage Labels’. The new tab would work in a similar way to this (see https://anglo-norman.net/search/), with a list of languages and a count of the number of associated entries. We could either then make a search run as soon as a language is clicked on, or we could allow multiple languages to be selected and then joined with Booleans as with the ‘Label’ search. The latter would be more consistent, but I’m not sure how useful it would be as there aren’t many entries that have multiple language tags (so ‘AND’ and ‘NOT’ would not be so helpful). I guess ‘OR’ would be more useful, but the user could just perform separate searches as there won’t be huge number of results anyway.
Secondly, we could add a language selector to the ‘Headwords & Forms’ search tab, underneath ‘Citation date’. We could provide a list of languages (including a note explaining how the languages are used and that they are not widely applied), with each language appearing as a checkbox (checking multiple will act as ‘OR’). The language search could then be used on its own (leaving ‘Headword’ and ‘Citation date’ blank) or in conjunction with the other search options.
So for example
After further consultation with the editor, Geert, we decided that I’d start by developing the separate tab option and if we may expand the headword search to incorporate language at a later date, if it’s still considered necessary.
Incorporating a language search is going to mean updating the database and the entry publication scripts (both in the management system and my batch scripts) to extract language data from the entries when they are edited or created. I’ll also need to update the ‘view entry’ page in the DMS so the language data is listed.
My plan of action is to do the following:
I managed to complete the first two tasks, extracting 3097 language tags and adding these to the database.
Also this week I had discussions with the Books and Borrowing people about the official launch of the resource that’s taking place in a couple of weeks. I’m going to be speaking at the launch so I needed to figure out what I should be talking about. I also returned to the ‘Browse book editions’ page on the website (https://borrowing.stir.ac.uk/books/) which was at this point taking a long time to load. This is because the page defaults to displaying all book editions in the system that have a title beginning with ‘A’- almost 3000 books. I did consider adding pagination to the facility, but I personally find it easier to scroll through a long page rather than flicking between many smaller pages, plus it means a user can use ‘Find’ in their browser to search the listing. Another option I considered was to limit the default display to a particular genre of book rather than all genres, but I decided that this might confuse people if they don’t notice the limit has been applied. Instead I set the page to not load a specific letter tab by default. The tabs load, but to view the content of one of them the user actually has to select one. This means the page now loads instantaneously and people get to choose what options they want to view without having a long wait.
Also this week I made some further updates to the Speech Star websites, adding in new ExtIPA animation videos to both the ‘pre’ and ‘post’ 2015 charts. This was a bit fiddly and took some time but we now have most of the animations in place. I also exported all of the Historical Thesaurus data for Fraser, as a project needs an up to date copy of it.
It was Easter Monday this week, and I’d taken the rest of the week except Friday off as holiday. I’ll be off for some of next week as well. After writing my blog post and catching up with emails on Friday I spent the rest of the day working for the Anglo-Norman Dictionary, as the editor was ready to launch a major update to the content – replacing every entry for the letter ‘T’, plus a number of updates to entries in other letters. I have a documented process that I created for such updates, so thankfully I could follow these instructions (which of course include making backups before changing anything), but it’s still a little scary when thousands of entries are getting deleted and replacement data is generated.
After backing everything up and making all necessary preparations I set the update scripts in motion and thankfully all went smoothly. The update replaced or created 4472 dictionary entries, of which 1805 were ‘main’ entries, with the rest being cross references. Of the ‘main’ entries these included 3310 main senses, 1286 subsenses, 2958 locutions, 2060 locution senses and 328 locution subsenses. Overall the entries included some 15380 citations.
However, during the afternoon one of the editors encountered a problem when updating one of the new entries via the Dictionary’s online management system. The script that publishes updates quit midway through with an SQL error, the entry could no longer be found on the live site by searching for the headword or variants and duplicates of the entry appearing in the ‘Browse’ pane.
This was clearly rather concerning so all work on the dictionary stopped whilst I investigated. Thankfully I managed to figure out what was going on. When subsequently editing the entry some UTF-8 characters had ended up in the attestation dates, but the corresponding columns in the database weren’t set up to store UTF-8. When the system attempted to insert the characters the database complained and the publication script stopped. What this meant was the remaining parts of the script didn’t get a chance to execute, and it was in these parts that older versions of the entry were archived, search terms were generated and everything was tidied up. As these parts didn’t run this meant duplicate entries crept in and search terms weren’t working properly. I therefore updated the database so all columns could handle UTF-8 data and removed the duplicate entries and thankfully all was working fine again afterwards.
This was a four-day week as Friday was the Good Friday holiday. I’ll also be on holiday for all of next week other than Friday, giving me a nice Easter break. I spent a lot of this week continuing to overhaul the Speech Star website (the version for speech therapists that hasn’t been publicly launched yet). I’d made a start with this last week but still had many updates to implement. This included incorporating the IPA and extIPA charts into the website, but ensuring that only the animation videos rather than the MRI scans were included. I had also been given a selection of new animations that needed to be incorporated into the IPA and extIPA charts on other websites such as Seeing Speech, which meant updating various databases, creating thumbnails and other such tasks.
The new website now features the IPA / extIPA charts and their animation videos, a selection of speech sound videos, such as animations showing the consonants of English and a page featuring ultrasound therapy videos, as the following screenshot shows. It’s a very different website to how it was a few weeks ago.
Also this week I was in communication with Susan Rennie regarding the old pilot Scots Thesaurus website, which launched almost ten years ago now. The domain for this website is due to expire soon and Susan is going to point the domain at a different website, with the original website being archived. This all required some communication with Susan, the University’s Hostmaster and the Research Data Management team, but I think we’re getting there now.
My final major task of this four-day week was to write an instruction manual for setting up a Speak For Yersel linguistic survey website using the tool I recently created. The 13-page, 3,500 word document provides step-by-step instructions for creating the data structures the tool requires, the setup facility, customisation and exporting data. It’s hardly the most riveting of reads, but it should be useful for anyone using the tool in future (including myself!).