This was a week of working on many different projects for me. I spent some time making final tweaks to the Speech Star websites (https://speechstar.ac.uk/ and https://www.seeingspeech.ac.uk/speechstar/) as the project officially came to an end this week. This included adding new videos, replacing existing videos, updating the metadata, updating the site text and adding in a few new images. Both resources have come together really well and I’m sure they will be hugely useful for speech therapy for a long time to come.
I also made a further update to the Anglo-Norman Dictionary resource related to the new ‘language’ search I added in last week. After this had gone live I had intended to make the languages in the entries link to the search results and I spent a bit of time getting this working. This involved updating the XSLT, the CSS and the JavaScript used to display the dictionary entries and the update is not live yet, but it’s in place on a test page and works pretty well. See the ‘(loanword….’ section in the following screenshot to see how it will work:
I also replied to a number of emails that had been sent to me from Ann Fergusson of the DSL regarding the new date search and sparklines that I developed for the project many months ago but is still to go live. Ann gave me some feedback on a number of issues and I spent some time making updates. The big change will be to the way dates of attestation that are before or after the active period of the dictionary in question. I’m going to have to update the way the sparkline data is generated, which may take some time. I’ll hopefully be able to look into this next week.
For the new regions of the Speak For Yersel project I updated the privacy policies and now everything is ready for us to begin sending out the URLs for the new areas for test purposes and for the Books and Borrowing project I wrote a couple of scripts to find and then generate some missing pages from the registers of Selkirk library. The first script identified page images that we have on the server for the two registers that do not have corresponding page records in our database. The script was set to output a 300px wide version of each missing image plus its filename and it was also possible to click on the image to view a full-size version. It turned out that the majority of the missing images were omitted for a reason – they were either blank or didn’t contain borrowing records. We decided that 70 images needed to be added in and 117 could be safely ignored. My second script then added in the missing pages. This took a bit of time to get working as not only did I need to create the pages, I also needed to ensure they were slotted into the correct place, updating the ‘next’ and ‘previous’ links and the page order as required. But the pages are now available and someone can begin the process of transcribing the borrowing records found on the pages.
I also spent about a day or so this week creating a content management system for the Burns Letter Writing Trail, which will be an interactive map of Burns’ correspondence that will eventually be added to https://www.burnsc21-letters-poems.glasgow.ac.uk/. I can’t really say much more about it at this stage, but the CMS is about 80% complete and I hope to finish the rest next week.
Finally, I spent about a day looking through the images and YAML files of playbills from the 19th century for Deven Parker. The images are generally a relatively small file size but they are actually pretty high resolution and I think they will work in a ‘zoom and pan’ interface. As an experiment I set up such an interface for one image, using the same JavaScript library I used for the Books and Borrowing project. I spent the rest of the day getting to grips with the YAMP files, building up my understanding of the textual data, sketching out an initial relational database structure that could be used for the project and compiling a list of questions for Deven. I’ll probably have to meet with her to go through these and see if or how my involvement with the project will develop.
I continued with the upgrade of the BARS review this week. I’d managed to complete the upgrade process last week, but discovered that the links to the actual text of journal articles as HTML and PDFs were not working. Further investigation this week revealed that this was a major problem. The way files are stored and referenced in the new version of the Open Journal System is entirely different to how things were in the original version. Previously there was an ‘article_files’ table where files associated with articles were located but the new version instead features a ‘files’ table that contains paths to files that are entirely different to the earlier version and the actual directory structure of the system. I realised that it was likely that the upgrade process not only upgraded the database but also moved files around too and as I never got the path to the files right on my Windows PC any upgrades that should have been applied to the files will have failed (although having said that I never saw any errors). I therefore had to begin the upgrade process again so that the files would actually get moved / renamed to match the updates to the database.
After further investigation it appeared that several people have experienced the same issue whilst upgrading, for example https://forum.pkp.sfu.ca/t/none-of-the-pdf-files-can-be-viewed-or-downloaded-after-upgraded-to-ojs-3/27381 and https://forum.pkp.sfu.ca/t/solved-cant-see-pdfs-after-upgrade-to-3-1-1-4/49382 and https://forum.pkp.sfu.ca/t/after-upgrading-to-ojs-3-1-1-4-files-are-not-found/48938/23.
I fired up the 2.4.8.5 version of the site that I had running on my old PC and this time ensured the config file included the correct path to the files. After doing so the instance managed to find the files, meaning I could restart the upgrade from this version. The first time I upgraded to 3.2.1 with the correct path the files were successfully indexed and added to the database, but no corresponding files were actually generated. It turned out that this was because my ‘files’ directory had somehow been set to read only. I fixed this an re-ran the upgrade to 3.2.1 and thankfully the files were all renamed and moved successfully. After that the upgrade to 3.4 worked fine.
However, the HTML version of the articles were still not opening in the browser but were instead getting downloaded. I found a forum post about this too: https://forum.pkp.sfu.ca/t/ojs-3-1-0-1-html-downloaded-it-automatically-from-articles/36024/8 which suggests that a plugin needs to be activated. After managing to log in as an admin user I managed to find and active the required plugin (HTML Article Galley). I also activated the ‘PDF.js PDF viewer’ plugin too. This now means you can view the HTML and PDF versions of the article in your browser, as with the old site. I also needed to update the permissions of a couple of directories and now the upgrade process is complete. For now the new version of the site is still running on a test server, so I’ll still need to replace the live site with the newly upgraded version once Matt is ready.
Also this week I investigated an issue with the Books and Borrowing server, as staff were unable to log into the CMS. As I suspected, this was because the server had run out of storage, meaning there was no space for a session variable to be stored. I contacted Stirling IT about this and thankfully they were able to free up some space. I also investigated some missing images and register pages from Selkirk library. It turns out that we only included pages that had been transcribed by a previous researcher, which meant that many of the register pages have been missed out. Thankfully we have the full set of digitised images and I’m therefore going to have to write a script that creates the missing pages, something I’ll try to tackle next week.
This week I finally moved back to my office in 13 University Gardens from the nicer office I’d been squatting in on Professors Square for the past year and a half. All my stuff had been moved over for me and when I was on campus on Tuesday I therefore had to spend a bit of time getting everything set up and in order.
Also on Tuesday, I had a meeting with Deven Parker, who has a Leverhulme funded position in English Literature and is working with Matt Sangster. We discussed a prospective project involving the digitisation of playbills from UK theatre from 1750 to 1843. There are currently around 300,000 digitised images and may be up to 500,000 in the end. Deven wants an online database to be made for these, featuring searchable text and the image and we discussed some of the possibilities. I’m going to have a look at some of the data next week and will think about what can be done.
Also this week we went live with the language search on for the Anglo-Norman Dictionary. This is now available as a tab on the advanced search page (https://anglo-norman.net/search/) and allows users to find dictionary entries that have been tagged as loanwords. It’s great to have this feature live. I also needed to update the bibliographical links to the DEAF website as they had changed their site, breaking all of the links we previously had. This required me to update the links in a few places, but all is working again now – for example see the DEAF links on this page: https://anglo-norman.net/bibliography/
I also made a couple of small fixes to the Emblems website (https://www.emblems.arts.gla.ac.uk/french/), ensured project images are not too large on the Glasgow Medical Humanities website (https://glasgowmedhums.ac.uk/projects/) and fixed a typo that had existed for many years on the New Modernist Editing ‘Digial Ode’ site (https://nme-digital-ode.glasgow.ac.uk) . I also had a chat with Craig Lamont about the interactive map of Burns correspondence that I will be developing. There will soon be an RA who will begin to compile the data and I will need to start creating a content management system for this in the next week or so.
I also continued to make updates to the new Speak For Yersel surveys (that are not yet quite ready to launch). This included adding in animated ‘bubble’ maps to the homepage and updating the survey tool to incorporate an optional question to the registration page about bilingualism, which I then incorporated into the surveys for Wales and the Republic of Ireland.
I then rounded off the week by making further updates to the Speech Star website, including adding text to the ‘About’ page, adding in some new videos, replacing some existing ones, updating metadata and ensuring all video popups are bookmarkable and have full citation information. For example, here’s a direct link to the ‘Lip consonants’ video in ‘Sound groups’: https://speechstar.ac.uk/speech-sound-animations/#location=65.
It was the May Day bank holiday on Monday this week. I then spent quite a bit of Tuesday and Wednesday completing several mandatory training courses that all University staff must take every few years. I had six to complete before the end of July so decided to get them all out of the way this week, each taking between 30 and 90 minutes to finish.
I spent most of the remainder of the week continuing to update the BARS review site for Matt Sangster. I’d made a bit of progress with this last week (upgrading from 2.4.3 to 2.4.8.5) but this week I needed to perform a major upgrade from 2.4.8.5 to 3.2.1 and then a further upgrade to the current version. Upon starting the process I quickly realised that version 3.2.1 is still not compatible with PHP 8 so I needed to downgrade PHP on my laptop to PHP7 to get things to work. It was also not documented that the config script’s database connection needed to be changed from ‘mysql’ to ‘mysqli’, and this took a while to figure out.
I managed to get the upgrade to start executing but the process was failing midway through with a very unhelpful ‘Specified key was too long; max key length is 1000 bytes’ error. This is a MySQL error but it was in no way clear exactly when and why the error was cropping up. I spent ages looking through numerous source code files to try and understand what was going on and then tried to modify things. But each time the error appeared I needed to delete and reinstall the database as it was then left in a half-upgraded state, which was a bit of a pain. I found this posting https://forum.pkp.sfu.ca/t/install-ojs-3-0-2-on-windows-with-php-7-1-8/32901/7 but it suggested upgrading MySQL would fix the issue and I was already running the most recent version and other information I found (such as https://stackoverflow.com/questions/1814532/mysql-error-1071-specified-key-was-too-long-max-key-length-is-767-bytes) didn’t really help much either.
I noticed that the existing tables were mixture of MyISAM and InnoDB storage engine tables so then decided to set them all to InnoDB (each needed to be done separately and there are almost 150 tables) but this still didn’t sort things. The main difficulty was that I didn’t know exactly which query the upgrade script was sending to the database was causing the error and looking at the code I couldn’t figure out how to add some kind of log to trace things. Thankfully one of the answers to this question https://stackoverflow.com/questions/4631133/mysql-log-of-invalid-queries noted that you can tell MySQL to log all queries that are passed to it and after doing so I could pinpoint exactly what in the upgrade script was causing the error.
The upgrade script created a number of new tables and one query was ‘ALTER TABLE email_templates_settings ADD UNIQUE INDEX email_settings_pkey (email_id, locale, setting_name)’ and it was this that caused the error and made the process fail. The problem was this appeared to be a perfectly valid query. Thankfully I then noticed that while existing tables were using the collation ‘utf8mb3_general_ci’ the tables created by the upgrade script were set to ‘utf8mb4_0900_ai_ci’ as this is the default collation now used by MySQL. Unfortunately this collation uses more bytes than the other one and was pushing the key length over the limit. After changing the database’s default collation and running things again the upgrade process finally completed successfully.
However, I was still unable to get the HTML and PDF versions of the articles to appear on my local PC. When I click on the links I just get a blank page, even though I’ve updated the path to files in the config file. This will require some further investigation.
I was then able to perform the final upgrade from 3.2.1 to 3.4. I was hoping this would be straightforward but unfortunately I still ran into a few problems. The upgrade failed with the error ‘Failed to open the referenced table ‘files’ (SQL: alter table submission_files add constraint submission_files_file_id_foreign foreign key (file_id) references files (file_id))’ and there was a single Google hit referring to this (https://forum.pkp.sfu.ca/t/upgrade-ojs-3-2-1-4-to-3-3-0-5-error/67785) but unfortunately the suggested solution (updating the database user privileges) did not work for me as my database user already had full privileges. Thankfully I spotted that during the previous upgrade some new tables had been created and these were using the ‘Myisam’ storage engine rather than ‘innodb’. I managed to update MySQL on my PC to ensure innodb was the default engine (see https://stackoverflow.com/questions/4199446/how-to-make-innodb-as-default-engine) and this fixed the issue. I then encountered some further errors relating to log files that needed to be deleted, but after doing so the upgrade process completed successfully. Unfortunately the HTML and PDF articles are still not loading, and I was hoping that this was an issue with running the software on my Windows PC. However, I then uploaded to newly upgraded site to a test server and the issue persisted. I’ll need to investigate this further next week – I just hope I don’t need to begin the upgrade process all over again to resolve it.
Also this week I made a number of further changes to the Speech Star website, including adding several new video clips and updating video metadata. There are still a few additional tasks I need to perform for this project and I’ll start to tackle them next week.
I worked for several different projects this week. On Monday I had a meeting with the Books and Borrowing PI and Co-I Katie and Matt and some researchers at UCL who are working on a project that deals with similar sorts of data to Books and Borrowing. They were interested in how we developed the B&B resource and we had a good discussion about various technical and organisational issues for about an hour. As a result of this meeting I decided to publish the development documentation for the project, which is now available here: https://borrowing.stir.ac.uk/documents/. This includes the data description document, the requirements documents for the project’s content management system, front-end and API, plus my ‘developer diary’, consisting of all of the sections relating to B&B taken from this blog. Hopefully the documents will be of some use to future projects.
Also for the Books and Borrowing project this week, the server that hosts the site at Stirling underwent a major upgrade and I therefore had to check through the site and the CMS to ensure everything still worked properly. There were a few issues with the CMS that I needed to fix, but other than that all would appear to be fine.
I also began the process of updating the BARS review site for Matt. This site is powered by the Open Journal System, but the version in use is now extremely out of date and needs an upgrade – a process that is not at all straightforward. I spent about a day working on the upgrade and the process is not even close to completion yet. It would appear that no 2.X versions of OJS will run on PHP 7, never mind the current PHP 8 and unfortunately my local server setup doesn’t function with anything before PHP 7. I managed to get a version of PHP5 running on an old PC I’ve got but after copying the OJS files and database I’d taken from the live hosting onto this PC (and updating the database connection settings) I couldn’t get the site to load – all I got was a strange browser error that I’d never encountered before.
This current live version of the site is running version 2.4.3 of OJS, and the upgrade pathway states that this would need to be upgraded to 2.4.8.5 (which will then need to be upgraded to 3.2.1, which can then be upgraded to the most recent 3.4 release). I therefore decided to try installing a fresh version of 2.4.8.5 to see if I had any better luck with that. Initially things didn’t look good as when attempting to access the locally hosted site all I got was a fatal PHP error. I managed to track this down and hacked about with the code to stop the error cropping up (I figured that since this version is only an interim installation it wouldn’t really matter). After that the fresh install (with no BARS data) appeared to run successfully.
The upgrade path documentation (https://docs.pkp.sfu.ca/dev/upgrade-guide/en/) states that to upgrade to a newer version you should set up the newer version and then point it at the older version’s database and reinstate the directories holding the old site’s files, so this is what I did. The upgrade script successfully noted the older database version and the upgrade process seemed to run successfully. At this point I then had what appeared to be the BARS site running on my old PC. However, there were some issues. The colour scheme appears to have been lost and there are some other differences in the layout. But more importantly, while the journals and articles all seem to be present and correct, pressing on the ‘HTML’ and ‘PDF’ links to access the content don’t actually load in any content. Thankfully Matt suggested that my local version of the site may need its config file updated to point to the correct files directory, and this was indeed the case, so progress is being made.
I then migrated everything onto my current PC, which currently has the PHP version set to 7, which OJS 2.4.8.5 should support. Unfortunately it doesn’t and when I try to access the site I just get fatal PHP errors relating to functions called in the code that have been removed from PHP 7. This is not necessarily a big issue, though, as now I have the site on my current PC I should be able to upgrade to OJS 3.X, a process I will continue with next week.
Also this week I made a few more tweaks to Matthew Creasy’s conference website and responded to a few emails that came in requesting my help with various things. I also made several more updates to the Speech Star website, which is now actually live (https://speechstar.ac.uk/). This included renaming the site and updating the homepage text, updating the favicon on the other Speech Star website to cut down on tab confusion (see https://www.seeingspeech.ac.uk/speechstar/) plus adding in a new menu item and link from this site to the other one. I also replaced the vocal tract video found here: https://speechstar.ac.uk/speech-sound-animations/ with a newer version.
I also returned to my work on the new language search for the Anglo-Norman Dictionary, completing an initial version of it. As with the label search, the languages are listed down the left and you can add or remove them from the panel on the right by clicking on them. Boolean options appear when multiple languages are selected and there’s an option to limit the search to compound words, exclude compound words or search for all. Below is a screenshot of how the search form currently looks:
The search is fully operational, but is not yet live as I need to hear back from the editor, Geert, about when he’d like the new feature to launch.
Finally this week I made some further updates to the Speak For Yersel follow-on projects. This included making a number of changes to the Welsh survey, adding in the site logos that Mary had created using ArcGIS and adding in the top-level tabs that will enable users to switch between survey areas. I also created a new ‘speech bubbles’ animated GIF for Northern Ireland based on images Mary sent me. The new survey areas are not yet publicly available but below is a screenshot of the Northern Ireland area, showing the logo, the top-level tabs and the animated GIF. We’re getting there!
