Week Beginning 23rd May 2022

I’d completed all of the outstanding tasks for ‘Speak For Yersel’ last week so this week I turned my attention to several other projects.  For the Books and Borrowing project I wrote a script to strip out duplicate author records from the data and reassign any books associated with the duplicates to the genuine author records.  The script iterated through each author in the ‘duplicates’ spreadsheet, found all rows where the ‘AID’ did not match the ‘AID to keep’ column, reassigned any book author records from the former to the latter and then deleted the author record.  The script deleted 310 duplicate authors and reassigned 735 books to other authors, making the data in the content management system a lot cleaner.

I then migrated the Uist Saints website to a server at Glasgow and got everything working at a temporary URL.  All looked fine to me, although there was an issue with the homepage that needed investigating.  This issue was present on the live site too, resulting in the page content cutting off and displaying a lot of blank space and no footer, with lots of errors being displayed in the console.  I did some investigation into the errors and discovered that these were being caused by some JavaScript embedded in the homepage that has been treated like HTML by WordPress.  It has added HTML line breaks (<br>) wherever there is a line break in the code, thereby breaking the JavaScript.  I updated the page to strip out all of the <br> tags and it now loads without any errors in the console but whatever the JavaScript is supposed to be doing still isn’t working and there’s still a huge expanse of empty space and then no footer.

The JavaScript appears to be attempting to display a map using the Leaflet mapping library, but using some sort of WordPress plugin to do so.  There are over 3000 lines of JavaScript code in the page, which is really crazy.  Every single marker on the map (e.g. “Cladh Choinnich (burial ground and site of chapel)” at [57.157715,-7.301283]) has its own script comprising around 70 lines of code.  Sofia, the project RA looked at the page and decided to try deleting the blocks of JavaScript, and this then seemed to solve to problem, which was great, as I was thinking I’d need to create a new map after somehow extracting all of the data.

I then moved on to the Ramsay ‘Gentle Shepherd’ data, and this week tackled the issue of importing the code I’d written into the University website’s T4 content management system.  I created a ‘one file’ version of the page that has everything incorporated in one single file – all the scripts, the data and the styles.  I was hoping I’d then be able to just upload this to T4 but I ran into a problem:

I selected the ‘Standard plain’ content type as I did for the Enlightenment map I created in T4 many years ago, but the ‘content’ box can only accept a maximum of 80,000 characters.  My ‘one file’ approach is around 404,000 characters so I can’t upload it.  I then wondered about using separate files, as I had done with the Enlightenment map, but the JSON data for the performances on its own is over 227,000 characters.  This data needs to be a single thing and can’t be split up into smaller chunks (at least not without then having to stitch the data back together in the JavaScript before it can be used every time someone loads the page which would have an impact on the speed of the page).

I notice that the Enlightenment map has a further content type called ‘_blank’ that isn’t available to me where the performance data is to go.  This type allows up to 150,000 characters.  Unfortunately this is still not big enough.  The Leaflet JavaScript library which I also need to upload is 141,000 characters so currently can’t be uploaded either. I then looked into uploading the JSON data as a media file and I managed to upload it, but apparently media files only become active in the system when they are linked to from a T4 page using T4’s method of linking to a file.  The JSON file would only ever be loaded in via an AJAX call from the JavaScript code so would never work.  However, I did realise that I could upload the JavaScript file with the JSON data stored directly within it as a media file and then link to this (and also the leaflet JavaScript file and the CSS files) from the T4 HTML file.  However, this wouldn’t work when using regular HTML tags to link to scripts and CSS files as T4 only activates media files when linked to using its own special way of inserting links.

A helpful guy called Rick in the Web Team suggested using the ‘standard’ content type and T4’s way of linking to files to get things working, and this did sort of work, but while the ‘standard’ content type allows you to manually edit the HTML, T4 then processes any HTML you enter, which included stripping out a lot of tags my code needed and overwriting other HTML tags, which was very frustrating.

However, I was able to view the source for the embedded media files in this template and then copy this into my ‘standard plain’ section and this seems to have worked.  There were other issues, though, such as that T4 applies its CSS styles AFTER any locally created styles meaning a lot of my custom styles were being overwritten.  I managed to find a way around this and the section of the page is now working if you preview it in T4.

Unfortunately to get this to work the JSON data needed to be embedded in the JavaScript file rather than loaded in as a separate file.  This is going to make it more difficult for non-technical people to edit the data directly in T4.  In order to do so someone would need to:  Download the ‘gspCode’ file in the Media Library, which T4 unhelpfully converts into a .txt file then rename the file to remove the .txt extension (so it ends in .js instead).  Then find the data array in the file, make the changes to it and then validate it in the handy JSON validator https://jsonlint.com/ before saving the JS file and uploading it as a replacement for the item in the Media Library.

With all of this out of the way I was hoping to begin work on the API and front-end for the Books and Borrowing project, and I did manage to make a start on this.  However, many further tweaks and updates came through from Jennifer Smith for the Speak For Yersel system, which we’re intending to sent out to selected people next week, and I ended up spending most of the rest of the week on this project instead.  This included several Zoom calls and implementing countless minor tweaks to the website content, including homepage text, updating quiz questions and answer options, help text, summary text, replacing images, changing styles and other such things.  I also updated the maps to set their height dynamically based on the height of the browser window, ensuring that the map and the button beneath it are visible without scrolling (but also including a minimum height so the map never gets too small).  I also made the maps wider and the question area narrower as there was previously quite a lot of wasted space with there was a 50/50 split between the two.

I also fixed a bug with the slider-based questions that was only affecting Safari that prevented the ‘next’ button from activating.  This was because the code that listened for the slider changing was set to do something when a slider was clicked on, but for it to work in Safari instead of ‘click’ the event needed to be ‘change’.  I also added in the new dictionary-based question type and added in the questions, although we then took these out again for now as we’d promised the DSL that the embedded school dictionary would only be used by the school children in our pilot.  I also added in a question about whether the user has been to university to the registration page and then cleared out all of the sample data and users that we’d created during our testing before actual users begin using the resource next week.