I was off sick from Tuesday to Thursday this week having caught some sort of virus that laid me low. However, on Monday I managed to complete the migration of all 156 poems in the Anthology of 16th and Early 17th Century Scots Poetry from ancient HTML to TEI XML. It’s something I’ve been working on since the New Year and it’s great to have finally completed it. The site is not yet live, though, as I need to wait until I receive feedback from the project PI. I’ll probably need to update and expand the information in the TEI header of each poem and I’d also like to make the XML files of the poems available as a downloadable ZIP file from the website, ideally under a Creative Commons license to enable researchers to reuse the data. I’ll also need to put in redirects from the old URLs, plus the PI has some updates to the poems and their glosses that I will need to incorporate. But until I hear back from him I can’t do anything else for the resource.
When I returned to work on Friday I had an update to the Anglo-Norman Dictionary waiting for me to process. The editor Geert had updated all entries beginning with ‘Y’, plus a handful of other related entries and I needed to run my scripts to delete the existing entries and replace them with the new versions. Thankfully all went smoothly.
I then moved onto a significant reworking of the Speech Star website (one of two websites for the project and the one that has not yet been officially launched). Previously this website featured access to several databases of ultrasound video files but the new update is going to shift the emphasis to a more focussed set animation video files, plus incorporating the IPA and extIPA charts and associated videos. There’s a lot to get through in this update and by the end of the day I had implemented only a part of it so I’ll continue with this next week.
I spent some time this week further tweaking the Speak For Yersel survey tool I’ve been working on recently. I completed an initial version of the tool last week, using it to publish a test version of the linguistic survey for the Republic of Ireland (not yet publicly available) and this week I ran the data for Northern Ireland through the tool. As I did so I began to think about the instructions that would be needed at each stage and I also reworked the final stages of the tool.
The final stage previously involved importing the area GeoJSON file and the settlement CSV file in order to associate actual settlements with larger geographical areas and to generate the areas within which survey responses will be randomly assigned a location. This was actually a two-step process and I therefore decided to split the stage in two, firstly ensuring the GeoJSON file is successfully parsed and imported and only then importing the settlement CSV file. I also created a final ‘setup complete’ stage as well, as previously the tool didn’t give any feedback that the process had completed successfully.
With the updates in place creating the Northern Ireland survey using the tool was a pretty straightforward process, taking a matter of minutes. I then moved on to creating our third and final survey for Wales, but unfortunately I soon realised that we didn’t have a top-level ‘regions’ GeoJSON file for this survey area. The ‘regions’ file provides the broadest level of geographical areas and are visible on the maps when you hover over areas. For example in the original SFY resource for Scotland there are 14 top-level regions such as ‘Fife’ or ‘Borders’ with labels that are visible when using the map such as the ones here: https://speakforyersel.ac.uk/explore-maps/lexical/.
Initially I tried creating my own regions in QGIS using the area GeoJSON file to group areas by the region contained in the settlement CSV files (e.g. ‘Anglesey’). However, this resulted in around 22 regions, which I think is too many for a survey area the size of Wales – for the Republic of Ireland we have 8 and for Northern Ireland we have 11. I asked the team about this and they are going to do some investigation so until I hear back from them Wales is on hold.
I also spent quite a bit of time this week continuing to migrate the Anthology of 16th and Early 17th Century Scots Poetry from ancient HTML to TEI XML. Previously the poems I’ve been migrating have varied from 14-line sonnets to poems up to around 200 lines in length. I’ve been manually copying each line into the Oxygen XML editor as I needed to check and replace ‘3’ characters that had been used to represent yoghs, add in the glosses and check for other issues. This week I reached the King’s Quair, which compared to the other poems is a bit of an epic, weighing in at over 1300 lines. I realised manually pasting in each line wasn’t an option if I wanted to keep my sanity and therefore I wrote a little jQuery script that extracted the text from the HTML table cell and generated the necessary XML line syntax. I was then able to run the script, make a few further tweaks to line groups and then paste the whole poem into Oxygen. This was significantly quicker than manual migration, but I did still need to add in the glosses, of which there were over 200, so it still took some time. I continued to import other poems using my new method and I really feel like I’ve broken the back of the anthology now – and by the end of the week I’ve completed the migration of 114 poems. Hopefully I’ll be able to launch the new site before Easter.
Also this week I began investigating the WCAG accessibility guidelines (https://www.w3.org/WAI/WCAG22/quickref/?versions=2.1) after we received a request about an accessibility statement for the Anglo-Norman Dictionary. I spoke to a few people in the University who have used accessibility tools to validate websites and managed to perform an initial check of the AND website, which held up pretty well. I’m intending to look through the guidelines and tools in greater details and hopefully update the sites I manage to make them more accessible after Easter.
Also this week I spoke to Susan Rennie about transferring ownership of the Scots Thesaurus domain to her after the DNS registration expires in April, added some statements to a couple of pages of the Books and Borrowing website referencing the project’s API and giving some information about it, and spoke to B&B project PI Katie Halsey about creating a preservation dataset for the project and depositing it with a research repository.
I continued to work on the new Speak For Yersel survey creation tool this week, using the Republic of Ireland as my test area. I managed to complete the ‘maps’ pages, which let users view all of the survey data on interactive maps. We’re still testing the new system so there’s not much data to actually view, but below is a screenshot showing one of the maps:
I also tweaked things to make it possible to cite / share / bookmark a specific map. This does mean that each time you select a different map the entire page needs to reload (as opposed to replacing the map only) but I think it’s handy to be able to link to specific maps and reloading the page is not a massive issue. The ‘attribution and copyright’ link in the bottom right of the map also now works, with the content of this being set in the config file so it’s easy to change.
I then moved onto the stats and data download page for project staff. As with SFY you can specify a period to limit the stats and data to the period given. As with SFY, the number of users and answers for each survey (within and beyond the area of study) are listed. Unlike SFY, there are also options to download this data as CSV files. You can download the users, or you can download the answers for a chosen survey, with options for downloading data for users within the area of study or outside it. Answer downloads also include fields about the user (and the question). Here’s a screenshot of how the interface looks:
That’s the tool pretty much completed now and I just need to see if there’s any feedback from the project team before I create similar resources for Northern Ireland and Wales using the tool.
I spent much of Monday going through the content management system for the Saints Places website, fixing errors that had appeared as a result of the migration to the site to a new server earlier in the year. I didn’t create this site or its CMS, but have inherited responsibility for it. The site (https://saintsplaces.gla.ac.uk/) launched more than ten years ago, with the project officially ending in 2013, but a researcher is still making updates to it so I agreed to get things working again. It took somewhat longer than I’d expected but at least it’s done now. Having said that it’s possible more errors lurk in places I’ve not been able to fully test so we’ll just need to see.
On Tuesday I participated in a meeting with Calum McMillan about ‘Change management’ in IT services at the University. This is about tracking what changes are made to IT systems, ensuring people are informed and strategies are in place in the event of changes failing. It was very interesting to hear more about this and I hope Arts developers will be kept informed of any changes that will be made to the servers on which our sites are hosted.
I had a further meeting on Friday with Matthew Creasy and his RA Jennifer Alexander about the James Joyce conference website (https://ijjf2024.glasgow.ac.uk/). I helped out with a couple of technical issues, gave some advice on how to present some materials and made a couple of minor tweaks to the website interface. I also updated the Seeing Speech (https://www.seeingspeech.ac.uk/), Dynamic Dialects (https://www.dynamicdialects.ac.uk/) and Speech Star (https://www.seeingspeech.ac.uk/speechstar/) websites to add in licensing and copyright statements.
I also returned to the new data and sparkline facilities for the Dictionaries of the Scots Language. I’d developed these features last year and had been waiting for feedback from the DSL people, which finally came last week. One thing they had agreed on was that we should limit the start date of the sparklines to 1375. When a refamiliarized myself with the work I’d previously done on the sparklines several months ago I realised I still needed to regenerate the data to make the DOST sparklines begin in 1375, and I also realised that the data displayed on our test server was not the current version (featuring the 50 year cut-off and SND extended to 2005) and that this data only existed on my laptop. I then had a moment of panic when I realised I’d deleted data from my laptop since I completed work on the date features last year, but thankfully I’ve managed to reinstate it.
I also realised that the way dates outside the sparkline’s range are handled will need to be changed. The way things are currently set up is that any dates that are beyond the scope of the sparkline result in the sparkline being ‘on’ at the start or end date to demonstrate that the entry is attested beyond the sparkline range. Dates before / after the sparkline start / end date then become the start / end date and are not individually displayed in the sparkline text, and the new start date is then treated as the start date for the purpose of the 50 year rule (for generating blocks of attestation where individual dates are found within 50 years of each other).
This currently happens on our test server for an SND entry where the earliest citation date is 1568 and the second earliest is 1721. When the sparkline data is generated 1568 becomes 1700 (the start year for SND) and as the gap between this and the next citation is less than the rule the sparkline displays a block from 1700-1721. The ‘dates of attestation’ hover-over and in-page text then display ‘1700-1721’ which is not at all accurate.
We need to have some kind of line on the sparkline at the start / end to demonstrate an entry’s dates of attestation continue beyond the scope, so for the above example there must be a line at 1700, even though this actually represents the year 1568. However, such a line needs to be flagged in the system as to be ignored for the purposes of building the blocks so that when the system finds the next date (1721) it compares this to the original date (1568) and not the line created at 1700. The system would therefore not generate a block from 1700-1721 but instead would create an individual line at 1721 (with an individual line at 1700 as well).
We will also need to ensure that the sparkline text includes all attestations before the sparkline’s start year rather than bundling these all up as ‘1700’. Actually, I guess there are two options here: We could just bundle them all up and display ‘<1700’ or we could display them all. It depends how verbose we want to be and how important it is to list all of the dates beyond the scope of the sparkline. For the example above the text would either start ‘<1700, 1721, 1773-1825’ or it would start ‘1568, 1721, 1773-1825’. The same process would also need to happen at the end of the sparkline too. This needs further discussion with the team before I proceed further.
Despite this issue needing further consideration, I did manage to make a number of other updates to the test site based on the team’s feedback. This included updating the CSV download on the search results page to add in the search criteria, changing the wording of the advanced search results page, removing date filter options from the search results pages in many situations (e.g. quotation searches) and changing how the filter options are displayed, moving from an always visible approach to a drop-down section that appears only when a user chooses to open it.
Finally this week I continued to work through the Anthology of 16th and Early 17th Century Scots Poetry and I’m now slightly over half-way through the migration of the ancient HTML to TEI XML.
I spent much of Monday this week continuing to migrate the Anthology of 16th and Early 17th Century Scots Poetry to TEI XML. I’ve now migrated 43 poems, so I think I’m about a third of the way there. It’s still going to take quite some time, but I’ll just keep tackling a few when I find the time, until they’re all done.
I spent the rest of the week continuing to work on the new Speak For Yersel regions. I’m starting with the Republic of Ireland as my test area for the development of a more generic tool that can then be used to create similar surveys in other geographical areas. Last week my data import script spotted a few errors with the geographical data for ROI and thankfully the team was able to address these and get an updated dataset to me this week. When I ran my setup script again the areas (used for deciding where a user’s answer marker should appear on the maps) imported successfully and I could then move onto developing the front-end.
Developing the front-end has involved rationalising the code I’d originally written for the Speak For Yersel website (https://speakforyersel.ac.uk/). This website has a broader focus than the new surveys will have, involving two additional survey types, plus quizzes and additional activities. It was also very much an active research project and the code I wrote needed to be updated significantly as the project developed and our ideas and understanding changes. This meant the code ended up a bit tangled and messy.
My task in developing the front-end for the new survey areas was to extract only those parts of the code that were relevant to the three survey types that will be included and to rationalise it – straightening it out and making it less tangled. The new code also had to work with configuration variables set during the creation of the resource (e.g. database table prefixes, site subdirectories).
I began developing the scripts to handle the layout and the connections to the config options and the database, generating placeholder pages for things like the homepage and the ‘About’ pages. I then developed the user registration facility, which I connected to the ROI geographical data thus enabling a user to begin typing the name of their settlement and for possible matches to be displayed in a drop-down list. Users are then saved in the database and stored using HTML5 Local Storage, including the GeoJSON shape associated with their chosen location, thus enabling markers to be generated within this area at random locations each time the user answers a survey question. The screenshot below shows the user details registered for a test user I created:
I then began working on the display of the surveys, completing the scripts that list the available surveys, display the intro text for a selected survey, load and display a question for each survey type and handle the progress bar. I also ensured that images can be displayed for questions in each survey type (if supplied) and have ensured that any associated audio files will play.
The screenshot below shows a ‘phonology’ question, with an associated image, just for test purposes (it’s of no relevance to the question). I also decided to move the ‘question number’ to the top right of the main pane, which I think helps declutter things a bit (for SFY it was below the progress bar and above the question). I also made the audio ‘play’ button a fixed width as previously the width was slightly different depending on whether ‘play’ or ‘stop’ was displayed, which made the adjacent button jump slightly during playback.
With this in place I then moved onto the processing of submitted answers. This has included saving all answers (including where multiple answer options are allowed), displaying the maps, dealing with map filters (e.g. respondent age group) and loading the next question, as you can see in the following screenshot:
I also created the ‘survey complete’ page and have ensured that the system logs which surveys a user has completed (including adding a tick to the survey button of completed surveys on the survey index page). I still need to create the maps page, add in the map attribution popup and develop the staff page with CSV download options, which I will start on next week.