Category: Speak for Yersel

Week Beginning 21st April 2025

Posted on

It was another four-day week due to Easter Monday, and I spent a fair amount of my time working for Speak For Yersel.  For our abstract for DH2025, I received some new, high-resolution images of the graphs from co-author Marc Barnard and after a little more tweaking and dealing with the submission process I was able to complete the submission process.  I still need to actually sign up for the conference, though, which is something I’m hoping to be able to do next week.

One of the reviewers of our abstract had made a comment about user fatigue and enquired about how many users actually completed the surveys and this prompted me to undertake some investigation.  I created a series of database queries that for each user extracted counts of the number of submitted answers in the morphology, lexis and phonology surveys.  I then exported the data into Excel spreadsheets, one per region (Scotland, Northern Ireland, Ireland and Wales) and sent this data on to Jennifer in case it was of some use.

However, I did have to point out that the counts for each user are of answers submitted and some questions allow for multiple answers to be selected so there’s not an exact 1:1 relationship between the figures and the number of questions.  Also, a user can complete a questionnaire more than once (or quit mid-way and then begin a second time).   I noticed that there were a few users who have submitted many more answers than there are questions.  For example, in the NI data there is a user who has submitted 102 morphology, 105 lexis and 76 phonology answers, even though there are only 34, 35 and 29 questions respectively.  It’s not possible to ascertain exactly what prompted the user to submit so many answers, but it is perhaps a case of someone passing an iPad round a group of people.

I then decided to experiment with visualising the data, and for this I used the Highcharts library (https://www.highcharts.com/).  There are lots of interesting visualisation that can be made with the data, and my first attempt was to generate a stacked column chart (https://www.highcharts.com/demo/highcharts/column-stacked).  I decided that it would be useful to visualise all of the answers submitted by each user over time so wanted to plot each user as a column, with columns on the x-axis arranged chronologically by date of user account creation.  Each column would then be split into three coloured sections showing the number of answers submitted across the three surveys (morphology, lexis and phonology).

I wrote a little script to generate the JSON data that Highcharts can easily work with, and adapted the Highcharts example linked to above to work with the new data on my local PC.  The resulting graphs include a lot of columns (6379 for Scotland, 785 for Northern Ireland, 350 for Ireland and 1487 for Wales) and everything does get rather squashed together and some gaps can get lost, but what is interesting is how these graphs show the overall pattern of submissions.  On the whole most users across all regions made a decent stab at completing all surveys, as there are clear bands of colours, admittedly with some variation and gaps.  The graphs also show the outliers, especially the people who submitted many more answers than there are questions.  One user in Scotland in particular seems to have gone a bit crazy, and this then causes the rest of the graph to be rather squashed.

Here’s the graph for Northern Ireland, showing a few users who submitted many more answers than there are questions, a few users who submitted very little, and an overall pattern showing users making a decent stab at completing all three surveys.

The pattern for Ireland is broadly similar, and with less users it’s easier to view the individual columns:

Wales has considerably more users, which means individual columns can get lost, but it’s still possible to get an overall sense of user completion rates:

For Scotland we have an awful lot more users, plus as mentioned earlier one user who submitted huge numbers of answers, which results in a graph that squashes up all of the other data:

I had to regenerate the Scotland data and graph as after my first attempt I realised that the Scotland resource features more than just the surveys but also features follow-on questions, quizzes and other activities and I hadn’t filtered all of this out.  I was also interested to expand the x-axis to allow a more nuanced view of the data, and also to place a maximum extent on the y-axis to avoid one user affecting the display of data for all other users:

The resulting graph above (which you’ll need to open to view properly) demonstrates a lot more variation in user submissions that are lost from the smaller graph and demonstrates how the smaller graph with its big blocks of solid submissions doesn’t reflect reality.  Having said that, the overall picture still shows a decent number of users submitting a complete or near-complete set of answers.  Of course there is still much that could be done with the data.  Even using the same stacked column graph, rearranging the users by number of submissions would be interested, and may demonstrate how the overall picture when presented in date order could possibly obscure the number of users who didn’t submit much data.  But that’s for another time.

Also this week I made a few changes to the Speech STAR resource including adding a link to the ‘in clinic’ site form the top level tabs for Seeing Speech, Dynamic Dialects and the STAR site, as you can see here: https://www.seeingspeech.ac.uk/speechstar/.  I also had a chat with Eleanor about the visibility of the STAR sites in search results.

I also made a few further updates to the VARICS lookup test that’s still in development, including updating the ‘how to’ guides and linking to them from the longer textual descriptions of measurements.

On Friday I had a Teams meeting with Katie, Matt and Kitt from the Books and Borrowing project to discuss the new versioning system I’m going to develop.  We agreed that I will develop the full versioning system I specified in the document I sent around two weeks ago, and I’ll aim to get started on this in the next week or two.

Week Beginning 14th April 2025

Posted on

This was a four-day week as Friday was Good Friday.  My biggest task of the week was to finally launch the new version of the DSL website that I’d been working on for at least 18 months now.  Although this version does not look massively different to the previous version, there are a lot of changes under the bonnet.  As well as incorporating updated data, this version features an entirely new search facility that routes all searches through the Solr indexes.  Quotation searches now allow users to filter their search by date of quotation and both search results and dictionary entry pages now feature ‘sparklines’ (which are not really sparklines but mini-bar chart) showing the periods of attestation, as you can see here: https://dsl.ac.uk/results/dreich.

I began the process of making the new version live on Monday, copying the new version of the online database across from our test server.  On Tuesday I completed the process and by Tuesday afternoon the new version was in place.  The process all went very smoothly, but there were some tweaks that needed to be made after the new version went live.  The editors had noticed that there were some occurrences where the entry XML had included self-closing empty bold and italic tags (e.g. <b/> and </i>).  These are not valid in HTML and browsers interpret them as being opening tags with no closing tags, meaning when the pages are rendered all text from that point onwards is displayed as bold or italic.  The editors realised there were about 40 entries that featured such tags, and I had to manually fix them in our online systems, while the editors made the updates within their editing system.

Also this week I worked on a final version of our ‘Speak for Yersel’ abstract for the DH2025 conference in Lisbon.  I’m still hoping to source some higher resolution images for the abstract so I haven’t submitted it yet, but other than that it’s pretty much ready to go, which is good as it has to be submitted by the end of the month.

I also made some updates to the HiMuJe Malabar website (https://himuje-malabar.glasgow.ac.uk/) for Ophira Gamliel and our German co-I Innes Weinrich, including updating the footer to add in a further institution and giving advice on editing WordPress pages to a new project RA.

Finally, I spent some further time considering the new versioning system for the Books and Borrowing data, which involved discussing the possibilities with the team and making further plans for the implementation of the feature.  It now looks like the team will require the full-on versioning system that I wrote a specification document for last week rather than the simpler, easier and quicker approach I’d also suggested.  This will mean pulling every part of the existing Books and Borrowing system to bits and implementing major changes.  It’s going to be pretty heavy going, but hopefully it will be worth it.  I still need to hear back from the team about how they would prefer the front-end to be updated, and it’s likely that I will start on this major update over the summer.

Week Beginning 10th March 2025

Posted on

I spent most of this week continuing to implement the new cognate reference search facility for the Anglo-Norman Dictionary.  When I left off last week I was midway through implementing a script to generate search form variants for all of the cognate reference forms found in the dictionary entries.  These are the words found in the grey box towards the top of an entry page such as https://anglo-norman.net/entry/asseger.  My script splits up multiple forms and bracketed sub-headwords, where brackets are found within a form the script generates variants with and without the bracketed text, multi-word forms have each word stored separately (in addition to the full string) to enable each word to be searched for, and where a dash is used variants are generated with the dash, with a space in place of the dash and with no space at all.

After fully testing the script and its output I then needed to work on the scripts that update dictionary entries to ensure that they also generate or regenerate the cogref search forms whenever they execute.  This included the batch upload process I use when handling large-scale updates to the database and the entry publication script found in the dictionary’s content management system, as used by the editorial team when updating individual entries.  I worked with the editor Geert to test out these updates and with a bit of tweaking I got everything working.

I then updated the ‘View entry’ page in the dictionary’s content management system so that all of the cogref search terms are displayed, so the team can check what can be searched for.  As mentioned previously, cogrefs are split into individual search terms on the space character, allowing individual words to be searched and this does mean that parts of speech and numerals also get stored as individual search terms, but I don’t think this will be a problem – no-one will see these search terms, only the listed ‘Display form’ will be shown in the results and the ‘search forms’ just show the variants that are stored behind the scenes.  I did consider stripping out all numerals and parts of speech but there are just so many possible permutations that it would be very tricky to catch them all, and storing them is not a big issue.  So for example for the entry ‘asseger’ linked to above, the following forms are generated:

Where there are multiple forms (e.g. Gdf) these are stored as separate search forms.  Where there are subheadwords (e.g. DEAF) these are similarly split into separate search forms.  Where there are forms with spaces (e.g. OED) the full string is saved as a search form, but so too are each individual word.  Where there are brackets within a word (e.g. MED) then forms are generated both with and without the bracketed text.  This also works where there’s only an open bracket, as happens quite often with MED headwords.  There were also a handful of MED cogref forms that had multiple brackets (around 13, I think).  In such cases the generated forms only include versions with all of the bracketed text and none of the bracketed text, as the generated forms for the entry ‘Oxherd’ demonstrate:

He above also demonstrates the variants that are created when a dash is used, and any forms with slashes are treated in the same way.

With all of the cogref search data in place and facilities established to ensure the data will be updated when entries are edited I then moved onto the creation of the new search facility itself.  The first step was to create a new endpoint in the API through which the search requests will be processed.  It took a while to implement and test this, ensuring that wildcards can be used in the search and enabling the search to be limited to one or more cognate dictionaries.  The search results initially brought back rows for each cogref form, so if (for example) the search was for ‘siege’ as found in the above screenshot there would be three individual DEAF search results, each with information about the associated entry.  However, what the search needed to return was entries and then within each entry a further section listing the individual search results and I therefore needed to switch things around slightly.

With the API updated and the new search endpoint working I then needed to update the front-end to add the new search form to the advanced search page, connect the form to the API and ensure that any returned results were processed in the same way as the other advanced search options (for example with pagination, links to entries and a way to return to the form with previously entered options remembered).  I created all of this in a test version of the page so the editors and I could test things out without making the update publicly available.

The new Cogref search appears as a new tab in the Advanced Search after ‘Languages’.  It consists of some introductory text, a search box where a cognate form can be entered (with or without wildcards), and a section where the cognate dictionaries you want to search can be selected.  These appear as checkboxes which are all checked by default.  There’s an ‘All’ checkbox that allows you to select / deselect all of the other checkboxes, so to limit your search to just DEAF, for example, press ‘All’ to deselect everything then press ‘DEAF’.  The cognate dictionaries have tooltips that display the full names, as the following screenshot demonstrates:

As with the headword search, by default any text you enter into the ‘Cognate Form’ box matches any corresponding string in the search forms – so for example ‘noble’ is actually treated as ‘*noble*’.  To perform an exact search use double quotes.  As previously mentioned, search forms have been created for many different variants and where a form has multiple words or is split with dashes each individual word is treated as a separate search form.  So, for example, if you perform an exact search for “herd” you will find things like ‘nout-herd’, ‘shep-herd’ etc.

The search results display details of what you’ve searched for, the entries that feature a cogref form that matches and the specific cogrefs that feature a form that matches your search criteria, as the following screenshot demonstrates:

The editors are now testing out the new search and hopefully we’ll be able to go live with it in the next week or so.

Also for the AND this week, the editor Karen spotted an issue with the cross references between entries, which are generated whenever an entry is updated.  Some cross references weren’t being found and it turns out that this was because the homonym number is separated from the lemma with a hash in the XML, but with an underscore in the ‘slug’ used in the website (this is correct and how things should be). What this meant was that the cross reference generator was taking the slug (e.g. Partein_1) and looking for that in the XML files and obviously not finding any matches as it’s recorded as ‘Partein#1’ in the XML.  I therefore updated the code so that the xref search turns ‘_’ into ‘#’ and the xrefs are now being found and generated.

This week I also did a bit of work for the VARICS project, adding in a new batch of short descriptions for measurements in the lookup feature I’m developing and discussing the ancillary text with members of the project team.  I also had a chat with Jennifer Smith about our Speak For Yersel paper for DH2025 and gave some feedback to Matthew Creasy about his Mallarmé Chancellor’s Fund application, which he submitted on Friday.

I also responded to a Query from William Ashford of the DSL about Google Analytics, mobile versus desktop users and search engine optimisation and responded to a query from Ann Ferguson regarding a couple of issues relating to the bibliography search.

Week Beginning 3rd March 2025

Posted on

The team meeting for the Dictionaries of the Scots Language took place in Edinburgh this week, and I attended on Wednesday.  At the meeting I led a session where I demonstrated the potential new interface I’d developed for the DSL website last summer.  I gave a walkthrough of all of the mock-up pages I’d created, such as the homepage, the search results and the entry page, demonstrating the interface on both desktop and mobile devices.  I can’t really say too much about the interface at this stage, or post any screenshots, but the demonstration was very well received by the team and on the whole they are keen to take it forwards.  We spent an hour or so discussing the interface and there were a number of helpful suggestions and new ideas proposed, and I’ll definitely take these into consideration.  There was some talk of launching the new interface next year, so sometime this year (possibly in the summer) I’m intending to replace the interface for our test version of the DSL website with the new interface.  This will involve creating a WordPress theme from the interface, ensuring it is fully integrated with the WordPress platform and also connecting the interface to the DSL’s API in order to pull in real data rather than displaying static mock-ups.  It’s a task that will likely take a few weeks to fully sort out, but once it’s done then the team can fully test out the interface and give further feedback.  I spent some time before the meeting preparing for my session and I also needed to fix a couple of issues with the mock-ups that I’d spotted – for example the tooltips were not triggering on touchscreens.

Also this week I learnt that our paper about Speak For Yersel that I submitted to DH2025 had been accepted, and I had an email conversation with Jennifer Smith about conference arrangements and the presentation.  I also spoke to people at The National Archives about the transferral of the OHOS domain to TNA and updated a few St Andrews records in the Books and Borrowing website whose URLs couldn’t be fixed in the batch update I executed last week.  Someone at St Andrews library made some very helpful suggestions for the books and I manually updated them.  I also heard this week that the Books and Borrowing resource has won the British Society for Eighteenth Century Studies’ 2025 digital prize, which is fantastic news.

I spent the rest of my week planning and developing a new search option for the Anglo-Norman Dictionary.  The editor, Geert, wanted a new advanced search facility that would enable users to select one or more cognate dictionaries and then search for a headword found in these dictionaries, returning a list of all AND entries that feature cognate references that match the criteria.  The cognate references are found in the grey box towards the top of every AND entry, for example on this page: https://anglo-norman.net/entry/noble.  There are ten cognate dictionaries (e.g. OED, DEAF) and the forms of the headword as found in these dictionaries are listed.  It is these forms that the new search will target.

I wrote an initial specification for the new feature, the data structures that would be required and the stages that would need to be followed to generate the necessary data and shared it with Geert.  After several email exchanges I wrote a second version of the document and then began work implementing the feature.  The biggest issue will be processing the cognate reference forms, as these will need to be split up in various different ways.  For some dictionaries there are headwords and also subheadwords in brackets.  Other dictionaries use brackets to denote variant forms, sometimes without supplying a closing bracket.  In other cases slashes are used to indicate separate headwords and then there are the issues of parts of speech and numerals appearing at the end of forms.

My first job was to write a script that would go through all active main AND entries and extract all of the cognate reference forms.  In doing so I discovered that there are 634 such entries that don’t include the cognate reference section, and I sent the list to Geert for him to investigate.  I also generated a spreadsheet containing every cognate reference for every entry, which was useful for identifying all of the variety contained in the fields.  I then began writing a script that would generate individual search forms for each cognate reference, generating alternative spellings based on brackets, for example.  By the end of the week I was still in the middle of this and I’ll continue with it next week.

Week Beginning 18th November 2024

Posted on

I spent a fair amount of time this week working on a paper about Speak For Yersel and its new regions for the DH2025 conference.  I worked on this with the PI Jennifer Smith and after several revisions and a meeting on Tuesday we had something that was ready to submit.  I also spent some time finalising the code for the tool that I created to generate the new survey areas.  We’ve decided to make this publicly available for others to download and reuse via GitHub but before I could add the code I needed to get it ready for publication – doing things like ensuring libraries like Bootstrap and jQuery are referenced externally rather than including them directly, ensuring no passwords and keys are included, updating the setup instructions document, preparing the example data and running through a few test installations to make sure the thing actually works.  For sample data we’re making the Wales data available.  This consists of not only the questions and answer options for the survey but all of the geolocational data that is needed in order to ensure submitted answers can be correctly pinned onto the map.  You can now access all of the code for SFY here: https://github.com/baitken/sfy.

Also this week I spent some time going through the website mockups and sample data that I’d been sent for the VARICS project.  They are creating a database of speech files with filters, similar in many respects to the facilities I created for the STAR project and I’ll be developing a resource for the project over the coming months.  We have a project meeting scheduled in a couple of weeks, and I have a lot of questions to ask when we meet.

I heard some good news this week regarding project funding.  The Anglo-Norman Dictionary has had its final phase approved by the AHRC, after a few weeks of uncertainty, which is fantastic.  Also Joanna Kopaczyk’s DOST project has received funding and I’ll be working with her on this in the new year, which is great.

Other tasks this week included updating the embedded Scots School Dictionary in Speak For Yersel to replace this with the new Concise Scots Dictionary that is now available from the DSL, plus giving some advice to the DSL editor William Ashford about which versions of the ancillary pages of the DSL site and its test equivalent to update.  I also updated the links from the Thesaurus of Old English to the Dictionary of Old English as their URLs have changed, and discussed the creation of a new place-names site for Nairn with Thomas Clancy, submitting the request to have a subdomain created for this.

 

Week Beginning 28th October 2024

Posted on

On Thursday this week the three new Speak For Yersel regions were successfully launched.  These can now be accessed here:  Northern Ireland: https://speakforyersel.ac.uk/ni/, Republic of Ireland: https://speakforyersel.ac.uk/roi/, Wales: https://speakforyersel.ac.uk/wales/.

I spent most of my week working on dictionary projects.  Last week I was given access to a new export of the Dictionaries of the Scots Language data that the team wanted me to import into a new test version of the website.  The aim was to check how the work on the dates was progressing and if there were any issues.  Setting up a new test instance of the website and importing all of the new data took a day or so and involved setting up new Solr caches, running many different scripts to process the data and generate the data for Solr and test everything out on a temporary version of the website running on my local PC.

As I prepared to process the bibliographical data I noticed that the XML now appears to have a different structure and I made the decision to hold off importing it until I hear more from the team.  Running the XML data through my existing scripts would not extract anything as the expected structure is not found.  Previously the source dictionary was specified in an attribute of the ‘record’ element, e.g.

<record id=”bib014623″ oldid=”sb1″ sourceDict=”snd”><bibl><author>Abel, George</author>. <title>Wylins fae my Wallet</title>. 1916 (Abd.)</bibl></record>

But in the new data the ‘bibl’ tag is wrapped in either a ‘snd’ or ‘dost’ tag, e.g.

<record oldid=”sb1″ id=”bib014623″><snd><bibl oldid=”sb1_1″ id=”bib014623_001″><author>Abel, George</author>. <title>Wylins fae my Wallet</title>. 1916 (Abd.)</bibl></snd></record>

I can update my extraction script to deal with this change, but it looks like there are other changes too (e.g. the ‘bibl’ element now having its own ID) so I’ll need to talk through all of the changes with the team before I implement any updates.

When it came to processing the dictionary entry data, the import all went very smoothly.  There were the 11 ‘PROBLEM’ dates (citation dates that give errors when passed through the date normalisation scripts that are part of the process that exports the data from the DSL’s editing software).  I extracted the citations that feature these and their corresponding entry IDs and sent them to the team;  it looks like these will be pretty easy to fix.

There are, however, some 28,730 citations for which no date has been extracted.  This is down from 37,010 in the previous version of the data.  I also extracted these and sent them to the team for consideration.  Looking at a few of these it would appear that the issue is that the citations don’t feature ‘date’ tags but only feature a ‘bibDate’ attribute for the ‘cref’ element, for example:

<cit> <cref refid=”bib012649″ bibDate=”1375″><title>Ib.</title> <sc>iii</sc>. 429; etc.</cref><q>In a nycht and in a day</q></cit>

I recall that we’d discussed these previously and I was told to ignore these as they’re still being worked on.  As they stand I can’t use them without further processing as they often include affixes, for example:

 

<cit> <cref refid=”bib013662″ bibDate=”a1400″><title>Ib.</title> xli. 294; etc.</cref><q>Of a mylk and of a clath</q></cit>  <cit>

There are also citations that feature no actual quote text.  There are 26,672 such citations in the new data, up from 25,929 in the previous dataset.  I’ve informed the team about these.

In the new dataset DOST has 48,349 entries containing 200,026 forms and 577,528 citations.  This compares to 50,212 entries containing 200,202  forms and 577,265 citations in the previous dataset.  SND has 29,981 entries in the new dataset, containing 67,080 forms and 171,388 citations, compared to 30,107 entries containing 67,414 forms and 170,605 citations in the previous dataset.

With all of the data extraction (other than the bibliographies) complete I contacted the team and will await to hear back from them before I proceed further.

I then moved onto the Anglo-Norman Dictionary, for which I updated the CSV output functionality of the API.  When I originally developed the API I’d left these unfinished as it didn’t look like the options would end up being used anyway.  In fact, the CSV output of single entries was entirely broken.  However, CSV output is now going to be used by the team in order to share the project’s data with third parties so I needed to get the functionality working.

It was slightly complicated to deal with the part of speech, label and form fields in the search results output as each entry can have any number of such fields, each of which can comprise of multiple types of data (e.g. each label has five fields).  For the data to be useable in a spreadsheet each bit of data needs to occupy its own cell, but across multiple search results the columns of data need to be in alignment.  For example, one entry may have one label and another may have ten.  Forms are listed as columns after labels so in order for the form columns to align for all entries the code needs to ascertain which entry features the most labels, create headings for all of these and then pad out the entries that feature less labels so that their form columns still appear in the correct place.

And some entries have a LOT of parts of speech, labels and forms.  For test purposes I was using a headword search for ‘*ment’, which gives 1924 results.  At least one entry in these results features 7 different parts of speech, 17 different labels and 442 forms (not necessarily the same entries in each case).

The CSV output now handles all of these correctly, rather than just displaying ‘array’ in the pos, label and forms columns.  It now means you can order the resulting spreadsheet by the columns to figure out which entries have the most pos, labels or forms, for example.  Plus if you’re really not interested in all of the separate pos, labels and forms you can just ignore or delete the columns.

After dealing with the pos, label and forms columns I thought I’d sorted the search results CSV output, but it turned out that there was rather more still to do.  We actually offer several different search types, such as citations and languages and these bring back different types of data.  Some of these include different array-based fields such as translation and I also needed to sort these out in the CSV output.  Thankfully I could apply the same methodology I’d used for fields such as pos so it didn’t take too long to sort this out.

For the rest of the week I returned to working on the interactive map of the correspondence of Robert Burns.  I haven’t worked on this since before my holiday and it took a bit of time to familiarise myself with the resource again.  Whilst doing so I spotted that when viewing the map in full-screen mode the autocomplete and tooltips in the filter popup were not working.  I’d used a similar approach to the new ‘map first’ place-names interface so checked these resources too and discovered they were similarly affected.  It took an awfully long time to get to the bottom of this.  Initially I had thought it was a simple ‘z-index’ issue but this was not the case.  When the map is in full-screen mode it’s treated as a separate view to the standard page, and the autocomplete and tooltips were still associated with the ‘in-page’ view and therefore not appearing.

The autocomplete and tooltips are initialised when the user opens the filter form modal by pressing on a button.  When in full-screen view the autocomplete and tooltips were still being attached to the ‘in-page’ view, which wasn’t visible, rather than the ‘full-screen’ view.  For the jQuery-UI auto-complete widget there is an ‘appendTo’ option that allows you to specify which element the autocomplete is attached to.  By specifying this I was able to get the autocomplete to work.

Unfortunately the jQuery-UI tooltips didn’t appear to have have a similar option and I went round in circles for hours trying to figure out how to get the tooltips to appear.  I thankfully got there in the end, though.  It turned out that it was possible to append the tooltips to an element, but this was only required when the map was in full-screen mode and would break the tooltips if the map wasn’t in full-screen.  The code that sorted it is:

$(".tip").tooltip({
  open: function (e, ui) {
    if(document.fullscreen)
      ui.tooltip.appendTo("#map");
    }
});

It was such a relief to finally get this working, and with this in place I then began working on the major update to allow specific views of the map to be bookmarked / shared / cited.  I’m still in the middle of this as it requires a significant number of changes to be implemented. I added in the Leaflet hash plugin that stores the location and zoom level in the hash of the page URL.  I also ensured that the currently active base map (e.g. satellite view) is also similarly remembered, as it the currently active tab in the map menu.  It’s taken most of the day to get this far, and there’s still more I need to do, namely ensuring any selected filters are remembered and any selected location is highlighted and its correspondence loaded into the correspondence tab.

I’ll also need to see what I can do about remembering the position in the timeline.  This is going to be tricky as currently the data is loaded in sequentially each time the user presses the ‘Next’ button.  If we want to remember that someone is (for example) viewing item 13 when they bookmark the map then I’ll need to also ensure that the previous 12 items are loaded.  I’ll see what I can do, but it might end up that the timeline always reverts to the first item.  I’ll hopefully continue with this next week.

Week Beginning 21st October 2024

Posted on

I was back at work on Monday this week after being on holiday the week before.  Whilst I was away the external hosting company where many of our sites are now hosted performed a major upgrade to the default PHP version our sites run on, from 7.X to 8.X.  I hadn’t been given an advanced warning of this significant update and unfortunately when it took effect many of our websites stopped working, and the first I was aware of this was when people started sending me frantic emails.  Thankfully I was checking my emails whilst I was away and I spent a bit of my holiday manually reverting the sites back to 7.X, a temporary measure that got the sites up and running again.

On my return to work I spent all of Monday going through every site in order to upgrade any parts of the code that were causing problems with PHP 8.  I had to go through 24 sites, but thankfully in most cases it was the same issue that was causing problems each time, and once identified it was relatively easy to fix.  By the end of the day all of the sites were working properly with PHP 8.X, which was a relief.

The Anglo-Norman Dictionary people had also alerted me to an issue with the dictionary management system whilst I was away too.  When editing entries any language tags were being lost, not from the XML itself, but the entries were no longer being picked up by the language search facility.  I spent a bit of time investigating what was going on here.  When a new or edited entry is published search terms such as language tags are extracted from the entry’s XML.  When I created the new language tag system a few months ago I first applied it to the batch processing scripts that handle an entire batch of entries and once this was tested and working I then I copied the code into the script that handles the publication of entries via the holding area in the DMS.

In each script the extraction of language tags only applies to ‘main’ entries rather than xref entries and there’s a check to ensure the XML file in question is the correct type.  In the batch processing scripts this check uses a variable named $entrytype but in the DMS publication script it’s $pub[“entrytype”] and unfortunately I overlooked this.  The result was that the DMS script was assuming (when processing language tags) that every entry wasn’t a ‘main’ entry and therefore the language extraction section of the script didn’t execute and no language tags were extracted.

It turned out that many entries had been edited since I’d added in the language search, meaning all of these had stopped being findable in the language search.  Thankfully I still had the script I’d originally written to extract language tags from all active entry XML files and populate the language search system, so I was able to re-run this and restore the data.

Also for the AND people this week I had a Teams meeting to discuss the new funding application we’re putting together, and also engaged in several email conversations about this.  I can’t say much more about the application at this stage, but it will involve adding lots of nice visualisations to the site (amongst other things).  I had to spend some time this week researching and considering the technical aspects of the new proposal in addition to the time spent in the meeting and writing emails.

I also spent some further time this week working on the new place-names proposal to the BA with Thomas Clancy and gave some advice to Gavin Miller about a new AHRC proposal he’s putting together.  I also made some minor updates to the Iona place-names map and fixed a bug in the interface that was causing the map legend to appear multiple times if a user clicked the ‘reset map’ several times in quick succession.  When this situation arose, the new legend ended up loading in before the old one was fully removed, causing multiple legends to be displayed.  It took a little time to investigate and my solution is something of a hack but it works:  when you press the reset map button any subsequent clicks in the following second are now ignored.  This gives the map time to sort itself out without getting itself in a tangle.

Whilst investigating this issue I also spotted some issues with the ‘browse placenames’ feature.  When resetting the map the contents of the browse pane were not being reset, meaning the list of browse options would not necessarily match the heading shown.  Also after resetting the map the drop-down list to change the visible browse options was no longer triggering.  I managed to fix both these issues in the Iona map and then roll the changes out to the Ayr map too.

I also discovered this week that LiDAR map layers for some areas of Scotland are publicly available for use in map-based resources.  See this page on the NLS site: https://maps.nls.uk/guides/lidar and specifically the links to the available phases listed here: https://maps.nls.uk/guides/lidar/#re-use.

Unfortunately Iona has not yet been covered, but Phase 3 covers much of the Berwickshire area and KCB while Phase 4 covers much of Ayrshire and I contacted the place-names people to see whether we should think about incorporating the LiDAR data into some of the online resources.  This would need to be handled differently to the other map layers and I think would benefit from being an additional layer that users could turn on or off, sitting atop the base map a user has chosen and including an opacity slider.  This would enable the user to easily compare the LiDAR data to features shown on whichever map they’ve selected.  I reckon it has interesting research potential and the place-names people agreed.  I’ll just need to find the time to add this in.

Also this week I was in communication with the Speak For Yersel people as the new survey areas are going to launch in the next week or so.  We agreed on the steps that would need to be taken (from a technical point of view mainly just updating the live Scotland survey to link through to the other areas) and I also added in some social media links and icons for two of the three new surveys.

Finally, I heard back from Deven Parker this week about her Playbills project and the experiments that have been undertaken by someone in Computing Science using ChatGPT to extract data from the images.  This is all looking very promising.  I can’t say too much more about it at this stage, but I spent some time looking through the sample output and giving some feedback and suggestions to Deven.

Weel Beginning 23rd September 2024

Posted on

I spent the majority of this week continuing to work on the interactive map of Burns correspondence, most of it working on implementing the filter options.  I started developing the filters form within the map menu, but I came to the conclusion that the menu was just too small for the large number of filter options that we want to offer.  The following screenshot shows my initial attempt, with hardly any of the filter form visible in the menu.

Instead I decided to set the filters tab of the menu to display a summary of the currently selected filters and a count of the matching correspondence.  I then added in a button that opens up the filter options in a popup similar to the one used to display the biographical information about correspondents, as the following screenshot demonstrates:

The filter options are split into two columns, with the right column containing the categories in a scrollable area.  Both hierarchical levels are represented, and you can select / deselect all, or select / deselect an entire category group by pressing on the checkbox next to its parent.  On the left there are filters for correspondent and correspondence.  For correspondent you can search for an individual person.  This is an ‘autocomplete’ and if you start typing (e.g. ‘rob’) then a list of possible matches appear, from which you can select one.  I figured it wasn’t really necessary to allow users to search for multiple people as this would increase the complexity and correspondence only ever involve two people anyway.  In addition to selecting a person you can choose to filter by gender (e.g. all correspondence involving women).  You can also specify the role of the correspondent, e.g. limiting the map to showing correspondence that were sent by women, or received by ‘Clarinda’.  Leaving this option blank means the filter matches both sender and recipient.

For correspondence you can limit to a single date or a range.  I’ve added tooltips to give information about what exactly is possible, but you can search for a year, a year and a month or an exact day in each field.  You can also limit by location.  This isn’t an autocomplete but instead matches whatever you enter against the full location field.  I figured this would be more useful as (for example) you can enter ‘Edinburgh’ and find all locations in Edinburgh, or you can enter a specific place in Edinburgh.

Any combination of these filters can be chosen, although not all combinations will make sense or give results.  Pressing on the ‘Filter’ button at the bottom of the popup closes the popup and performs the search, limiting the data on the map accordingly.  When you press on a map marker the correspondence listed are similarly filtered.  After performing a search and closing the filter popup a summary of your filters is displayed in the tab (as mentioned earlier).  If you re-open the filter popup your selected filters remain in the form and you can either edit your selection or press the ‘Reset filters’ at the top of the popup to start afresh.

I had to do a lot of behind the scenes work to get all of this operational – lots of updates to the API to process the filter selection for both the map data and the list of correspondence.  I think it’s working pretty well and it’s much nicer to use the map when it’s filtered rather than showing all of the data.  Next week I’ll continue with the outstanding tasks mentioned last week:  the timeline, citation and bookmarking etc.

Other than working on the correspondence map I spent some further time on the article for the Books and Borrowers project, adding in new sections on the four-tier book structure, authors, genre and borrower occupations.  I also spotted an issue with the online resource’s advanced search, whereby the library registers were not getting added to the multi-select area.  I realised this was an unintended consequence of an update to the API I’d made a couple of weeks back to cut down on complex queries.  The call to return registers was running several subqueries to work out counts of things like associated borrowers and borrowings.  I had thought the call was only ever used when viewing registers for a specific library, and in this use case the query was pretty rapid as only a few registers at a time were ever involved.  I therefore blocked a call to the API that didn’t specify a library, not realising this call was actually used to generate the list of registers for the advanced search form.

Once I realised that the call was actually needed I decided to create static cached fields in the register table for the counts, rather than conducting the counts on the fly in subqueries.  With the new fields in place I then created a script that would populate the fields and updated the API to reference these fields rather than perform the additional queries.  This resulted in a significantly faster query time and the advanced search form now includes registers again.

Also this week I had an email conversation with Matthew Creasy about an interactive map we would like me to build.  I can’t say much more about it at this stage, but we’ve arranged to meet the week after next to talk about it in more detail.  I also spoke to Thomas Clancy about a possible funding application and made a few minor tweaks to the Speak For Yersel region resources, which will hopefully be going live fairly soon.

Week Beginning 9th September 2024

Posted on

I spent a fair amount of time this week continuing to work on the Burns Correspondence map.  Map markers are now currently all in grey by default, and I’ve added in grey dotted lines showing the connections between places.  This results in a very cluttered map with all data visible, but of course once filters are applied (not yet developed) the map will contain less data.  It is also now possible to press on a marker to highlight the place, the connections and the places connected to.  This highlights the pressed on place with a red border, turns the connecting lines red and highlights the connected locations with a black border.  I had previously allowed the user to select multiple locations but I decided this just made things messy so now if you select a different location your previous selection is unselected and replaced with the new selection.

I also worked on a map menu, which now has four tabs:  Home (which will include an introduction to the resource and a ‘reset map’ button (not currently operational); Correspondence, which will contain all information pertaining to the location the user presses on; Filters which will contain the filter options and Timeline, which will include the facility to animate the correspondence over time.

I’m still in the middle of working on the Correspondence tab.  When you press on a location marker this tab gets selected (if the menu is currently hidden it opens) and all of the correspondence sent from/to this location will be listed, in date order.  For now only the dates are displayed, with correspondence items split into sections by year.  There is going to be rather a lot of data contained in each of these sections, and I might add in an option to further expand the menu to help people view the data.  We might want to give the option of switching from a ‘compact’ to an ‘expanded’ view of the data, with things like the bibliographies only appearing in the latter.  I did this for the Books and Borrowing project and it worked pretty well – see https://borrowing.stir.ac.uk/search/0/0/0/advanced/bhids|9803 with the ‘switch to…’ option in the menu bar to the right of the ‘Search Results’ text.  I might also add in a further highlighting option whereby people can press on the correspondence item in the menu and see its corresponding line on the map.  Below is a screenshot of the ‘full screen’ view with ‘Lochlie’ selected so you can see how things currently look.  There’s clearly still a lot to do, but I feel that I’ve made some pretty good progress this week.

I participated in two meeting this week with researchers who are putting together proposals that I’ll be involved with.  The first was with Garrick Allen in Theology and the second was with Emily Pickard in Scottish Literature.  I needed to spend some time prior to each meeting reading through the proposal documentation and preparing some notes and questions, and then some further time after each meeting writing some materials to be included and responding to questions raised.  Both meetings went very well and hopefully both researcher will secure their respective funding.

I also had a further email conversation with Gerry Carruthers and members of his team about their new proposal and fixed a small bug in the new DSL sparklines that I’d introduced last week.  I also removed a video that needed to be taken down from the Speech Star website, made a few further tweaks to the contents of the Irish and Northern Irish surveys for Speak For Yersel and make a few minor changes to the Anthology of 16th and Early 17th Century Scots Poetry.

I spent the remainder of the week working on the article about the development of the Books and Borrowing resource that I’m writing with Katie and Matt.  I made good progress on my sections and I hope to get a first draft finished next week.

Week Beginning 26th August 2024

Posted on

This week I returned to working for the Books and Borrowing project, spending about a day creating a new version of the public data that incorporated some large additions to the Orkney dataset that one of the researchers has been working on over the past few months.  It takes quite a bit of time to prepare the data for publication, which results in a new version of the site’s Solr index plus the generation of several other cached files.  I also had a chat with project PI Katie about a potential publication about the project’s online resource.  We’re probably going to speak more about this next week.

Also this week the updates I’d made to the DSL’s Solr index last week went live and I also fixed an issue with the sparkline tooltip text in the search results on touchscreens, which one of the DSL team had spotted.  This issue was that the search results are block-level links – pressing on anywhere within the area of a result triggers the loading of the entry.  As the sparkline is within this block pressing on it was also triggering the load.  As there is no ‘hover-over’ on touchscreens the sparkline tooltip can only be triggered by pressing on it, which also triggers the page load.

Thankfully there was a quick fix for this:  a small amount of JavaScript that ensured that pressing on the sparkline no longer triggers the page load.  You can still press anywhere else in the result block to load the entry, while pressing on the sparkline now makes the sparkline tooltip appear and nothing more.  I tested this out on my iPad and Android phone and it’s working fine for me.

I also spent about a day making updates to the new Speak For Yersel regions (these regions are not yet publicly available).  Following pilot testing of the regions I’d been given rather lengthy lists of things to change for Ireland and Northern Ireland.  These ranged from changing the question and answer option text to replacing or deleting entire questions or swapping the sound files for new versions.

One of the researchers also wanted the order for the Ireland survey questions to be randomised each time the survey loads and in order to implement this I had to update the database structure and the survey setup script to add in an ‘random question order’ option when creating each survey.  I then needed to update the script that pulls questions into the survey so that the order was randomised when this option has been set.  And as the question number that is displayed in the top-right of the question pane was dependent on the fixed ‘question order’ field stored in the database I had to ensure this data was reset after the question data was randomised and before it was used in the front-end.  So there was quite a lot to update, but it’s all working now.

Also this week I responded to questions about WordPress plugins from Petra Poncarova about her Erskine site (https://erskine.glasgow.ac.uk/) and Amy Wilcockson about the Robert Fergusson site (https://robert-fergusson.glasgow.ac.uk/).

I spent the remainder of the week beginning work on the front-end for the Robert Burns correspondence map.  I’d completed work on the CMS for this project several months ago and since then a researcher has populated it with data about locations, people and correspondence and it’s now time to see how all of this will work on an interactive map.  Some of my time was devoted to planning how the resource will function and some further time was spent setting up the basic infrastructure and an initial map.

For now the map uses the  six-inch Ordnance Survey maps (1843-1882) from the NLS and the same base-map as I used on Gerry McKeever’s Regional Romanticism map (https://regionalromanticism.glasgow.ac.uk/paul-jones/), although this may change.  Currently the map appears within the layout of the site, with a ‘full screen’ option in the top left, but I may change this to make the default view take up the entire browser window, as with the Paul Jones map linked to above.

I decided that I wanted to try and have a map menu that appeared across the full width of the bottom of the map, but was unable to find a Leaflet plugin that offered this.  It’s likely that this is because Leaflet plugins have to be added to a specific container that is associated with either the top left or right or bottom left or right of the Leaflet map – there doesn’t appear to be anything that expands across the full width of the map.  I tried to implement a layout that would break out of these confines but didn’t manage to find a CSS solution within the Leaflet framework so instead I created my own pane using jQuery.  This currently takes up the bottom third of the map, has a semi-transparent background and has a button that slides the panel down and hides it, and another button that slides the panel up and reinstates it, as you can see in the following screenshot:

The menu is currently empty but it’s here that I will place the timeslider to limit the data to specific dates, plus options to filter the map by categories and people.  Currently every location in the system is displayed as a red dot on the map, but as of yet there is no information about correspondents and correspondence.  I will add lines between locations to show correspondence and I still need to figure out how I’m going to add correspondents and display the information about each correspondence.  I’ll continue with this next week.