I was on holiday last week, although I kept up to date with my emails whilst I was away, so at least I didn’t have a backlog waiting for me when I got back. I did have quite a lot to do this week, though, and the most pressing was to complete the rollout of the new XML entry structure for the Dictionaries of the Scots Language. I’d begun this process the week before my holiday, but had run into difficulties getting the Solr indexes set up on the server, and while I attempted to resolve this with Andy in IT Services we ran into difficulties were were unable to resolve before I headed off.
After further investigation and discussions with Andy this week it turned out that there were two issues. Firstly the ‘conf’ directory for Solr indexes needed to go into a different directory on the server compared to how things work on my laptop, and how things used to work on the server. Rather than appearing in the index’s own directory, the ‘conf’ directories need to be stored in directories that have the index name within the configsets directory, otherwise the creation of the index fails. Secondly, when Andy did manage to successfully create the first index the week before last he unfortunately got the name slightly wrong, which neither of us spotted in the rush to get things finished on the Friday afternoon before my holiday. This then prevented the command to index content from working, as Solr was attempting to post files to an index that didn’t actually exist.
Thankfully these issues have now been sorted, the ‘entries’ and ‘quotations’ indexes now work. I was then able to update our test website to use the new indexes, meaning the website now includes the display of the new entry structure, data extracted from the new XML structure and also the map popups. The DSL team can now experiment with all of this and get back to me with any tweaks or updates that they require.
Also for the DSL this week, I dealt with the last item on my list of feedback for the map of regions and dialect areas. I’d suggested that we should include facilities to cite or share views of the map, and also make use of the URL shortener I’d developed for the place-names resources and this was something the team wanted me to implement. By the end of the week I’d completed a first version of this and had added it to the map.
Initially I had intended to have a ‘cite’ popup as I use on the place-name resources, but then I thought it would be better to use the same ‘share’ options that we use elsewhere on the DSL site for consistency. I added the ‘share’ buttons in a row at the bottom of the left-hand menu and got things working with the full page URLs appearing in the ‘share’ output.
However, I ran into something of a brick wall when implementing the URL shortener. I tried many different approaches to get the shortened URLs to replace the default page URLs in the ‘share’ options but nothing would work. The reason is that the URL shortener posts data to the server to generate the shortened code and then saves this with the full URL in the database, but despite the ‘share’ plugin allowing custom URLs to be added in, there does not appear to be any way to make its code hang around until my script connects to the server and generates the shortened URL – it just jumps ahead, thinks no custom URL exists and uses the default page URL regardless.
After attempted many different ways to get around this, but failing to find anything that would work, I then decided to revert back to my ‘cite’ popup option, as this would enable me to generate the shortened URL when the popup is opened, meaning it’s ready and waiting when the user clicks the ‘share’ option contained within the popup. Plus in addition to the ‘share’ options we can also include the citation styles for people who want them. Thankfully this approach has worked, as the screenshot below demonstrates:
You can see the shortened URL in the ‘cite’ options (redacted as the resource is not yet live), and this shortened version is also used in the ‘Share’ options. This is still just a first version of the ‘cite / share’ feature, however, and further changes may be required. For now there is no custom text in the ‘share’ or ‘site’ options describing what the map shows. It would be quite a lot of work to add this in, so I’ll just see if the DSL team want me to do it. It would mean that rather than just saying ‘Map of regions and dialect areas’ a citation might say something like ‘Map of the regions of Scotland centred on [lat,lon here] with regions Ayrshire, North Ayrshire and West of Scotland selected’.
Also this week I had a meeting with the HiMuJe Malabar team to discuss the development of the interactive map. It was a very useful meeting and we have a much clearer idea of how things will develop now, and how the map will relate to the digital edition, which is something that is being managed at a partner institution. This led on to discussions about issues Ophira was having with displaying the digital edition files in Oxygen on her PC. The digital edition is one long block of prose and this is causing Oxygen to get rather laggy when editing it in the ‘author’ view. I asked my colleague Luca to have a look at this, as he has more experience with Oxygen than I do, and he was able to come up with a solution that splits the single XML file into multiple smaller files. This approach seems to have worked very well, but we need to see what the project partners make of this change before it can be adopted by the project.
I also met with Henry Ivry this week to discuss the development of a new online resource for the Beniba Centre. We had a good chat about the various options and what might need to be done and Henry is going to get back to me with some further information that I can then use to get things started. I also made a bit of progress with the place-names of Armagh project with Mícheál Ó Mainnín at Queens University Belfast. We now have a request submitted for the domain and hosting and hopefully this will all get set up in the next couple of weeks.
I also fixed an issue with the display of an entry in the Anglo-Norman Dictionary. A malformed <link_loc> was causing issues with the page. The <link_loc> should have a comma (1,175c), but this was missing. The JavaScript that processes the references expects to find a comma and when it doesn’t it throws an error, meaning all of the other JavaScript stops working (including the parts that rearrange the author’s initials and numbers). I offered to update the code to deal with this a little more gracefully, but the editor Geert didn’t think it was an issue that would crop up very often and didn’t think it was worth doing.
Also this week I was contacted by Alison Wiggins to ask me to do some work on her Mary Queen of Scots Letters project, which has lain dormant for several years. Alison now has some time to work with the data and get things moving again and she had a few questions and requests, which I attended to.
This week I continued to make the updates required to support the new entry XML structure for the Dictionaries of the Scots Language. I put in a request for the new Solr indexes to be set up on the server hosted by the University, and put in further requests to upload the new database tables and to replace the WordPress uploads folder on our test site with the folder from our live site so that the test site would more closely reflect the content and structure of the live site. The last two items were sorted pretty rapidly, although they did require some input from me. Unfortunately the first item has proven to be very tricky, and despite submitting the job on Monday afternoon we still didn’t have the new Solr indexes in place by the end of the week.
It would appear that the version of Solr running on the server requires different commands to set up and index new indexes compared to the commands we previously used. It’s all a bit frustrating as the documentation is not as clear as it could be, and the examples that are given are simply not working when we use them with our data and configuration files. I was really hoping to have the new version of the site up and running on our test server by the end of the week, as I’ll be on holiday next week, but unfortunately despite the best efforts of Andy in IT Services and a lot of input from me we were unable to get this completed. We’ll have to get back into it the week after next, once I’m back from my holiday.
However, I did manage to complete the other updates to the systems that were needed to support the new XML data structure, such as updating the XSLT files to format the new elements as requested. This included adding in formatting for new sensitivity labels (which are also added to the ‘sticky’ header that stays on screen when you scroll down the entry) and ensuring the new nested <combGrp> data are displayed, as well as sorting out numbers, treatments and a bunch of other elements. And example entry showing the new labels can be seen below:
I also helped to sort out some issues with the DSL’s mailing lists by checking their DNS settings and requesting some changes to the records, which are managed by the University. And I devoted some time to making updates to the new region and area maps based on feedback that had been sent to me by Vasilis this week. This included updating the ‘info panel’ to ensure it can be hidden whilst still retaining the selected content, adding in a visible and distinctly styled border between Scotland and England and ensuring the zooming and panning between areas is handled more smoothly when content is selected from the drop-down lists or as a result of a search, amongst other things. I still need to add in a ‘cite/share’ option, although this is going to possibly take some time as I’ll need to decide what information needs to get displayed in the cite text and how this should be generated.
Also this week I met with Ophira and newly appointed RA Renu to discuss the interactive map I’ll be developing for their project. Details are still a little vague at this stage, but I game them some hopefully helpful suggestions and we’re going to meet again the week after I’m back from my holiday. I was also supposed to meet Henry Ivry this week to discuss the new project website I’ve set up for him, but unfortunately he needed to postpone this. I did manage to get a bare-bones version of the new website set up for him, though.
Also this week I continued to work with Andrew McHugh and Luca to fix the issues that were affecting the Emblems websites. The sites have been swamped by bot traffic and other malicious requests which have been bringing the sites down. Thankfully Luca was able to come up with some solutions to address this and by the end of the week the sites were a lot more stable, which is quite a relief. Also this week I arranged an Arts Developers coffee and catch-up, this time also involving Paul Grimwood, who is working as a systems developer for Andrew’s RCaaS unit. It was very interesting to learn more about what this group is up to and how things might develop in future.
I also replied to a request from Geert, the editor of the Anglo-Norman dictionary, to output all of the entries beginning with ‘A’ as a Word file. It took quite a bit of time to get a script to output all of the entries beginning with ‘A’, as the processes that are normally applied to one entry instead had to be applied to 4070 entries. I created a static web page that lists all of these. I attempted to copy and paste the output into Word, a process which did eventually work after about half an hour. However, for me at least this file is practically unusable as Word becomes unstable when the file is open, as it is several thousand pages long. Geert had been intending to print off the output to work through it all in hard copy, but thankfully he reconsidered this idea!
I spent a lot of my time this week working with the new entry XML data structure for the Dictionaries of the Scots Language. I received an export of all of the XML data from the DSL’s editing system on Monday and set to work updating my data processing scripts, the database and the Solr index structure, all on my laptop at this stage. I have updated the citation extraction script so that it now extracts senses at all levels throughout the entry document. I’ve also updated the script to deal with <cit> elements that contain multiple <cref> elements, as previously only the first was being extracted. The number of entries in the newly processed data has gone down from 78,165 to 78,141. The number of extracted citations has gone up from 748,947 to 752,250.
I also created a new ‘citations_geo’ table that holds information about the <geo> tags found within each citation. This is a separate table as a citation may have more than one <geo> tag. After extracting the data this table now contains 172,018 rows relating to 171,290 citations from 22,847 entries (all SND). I then updated the Solr indexes (both ‘entries’ and ‘quotations’) to include new fields for the <geo> data. These allow multiple <geo> information to be stored. I then needed to update the scripts that process the data for ingest into Solr to incorporate these fields, using the new table mentioned above. These new fields in Solr will be used for region searching and filtering. If we’re only wanting to add region searches to the quotations search the updates to the ‘entries’ index are not strictly necessary, but I thought I’d add this in just in case we want to use it in future. For example “show me all of the entries that have at least one quotation from Edinburgh and at least one quotation from Shetland”.
I’ve also begun to update the XSLT to format the new elements in the entry page. So far all I’ve done is update the <geo> tag to add in the mini-map popups, as you can see in the following screenshot:
These use the same tooltips as we use for the sparklines etc. I’m not going to do much more with the styling of them for now as these tooltips will be replaced by a different library when I work on the new site design anyway. Next week I’ll continue to make the necessary updates to the XSLT and I’ll then get everything transferred from my laptop to our test server. I may also begin implementing the region search / filter option, but I need to get some feedback from the DSL team before I can so this.
The DSL ream had also spotted an issue with the dictionary browse feature missing off the first few entries from the dictionaries. This was a strange one as I was unable to replicate the issue, and it didn’t seem to be related to browser or operating system. After a lot of testing I finally managed to replicate it on my iPad, and after that I was able to figure out how it was being triggered. The issue was happening on screens that are not tall enough to display the full-size browse pane. The reason this wasn’t being triggered on my phone is because the screen was narrow enough to make the browse appear underneath the entry, meaning there was enough space to display the full browse. On iPads and other displays that are wide but not so tall a compact version of the browse that contains less entries is displayed. The code that returned the limited list was chopping five items off the start and end of the full list, meaning at the start of the dictionary the first five items were never getting displayed. This was only affecting the start of the dictionary and not the end – I’ve just checked with ‘zulu’ – and I’ve now fixed it.
Also this week I created all of the necessary pages and uploaded all necessary files for a new edition of Johann Nepomuk Hummel’s Scottish Songs for George Thomson that will be added to the ‘Editing Burns’ website. These pages are not yet live, but I completed work on them, and met with the researcher, Marjory Rycroft, who compiled the data and undertook the research.
I also had an email conversation with Henry Ivry about a new website he wants to set up. I’m meeting with him to discuss this further next week. I also had to spend some time with Luca and Andrew trying to fix the problem of the Emblems websites being swamped by non-legitimate page requests. By the end of the week we’d managed to address a lot of these, but there is still more that needs to be done to improve the site’s stability.