I divided my time this week primarily between three projects: SCOSYA, RNSN and the Historical Thesaurus. For SCOSYA I made some changes to the ‘story’ interface to the atlas that I began work on the week before last. Feedback from the project team suggested that cropping the tessellation would be a good thing, as with the exiting prototype the polygons and lines expand beyond Scotland and into England and Ireland, which would give users the wrong impression. My suggestion of incorporating a slider to change the opacity of the polygons was also seen as something that should be implemented. It turned out that implementing these two changes would be rather more time consuming and tricky than I had anticipated. First of all I tackled the cropping of the tessellation. After doing some research it would appear that there is no easy way to specify a boundary for a Voronoi map, beyond which the cell borders aren’t shown. However, it is possible to place a mask over areas of the map that are not relevant. This cuts off the Voronoi cells, but does also completely obscure any part of the map covered by the mask. Making the mask transparent wouldn’t work as then both the map and the cells would show through. I’ve created a mask and made it grey. The edges of the mask are possibly a bit clunky and I should maybe refine them further (e.g. making them less pointy). Here is an example of the mask in place:
It possibly looks like it would have been the work of about 5 minutes to add this mask in, but it was actually pretty complicated to get it to work. Initially I tried adding the polygon as a layer in Leaflet / MapBox, but unfortunately this then appeared beneath the Voronoi cells, as the Voronoi layer is generated by D3.js and is placed on top of the map. I’m stuck (for the time being) using an old version of Leaflet that doesn’t provide the facility to change the z-index of layers, so instead I looked to create the mask using D3. Every time the map is zoomed or panned the mask needs to be recalculated and redrawn, otherwise it just stays at the same dimensions and obscures the wrong things. I spent some infuriating hours with the mask appearing in the wrong place, moving to the wrong place, failing to change size on zoom and other such things. There were also issues relating to being able to click on the cells, as the mask sits on a separate layer, which initially meant even though the cells showed through the hole in the mask, clicking on them didn’t work as clicks were being blocked by the mask’s layer. However, I eventually managed to get things working, and the mask no behaves as intended.
I then looked to implement the opacity slider, which also proved to be tricky to get working. Implementing the slider itself was very straightforward – I just used the jQuery UI library which includes a slider. This was then set up to change the transparency of certain elements when the user interacted with it. I’m using the transparency option in RGBA in CSS (A is the ‘alpha channel’, which is the transparency). This is better than using opacity in CSS, as changing the alpha channel affects the individual element along whereas updating the opacity affects both the current element and everything contained within the element. However, one slight annoyance is it’s not (currently) possible to just update the alpha channel via JavaScript, instead you have to replace the entire RGBA statement. As I have 6 different colours (plus 6 different ‘selected’ colours) all of these need to be individually updated each time the slider moves. E.g. Rating level 5 has the fill colour ‘rgba(189,0,38,0.5)’ by default. If the slider is moved to make the cells fully opaque the code can’t just say ‘update all classes that have an alpha channel of 0.5 to make this 1’, but instead I need to redefine each individual colour, e.g. change ‘rgba(189,0,38,0.5)’ to ‘rgba(189,0,38,1)’. It’s a shame there isn’t a quicker way of doing this.
The biggest difficulty I encountered with implementing the slider was getting the cells to actually update. I could get the slider to change the transparency of elements elsewhere on the page, but no matter what I did I just couldn’t get the change to update the SVG elements on the map (i.e. the Voronoi cells). I spent ages trying to get to the bottom of this. The jQuery UI code could interact with the cells, turn them off, find their contents etc, but updating their styles just did nothing. It turned out that the reason was the map was set to constantly refresh itself (to ensure that the Voronoi layer always appeared in the right place), and therefore any changes to the opacity made by the slider were immediately being overwritten by the code that displays the map. Once I figured this out I could instead update the code that refreshes the map, and the slider thankfully could then control the opacity of the cells. The final implementation of this was very straightforward, but it took an awfully long time to reach the point where I knew where and how to update the code. You can see an example of the slider below:
I also spotted some issues when loading a new dataset into the map (e.g. going from the H1 data to the H2 data when navigating from one slide to another). In such cases the H1 data wasn’t being removed, and the H2 data then sat on top of the H1 data for each cell. As the colours of each cell might be different depending on the data this introduced quite a lot of flickering and unexpected changes of cell colour on pan and zoom. Again, this was tied into the frequency of map refreshes, with the map getting refreshed with the ‘old data’ after the point when the ‘new data’ was first loaded in. By ensuring the map only redraws itself when anything actually changes I managed to fix this. I really ought to completely rewrite this prototype at some point as the current implementation is a bit of a hack – cobbled together from many existing libraries that probably shouldn’t work together and using older versions of libraries because the newer versions don’t work with example code I started with. Currently it uses an older version of D3, an older version of MapBox, which in turn uses an older version of Leaflet, together with jQuery and jQuery UI. The example I started from is https://chriszetter.com/voronoi-map/examples/uk-supermarkets/ which uses these older versions and won’t work with newer versions due to there being major changes between versions of D3 and Leaflet. A rewrite is going to be a fairly major job.
For RNSN I spent some time reworking the song stories as we’re getting pretty close to publication time for these. Previously the pages had no titles, which made them tricky to differentiate in the WordPress Admin interface, had no ‘Featured Image’, so didn’t really fit in with the other pages of the site, had no footer (well, the footer was there but was hidden by the story), and the story pane became obscured by the site title on narrow screens. I have now fixed all these issues. All story pages now have titles, which appear above the story panes on the public pages and appear in the list of pages and the ‘add to menu’ page in the admin interface. All story pages now have the ‘featured image’ associated with their nation at the top of the page, except the ‘over-arching timeline’ which currently features the ‘dancing’ image. The story pane now has a bit of a margin to the left and right, whereas previously it extended to the very edge of the browser window, and I’ve given the pane a border, which I think works better as it shows where the story area ends. I’ve also tweaked the text areas in the timeline to make the area a bit wider. The footer now appears below the story pane, which I think is pretty important as it contains partner / funder logos. Plus without the footer the pages looked rather incomplete. Finally, on narrow screens the story pane is never obscured by the title, thanks to the inclusion of the featured image.
For the Historical Thesaurus I returned to the OED / HT linking task. Marc had requested some updates to the lists of unmatched categories. The initial summary page now displays the total number of words found in unmatched categories, plus a breakdown of how many categories feature how many words. E.g. in the 1894 unmatched OED categories that are not empty there are 3451 words, and 1136 of these categories contain one word each. This information is provided for both OED and HT datasets (for HT the total number of words refers to non-OE words). I also created a new script that goes through all of the matched categories and works out how many lexemes can be automatically matched on word form only. The script lists all 222433 matched categories in a table with one row per category. Each row displays the information about the HT and OED category, plus the ‘match process’ that was used to make the match. It also has columns listing counts of the HT and OED words contained in the categories and full lists of these words. For each of the matched categories, the script compares each HT and OED word (the ‘word_stripped’ and ‘lemma_stripped’ fields) and makes a count of the number of words that match. This count is then displayed in the final column. Where the number of matches is the same as the number of OED words the entire row is given a green background. Where there are words that don’t match these are highlighted with a red background in the lists of words, with words that do match appearing with a green background.
Fraser also got back to me with a further batch of categories that could be ticked off, reducing the number of unmatched OED categories (with a POS and containing words) from 1894 to 1264. I also spent a bit of time investigating travel arrangements for the Bergamo conference in April.
In addition to these tasks I met with Matthew Creasy on Wednesday to discuss the digital edition interface I’d created for his Decadence and Translation project. He’s very happy with how things look and will try to supply me with some updated content before he demonstrates the interface to project partners in February, after which I’ll update the interface if we get any feedback. I also met with Charlotte Methuen to discuss a project she’s currently putting together. I agreed to help out with the technical side of the proposal and will no doubt be writing a Data Management Plan for her next week.
I worked on a lot of different projects this week, which was a bit of a change from last week’s focus on a single project. For the RNSN project I investigated how to create an over-arching timeline that would pull in data from all of the individual timelines I’d created previously. I had investigated a variety of timeline libraries, but none of them really offered as good a level of functionality as the timeline.js library I was already using for the individual timelines, so I decided to focus on adapting this library to work with multiple timelines. Thankfully the library already included support for ‘groups’, and by assigning slides to the same group they would appear on different ‘tracks’ in the timeline summary section. For my initial version I created a ‘group’ for each of the individual timelines, and a further ‘group’ for the spreadsheet of ‘major events’ that Kirsteen had previously sent me to give context to the individual stories. This initial version mostly worked pretty well, although I needed to tweak the CSS and some of the library’s JavaScript somewhat in order to make the group names more clearly visible in the summary section and to make the group names appear within the main body of the slides too. On Friday I met with Kirsteen and Brianna, the project RA to discuss our future plans for the project and also to discuss the new timeline. Kirsteen was very happy with how the timeline was coming along, and during the meeting I proposed some further changes. In order to reduce the amount of space taken up by the summary section I suggested we have a ‘track’ for each nation rather than each song, and I suggested we ensure there is a link to each song’s full timeline from every slide as well. Kirsteen suggested changing the colour of the contextual slides to more clearly differentiate them from the regular slides too. I spent some further time implementing these updates and by the end of the week things were looking good. I’ll still need to add in more content and to check how the interface works on narrower screens, but I’m pretty happy with how things are working out. Here’s a screenshot of the timeline as it currently looks:
I also spent a bit of time working on Matthew Creasy’s Decadence and Translation project. Matthew had digitised a number of pages of short French poems that he wanted to make available as a sort of mini digital edition that would involve a zoomable / panable interface to the images, plus textual transcriptions and translations with embedded notes. Thankfully I’ve had quite a bit of previous experience with such interfaces (e.g. the ‘Digital Ode’ for the New Modernist Editing project) and could use a lot of existing code. I couldn’t make things exactly the same as Matthew wanted people to be able to comment on each page, and as the rest of the project website is a WordPress site, it made sense to use WordPress’s commenting features. For this reason I created a WordPress page for each digitised image, and embedded a small amount of code within each page. The actual content was stored in a JSON file I created, and the only change needed between individual WordPress pages was to change a numerical ID. My JavaScript code would then grab this ID, pull in the JSON file and extract the entry with the corresponding number, find the image tiles and set up an OpenLayers interface to allow the image to be zoomed, pull in the translations, transcriptions, notes and other things as HTML text and format everything for display. WordPress would then add in its regular comment boxes and that would be that. Matthew had sent me test information about six images and I added all of this to my JSON file. Everything seems to be working pretty well so far, and I just need further input from Matthew now. Here’s how things currently look:
Also this week I created an updated of a Data Management Plan for Thomas Clancy (the fourth version and possibly the last), updated the test version of the SCOSYA atlas to limit the attributes contained in it before a class uses the interface next week, made some further (and possibly final) tweaks to the Bilingual Thesaurus and migrated the Thesaurus of Old English to our new Thesaurus domain and ensured the site and its data all worked in the new location, which thankfully it did. I’ll need to hear back from Marc and Fraser before I make this new version ‘live’, though. I also made a small tweak to the DSL website, started to think about the next batch of updates that will be required to link up the HT and OED data, and did some App store related duties for someone elsewhere in the University. So all in all a pretty busy week.
This was my first week back after an enjoyable but all too rapid Christmas holiday. I spent most of Monday catching up with emails, writing a new and up to date ‘to do’ list and upgrading all of the WordPress sites I manage to the new Version 5 of WordPress, which has introduced a horrible new editor in place of the perfectly good one that was there before. I decided to do a full backup of all of the sites before I upgraded them, which took some time, but it’s good to have a local backup of everything, just in case the main backups go wrong. Rather annoyingly when I upgraded WordPress on Monday it was version 5.0.2 any by the end of the week 5.0.3 had been released, so I’ll have to upgrade all the sites again sometime.
I also spent a bit of time this week making further updates to the RNSN song stories, based on feedback received from the extended project team over the past few weeks. Thankfully these were mostly minor tweaks to content rather than big requests for major changes. I also made some updates to the Data Management Plan for Thomas Clancy’s Iona project and met briefly with him to discuss some related matters. I also gave some advice to Meg MacDonald and Bryony Randall about the proposal Bryony is putting together and had a chat to Luca about one of Graeme Cannon’s old projects that Luca is helping to update.
Other than these issues I spent the rest of my week working on the SCOSYA project. I had been hoping to get started on the public interface for the Atlas in the run-up to Christmas, but had been unable to find the time to work on it due to other work commitments. Somewhat amazingly, this week proved to be rather quiet (not having anything to do for the Historical Thesaurus for once certainly made a big difference) and it was great to have a nice big block of time to devote to the project.
Previously we had agreed that several guided ‘stories’ through views of the data would be good to have, and E, the new RA on the project sent me a draft version of a story based on the data for ‘Gonnae’. I’d suggested using the ‘Storymap’ interface (see https://storymap.knightlab.com/) as a means of navigating through different views of map-based data, and E and I had agreed that I’d also investigate ‘Voronoi’ maps as a means of extrapolating point-based data into broader areas.
I had found a nice example of a Voronoi map that was created using the d3.js library, which I’ve had quite a lot of experience with. It’s a map of supermarkets in the UK, and you can view it here: https://chriszetter.com/voronoi-map/examples/uk-supermarkets/.
I decided to try and get a Voronoi interface working with our data as a first step, using the source code for the ‘supermarkets’ map as a starting point. Below is a screenshot of my first attempt:
This version uses the ‘Gonnae’ data (an adapted version of a CSV file generated by the project’s API) and our base map. You can see in the above screenshot how the ‘cells’ are generated based on our questionnaire locations, but no colours are used to differentiate rating levels, the checkboxes in the top right don’t actually do anything and the text in the bottom left still says ‘explore supermarkets in the UK’ until you click on a cell, at which point the location name is displayed.
For the next version I wanted to properly integrate the map with our website interface, investigate how to shade the cells, and replace the ‘supermarket’ text in the bottom left with something more useful. Below is a screenshot of this version:
This version also pulls its data directly from our API whereas the previous version used a stripped-down CSV file I’d manually made from the API data. This is why this version has slightly different cell layouts (e.g. it includes cells for where there is no data for this question). Cells are given different colours depending on the average rating (the same colours as on the main point-based map), with grey used for locations where there is no data for this question. Information about what the map shows (‘Gonnae’) is found in the bottom left, and clicking on a cell highlights it and displays the location name, average score and what this score means in the bottom left too. The cell colours are possibly not different enough to easily distinguish them, and the opacity level does make it rather difficult to make out the underlying map, but these are things that can be tweaked. I also included points to show where the actual locations are. I personally find these quite helpful, but they can always be removed.
With this version in place I then decided to try and get the storymap interface working with our base map. This is where things started going wrong. The Voronoi script uses a rather old version of the MapBox library, which in turn uses an older version of the Leaflet mapping library. I tried switching to a more recent version but unfortunately this breaks the code. I could (and I still might) go through the code line by line to figure out what function calls need to be updated for the script to work with a more recent version, but this is likely to be a tricky and time-consuming process, and one that might not actually be successful. The issue is that storymap requires a more up to date version of the MapBox library to function, and even getting it to work with our basemap in a way that would be compatible with the Voronoi code took a lot of hacking about with the storymap source code. In the end I managed to get storymap to work with our base map, but failed to get the Voronoi cells to actually display. Here is a screenshot of this version, to get an example of how the story would work:
As you can see, the information from the ‘Gonnae’ document is there, as is our base map. A Youtube clip from Chewin the Fat is embedded in the first slide and as you navigate between slides the map pans and zooms. It’s obviously not that great because the map doesn’t display any data. But in addition I realised that the storymap interface might not be all that suitable anyway. The ‘slide’ part takes up half the width of the browser window, which obscures far too much of the map. Also, the map icons for the slides and the dotted lines aren’t really much use to us either.
I therefore decided I’d write my own version of the storymap library from scratch – a version that would work with the Voronoi script, would take up less room on the map, and would only include features we actually needed, plus new features that storymap didn’t include that I would otherwise have had to have added in (e.g. triggers to change the data displayed on the underlying map when navigating from slide to slide). I say ‘from scratch’ but it wasn’t really as bad as all that as I could reuse a lot of code I’d written for previous projects.
It’s taken quite a while to implement all of the features, but I managed to complete a working version by Friday. It’s still just a proof of concept and will need further work, but here is a screenshot of it:
And here is some information about how it works:
That’s as far as I’ve got for now. I have sent the URLs off to E and Jennifer Smith for feedback, and will probably not do any further work on this until I hear back from them. There are other features we might also want to implement, e.g. an option to switch from Voronoi cells to points, or a slider to dynamically change the opacity of the cells, or options to allow us to highlight a particular group of cells for a slide. I also still need to make this interface work better on mobile phones (tablets should already be ok, though) and try to add in a ‘full screen’ option, which might prove to be tricky. But all of that is for another week.