I divided my time this week primarily between three projects: SCOSYA, RNSN and the Historical Thesaurus. For SCOSYA I made some changes to the ‘story’ interface to the atlas that I began work on the week before last. Feedback from the project team suggested that cropping the tessellation would be a good thing, as with the exiting prototype the polygons and lines expand beyond Scotland and into England and Ireland, which would give users the wrong impression. My suggestion of incorporating a slider to change the opacity of the polygons was also seen as something that should be implemented. It turned out that implementing these two changes would be rather more time consuming and tricky than I had anticipated. First of all I tackled the cropping of the tessellation. After doing some research it would appear that there is no easy way to specify a boundary for a Voronoi map, beyond which the cell borders aren’t shown. However, it is possible to place a mask over areas of the map that are not relevant. This cuts off the Voronoi cells, but does also completely obscure any part of the map covered by the mask. Making the mask transparent wouldn’t work as then both the map and the cells would show through. I’ve created a mask and made it grey. The edges of the mask are possibly a bit clunky and I should maybe refine them further (e.g. making them less pointy). Here is an example of the mask in place:
It possibly looks like it would have been the work of about 5 minutes to add this mask in, but it was actually pretty complicated to get it to work. Initially I tried adding the polygon as a layer in Leaflet / MapBox, but unfortunately this then appeared beneath the Voronoi cells, as the Voronoi layer is generated by D3.js and is placed on top of the map. I’m stuck (for the time being) using an old version of Leaflet that doesn’t provide the facility to change the z-index of layers, so instead I looked to create the mask using D3. Every time the map is zoomed or panned the mask needs to be recalculated and redrawn, otherwise it just stays at the same dimensions and obscures the wrong things. I spent some infuriating hours with the mask appearing in the wrong place, moving to the wrong place, failing to change size on zoom and other such things. There were also issues relating to being able to click on the cells, as the mask sits on a separate layer, which initially meant even though the cells showed through the hole in the mask, clicking on them didn’t work as clicks were being blocked by the mask’s layer. However, I eventually managed to get things working, and the mask no behaves as intended.
The biggest difficulty I encountered with implementing the slider was getting the cells to actually update. I could get the slider to change the transparency of elements elsewhere on the page, but no matter what I did I just couldn’t get the change to update the SVG elements on the map (i.e. the Voronoi cells). I spent ages trying to get to the bottom of this. The jQuery UI code could interact with the cells, turn them off, find their contents etc, but updating their styles just did nothing. It turned out that the reason was the map was set to constantly refresh itself (to ensure that the Voronoi layer always appeared in the right place), and therefore any changes to the opacity made by the slider were immediately being overwritten by the code that displays the map. Once I figured this out I could instead update the code that refreshes the map, and the slider thankfully could then control the opacity of the cells. The final implementation of this was very straightforward, but it took an awfully long time to reach the point where I knew where and how to update the code. You can see an example of the slider below:
I also spotted some issues when loading a new dataset into the map (e.g. going from the H1 data to the H2 data when navigating from one slide to another). In such cases the H1 data wasn’t being removed, and the H2 data then sat on top of the H1 data for each cell. As the colours of each cell might be different depending on the data this introduced quite a lot of flickering and unexpected changes of cell colour on pan and zoom. Again, this was tied into the frequency of map refreshes, with the map getting refreshed with the ‘old data’ after the point when the ‘new data’ was first loaded in. By ensuring the map only redraws itself when anything actually changes I managed to fix this. I really ought to completely rewrite this prototype at some point as the current implementation is a bit of a hack – cobbled together from many existing libraries that probably shouldn’t work together and using older versions of libraries because the newer versions don’t work with example code I started with. Currently it uses an older version of D3, an older version of MapBox, which in turn uses an older version of Leaflet, together with jQuery and jQuery UI. The example I started from is https://chriszetter.com/voronoi-map/examples/uk-supermarkets/ which uses these older versions and won’t work with newer versions due to there being major changes between versions of D3 and Leaflet. A rewrite is going to be a fairly major job.
For RNSN I spent some time reworking the song stories as we’re getting pretty close to publication time for these. Previously the pages had no titles, which made them tricky to differentiate in the WordPress Admin interface, had no ‘Featured Image’, so didn’t really fit in with the other pages of the site, had no footer (well, the footer was there but was hidden by the story), and the story pane became obscured by the site title on narrow screens. I have now fixed all these issues. All story pages now have titles, which appear above the story panes on the public pages and appear in the list of pages and the ‘add to menu’ page in the admin interface. All story pages now have the ‘featured image’ associated with their nation at the top of the page, except the ‘over-arching timeline’ which currently features the ‘dancing’ image. The story pane now has a bit of a margin to the left and right, whereas previously it extended to the very edge of the browser window, and I’ve given the pane a border, which I think works better as it shows where the story area ends. I’ve also tweaked the text areas in the timeline to make the area a bit wider. The footer now appears below the story pane, which I think is pretty important as it contains partner / funder logos. Plus without the footer the pages looked rather incomplete. Finally, on narrow screens the story pane is never obscured by the title, thanks to the inclusion of the featured image.
For the Historical Thesaurus I returned to the OED / HT linking task. Marc had requested some updates to the lists of unmatched categories. The initial summary page now displays the total number of words found in unmatched categories, plus a breakdown of how many categories feature how many words. E.g. in the 1894 unmatched OED categories that are not empty there are 3451 words, and 1136 of these categories contain one word each. This information is provided for both OED and HT datasets (for HT the total number of words refers to non-OE words). I also created a new script that goes through all of the matched categories and works out how many lexemes can be automatically matched on word form only. The script lists all 222433 matched categories in a table with one row per category. Each row displays the information about the HT and OED category, plus the ‘match process’ that was used to make the match. It also has columns listing counts of the HT and OED words contained in the categories and full lists of these words. For each of the matched categories, the script compares each HT and OED word (the ‘word_stripped’ and ‘lemma_stripped’ fields) and makes a count of the number of words that match. This count is then displayed in the final column. Where the number of matches is the same as the number of OED words the entire row is given a green background. Where there are words that don’t match these are highlighted with a red background in the lists of words, with words that do match appearing with a green background.
Fraser also got back to me with a further batch of categories that could be ticked off, reducing the number of unmatched OED categories (with a POS and containing words) from 1894 to 1264. I also spent a bit of time investigating travel arrangements for the Bergamo conference in April.
In addition to these tasks I met with Matthew Creasy on Wednesday to discuss the digital edition interface I’d created for his Decadence and Translation project. He’s very happy with how things look and will try to supply me with some updated content before he demonstrates the interface to project partners in February, after which I’ll update the interface if we get any feedback. I also met with Charlotte Methuen to discuss a project she’s currently putting together. I agreed to help out with the technical side of the proposal and will no doubt be writing a Data Management Plan for her next week.
Also this week I created an updated of a Data Management Plan for Thomas Clancy (the fourth version and possibly the last), updated the test version of the SCOSYA atlas to limit the attributes contained in it before a class uses the interface next week, made some further (and possibly final) tweaks to the Bilingual Thesaurus and migrated the Thesaurus of Old English to our new Thesaurus domain and ensured the site and its data all worked in the new location, which thankfully it did. I’ll need to hear back from Marc and Fraser before I make this new version ‘live’, though. I also made a small tweak to the DSL website, started to think about the next batch of updates that will be required to link up the HT and OED data, and did some App store related duties for someone elsewhere in the University. So all in all a pretty busy week.
This was my first week back after an enjoyable but all too rapid Christmas holiday. I spent most of Monday catching up with emails, writing a new and up to date ‘to do’ list and upgrading all of the WordPress sites I manage to the new Version 5 of WordPress, which has introduced a horrible new editor in place of the perfectly good one that was there before. I decided to do a full backup of all of the sites before I upgraded them, which took some time, but it’s good to have a local backup of everything, just in case the main backups go wrong. Rather annoyingly when I upgraded WordPress on Monday it was version 5.0.2 any by the end of the week 5.0.3 had been released, so I’ll have to upgrade all the sites again sometime.
I also spent a bit of time this week making further updates to the RNSN song stories, based on feedback received from the extended project team over the past few weeks. Thankfully these were mostly minor tweaks to content rather than big requests for major changes. I also made some updates to the Data Management Plan for Thomas Clancy’s Iona project and met briefly with him to discuss some related matters. I also gave some advice to Meg MacDonald and Bryony Randall about the proposal Bryony is putting together and had a chat to Luca about one of Graeme Cannon’s old projects that Luca is helping to update.
Other than these issues I spent the rest of my week working on the SCOSYA project. I had been hoping to get started on the public interface for the Atlas in the run-up to Christmas, but had been unable to find the time to work on it due to other work commitments. Somewhat amazingly, this week proved to be rather quiet (not having anything to do for the Historical Thesaurus for once certainly made a big difference) and it was great to have a nice big block of time to devote to the project.
Previously we had agreed that several guided ‘stories’ through views of the data would be good to have, and E, the new RA on the project sent me a draft version of a story based on the data for ‘Gonnae’. I’d suggested using the ‘Storymap’ interface (see https://storymap.knightlab.com/) as a means of navigating through different views of map-based data, and E and I had agreed that I’d also investigate ‘Voronoi’ maps as a means of extrapolating point-based data into broader areas.
I had found a nice example of a Voronoi map that was created using the d3.js library, which I’ve had quite a lot of experience with. It’s a map of supermarkets in the UK, and you can view it here: https://chriszetter.com/voronoi-map/examples/uk-supermarkets/.
I decided to try and get a Voronoi interface working with our data as a first step, using the source code for the ‘supermarkets’ map as a starting point. Below is a screenshot of my first attempt:
This version uses the ‘Gonnae’ data (an adapted version of a CSV file generated by the project’s API) and our base map. You can see in the above screenshot how the ‘cells’ are generated based on our questionnaire locations, but no colours are used to differentiate rating levels, the checkboxes in the top right don’t actually do anything and the text in the bottom left still says ‘explore supermarkets in the UK’ until you click on a cell, at which point the location name is displayed.
For the next version I wanted to properly integrate the map with our website interface, investigate how to shade the cells, and replace the ‘supermarket’ text in the bottom left with something more useful. Below is a screenshot of this version:
This version also pulls its data directly from our API whereas the previous version used a stripped-down CSV file I’d manually made from the API data. This is why this version has slightly different cell layouts (e.g. it includes cells for where there is no data for this question). Cells are given different colours depending on the average rating (the same colours as on the main point-based map), with grey used for locations where there is no data for this question. Information about what the map shows (‘Gonnae’) is found in the bottom left, and clicking on a cell highlights it and displays the location name, average score and what this score means in the bottom left too. The cell colours are possibly not different enough to easily distinguish them, and the opacity level does make it rather difficult to make out the underlying map, but these are things that can be tweaked. I also included points to show where the actual locations are. I personally find these quite helpful, but they can always be removed.
With this version in place I then decided to try and get the storymap interface working with our base map. This is where things started going wrong. The Voronoi script uses a rather old version of the MapBox library, which in turn uses an older version of the Leaflet mapping library. I tried switching to a more recent version but unfortunately this breaks the code. I could (and I still might) go through the code line by line to figure out what function calls need to be updated for the script to work with a more recent version, but this is likely to be a tricky and time-consuming process, and one that might not actually be successful. The issue is that storymap requires a more up to date version of the MapBox library to function, and even getting it to work with our basemap in a way that would be compatible with the Voronoi code took a lot of hacking about with the storymap source code. In the end I managed to get storymap to work with our base map, but failed to get the Voronoi cells to actually display. Here is a screenshot of this version, to get an example of how the story would work:
As you can see, the information from the ‘Gonnae’ document is there, as is our base map. A Youtube clip from Chewin the Fat is embedded in the first slide and as you navigate between slides the map pans and zooms. It’s obviously not that great because the map doesn’t display any data. But in addition I realised that the storymap interface might not be all that suitable anyway. The ‘slide’ part takes up half the width of the browser window, which obscures far too much of the map. Also, the map icons for the slides and the dotted lines aren’t really much use to us either.
I therefore decided I’d write my own version of the storymap library from scratch – a version that would work with the Voronoi script, would take up less room on the map, and would only include features we actually needed, plus new features that storymap didn’t include that I would otherwise have had to have added in (e.g. triggers to change the data displayed on the underlying map when navigating from slide to slide). I say ‘from scratch’ but it wasn’t really as bad as all that as I could reuse a lot of code I’d written for previous projects.
It’s taken quite a while to implement all of the features, but I managed to complete a working version by Friday. It’s still just a proof of concept and will need further work, but here is a screenshot of it:
And here is some information about how it works:
- There are (currently) two different slide types: ‘Full’ and ‘Side’. Full takes up the entire width of the map while side hovers in the top right of the map.
- The YouTube video is embedded in the first slide. Any content could be added to a slide as required.
- There are ‘next’ and ‘previous’ buttons at the edges of the map (so long as there is a next or previous slide). Pressing on one of these loads the appropriate slide, repositions / zooms the map (if necessary) and loads new map data (if necessary).
- When navigating to a different slide, the slide contents animate smoothly, swishing off to the left or right depending on which direction you’ve pressed. The map panning and zooming and the Voronoi data layer are unfortunately a bit clunkier but without upgrading the mapping library (and rewriting the Voronoi script) I’m not sure I’ll be able to make this any smoother.
- No matter what slide you’re on you can zoom and pan around the map as much as you want. You can also click on a cell to view its details.
- When a cell is clicked on you can now click it a second time to deselect it.
- The cell shading is slightly less intense in this version as opposed to the previous version, making the background map a bit easier to see.
- When you navigate to the ‘gonnae you leave us alone’ slide this triggers the loading of a different data view (H2 rather than H1). The information in the bottom left box updates to reflect this. I might temporarily highlight this box when the data changes to make it clearer that a change has been made. If you navigate back to the previous slide the ‘H1’ data loads in again. For other stories we can change the view of the data once per slide, and any searches can be incorporated (e.g. limits by age range or ratings). If we don’t change the position and zoom between slides but just change the data users will be able to very easily compare two datasets by navigating between the slides, which could be useful.
- The data for the ‘storymap’ is found in a JSON file that I created manually. Each slide contains the following fields:
- ID: a unique numerical identifier for the slide. The script expects the first slide to have an ID of 1
- Type: This is either ‘full’ for a full-screen slide or ‘side’ for a side-based slide. We can add more types if required.
- Title: An optional field for when a slide has a title. Only the first and last slides currently have titles.
- Body: The body of the slide, which is entered as HTML. Note that double quotes need to be escaped (\” not “) otherwise the JSON file breaks. Any required HTML content can go in here.
- mapData: The URL of the API search that will be used by the Voronoi script. Must be JSON data not CSV data. Can be left blank if the data is the same as the previous slide but bear in mind that when data changes the slide before the new data needs to feature the URL of the previous data for this data to be reloaded when a user presses the ‘previous’ button.
- dataDescription: The information about the data that gets displayed in the bottom left box. As with the mapData field this can be left blank if it’s the same as for other slides, but the same ‘previous slide’ issue needs to be considered.
- Lat: The latitude where the map should focus for this slide
- Lng: The longitude where the map should focus for this slide
- Zoom: The zoom level of the map for this slide
- Prev: The ID of the previous slide (can be left blank if no previous slide)
- Next: The ID of the next slide (can be left blank if no next slide)
- To create a new story all I’d need to do is create a new JSON file following the same layout as above, which should be fairly straightforward.
That’s as far as I’ve got for now. I have sent the URLs off to E and Jennifer Smith for feedback, and will probably not do any further work on this until I hear back from them. There are other features we might also want to implement, e.g. an option to switch from Voronoi cells to points, or a slider to dynamically change the opacity of the cells, or options to allow us to highlight a particular group of cells for a slide. I also still need to make this interface work better on mobile phones (tablets should already be ok, though) and try to add in a ‘full screen’ option, which might prove to be tricky. But all of that is for another week.