Week Beginning 21st January 2019

I divided my time this week primarily between three projects:  SCOSYA, RNSN and the Historical Thesaurus.  For SCOSYA I made some changes to the ‘story’ interface to the atlas that I began work on the week before last.  Feedback from the project team suggested that cropping the tessellation would be a good thing, as with the exiting prototype the polygons and lines expand beyond Scotland and into England and Ireland, which would give users the wrong impression.  My suggestion of incorporating a slider to change the opacity of the polygons was also seen as something that should be implemented.  It turned out that implementing these two changes would be rather more time consuming and tricky than I had anticipated.  First of all I tackled the cropping of the tessellation.  After doing some research it would appear that there is no easy way to specify a boundary for a Voronoi map, beyond which the cell borders aren’t shown.  However, it is possible to place a mask over areas of the map that are not relevant.  This cuts off the Voronoi cells, but does also completely obscure any part of the map covered by the mask.  Making the mask transparent wouldn’t work as then both the map and the cells would show through.  I’ve created a mask and made it grey.  The edges of the mask are possibly a bit clunky and I should maybe refine them further (e.g. making them less pointy).  Here is an example of the mask in place:

It possibly looks like it would have been the work of about 5 minutes to add this mask in, but it was actually pretty complicated to get it to work.  Initially I tried adding the polygon as a layer in Leaflet / MapBox, but unfortunately this then appeared beneath the Voronoi cells, as the Voronoi layer is generated by D3.js and is placed on top of the map.  I’m stuck (for the time being) using an old version of Leaflet that doesn’t provide the facility to change the z-index of layers, so instead I looked to create the mask using D3.  Every time the map is zoomed or panned the mask needs to be recalculated and redrawn, otherwise it just stays at the same dimensions and obscures the wrong things.  I spent some infuriating hours with the mask appearing in the wrong place, moving to the wrong place, failing to change size on zoom and other such things.  There were also issues relating to being able to click on the cells, as the mask sits on a separate layer, which initially meant even though the cells showed through the hole in the mask, clicking on them didn’t work as clicks were being blocked by the mask’s layer.  However, I eventually managed to get things working, and the mask no behaves as intended.

I then looked to implement the opacity slider, which also proved to be tricky to get working.  Implementing the slider itself was very straightforward – I just used the jQuery UI library which includes a slider.  This was then set up to change the transparency of certain elements when the user interacted with it.  I’m using the transparency option in RGBA in CSS (A is the ‘alpha channel’, which is the transparency).  This is better than using opacity in CSS, as changing the alpha channel affects the individual element along whereas updating the opacity affects both the current element and everything contained within the element.  However, one slight annoyance is it’s not (currently) possible to just update the alpha channel via JavaScript, instead you have to replace the entire RGBA statement.  As I have 6 different colours (plus 6 different ‘selected’ colours) all of these need to be individually updated each time the slider moves.  E.g. Rating level 5 has the fill colour ‘rgba(189,0,38,0.5)’ by default.  If the slider is moved to make the cells fully opaque the code can’t just say ‘update all classes that have an alpha channel of 0.5 to make this 1’, but instead I need to redefine each individual colour, e.g. change ‘rgba(189,0,38,0.5)’ to ‘rgba(189,0,38,1)’.  It’s a shame there isn’t a quicker way of doing this.

The biggest difficulty I encountered with implementing the slider was getting the cells to actually update.  I could get the slider to change the transparency of elements elsewhere on the page, but no matter what I did I just couldn’t get the change to update the SVG elements on the map (i.e. the Voronoi cells).  I spent ages trying to get to the bottom of this.  The jQuery UI code could interact with the cells, turn them off, find their contents etc, but updating their styles just did nothing.  It turned out that the reason was the map was set to constantly refresh itself (to ensure that the Voronoi layer always appeared in the right place), and therefore any changes to the opacity made by the slider were immediately being overwritten by the code that displays the map.  Once I figured this out I could instead update the code that refreshes the map, and the slider thankfully could then control the opacity of the cells.  The final implementation of this was very straightforward, but it took an awfully long time to reach the point where I knew where and how to update the code.  You can see an example of the slider below:

I also spotted some issues when loading a new dataset into the map (e.g. going from the H1 data to the H2 data when navigating from one slide to another).  In such cases the H1 data wasn’t being removed, and the H2 data then sat on top of the H1 data for each cell.  As the colours of each cell might be different depending on the data this introduced quite a lot of flickering and unexpected changes of cell colour on pan and zoom.  Again, this was tied into the frequency of map refreshes, with the map getting refreshed with the ‘old data’ after the point when the ‘new data’ was first loaded in.  By ensuring the map only redraws itself when anything actually changes I managed to fix this.  I really ought to completely rewrite this prototype at some point as the current implementation is a bit of a hack – cobbled together from many existing libraries that probably shouldn’t work together and using older versions of libraries because the newer versions don’t work with example code I started with.  Currently it uses an older version of D3, an older version of MapBox, which in turn uses an older version of Leaflet, together with jQuery and jQuery UI.  The example I started from is https://chriszetter.com/voronoi-map/examples/uk-supermarkets/ which uses these older versions and won’t work with newer versions due to there being major changes between versions of D3 and Leaflet.  A rewrite is going to be a fairly major job.

For RNSN I spent some time reworking the song stories as we’re getting pretty close to publication time for these.  Previously the pages had no titles, which made them tricky to differentiate in the WordPress Admin interface, had no ‘Featured Image’, so didn’t really fit in with the other pages of the site, had no footer (well, the footer was there but was hidden by the story), and the story pane became obscured by the site title on narrow screens.  I have now fixed all these issues.  All story pages now have titles, which appear above the story panes on the public pages and appear in the list of pages and the ‘add to menu’ page in the admin interface.  All story pages now have the ‘featured image’ associated with their nation at the top of the page, except the ‘over-arching timeline’ which currently features the ‘dancing’ image.  The story pane now has a bit of a margin to the left and right, whereas previously it extended to the very edge of the browser window, and I’ve given the pane a border, which I think works better as it shows where the story area ends.  I’ve also tweaked the text areas in the timeline to make the area a bit wider.  The footer now appears below the story pane, which I think is pretty important as it contains partner / funder logos.  Plus without the footer the pages looked rather incomplete.  Finally, on narrow screens the story pane is never obscured by the title, thanks to the inclusion of the featured image.

For the Historical Thesaurus I returned to the OED / HT linking task. Marc had requested some updates to the lists of unmatched categories.  The initial summary page now displays the total number of words found in unmatched categories, plus a breakdown of how many categories feature how many words.  E.g. in the 1894 unmatched OED categories that are not empty there are 3451 words, and 1136 of these categories contain one word each.  This information is provided for both OED and HT datasets (for HT the total number of words refers to non-OE words).  I also created a new script that goes through all of the matched categories and works out how many lexemes can be automatically matched on word form only.  The script lists all 222433 matched categories in a table with one row per category.  Each row displays the information about the HT and OED category, plus the ‘match process’ that was used to make the match.  It also has columns listing counts of the HT and OED words contained in the categories and full lists of these words.  For each of the matched categories, the script compares each HT and OED word (the ‘word_stripped’ and ‘lemma_stripped’ fields) and makes a count of the number of words that match.  This count is then displayed in the final column.  Where the number of matches is the same as the number of OED words the entire row is given a green background.  Where there are words that don’t match these are highlighted with a red background in the lists of words, with words that do match appearing with a green background.

Fraser also got back to me with a further batch of categories that could be ticked off, reducing the number of unmatched OED categories (with a POS and containing words) from 1894 to 1264.  I also spent a bit of time investigating travel arrangements for the Bergamo conference in April.

In addition to these tasks I met with Matthew Creasy on Wednesday to discuss the digital edition interface I’d created for his Decadence and Translation project.  He’s very happy with how things look and will try to supply me with some updated content before he demonstrates the interface to project partners in February, after which I’ll update the interface if we get any feedback.  I also met with Charlotte Methuen to discuss a project she’s currently putting together.  I agreed to help out with the technical side of the proposal and will no doubt be writing a Data Management Plan for her next week.