I continued with the HT / OED linking tasks for a lot of this week, dealing not only with categories but also the linking of lexemes within linked categories. We’d previously discovered that the OED had duplicated an entire branch of the HT: their 03.01.07.06 was structurally the same as their 03.01.04.06, but the lexemes contained in the two branches didn’t match up exactly due to subsequent revisions. We had decided to ‘quarantine’ the 03.01.07.06 so as to ensure no contents from this branch are accidentally matched up. I did so by adding a new ‘quarantined’ column to the ‘category_oed’ table. It’s ‘N’ by default and ‘Y’ for the 207 categories in this branch. All future lexeme matching scripts will be set to ignore this branch.
I also created a ‘gap matching’ script. This grabs every unmatched OED category that has a POS and contains words (not including the quarantined categories). There are 950 in total. For each of these the script grabs the OED categories with an ID one lower and one higher than the category ID and only returns them if they are both the same POS and contain words. So for example with OED 2560 ‘relating to dry land’ (aj) the previous category is 2559 ‘partially’ and the next category is 2561 ‘spec’. It then checks to see whether these are both matched up to HT categories. In this case they are, the former to 910 ‘partially’, the latter to 912 ‘specific’. The script then notes whether there is a gap in the HT numbering, which there is here. It also checks to make sure the category in the gap is of the same POS. So in this example, 911 is the gap and the category (‘pertaining to dry land’) is an Aj. So this category is returned in its own column, along with a count of the number of words and a list of the words.
There are, however, some things to watch out for. There are a few occasions where there is more than one HT category in the gap. For example, for the OED category 165009 ‘enter upon command’ the ‘before’ category matches HT category 157423 and the ‘after’ category matches 157445, meaning there are several categories in the gap. Currently in such cases the script just grabs the first HT category in the gap. Linked to this (but not always due to this) some HT categories in the gap are already linked to other OED categories. I’ve put in a check for this so they can be manually checked.
There are 169 gaps to explore and of these 14 HT categories in the gap are already matched to something else. There are also two categories where the identified HT category in the gap is the wrong POS, and these are also flagged. Many of the potential matches are ones that have fallen through the cracks due to lexemes being too different to automatically match up, generally due to there being only 1-3 matching words in the category. The matches look pretty promising, and will just need to be manually checked over before I tick a lot of them off.
Also this week, I updated the ‘match lexemes’ script output to ignore final ‘s’ and initial ‘to’. I also added in counts of matched and unmatched words. We were right to be concerned about duplicate words as the ‘total matched’ figures for OED and HT lexemes are not the same, meaning a word in OED matches multiple in HT (or vice-versa). After running the script here are some stats:
For ’01’ there are 347312 matched HT words and 40877 unmatched HT words, and 347947 matched OED words and 27840 unmatched OED words. For ’02’ there are 110510 matched HT words and 13184 unmatched HT words, and 110651 matched OED words and 9226 unmatched OED words. For ’03’ there are 201653 matched HT words and 22319 unmatched HT words, and 201994 matched OED words and 15777 unmatched OED words.
I then created a script that lists all duplicate lexemes in HT and OED categories. There shouldn’t really be any duplicate lexemes in categories as each word should only appear once in each sense. However, my script uncovered rather a lot of duplicates. This is going to have an impact on our lexeme matching scripts as our plans were based on the assumption that each lexeme form would be unique in a category. My script gives four different lists for both HT and OED categories: All categories comparing citation form, all categories comparing stripped form, matched categories comparing citation form and matched categories comparing stripped form. The output lists the lexeme ID and either fulldate in the case of HT or GHT dates 1 and 2 in the case of OED so it’s easier to compare forms.
For all HT categories there are 576 duplicates using citation form and 3316 duplicates using the stripped form. The majority of these are in matched categories (550 and 3264 respectively). In the OED data things get much, much worse. For all OED categories there are 5662 duplicates using citation form and 6896 duplicates using the stripped form. Again, the majority of these are in matched categories (5634 and 6868 respectively). This is going to need some work in the coming weeks.
As we can’t currently rely on the word form in a category to be unique, I decided to make a new script that matches lexemes in matched categories using both their word form and their date. It matches both stripped word form and start date (the first bit of HT fulldate against the GHT1 date) and is looking pretty promising, with matched figures not too far off those found when comparing stripped word form on its own. The script lists the HT word / ID and date and its corresponding OED word / ID and date in both the HT and OED word columns. Any unmatched HT or OED words are then listed in red underneath
Here are some stats (with those for the ‘only matching by stripped form’ in brackets for comparison)
01: There are 335195 (347312) matched HT words and 52994 (40877) unmatched HT words, and 335196 (347947) matched OED words and 40591 (27840) unmatched OED words.
02: There are 106845 (110510) matched HT words and 16849 (13184) unmatched HT words, and 106842 (110651) matched OED words and 13035 (9226) unmatched OED words.
03: There are 193187 (201653) matched HT words and 30785 (22319) unmatched HT words, and 193186 (201994) matched OED words and 24585 (15777) unmatched OED words.
I’m guessing that the reason the number of HT and OED matches aren’t exactly the same is because of duplicates with identical dates somewhere. But still, the matches are much more reliable. However, there would still appear to be several issues relating to duplicates. Some OED duplicates are carried over from HT duplicates – e.g. ‘stalagmite’ in HT 3142 ‘stalagmite/stalactite’. Duplicates appear in both HT and OED, and the forms in each set have matching dates so are matched up without issue. But sometimes the OED has changed a form, which has resulted in a duplicate being made. E.g. For HT 5750 ‘as seat of planet’ there are two OED ‘term’ words. The second one (ID 252, date a1625) should actually match the HT word ‘termin’ (ID 19164, date a1625). In HT 6506 ‘Towards’ the OED has two ‘to the sun-ward’, but the latter (ID 1806, date a1711) seems to have been changed from the HT’s ‘sunward’ (ID 20940, date a1711), which is a bit weird. There are also some cases where the wrong duplicate is still being matched, often due to OE dates. For example, in HT category 5810 (Sky, heavens (n)), ‘heaven’ (HT 19331 with dates OE-1860) is set to match OED 399 ‘heaven’ (with dates OE-). But HT ‘heavens’ (19332 with dates OE-) is also set to match OED 399 ‘heaven’ (as the stripped form is ‘heaven’ and the start date matches). The OED duplicate ‘heaven’ (ID 433, dates OE-1860) doesn’t get matched as the script finds the 399 ‘heaven’ first and goes no further. Also in this case the OED duplicate ‘heaven’ appears to have been created by the OED removing ‘final -s’ from the second form.
On Friday I met with Marc to discuss all of the above, and we made a plan about what to focus on next. I’ll be continuing with this next week.
Also this week I did some more work for the DSL people. I reviewed some documents Ann had sent me relating to IT infrastructure, spoke to Rhona about some future work I’m going to be doing for the DSL that I can’t really go into any detail about at this stage, created a couple of new pages for the website that will go live next week and updated the way the DSL’s entry page works to allow dictionary ID (e.g. ‘dost24821’) to be passed to the page in addition to the current way of passing dictionary (e.g. ‘dost’) and entry href (e.g. ‘milnare’).
I also gave some advice to the RA of the SCOSYA project who is working on reshaping the Voronoi cells to more closely fit the coastline of Scotland, gave some advice to a member of staff in History who is wanting to rework an existing database, spoke to Gavin Miller about his new Glasgow-wide Medical Humanities project and completed the migration of the RNSN timeline data from Google Docs to locally hosted JSON files.
Everyone in the College of Arts had their emails migrated to a new system this week, so I had to spend a little bit of time getting all of my various devices working properly. Rather worryingly, the default Android mail client told me I couldn’t access my emails until I allowed outlook.office365.com to remotely control my device, which included giving permissions to erase all data from my phone, control screen locks and control cameras. It seemed like a lot of control to be giving a third party when this is my own personal device and all I want to do is read and send emails. After some investigation would appear that the Outlook app for Android doesn’t require permission to erase all data or control the camera, just less horrible permissions involving setting password types and storage encryption. It’s only the default Android mail app that asks for the more horrible permissions. I therefore switched to using the Outlook app, although I also realised the default Android calendar app was also asking for the same permissions, so I’ve had to switch to using the calendar in the Outlook app as well.
With that issue out of the way, I divided my time this week primarily between three projects. First of all in SCOSYA. On Wednesday I met with E and Jennifer to discuss the ‘story atlas’ interface I’d created previously. Jennifer found the Voronoi cells rather hard to read due to the fact that the cells are overlaid on the map, meaning the cell colour obscures features such as place-names and rivers, and the cells extend beyond the edges of the coastline, which makes it hard to see exactly what part of the country each cell corresponds to. Unfortunately the map and all its features (e.g. placenames, rivers) are served up together as tiles. It’s not possible to (for example) have the base map, then our own polygons then place-names, rivers etc on the top. Coloured polygons are always going to obscure the map underneath as they are always added on top of the base tiles. Voronoi diagrams automatically generate cells based on the proximity of points, and this doesn’t necessarily work so well with a coastline such as Scotland’s that features countless islands and features. Some cells extend across bodies of water and give the impression that features are found in areas where they wouldn’t necessarily be found. For example, North Berwick appears in the cell generated by Anstruther, over the other side of the Firth of Forth. We decided, therefore, to abandon Voronoi diagrams and instead make our own cells that would more accurately reflect our questionnaire locations. This does mean ‘hard coding’ the areas, but we decided this wasn’t too much of a problem as our questionnaire locations are all now in place and are fixed. It will mean that someone will have to manually trace out the coordinates for each cell, following the coastline and islands, which will take some time, but we reckoned the end result will be much easier to understand. I found a very handy online tool that can be used to trace polygons on a map and then download the shapes as GeoJSON files: https://geoman.io/studio and I also investigated whether it might be possible to export the polygons generated by my existing Voronoi diagram to use these as a starting point, rather than having to generate the shapes manually from scratch.
I spent some time trying to extract the shapes, but I was unable to do so using the technologies used to generate the map, as the polygons are not geolocational shapes (i.e. with latitude / longitude pairs) but are instead SVG shapes with coordinates that relate to the screen, which then get recalculated and moved every time the underlying map moves. However, I then investigated alternative libraries and have come across one called turf.js (http://turfjs.org/) that can generate Voronoi cells that are actual geolocational shapes. The Voronoi bit of the library can be found here: https://github.com/Turfjs/turf-voronoi and although it rather worryingly is plastered with messages from 4 years ago saying ‘Under development’, ‘not ready for use!’ and ‘build failing’ I’ve managed to get it to work. By passing it our questionnaire locations as lat/lng coordinates I’ve managed to get it to spit out Voronoi polygons as a series of lat/lng coordinates. These can be uploaded to the mapping service linked to above, resulting in polygons as the following diagram shows:
However, the Voronoi shapes generated by this library are not the same dimensions as those generated by the other library (see an earlier post for an image of this). They are a lot spikier somehow. I guess the turf.js Voronoi algorithm is rather different to the d3.js Voronoi algorithm. Also, the boundaries between cells consist of lines for each polygon, meaning when dragging a line you’ll have to drag two or possibly three or more lines to fully update the positions of each cell. Finally, despite including the names of each location in the data that was inputted into the Turf.js Voronoi processor this data has been ignored, meaning the polygon shapes have no place-name associated with them. There doesn’t seem to be a way of getting these added back in automatically, so at some point I’m going to have to manually add place-names (and unique IDs) to the data. This is going to be pretty horrible, but actually I would have had to have done that with any manually created shapes too. It’s now over to other members of the team to tweak the polygons to get them to fit the coastline better.
Also for SCOSYA this week, the project’s previous RA, Gary Thoms, got in touch to ask about generating views of the atlas for publication. He was concerned about issues relating to copyright, issues relating to the resolution of the images and also the fact that the publication would prefer images to be in greyscale rather than colour. I investigated each of these issues:
Regarding copyright: The map imagery we use is generated using the MapBox service. According to their terms of service (see the ‘static images for print’ section here: https://docs.mapbox.com/help/how-mapbox-works/static-maps/) we are allowed to use them in academic publications: “You may make static exports and prints for non-commercial purposes such as flyers, posters, or other short publications for academic, non-profit, or personal use.” I’m not sure what their definition of ‘short’ is, though. Attribution needs to be supplied (see https://docs.mapbox.com/help/how-mapbox-works/attribution/). Map data (roads, place-names etc) comes from OpenStreetMap and is released via a Creative Commons license. This should also appear in the attribution.
Regarding resolution: The SCOSYA atlas maps are raster images rather than scalable vector images, so generating images that are higher than screen resolution is going to be tricky. There’s not much we can do about it without generating maps in a desktop GIS package, or some other such software. All online maps packages I’ve used (Google Maps, Leaflet, MapBox) use raster image tiles (e.g. PNG, JPEG) rather than vector images (e.g. SVG). The page linked to above states “With the Mapbox Static API, image exports can be up to 1,280 px x 1,280 px in size. While enabling retina may improve the quality of the image, you cannot export at a higher resolution using the Static API, and we do not support vector image formats.” And later on: “The following formats are not supported as a map export option and are not currently on our road map for integration: SVG, EPS, PDF”. The technologies we’re using were chosen to make an online, interactive atlas and I’m afraid they’re not ideally suited for producing static printed images. However, the ‘print map to A3 Portrait image’ option I added to the CMS version of the atlas several months ago does allow you to grab a map image that is larger than your screen. Positioning the map to get what you want is a bit hit and miss, and it can take a minute or so to process once you press the button, but it does then generate an image that is around 2440×3310 pixels, which might be good enough quality.
Regarding greyscale images: I created an alternative version of the CMS atlas that uses a greyscale basemap and icons (see below for an example). It is somewhat tricky to differentiate the shades of grey in the icons, though, so perhaps we’ll need to use different icon shapes as well. I haven’t heard back from Gary yet, so will just need to see whether this is going to be good enough.
The next project I focussed on this week was the Historical Thesaurus, and the continuing task of linking up the HT and OED categories and lexemes. I updated one of the scripts I wrote last week so that the length of the subcat is compared rather than the actual subcat (so 01 and 02 now match, but 01 and 01.02 don’t). This has increased the matches from 110 to 209. I also needed to rewrite the script that outputted all of the matching lexemes in every matched category in the HT and OED datasets as I’d realised that my previous script had silently failed to finish due to its size – it just cut off somewhere with no error having been given by Firefox. The same thing happened in Firefox again when I tried to generate a new output, and when trying in Chrome it spent about half an hour processing things then crashed. I’m not sure which browser comes out worse in this, but I’d have to say Firefox silently failing is probably worse, which pains me to say as Firefox is my browser of choice.
Anyway, I have since split the output into three separate files – one each for ‘01’, ‘02’ and ‘03’ categories, and thankfully this has worked. There are a total of 223,182 categories in the three files, up from the 222433 categories in the previous half-finished file. I have also changed the output so that OED lexemes that are marked as ‘revised’ in the database have a yellow [R] after them. This applies to both matched and unmatched lexemes, as I thought it might be useful to see both. I’ve also added a count of the number of revised forms that are matched and unmatched. These appear underneath the tables. It was adding this info underneath the tables that led me to realise the data had failed to fully display – as although Firefox said the page was loaded there was nothing displaying underneath the table. So, for example, in the 114,872 ‘01’ matched categories there are 122,196 words that match and are revised and 15,822 words that don’t match and are revised.
On Friday I met with Marc and Fraser to discuss the next steps for the linking process and I’ll be focussing on this for much of the next few weeks, all being well. Also this week I finally managed to get my travel and accommodation for Bergamo booked.
The third main project I worked on this week was RNSN. For this project I updated our over-arching timeline to incorporate the new timeline I created last week and the major changes to an existing timeline. I also made a number of other edits to existing timelines. One of the project partners had been unable to access the timelines from her work. The timeline page was loading, but the Google Doc containing the data failed to load. It turned out that the person’s work WiFi was blocking access to Google Docs, as when the person checked via the mobile network the full timeline loaded without an issue. This got me thinking that hosting data for the timelines via Google Docs is probably a bad idea. The ‘storymap’ data is already hosted in JSON files hosted on our own servers, but for the timelines I used the Google Docs approach as it was so easy to add and edit data. However, it does mean that we’re relying on a third party service to publish our timelines (all other code for the timelines is hosted at Glasgow). If providers block access to Google Doc hosted spreadsheets, or Google decides to remove free access to this data (as it recently did for Google Maps) then all our timelines break. In addition, the data is currently tied to my Google account, meaning no-one else can edit it or access it.
After a bit of investigation I discovered that you can just store timeline data in locally hosted JSON files, and read these into the timeline script in a very similar way to a Google Doc. I therefore created a test timeline in the JSON format and everything worked perfectly. I migrated two timelines to this format and will need to migrate the remainder in the coming weeks. It will be slightly time consuming and may introduce errors, but I think it will be worth it.
Also this week I made a couple of small tweaks to the Decadence and Translation transcription pages, including reordering the pages and updating notes and explanatory texts, upgraded WordPress to the latest version for all the sites I manage and fixed the footer for the DSL WordPress site.
Last Friday afternoon I met with Charlotte Methuen to discuss a proposal she’s putting together. It’s an AHRC proposal, but not a typical one as it’s in collaboration with a German funding body and it has its own template. I had agreed to write the technical aspects of the proposal, which I had assumed would involve a typical AHRC Data Management Plan, but the template didn’t include such a thing. It did however include other sections where technical matters could be added, so I wrote some material for these sections. As Charlotte wanted to submit the proposal for internal review by the end of the week I needed to focus on my text at the start of the week, and spent most of Monday and Tuesday working on it. I sent my text to Charlotte on Tuesday afternoon, and made a few minor tweaks on Wednesday and everything was finalised soon after that. Now we’ll just need to wait and see whether the project gets funded.
I also continued with the HT / OED linking process this week as well. Fraser had clarified which manual connections he wanted me to tick off, so I ran these through a little script that resulted in another 100 or so matched categories. Fraser had also alerted me to an issue with some OED categories. Apparently the OED people had duplicated an entire branch of the thesaurus (03.01.07.06 and 03.01.04.06) but had subsequently made changes to each of these branches independently of the other. This means that for a number of HT categories there are two potential OED category matches, and the words (and information relating to words such as dates) found in each of these may differ. It’s going to be a messy issue to fix. I spent some time this week writing scripts that will help us to compare the contents of the two branches to work out where the differences lie. First of all I wrote a script that displays the full contents (categories and words) contained in an OED category in tabular format. For example, passing the category 03.01.07.06 then lists the 207 categories found therein, and all of the words contained in these categories. For comparison, 03.01.04.06 contains 299 categories.
I then created another script that compares the contents of any two OED categories. By default, it compares the two categories mentioned above, but any two can be passed, for example to compare things lower down the hierarchy. The script extracts the contents of each chosen category and looks for exact matches between the two sets. The script looks for an exact match of the following in combination (i.e. all must be true):
In such cases the categories are listed in a table together with their lexemes, and the lexemes are also then compared. If a lexeme from cat1 appears in cat2 (or vice-versa) it is given a green background. If a lexeme from one cat is not present in the other it is given a red background, and all lexemes are listed with their dates. Unmatched categories are listed in their own tables below the main table, with links at the top of the page to each. 03.01.04.06 has 299 categories and 03.01.07.06 has 207 categories. Of these there would appear to be 209 matches, although some of these are evidently duplicates. Some further investigation is required, but it does at least look like the majority of categories in each branch can be matched.
I also updated the lists of unmatched categories to incorporate the number of senses for each word. The overview page now gives a list of the number of times words appear in the unmatched category data. Of the 2155 OED words that are currently in unmatched OED categories we have 1763 words with 1 unmatched sense, 232 words with 2 unmatched senses, 75 words with 3 unmatched senses, 18 words with 6 unmatched senses, 36 words with 4 unmatched senses, 15 words with 5 unmatched senses and 16 words with 8 unmatched senses. I also updated the full category lists linked to from this summary information to include the count of senses (unmatched) for each individual OED word, so for example for ‘extra-terrestrial’ the following information is now displayed: extra-terrestrial (1868-1969 [1963-]) [1 unmatched sense].
Also this week I tweaked some settings relating to Rob Maslen’s ‘Fantasy’ blog, investigated some categories that had been renumbered erroneously in the Thesaurus of Old English and did a bit more investigation into travel and accommodation for the Bergamo conference.
I split the remainder of my time between RNSN and SCOSYA. For RNSN I had been sent a sizable list of updates that needed to be made to the content of a number of song stories, so I made the necessary changes. I had also been sent an entirely new timeline-based song story, and I spent a couple of hours extracting the images, text and audio from the PowerPoint presentation and formatting everything for display in the timeline.
For SCOSYA I spent some time further researching Voronoi diagrams and began trying to update my code to work with the current version of D3.js. It turns out that there have been many changes to the way in which D3 implements Voronoi diagrams since the code I based my visualisations on was released. For one thing, ‘d3-voronoi’ is going to be deprecated and replaced by a new module called d3-delaunay. Information about this can be found here: https://github.com/d3/d3-voronoi/blob/master/README.md. There is also now a specific module for applying Voronoi diagrams to spheres using coordinates, called d3-geo-voronoi (https://github.com/Fil/d3-geo-voronoi). I’m now wondering whether I should start again from scratch with the visualisation. However, I also received an email from Jennifer raising some issues with Voronoi diagrams in general so we might need an entirely different approach anyway. We’re going to meet next week to discuss this.
I divided my time this week primarily between three projects: SCOSYA, RNSN and the Historical Thesaurus. For SCOSYA I made some changes to the ‘story’ interface to the atlas that I began work on the week before last. Feedback from the project team suggested that cropping the tessellation would be a good thing, as with the exiting prototype the polygons and lines expand beyond Scotland and into England and Ireland, which would give users the wrong impression. My suggestion of incorporating a slider to change the opacity of the polygons was also seen as something that should be implemented. It turned out that implementing these two changes would be rather more time consuming and tricky than I had anticipated. First of all I tackled the cropping of the tessellation. After doing some research it would appear that there is no easy way to specify a boundary for a Voronoi map, beyond which the cell borders aren’t shown. However, it is possible to place a mask over areas of the map that are not relevant. This cuts off the Voronoi cells, but does also completely obscure any part of the map covered by the mask. Making the mask transparent wouldn’t work as then both the map and the cells would show through. I’ve created a mask and made it grey. The edges of the mask are possibly a bit clunky and I should maybe refine them further (e.g. making them less pointy). Here is an example of the mask in place:
It possibly looks like it would have been the work of about 5 minutes to add this mask in, but it was actually pretty complicated to get it to work. Initially I tried adding the polygon as a layer in Leaflet / MapBox, but unfortunately this then appeared beneath the Voronoi cells, as the Voronoi layer is generated by D3.js and is placed on top of the map. I’m stuck (for the time being) using an old version of Leaflet that doesn’t provide the facility to change the z-index of layers, so instead I looked to create the mask using D3. Every time the map is zoomed or panned the mask needs to be recalculated and redrawn, otherwise it just stays at the same dimensions and obscures the wrong things. I spent some infuriating hours with the mask appearing in the wrong place, moving to the wrong place, failing to change size on zoom and other such things. There were also issues relating to being able to click on the cells, as the mask sits on a separate layer, which initially meant even though the cells showed through the hole in the mask, clicking on them didn’t work as clicks were being blocked by the mask’s layer. However, I eventually managed to get things working, and the mask no behaves as intended.
I then looked to implement the opacity slider, which also proved to be tricky to get working. Implementing the slider itself was very straightforward – I just used the jQuery UI library which includes a slider. This was then set up to change the transparency of certain elements when the user interacted with it. I’m using the transparency option in RGBA in CSS (A is the ‘alpha channel’, which is the transparency). This is better than using opacity in CSS, as changing the alpha channel affects the individual element along whereas updating the opacity affects both the current element and everything contained within the element. However, one slight annoyance is it’s not (currently) possible to just update the alpha channel via JavaScript, instead you have to replace the entire RGBA statement. As I have 6 different colours (plus 6 different ‘selected’ colours) all of these need to be individually updated each time the slider moves. E.g. Rating level 5 has the fill colour ‘rgba(189,0,38,0.5)’ by default. If the slider is moved to make the cells fully opaque the code can’t just say ‘update all classes that have an alpha channel of 0.5 to make this 1’, but instead I need to redefine each individual colour, e.g. change ‘rgba(189,0,38,0.5)’ to ‘rgba(189,0,38,1)’. It’s a shame there isn’t a quicker way of doing this.
The biggest difficulty I encountered with implementing the slider was getting the cells to actually update. I could get the slider to change the transparency of elements elsewhere on the page, but no matter what I did I just couldn’t get the change to update the SVG elements on the map (i.e. the Voronoi cells). I spent ages trying to get to the bottom of this. The jQuery UI code could interact with the cells, turn them off, find their contents etc, but updating their styles just did nothing. It turned out that the reason was the map was set to constantly refresh itself (to ensure that the Voronoi layer always appeared in the right place), and therefore any changes to the opacity made by the slider were immediately being overwritten by the code that displays the map. Once I figured this out I could instead update the code that refreshes the map, and the slider thankfully could then control the opacity of the cells. The final implementation of this was very straightforward, but it took an awfully long time to reach the point where I knew where and how to update the code. You can see an example of the slider below:
I also spotted some issues when loading a new dataset into the map (e.g. going from the H1 data to the H2 data when navigating from one slide to another). In such cases the H1 data wasn’t being removed, and the H2 data then sat on top of the H1 data for each cell. As the colours of each cell might be different depending on the data this introduced quite a lot of flickering and unexpected changes of cell colour on pan and zoom. Again, this was tied into the frequency of map refreshes, with the map getting refreshed with the ‘old data’ after the point when the ‘new data’ was first loaded in. By ensuring the map only redraws itself when anything actually changes I managed to fix this. I really ought to completely rewrite this prototype at some point as the current implementation is a bit of a hack – cobbled together from many existing libraries that probably shouldn’t work together and using older versions of libraries because the newer versions don’t work with example code I started with. Currently it uses an older version of D3, an older version of MapBox, which in turn uses an older version of Leaflet, together with jQuery and jQuery UI. The example I started from is https://chriszetter.com/voronoi-map/examples/uk-supermarkets/ which uses these older versions and won’t work with newer versions due to there being major changes between versions of D3 and Leaflet. A rewrite is going to be a fairly major job.
For RNSN I spent some time reworking the song stories as we’re getting pretty close to publication time for these. Previously the pages had no titles, which made them tricky to differentiate in the WordPress Admin interface, had no ‘Featured Image’, so didn’t really fit in with the other pages of the site, had no footer (well, the footer was there but was hidden by the story), and the story pane became obscured by the site title on narrow screens. I have now fixed all these issues. All story pages now have titles, which appear above the story panes on the public pages and appear in the list of pages and the ‘add to menu’ page in the admin interface. All story pages now have the ‘featured image’ associated with their nation at the top of the page, except the ‘over-arching timeline’ which currently features the ‘dancing’ image. The story pane now has a bit of a margin to the left and right, whereas previously it extended to the very edge of the browser window, and I’ve given the pane a border, which I think works better as it shows where the story area ends. I’ve also tweaked the text areas in the timeline to make the area a bit wider. The footer now appears below the story pane, which I think is pretty important as it contains partner / funder logos. Plus without the footer the pages looked rather incomplete. Finally, on narrow screens the story pane is never obscured by the title, thanks to the inclusion of the featured image.
For the Historical Thesaurus I returned to the OED / HT linking task. Marc had requested some updates to the lists of unmatched categories. The initial summary page now displays the total number of words found in unmatched categories, plus a breakdown of how many categories feature how many words. E.g. in the 1894 unmatched OED categories that are not empty there are 3451 words, and 1136 of these categories contain one word each. This information is provided for both OED and HT datasets (for HT the total number of words refers to non-OE words). I also created a new script that goes through all of the matched categories and works out how many lexemes can be automatically matched on word form only. The script lists all 222433 matched categories in a table with one row per category. Each row displays the information about the HT and OED category, plus the ‘match process’ that was used to make the match. It also has columns listing counts of the HT and OED words contained in the categories and full lists of these words. For each of the matched categories, the script compares each HT and OED word (the ‘word_stripped’ and ‘lemma_stripped’ fields) and makes a count of the number of words that match. This count is then displayed in the final column. Where the number of matches is the same as the number of OED words the entire row is given a green background. Where there are words that don’t match these are highlighted with a red background in the lists of words, with words that do match appearing with a green background.
Fraser also got back to me with a further batch of categories that could be ticked off, reducing the number of unmatched OED categories (with a POS and containing words) from 1894 to 1264. I also spent a bit of time investigating travel arrangements for the Bergamo conference in April.
In addition to these tasks I met with Matthew Creasy on Wednesday to discuss the digital edition interface I’d created for his Decadence and Translation project. He’s very happy with how things look and will try to supply me with some updated content before he demonstrates the interface to project partners in February, after which I’ll update the interface if we get any feedback. I also met with Charlotte Methuen to discuss a project she’s currently putting together. I agreed to help out with the technical side of the proposal and will no doubt be writing a Data Management Plan for her next week.
I worked on a lot of different projects this week, which was a bit of a change from last week’s focus on a single project. For the RNSN project I investigated how to create an over-arching timeline that would pull in data from all of the individual timelines I’d created previously. I had investigated a variety of timeline libraries, but none of them really offered as good a level of functionality as the timeline.js library I was already using for the individual timelines, so I decided to focus on adapting this library to work with multiple timelines. Thankfully the library already included support for ‘groups’, and by assigning slides to the same group they would appear on different ‘tracks’ in the timeline summary section. For my initial version I created a ‘group’ for each of the individual timelines, and a further ‘group’ for the spreadsheet of ‘major events’ that Kirsteen had previously sent me to give context to the individual stories. This initial version mostly worked pretty well, although I needed to tweak the CSS and some of the library’s JavaScript somewhat in order to make the group names more clearly visible in the summary section and to make the group names appear within the main body of the slides too. On Friday I met with Kirsteen and Brianna, the project RA to discuss our future plans for the project and also to discuss the new timeline. Kirsteen was very happy with how the timeline was coming along, and during the meeting I proposed some further changes. In order to reduce the amount of space taken up by the summary section I suggested we have a ‘track’ for each nation rather than each song, and I suggested we ensure there is a link to each song’s full timeline from every slide as well. Kirsteen suggested changing the colour of the contextual slides to more clearly differentiate them from the regular slides too. I spent some further time implementing these updates and by the end of the week things were looking good. I’ll still need to add in more content and to check how the interface works on narrower screens, but I’m pretty happy with how things are working out. Here’s a screenshot of the timeline as it currently looks:
I also spent a bit of time working on Matthew Creasy’s Decadence and Translation project. Matthew had digitised a number of pages of short French poems that he wanted to make available as a sort of mini digital edition that would involve a zoomable / panable interface to the images, plus textual transcriptions and translations with embedded notes. Thankfully I’ve had quite a bit of previous experience with such interfaces (e.g. the ‘Digital Ode’ for the New Modernist Editing project) and could use a lot of existing code. I couldn’t make things exactly the same as Matthew wanted people to be able to comment on each page, and as the rest of the project website is a WordPress site, it made sense to use WordPress’s commenting features. For this reason I created a WordPress page for each digitised image, and embedded a small amount of code within each page. The actual content was stored in a JSON file I created, and the only change needed between individual WordPress pages was to change a numerical ID. My JavaScript code would then grab this ID, pull in the JSON file and extract the entry with the corresponding number, find the image tiles and set up an OpenLayers interface to allow the image to be zoomed, pull in the translations, transcriptions, notes and other things as HTML text and format everything for display. WordPress would then add in its regular comment boxes and that would be that. Matthew had sent me test information about six images and I added all of this to my JSON file. Everything seems to be working pretty well so far, and I just need further input from Matthew now. Here’s how things currently look:
Also this week I created an updated of a Data Management Plan for Thomas Clancy (the fourth version and possibly the last), updated the test version of the SCOSYA atlas to limit the attributes contained in it before a class uses the interface next week, made some further (and possibly final) tweaks to the Bilingual Thesaurus and migrated the Thesaurus of Old English to our new Thesaurus domain and ensured the site and its data all worked in the new location, which thankfully it did. I’ll need to hear back from Marc and Fraser before I make this new version ‘live’, though. I also made a small tweak to the DSL website, started to think about the next batch of updates that will be required to link up the HT and OED data, and did some App store related duties for someone elsewhere in the University. So all in all a pretty busy week.
I took Friday off this week, in the run-up to Christmas, and spent the remaining four days trying to finish some of my outstanding tasks before the holidays begin. This included finishing the ‘song story’ I’d left half finished last week for the RNSN project, starting and completing the other ‘song story’ that I had in my ‘to do’ list and updating two other stories to add in audio files. All this required a lot of adapting images, uploading and linking to files, writing HTML and other such trivial but rather time-consuming tasks. It probably took the best part of two days to get it all done, but things are looking great and I reckon the bulk of the work on these song stories is now complete. We’re hoping to launch them early next year, at which point I’ll be able to share the URLs.
I also continued to talk Joanna Kopacyk with the proposal she’s putting together. This included having a long email conversation with both her and Luca, plus meeting in person with Joanna to go through some of the technical aspects that still needed a bit of thought. Things seem to be coming together well now and hopefully Joanna will be able to submit the proposal in the new year.
Bryony Randall is also working on a proposal, this time a follow-on funding bid. She’d set up a project website on WordPress.com, but it was filled with horribly intrusive adverts, and I thought it would give a better impression to reviewers if we migrated the site to a Glasgow server. I started this process last week and completed it this week. The new website can be found here: https://newmodernistediting.glasgow.ac.uk/
I also spent a couple of hours on the Bilingual Thesaurus, changing the way the selection of languages of origin and citation are handled on the search page. There are so many languages and it had been suggested that higher-level groupings could help ensure users selected all of the appropriate options. So, for example, a new top level group would be ‘Celtic’ and then within this there would be Irish, Old Irish, Scots Gaelic etc. Each group has a checkbox and if you click on it then everything within the group is checked. Clicking again deselects everything, as the screenshot below demonstrates. I think it works pretty well.
I could hardly let a week pass without continuing to work on the HT / OED category linking task, and therefore I spent several further hours working on this. I completed a script that compares all lexemes and their search terms to try and find matches. For this to work I also had to execute a script to generate suitable search terms for the OED data (e.g. variants with / without brackets). The comparison script takes ages to run as it has to compare every word in every unmatched OED category to every word in every unmatched HT category. The script has identified a number of new potential matches that will hopefully be of some use. It also unfortunately identified many OED lexemes that just don’t have any match in the HT data, despite having a ‘GHT date’, which means there should be a matching HT word somewhere. It looks like some erroneous matches might have crept into our matching processes. In some cases the issue is merely that the OED have changed the lexeme so it no longer matches (e.g. making a word plural). But in other cases things look a little weird.
For example, OED 231036 ‘relating to palindrome’ isn’t matched and contains 3 words, none of which are found in the remaining unmatched HT categories (palindrome, palindromic, palindromical). I double-checked this in the database. The corresponding HT category is 219358 ‘pertaining to palindrome’, which contains four words (palinedrome, cancrine, palindromic, palindromical). This has been matched to OED category 194476 ‘crab-like, can be read backwards and forwards’, which contains the words ‘palinedrome, cancrine, palindromic, palindromical’. on further investigation I’d say OED category 194476 ‘crab-like, can be read backwards and forwards’ should actually match HT category 91942 ‘having same form in both directions’ which contains a single word ‘palindromic’. I get the feeling the final matching stages are going to get messy. But this is something to think about next year. That’s all from me for 2018. I wish anyone who is reading this a very merry Christmas.
I continued to work on the outstanding ‘stories’ for the Romantic National Song Network this week, completing work on another story using the storymap.js library. I have now completed seven of these stories, which is more than half of the total the project intends to make.
On Monday I met with E Jamieson, the new RA on the SCOSYA project, to discuss the maps we are going to make available to the public. We discussed various ways in which the point-based data might be extrapolated, such as heat maps and Voronoi diagrams. I found a nice example of a Leaflet.js / D3.js based Voronoi diagram that I think could work very well for the project (see https://chriszetter.com/voronoi-map/examples/uk-supermarkets/) so I might start to investigate using such an approach. I think we’d want to be able to colour-code the cells, although other D3.js examples of Voronoi diagrams suggest that this is possible (see this one: ). We also discussed how the more general public views of the data (as opposed to expert view) might work. The project team like the interface offered by this site: https://ygdp.yale.edu/phenomena/done-my-homework, although we want something that presents more of the explanatory information (including maybe videos) via the map interface itself. It looks like the storymap.js library (https://storymap.knightlab.com/) I’m using for RNSN might actually work very well for this. For RNSN I’m using the library with images rather than maps, but it was primarily designed for use with maps, and could hopefully be adapted to work with a map showing data points or even Voronoi layers.
I spent a further couple of days this week working on the HT / OED data linking task. This included reading through and giving feedback on Fraser’s abstract for DH2019 and updating the v1 / v2 comparison script to add in an additional column to show whether the v1 match was handled automatically or manually. I also created a new script to look at the siblings of categories that contain monosemous forms to see whether any of these might have matches at the same level. This script takes all of the monosemous matches as listed in the monosemous QA script and for each OED and HT category finds their unmatched siblings that don’t otherwise also appear in the list. The script then iterates through the OED siblings and for each of these compares the contents to the contents of each of the HT siblings. If there is a match (matches for this script being anything that’s green, lime green, yellow or orange) the row is displayed on screen. Where there are multiple monosemous categories at the same level the siblings will be analysed for each of the categories, so there is some duplication. E.g. the first monosemous link is ‘OED category 2797 01.02.06.01|03 (n) deep place or part matches HT category 1017 01.02.06.01.01|03 (n) deep place/part’ and there are two unmatched OED siblings (‘shallow place’ and ‘accumulation of water behind barrier’), so these are analysed. But the next monosemous category (OED category 2803 01.02.06.01|07 (n) bed of matches HT category 1024 01.02.06.01.01|07 (n) bed of) is at the same level, so the two siblings are analysed again. This happens quite a lot, but even so there are still some matches that this script finds that wouldn’t otherwise have been found due to changes is category number. I’ve made a count of the total unique matches (all colours) and it’s 162. I fear we are getting to the point where the amount of time it takes to write scripts to identify matches is taking longer than the time it would take to manually identify matches, though. It took several hours to write this script for 162 potential matches.
I also created a script that lists all of the non-matched OED and HT categories, split into various smaller lists, such as main categories or sub-categories, and on Wednesday I attended a meeting with Marc and Fraser to discuss our next steps. I came out of the meeting with another long list of items to try and tackle, and I spent some of the rest of the week going through the list. I ticked off the outstanding green, lime green and yellow matches on the lexeme pattern matching, sibling matching and monosemous matching scripts.
I then updated the sibling matching script to look for matches at any subcat level, but unfortunately this didn’t really uncover much new, at least initially. It found just one extra green and three yellows, although the 86 oranges look like they would mostly be ok too, with manual checking. I went over my script and it was definitely doing what I’m expecting it to do, namely: Get all of the unmatched OED cats (e.g. 03.05.01.02.01.02|05.04 (vt)); for subcats get all of the unmatched HT subcats of the maincat in the same POS (e.g. all the unmatched subcats of 03.05.01.02.01.02 that are vt); list all of the subcats; if one of the stripped headings matches or has a Levenshtein score of 1 then this is highlighted in green and its contents are compared.
I then updated the script so that it didn’t compare category headings at all, but instead only looked at the contents. In this script each possible match appears in its own row (e.g. cat 120031 appears 4 times, once as an orange, 3 times as purple). It has brought back 8 greens, 1 lime green, 4 yellows and 1617 oranges.
I then updated the monosemous QA script to identify categories where the monosemous form has dates that match and one further date matches, the idea being if these criteria are met the category match is likely to be legitimate. This was actually really difficult to implement and took most of a day to do. This is because the identification of monosemous forms was done at a completely different point (and actually by a completely different script) to the listing and comparing of the full category contents. I had to rewrite large parts of the function that gets and compares lexemes in order to integrate the monosemous forms. The script now makes all monosemous forms in the OED word list for each category bold and compares these forms and their dates to all of the HT words in the category. A count of all of the monsemous forms that match an HT form in terms of stripped / pattern matched content and start date is stored. If this count is 1 or more and the count of ‘Matched stripped lexemes (including dates)’ is 2 or more then the match is bumped up to yellow. This has identified 512 categories, which is about a sixth of the total OED unmatched categories with words, which is pretty good.
Other tasks this week included creating a new (and possibly final) blog post for the REELS project, dealing with some App related questions from someone in MVLS, having a brief meeting with Clara Cohen from English Language to discuss the technical aspects of a proposal she’s putting together and making a few further tweaks to the Bilingual Thesaurus website.
On Friday I attended the Corpus Linguistics in Scotland event at Edinburgh University. There were 12 different talks over the course of the day on a broad selection of subjects. As I’m primarily interested in the technologies used rather than the actual subject matter, here are some technical details. One presenter used Juxta (https://www.juxtaeditions.com/) to identify variation in manuscripts. Another used TEI to mark up pre-modern manuscripts for lexicographical use (looking at abbreviations, scribes, parts of speech, gaps, occurrences of particular headwords). Another speaker had created a plugin for the text editor Emacs that allows you to look at things like word frequencies, n-grams and collocations. A further speaker handled OCR using Google Cloud Vision (https://cloud.google.com/vision/) that can take images and analyse them in lots of ways, including extracting the text. A couple of speakers used AntConc (http://www.laurenceanthony.net/software/antconc/) and another couple used the newspaper collections available through LexisNexis (https://www.lexisnexis.com/ap/academic/form_news_wires.asp) as source data. Other speakers used Wordsmith tools (https://www.lexically.net/wordsmith/), Sketch Engine (https://www.sketchengine.eu) and WMatrix (http://ucrel.lancs.ac.uk/wmatrix/). It was very interesting to learn about the approaches taken by the speakers.
This week I mainly working on three projects: The Historical Thesaurus, the Bilingual Thesaurus and the Romantic National Song Network. For the HT I continued with the ongoing and seemingly never-ending task of joining up the HT and OED datasets. Marc, Fraser and I had a meeting last Friday and I began to work through the action points from this meeting on Monday. By Wednesday I had ticked off most of the items, which I’ll summarise here.
Whilst developing the Bilingual Thesaurus I’d noticed that search term highlighting on the HT site wasn’t working for quick searches, only advanced searches for words, so I investigated and fixed this. I then updated the lexeme pattern matching / date matching script to incorporate the stoplist we’d created during last week’s meeting (words or characters that should be removed when comparing lexemes, such as ‘to ‘ and ‘the’). This worked well and has bumped matches up to better colour levels, but has resulted in some words getting matched multiple times. E.g. when removing ‘to’, ‘of’ etc this results in a form that then appears multiple times. For example, in one category the OED has ‘bless’ twice (presumably an erro?) and HT has ‘bless’ and ‘bless to’. With ‘to’ removed there then appear to be more matches that there should be. However, this is not an issue when dates are also taken into consideration. I also updated the script so that categories where there are 3 matches and at least 66% of words match have been promoted from orange to yellow.
When looking at the outputs at the meeting Marc wondered why certain matches (e.g. 120202 ‘relating to doctrine or study’ / ‘pertaining to doctrine/study’ and 88114 ‘other spec.’ / ‘other specific’) hadn’t been ticked off and wondered whether category heading pattern matching had worked properly. After some investigation I’d say it has worked properly – the reason these haven’t been ticked off is they contain too few words to have reached the criteria for ticking off.
Another script we looked at during our meeting was the sibling matching script, which looks for matches at the same hierarchical level and part of speech, but different numbers. I completely overhauled the script to bring it into line with the other scripts (including recent updates such as the stoplist for lexeme matching and the new yellow criteria). There are currently 19, 17 and 25 green, lime green and yellow matches that could be ticked off. I also ticked off the empty category matches listed on the ‘thing heard’ script (so long as they have a match) and for the ‘Noun Matching’ I ticked off the few matches that there were. Most were empty categories and there were less than 15 in total.
Another script I worked on was the ‘monosemous’ script, which looks for monosemous forms in unmatched categories and tries to identify HT categories that also contain these forms. We weren’t sure at the meeting whether this script identified words that were fully monosemous in the entire dataset, or those that were monosemous in the unmatched categories. It turned out it was the former, so I updated the script to only look through the unchecked data, which has identified further monosemous forms. This has helped to more accurately identify matched categories. I also created a QA script that checks the full categories that have potentially been matched by the monosemous script.
I also worked on the date fingerprinting script. This gets all of the start dates associated with lexemes in a category, plus a count of the number of times each date appears, and uses these to try and find matches in the HT data. I updated this script to incorporate the stoplist and the ‘3 matches and 66% match’ yellow rule, and ticked off lots of matches that this script identified. I ticked off all green (1556), lime green (22) and yellow (123) matches.
Out of curiosity, I wrote a script that looked at our previous attempt at matching the categories, which Fraser and I worked on last year and earlier this year. The script looks at categories that were matched during this ‘v1’ process that had yet to be matched during our current ‘v2’ process. For each of these the script performs the usual checks based on content: comparing words and first dates and colour coding based on number of matches (this includes the stoplist and new yellow criteria mentioned earlier). There are 7148 OED categories that are currently unmatched but were matched in V1. Almost 4000 of these are empty categories. There are 1283 ‘purple’ matches, which means (generally) something is wrong with the match. But there are 421 in the green, lime green and yellow sections, which is about 12% of the remaining unmatched OED categories that have words. It might also be possible to spot some patterns to explain why they were matched during v1 but have yet to be matched in v2. For example, 2711 ‘moving water’ has 01.02.06.01.02 and its HT counterpart has 01.02.06.01.01.02. There are possibly patterns in the 1504 orange matches that could be exploited too.
Finally, I updated the stats page to include information about main and subcats. Here are the current unmatched figures:
Unmatched (with POS): 8629
Unmatched (with POS and not empty): 3414
Unmatched Main Categories (with POS): 5036
Unmatched Main Categories (with POS and not empty): 1661
Unmatched Subcategories (with POS): 3573
Unmatched Subcategories (with POS and not empty): 1753
So we are getting there!
For the Bilingual Thesaurus I completed an initial version of the website this week. I have replaced the original colour scheme with a ‘red, white and blue’ colour scheme as suggested by Louise. This might be changed again, but for now here is an example of how the resource looks:
The ‘quick’ and ‘advanced’ searches are also now complete, using the ‘search words’ mentioned in a previous post, and ignoring accents on characters. As with the HT, by default the quick search matches category headings and headwords exactly, so ‘ale’ will return results as there is a category ‘ale’ and also a word ‘ale’ but ‘bread’ won’t match anything because there are no words or categories with this exact text. You need to use an asterisk wildcard to find text within word or category text: ‘bread*’ would find all items starting with ‘bread’, ‘*bread’ would find all items ending in ‘bread’ and ‘*bread*’ would find all items with ‘bread’ occurring anywhere.
The ‘advanced search’ lets you search for any combination of headword, category, part of speech, section, dates and languages or origin and citation. Note that if you specify a range of years in the date search it brings back any word that was ‘active’ in your chosen period. E.g. a search for ‘1330-1360’ will bring back ‘Edifier’ with a date of 1100-1350 because it was still in use in this period.
As with the HT, different search boxes are joined with ‘AND’ – e.g. if you tick ‘verb’ and select ‘Anglo Norman’ as the section then only words that are verbs AND Anglo Norman will be returned. Where search types allow multiple options to be selected (i.e. part of speech and languages of origin and citation) if multiple options in each list are selected these are joined by ‘OR’. E.g. if you select ‘noun’ and ‘verb’ and select ‘Dutch’, ‘Flemish’ and ‘Italian’ as languages or origin this will find all words that are either nouns OR verbs AND have a language of origin of Dutch OR Flemish OR Italian.
Search results display the full hierarchy leading to the category, the category name, plus the headword, section, POS and dates (if the result is a ‘word’ result rather than a ‘category’ result). Clicking through to the category highlights the word. I also added in a ‘cite’ option to the category page and updated the ‘About’ page to add a sentence about the current website. The footer still needs some work (e.g. maybe including logos for the University of Westminster and Leverhulme) and there’s a ‘terms of use’ page linked to from the homepage that currently doesn’t have any content, but other than that I think most of my work here is done.
For the Romantic National Song Network I continued to create timelines and ‘storymaps’ based on powerpoint presentations that had been sent to me. This is proving to be a very time-intensive process, as it involves extracting images, audio files and text from the presentations, formatting the text as HTML, reworking the images (resizing, sometimes joining multiple images together to form one image, changing colour levels, saving the images, uploading them to the WordPress site), uploading the audio files, adding in the HTML5 audio tags to get the audio files to play, creating the individual pages for each timeline entry / storymap entry. It took the best part of an afternoon to create one timeline for the project, which involved over 30 images, about 10 audio files and more than 20 Powerpoint slides. Still, the end result works really well, so I think it’s worth putting the effort in.
In addition to these projects I met with a PhD student, Ewa Wanat, who wanted help in creating an app. I spent about a day attempting to make a proof of concept for the app, but unfortunately the tools I work with are just not very well suited to the app she wants to create. The app would be interactive and highly dependent on logging user interactions as accurately as possible. I created looked into using the d3.js library to create the sort of interface she wanted (a circle that rotates with smaller circles attached to it, that the user should tap on when a certain point in the rotation is reached), but although this worked, the ‘tap’ detection was not accurate enough. In fact on touchscreens more often than not a ‘tap’ wasn’t even being registered. D3.js just isn’t made to deal with time-sensitive user interaction on animated elements and I have no experience with any libraries that are made in this way, so unfortunately it looks like I won’t be able to help out with this project. Also, Ewa wanted the app to be launched in January and I’m just far too busy with other projects to be able to do the required work in this sort of timescale.
Also this week I helped extract some data about the Seeing Speech and Dynamic Dialects videos for Eleanor Lawson, I responded to queries from Meg MacDonald and Jennifer Nimmo about technical work on proposals they are involved with, I responded to a request for advice from David Wilson about online surveys, and another request from Rachel Macdonald about the use of Docker on the SPADE server. I think that’s just about everything to report.
After a rather hectic couple of weeks this was a return to a more regular sort of week, which was a relief. I still had more work to do than there was time to complete, but it feels like the backlog is getting smaller at least. As with previous weeks, I continued with the HT / OED linking of categories processes this week, following on from the meeting Marc, Fraser and I had the Friday before. For the lexeme / data matching script I separated out categories with zero matches that have words from the orange list into a new list with a purple background. So orange now only contains categories where at least one word and its start date match. The ones now listed in purple are almost certainly incorrect matches. I also changed the ordering of results so that categories are listed by the largest number of matches, to make it easier to spot matches that are likely ok.
I also updated the ‘monosemous’ script, so that the output only contains OED categories that feature a monosemous word and is split into three tables (with links to each at the top of the page). The first table features 4455 OED categories that include a monosemous word that has a comparable form in the HT data. Where there are multiple monosemous forms they each correspond to the same category in the HT data. The second table features 158 OED categories where the linked HT forms appear in more than one category. This might either be because the word is not monosemous in the HT data and appears in two different categories (these are marked with the text ‘red|’ they can be search for in page. An OED category can also appear in this table even if there are no red forms if (for example) one of the matched HT words is in a different category to all of the others (see OED catid 45524) where the word ‘Puncican’ is found in a different HT category to the other words). The final table contains those OED categories that feature monosemous words that have no match in the HT data. There are 1232 of these. I also created a QA script for the 4455 matched monosemous categories, which applies the same colour coding and lexeme matching as other QA scripts I’ve created. On Friday we had another meeting to discuss the findings and plan our next steps, which I will continue with next week.
Also this week I wrote an initial version of a Data Management Plan for Thomas Clancy’s Iona project, and commented on the DMP assessment guidelines that someone from the University’s Data Management people had put together. I can’t really say much more about these activities, but it took at least a day to get all of this done. I also did some app management duties, setting up an account for a new developer, and made the new Seeing Speech and Dynamic Dialects websites live. These can now be viewed here: https://www.seeingspeech.ac.uk/ and here: https://www.dynamicdialects.ac.uk/. I also had an email conversation with Rhona Alcorn about Google Analytics for the DSL site.
With the REELS project’s official launch approaching, I spent a bit of time this week going through the 23 point ‘to do’ list I’d created last week. In fact, I added another three items to it. I’m going to tackle the majority of the outstanding issues next week, but this week I investigated and fixed an issue with the ‘export’ script in the Content Management System. The script is very memory intensive and it was exceeding the server’s memory limits, so asking Chris to increase this limit sorted the issue. I also updated the ‘browse place-names’ feature of the CMS, adding a new column and ordering facility to make it clearer which place-names actually appear on the website. I also updated the front-end so that it ‘remembers’ whether you prefer the map or the text view of the data using HTML5 local storage and added in information about the Creative Commons license to the site and the API. I investigated the issue of parish boundary labels appearing on top of icons, but as of yet I’ve not found a way to address this. I might return to it before the launch if there’s time, but it’s not a massive issue. I moved all of the place-name information on the record page above the map, other than purely map-based data such as grid reference. I also removed the option to search the ‘analysis’ field from the advanced search and updated the element ‘auto-complete’ feature so that it only now matches the starting letters of an element rather than any letters. I also noticed that the combination of ‘relief’ and ‘water’ classifications didn’t have an icon on the map, so I created one for it.
I also continued to work on the Bilingual Thesaurus website this week. I updated the way in which source links work. Links to dictionary sources now appear as buttons in the page, rather in a separate pop-up. They feature the abbreviation (AND / MED / OED) and the magnifying glass icon and if you hover over a button the non-abbreviated form appears. For OED links I’ve also added the text ‘subscription required’ to the hover-over text. I also updated the word record so that where language of origin is ‘unknown’ the language of origin no longer gets displayed, and I made the headword text a bit bigger so it stands out more. I also added the full hierarchy above the category heading in the category section of the browse page, to make it easier to see exactly where you are. This will be especially useful for people using the site on narrow screens as the tree appears beneath the category section so is not immediately visible. You can click on any of the parts of the hierarchy here to jump to that point.
I then began to work on the search facility, and realised I needed to implement a ‘search words’ list that features variants. I did this for the Historical Thesaurus and it’s really useful. What I’ve done so far is generate alternatives for words that have brackets and dashes. For example, the headword ‘Bond(e)-man’ has the following search terms: Bond(e)-man, Bond-man, Bonde-man, Bond(e) man, Bond man, Bonde man, Bond(e)man, Bondman, Bondeman. None of these varieties will ever appear on the website, but instead will be used to find the word when people search. I’ll need some feedback as to whether these options will suffice, but for now I’ve uploaded variants to a table and began to get the quick search working. It’s not entirely there yet, but I should get this working next week. I also need to know what should be done about accented characters for search purposes. The simplest way to handle them would be to just treat them as non-accented characters – e.g. searching for ‘alue’ will find ‘alué’. However, this does mean you won’t be able to specifically search for words that include accented characters – e.g. a search for all the words featuring an ‘é’ will just bring back all characters with an ‘e’ in them.
I was intending to add a count of the number of words in each hierarchical level to the browse, or at least to make hierarchical levels that include words bold in the browse, so as to let users know whether it’s worthwhile clicking on a category to view the words at this level. However, I’ve realised that this will just confuse users as levels that have no words in them but include child categories that do have words in them would be listed with a zero or not in bold, giving the impression that there is no content lower down the hierarchy.
My last task for the week was to create a new timeline for the RNSN project based on data that had been given to me. I think this is looking pretty good, but unfortunately making these timelines and related storymaps is very time-intensive, as I need to extract and edit the images, upload them to WordPress, extract the text and convert it into HTML and fill out the template with all of the necessary fields. It took about 2 and a half hours to make this timeline. However, hopefully the end result will be worth it.
I continued to work on the HT / OED data alignment for a lot of this week. I updated the matching scripts I had previously created so that all matches based on last lexeme were removed and instead replaced by a ‘6 matches or more and 80% of words in total match’ check. This was a lot more effective that purely comparing the last word in each category and helped match up a lot more categories. I also created a QA script to check the manual matches that were made during our first phase of matching. There are 1407 manual matches in the system. The script also listed all the words in each potential matched category to make it easier to tell where any potential difficulties were. I also updated the ‘pattern matching’ script I’d created last week to list all words and include the ‘6 matches and 80%’ check and changed the layout so that separate groupings now appear in different tables rather than being all mixed up in one table. It took quite a long time to sort this out, but it’s going to be much more useful for manual checking.
I then moved on to writing a new ‘sibling matching’ script. This script goes through all unmatched OED categories (this includes all that appear in other scripts such as the pattern matching one) and retrieves all sibling categories of the same POS. E.g. if the category is ‘01.01.01|03 (n)’ then the script brings back all HT noun subcats of ’01.01.01’ that are ‘level 1’ subcats and compares their headings. It then looks to see if there is a sibling category that has the same heading – i.e. looking for when a category has been renumbered within the same level of the thesaurus. This has uncovered several hundred such potential matches, which will hopefully be very helpful. I also then created a further script that compares non-noun headings to noun headings at the same level, as it looked like a number of times the OED kept the noun heading for other parts of speech while the HT renamed them. This identified a further 65 possible matches, which isn’t too bad.
I met with Marc and Fraser on Wednesday to discuss the recent updates I’d made, after which I managed to tick off 2614 matched categories, taking our total of unmatched OED categories that have a part of speech and are not empty down to 10,854. I then made a start on a new script that looks at pattern matching for category contents (i.e. words), but I didn’t have enough time to make a huge amount of progress with this.
I also fixed an issue with the HT’s Google Analytics not working properly. It looks like the code stopped working around the time we shifted domains in the summer, but it was a bit of a weird one as everything was apparently set up correctly – the ‘Send test traffic’ option on the .JS tracking code section successfully navigated through to the site and the tracking code was correct, but nothing was getting through. However, I replaced the existing GA JavaScript that we had on our page with a new snippet from the JS section of Google Analytics and this seems to have done the trick.
However, we also have some calls to GA in our JavaScript so that loading parts of the tree, changing parts of speech, selecting subcats etc are considered ‘page hits’ and were reported to GA. None of these worked after I changed the code. I followed some guidelines here:
https://developers.google.com/analytics/devguides/collection/analyticsjs/sending-hits
to try and get things working but the callbacks were never being initiated – i.e. data wasn’t getting through to Google. Thankfully Stack Overflow had an answer that worked (After trying several that didn’t):
https://stackoverflow.com/a/40761709
I’ve updated this so that pageviews rather than events are sent and now everything seems to be working again.
I spent a bit more time this week working on the Bilingual Thesaurus project, focussing on getting the front end for the thesaurus working. I’ve reworked the code for the HT’s browse facility to work with the project’s data. This required quite a lot of work as structurally the datasets are quite different – the HT relies in its ‘tier’ numbers for parent / child / sibling category relationships, and also has different categories for parts of speech and nested subcategories. The BTH data is much simpler (which is great) as it just has parent and child categories, with things like part of speech handled at word level. This meant I had to strip a lot of stuff out of the code and rework things. I’m also taking the opportunity to move to a new interface library (Bootstrap) so had to rework the page layout to take this into consideration too. I managed to get an initial version of the browse facility working now, which works in much the same way as the main HT site: clicking on a heading allows you to view its words and clicking on a ‘plus’ sign allows you to view the child categories. As with the HT you can link directly to a category too. I do still need to work on the formatting of the category contents, though. Currently words are just listed all together, with their type (AN or ME) listed first, then the word, then the POS in brackets, then dates (if available). I haven’t included data about languages of source or citation yet, or URLs. I’m also going to try and get the timeline visualisations working as well. I’ll probably split the AN and ME words into separate tabs, and maybe split the list up by POS too. I’m also wondering whether the full category hierarchy should be represented above the selected category (the right pane), as unlike the HT there’s no category number to show your position in the thesaurus. Also, as a lot of the categories are empty I’m thinking of making the ones with words in them bold in the tree, or even possibly adding a count of words in brackets after the category heading. I’ve also updated the project’s homepage to include the ‘sample category’ feature, allowing you to press the ‘reload’ icon to load a new random category.
I also made some further tweaks to the Seeing Speech data (fixing some of the video titles and descriptions) and had a chat with Thomas Clancy about his Iona proposal, which is starting to come together again after several months of inactivity. Also for the ‘Books and Borrowing’ proposal I replied to a request for more information on the technical side of things from Stirling’s IT Services, who will be hosting the website and database for the project. I met with Luca this week as well, to discuss how best to grab XML data via an AJAX query and process it using client-side JavaScript. This is something I had already tackled as part of the ‘New Modernist Editing’ project, so was able to give Luca some (hopefully) useful advice. I also continued an email conversation with Bryony Randall and Ronan Crowley about the workshop we’re running on digital editions later in the month.
On Friday I spent most of the day working on the RNSN project, adding direct links to the ‘nation’ introductions to the main navigation menu and creating new ‘storymap’ stories based on Powerpoint presentations that had been sent to me. This is actually quite a time-consuming process as it involves grabbing images from the PPT, reformatting them, uploading them to WordPress, linking to them from the Storymap pages, creating Zoomified versions of the image or images that will be used as the ‘map’ for the story, extracting audio files from the PPT and uploading them, grabbing all of the text and formatting it for display and other such tasks. However, despite being a long process the end result is definitely worth it as the stroymaps work very nicely. I managed to get two such stories completed today, and now I’ve re-familiarised myself with the process it should be quicker when the next set get sent to me.
I’m going to be on holiday next week so there won’t be another report from me until the week after that.