I spent the majority of this week working on the new, interactive region and dialect area map for the Dictionaries of the Scots Language. DSL editor Vasilis had created a prototype interface and geoJSON data before the summer and a few weeks ago I had a meeting with a few members of the DSL team about creating an new version of this that would be more suited to the DSL website and would be optimised for mobile use. I then created a specification document which I shared with the group and after some feedback we reached agreement on what needed to be developed.
I was able to base the new interface on my previous map-based projects such as the place-name resources and the Scots Syntax Atlas, and as with these I decided to use the Leaflet.js mapping library. My first task was to incorporate all of the geoJSON data into an interactive map to get an idea of how it might work and whether there would be too many areas for people to make sense of. The first issue I encountered was that the geoJSON data had been created using a projection that Leaflet did not support. I’m not exactly a GIS expert, but apparently Leaflet expects all coordinates to be latitude and longitude using the Coordinate Reference System (CRS) WGS84. The DSL regions geoJSON data stored its coordinates as EPSG 3857, so the coordinates were things like ‘[ -203142.297936542046955, 7877431.976650067605078 ]’ rather than ‘[-1.804316665436796,57.55628903006679]’. This meant that no data was getting displayed on the map. Thankfully I’d encountered this issue before whilst working on the Speak For Yersel project, and I was able to use QGIS to convert the data into the alternative CRS, after which it appeared on the map correctly.
The next issue was that the geoJSON was around 11MB in size, which is far too big for downloading into a webpage. Again, we’d dealt with this issue with Speak For Yersel, where we’d used an algorithm to simplify the polygons and massively reduce the file size. Unfortunately this was a task that another member of the team had performed and going through my blogs and emails I was unable to find details of exactly how this had been accomplished. I tried a couple of approaches in QGIS but they converted the polygons to really basic shapes that were totally unsuitable. Thankfully I found a really amazing resource called mapshaper (https://mapshaper.org/) that allows you to paste in a geoJSON file, visualise it and adapt it, including simplifying it. After some experimentation I exported a file that was just over 1MB in size, which was much more suitable whilst retaining sufficient detail.
I was then able to plug this data into a Leaflet map and visualise it. As agreed with the team, we would have separate maps for regions and dialect areas, so I updated my code to only display regions, stripping out the dialect areas. Initially I created an interface based on the Speak For Yersel maps, with polygons appearing with a faint grey dotted border and the ability to hover over a region, which would highlight it with a yellow border and display the region’s name in a box in the top-right of the screen. I realised that this approach was not going to work very well, firstly because hover-over doesn’t work very well on touchscreens and secondly because there are generally multiple layers present at any one point of the map and highlighting these and displaying all of their names as the cursor moves around the map would be messy.
Instead I decided to remove the hover-over and focus on the click event. We’d decided that when the user clicks a point on the map all regions represented at this point would be highlighted on the map, and information about them, plus the dialect areas would appear in a pane in the map, with options to turns on and off each layer.
In order to grab details about each layer that was represented at the point the user clicked on I needed to use a Leaflet plugin called ‘Leaflet point in polygon’ (https://github.com/mapbox/leaflet-pip). Using this gave my code access to as handy array of all of the layers found at the point where the user clicks, thus allowing me to highlight them on the map and access their details to present them in the information pane.
While Vasilis’ prototype had used solid blocks on the map to represent the regions, we’d decided instead to make these transparent so people could still make out the placenames and features of the underlying map. I also decided to replace the dotted lines with solid ones, as these looked a little neater where multiple polygons converged. I had been worried that there would be too much data and that the map would be too cluttered to be understandable, but I think the map works with all regions visible, and it helps give people a sense of which area might be interesting to click on.
I worked on the information pane, creating lists of regions and dialect areas found at the point clicked on and making it possible to turn these on or off, using a red border and transparent background for regions and a purple border and background for dialect areas, with dialect areas switched off by default. I also decided to add a map marker to where the user clicked to make it clearer how the highlighting has been calculated. I also added in the option to close the information pane (which would also remove all highlighting). This can also be achieved by pressing anywhere on the map that isn’t a region (e.g. in the sea). The screenshot below shows a section of the map with all regions located at the point clicked on highlighted in red and one dialect area (Northeastern Scots (a)) selected and shown in purple:
This is quite a lot of information, but note that the purple area is not visible by default and has to be manually selected. Also, using the options in the information pane you can very easily filter out any layers you’re not interested in. The following screenshot only has one layer (Moray Firth Fishing Villages) highlighted, for example:
There’s still work to be done with the information pane – for example the listed regions and dialect areas should ideally be hierarchically ordered, but I’m pretty happy with how things are developing so far.
I then decided to switch my focus to the map menu, which I wanted to add to the left of the map using the code I’d developed for other resources. This menu would provide lists of regions and dialect areas, allowing the user to select one to view it on the map rather than just clicking round the map. Vasilis had sent me lists of dialect areas and an XML file containing the hierarchical structure of regions and I spent quite a lot of time working with this data. I needed to ensure that the data presented in these files tied into the geoJSON data, and there were a few inconsistencies that took some time to sort out. For example, the ‘abbr’ element in the XML file did not always match up with the ‘abbr’ attribute in the geoJSON data and there were a few typos. Also, the regions XML file also contained dialect areas and I needed to strip these out. It took somewhat longer and was rather more tricky than anticipated to extract the data in a format that I could use, but I got there in the end. I think there’s still work to be done on both the hierarchical structure and the presentation, but it’s a start at least. The screenshot below shows the hierarchical layout of the dialect areas using a drop-down list:
I still need to add in a search option and I might replace the HTML select box with an alternative interface that can better display the hierarchy. As of yet it’s not possible to select an item and actually visualise it on the map, and this led onto another issue:
Selecting an entire region needs to behave differently to clicking on the map; when the user clicks on the map the regions that need to be highlighted are simply all of those regions that are present at this exact point on the map. But when an entire region is selected this may include several intersecting regions at various points within the region so (for example) ascertaining the centre of the selected region and working out which regions are present at this point would not give an accurate picture. I therefore needed a method to ascertain all regions that intersect with the selected region anywhere within the region.
I cam across someone else who had this same issue (https://gis.stackexchange.com/questions/170919/how-to-tell-if-a-geojson-path-intersects-with-another-feature-in-leaflet) and the answers pointed me towards the turf.js library, which I had also previously used for Speak For Yersel. I downloaded the most recent version of Turf (version 7) and wrote a test script to see whether it would work. However, I just couldn’t seem to get Turf to work with the geoJSON data – it just gave errors. I also tried plugging in the example given in the answer on the page linked to above and it also gave errors, which was very frustrating. Eventually I decided to try an older version of Turf (version 6) and both the example code and my geoJSON areas worked perfectly, so there much be something in Turf 7 that has changed. However, this means I can process all of the intersects using Turf 6 and store these as a cached list to be used whenever a region is selected from the lists. This will be my task of the start of next week.
Also this week I worked on the Books and Borrowing project. I now have access to the Stirling servers again and was therefore able to work through some of the outstanding tasks, although I won’t be able to update the online systems until the duplicate author work that another member of the team is working on has been completed (well, I could, but then I’d have to do it all again afterwards) so for now most of the updates are limited to the version of the site running on my laptop.
What I did manage to do was fix the weird apostrophe encoding issues across all data types, replacing the encoded apostrophes with regular straight ones. I’ve made this change on the live database, but until I regenerate the search indexes the online searches won’t be changed. I also replaced all curly apostrophes and double-quotation marks found in authors, borrowers, book editions, works, holdings and borrowing records with their straight equivalents. This took some time to implement and test as it involved updating several thousand records. All seems to have worked, though. Again, this change has been made in the online database and you can see the change when viewing records (see for example: https://borrowing.stir.ac.uk/search/0/0/0/advanced/brids|32203 where the text ‘Peveril of the Peak : by the author of “Waverley, Kenilworth”’ once had curly double-quotes).
I also removed ESTC from the search results filters (again, only on my laptop for now) and added in Book Work title to the search results filters (also only oh my laptop for now) as the following screenshot shows:
The book work titles are rather long, but I think it works ok and I’m sure will be a really useful addition. In order to implement this update I needed to update the Solr index structure to add a new field, reindex the data, update the API and the script that processes search results, so a fair amount of work, but worth it. I also still need to update the script that generates the Solr data to ensure it differentiates authors that have the same name, but I’ll wait to do this until the duplicate author work is complete. After that I’ll then grab a fresh copy of the online data, run it through the various stages needed to generate indexes and caches, test everything out on my laptop again, and after that I’ll need to get Stirling’s IT people to set up a new Solr index and import the new files. Once this has been completed I’ll then be able to update the online API and front-end.
Also this week I made some updates to the structure of the SpeechStar resource for Eleanor Lawson. These included some major changes of page URLs, which introduced some complexity as we don’t want people who may have cited a view of the old pages, or a specific video on one of those pages to end up with a ‘not found’. To avoid this I put in redirects from the old URLs, but this wasn’t enough as I also needed to ensure any filters were also passed over to the new pages, but this is all in place now. So for example, someone who has bookmarked or cited the ‘Child speech error database’, listing the videos by word, filtered by sound: k and sex: female with the video for ‘school’ open would have this link:
And rather than giving a ‘not found’ or redirecting to the default view of the page the above link continues to open the correct video with the correct filters.