I continued to work on the new region / dialect area map for the Dictionaries of the Scots Language for around half of this week.  I’d made really good progress with the map last week and by the end of the week I’d figured out a possible way to grab all areas that intersected a given GeoJSON area using the Turf.js library (version 6 as for some reason version 7 didn’t work).  This week I wrote a script to go through all of the regions, identify all intersecting regions using Turf.js and store these in a JSON file.

All went well to begin with, but midway through processing my script threw an error, with Turf unable to process a geoJSON aera.  It turned out that some of the coordinates in our region data use a lot of decimal places – more than Turf can handle, in fact.  Thankfully Turf has a helpful ‘truncate’ function that truncates the precision of a region, and once I applied this the script executed successfully.

Once I had my JSON file containing all of the intersects for every region and dialect area I could then work on what to do when the user selects a region / area from the drop-down lists.  When this happens my original intention was that the corresponding area would be highlighted in yellow, with all intersecting regions given the usual red colour, while intersecting dialect areas would be ‘off’ by default but the user could turn these on from the info pane.  I also wanted the map to zoom and pan to focus on the area that had been selected.

However, all of this brought some difficulties.  It turns out that there are a lot of intersections with our data.  Not all adjacent areas neatly line up and there are frequently very small (or not so small) overlaps.  These are of course considered intersects, meaning that many adjacent areas get added in when really they shouldn’t.  This meant that lots of areas were getting highlighted in red, and the yellow highlighting of the area the user actually selected was getting somewhat lost in the mix, due to the transparent nature of the area fills.  I therefore decided that I would only highlight the area the user actually selected (in the usual red, not yellow), and all other intersecting areas would be switched off by default, but would be selectable in the info pane, as the following screenshot demonstrates:

I think this works pretty well and is not too overwhelming for users.  The next issue I needed to address was zooming and panning to the selected area.  In order to do this I needed to use Leaflet’s ‘fitBounds’ method, which expects latitude and longitude values for the North East and South West corners of a rectangle.  Unfortunately all I had was very complex geoJSON polygons so I needed to investigate how to generate the ‘bounds’ from this.  Initially I thought I’d have to do this on the fly (i.e. when the user selected a region) and figuring out how to get Leaflet to grab a specific area within the geoJSON file and calculate this proved very tricky.  Eventually I realised that I didn’t actually need to do this:  I could simply store the ‘bounds’ for each area along with the intersects.  Turf has a plugin called ‘bbox’ (https://turfjs.org/docs/api/bbox) that can take a geoJSON shape and create a ‘bounding box’ from it.  After a bit of experimentation I realised that this solution would work perfectly, so I updated my script that generated the intersects to also generate and store the ‘bounds’ for each area.  I was then able to update my map code to grab these bounds when the user selects an area and pass them to Leaflet’s ‘fitBounds’ method, making the map show the desired location.  I updated this to add in a bit of padding and was very pleased with the result.

With this in place I then moved onto the region search.  In addition to selecting a region from a pretty sizeable drop-down list, we also wanted users to be able to search the areas, using an autocomplete textbox.  Vasilis’ prototype had used a neat little jQuery plugin called Select2 (https://select2.org/).  This included an autocomplete search box within the drop-down menu, which was nice and compact.  However, it isn’t immediately apparent to users that this search box is an option – both I and another member of the DSL team who tried out the prototype didn’t even notice this was an option until Vasilis pointed it out.  For this reason I decided to separate out the list and the search option, to hopefully make it clearer to users that a search option is even available.

I used the usual jQuery UI autocomplete that I’ve used many times before on other projects, connecting via AJAX to a PHP script that worked with the ‘intersects’ JSON file to find matching regions.  This appeared to work well, and on selection of a matching item in the autocomplete list the same code that handles the list executes to select the area on the map, as the screenshot below demonstrates.  However, I spotted that the autocomplete was not appearing when the map was in full-screen mode, and remembered I’d encountered a similar issue with the place-names projects.  For these projects the autocomplete was in a modal pop-up and using the autocomplete’s ‘appendTo’ option (https://api.jqueryui.com/autocomplete/#option-appendTo) to attach it specifically to this modal in full-screen mode did the trick.  For the current map, however, the autocomplete is in the map menu, which is a limited height based on the menu’s contents.  When the autocomplete was added it was too tall for the menu and caused awkward scrollbars to appear.  I tried several approaches to get around this issue, but in the end decided to just make the menu tall enough to hold the autocomplete when in full-screen mode, as the following screenshot demonstrates:

With this in place the ‘regions’ map was pretty much complete, and I moved onto the dialect area map.  I updated the code so that when the ‘Dialect Areas’ menu item is selected the map resets and displays the dialect areas rather than the regions.  Pressing on the ‘Regions’ menu swaps things back again, as does pressing ‘reset map’.  With the dialect areas visible, I then needed to update the code for clicking on the map so that it would display dialect areas rather than regions, and update the info pane so that dialect areas appear first, as you can see in the following screenshot, which shows one dialect area and several regions selected:

I now had the core functionality of the map in place and I then spent some time adding in more of the necessary features, including the help and about modal window, the attribution modal window and a ‘loading’ spinner that gets displayed until the geoJSON data has loaded.  A also added in the option to select a satellite base map.  I’m not sure how useful this will be, but it’s good to have it as an option that can be selected at any time from the ‘Home’ menu.  A satellite view of the previous map can be seen here:

I then moved on to what will hopefully be the final big issue relating to the development of the map: adding in the option to bookmark / cite / share a specific view of the map.  As with the place-names resources, this will involve not just tracking the position and zoom level, but also exactly which data is visible on the map as the user zooms, pans, clicks and selects things in the resource.  It’s hugely useful to be able to save or share a specific view of the map rather than only being able to link to the default view of the map, and even though it will take some time to implement such a feature I think it’s worth it.  The first thing I did was add in the leaflet hash plugin (https://github.com/mlevans/leaflet-hash) which tracks and processes position and zoom.  As with the place-name projects, I adapted this plugin to also track and process further data held in the hash (the part of the URL after that ‘#’ sign).  I decided on a structure for this data and began implementing the code that will process the various elements, ensuring that (for example) when the user has selected the satellite map this is picked up by the code and the relevant base map is shown.  By the end of the week I’d added in the code for processing the base map, the active map type (region or dialect area), the browse option (region search, region browse, dialect area browse, map click) and the browse data.  I still need to add in the selected items in the story pane, which I’ll do next week.  I also still need to update the code to add all of this information to the hash based on the user actions – for now all the code does is process information in the hash rather than add data to it.  Hopefully I’ll finish this and complete an initial version of the map next week.

My other big task for the week was to roll-out the new versioning system for Books and Borrowing, replacing all of the code and the data on the online server.  I’d got everything sorted on my laptop last week and had sent the data structure and files to Stirling’s IT people so they could set up the new Solr core that was required.  This was set up the day I asked for it, which was great, and I therefore began regenerating all of the database data on the live server.

Unfortunately I began to encounter some strange behaviour with the online version of the database, and as my contact in Stirling’s IT department is on holiday until Tuesday next week I had to abandon my attempts at rolling out the new system.

I first encountered the issue when running my script to populate the book holding totals cached data.  These are now stored in a separate table as there may be different totals for different versions of the data.  The script ran fine on my laptop, and appeared to run perfectly on the server, but when I checked the online admin interface to the database (PHPMyAdmin) the book holding totals table was empty.

I spent ages trying to work out why the ‘insert’ queries were not running, but also not giving any errors.  The exact query when entered directly into PHPMyAdmin worked and displayed the inserted row, but when executed through my script the row wasn’t shown in PHPMyAdmin.

Eventually I figured out that the queries WERE executing successfully in my script and WERE being interested into the database, as a test script I wrote connected to the database and listed the rows.  But still PHPMyAdmin was showing an empty table.  It was almost as if my scripts were connecting to a different version of the database to the one PHPMyAdmin was accessing.  However, this wasn’t the case as if I ran the ‘empty’ command in PHPMyAdmin to clear the data from the table my test script also then showed no data.  It was all very odd, but having identified the issue I could proceed to run my cache generation scripts (which included several more similar tables) and test that the data had been inserted using scripts rather than PHPMyAdmin.

However, the library cached data is pretty sizeable and instead of generating it on the server I decided to export the tables from the version of the database running on my laptop and import them into the server’s database using PHPMyAdmin.  This appeared to have worked and I could see the data in PHPMyAdmin.  I completed the data import scripts and then started on upgrading the online scripts to use my new versioning code.  I started by setting up the new ‘V2’ version of the API.  However, API calls in this that worked perfectly on my laptop were giving errors when run on the server.  After much more investigation it would appear that the library cache tables that I imported via PHPMyAdmin are coming back as empty when accessed via the API scripts, even though they contain data when viewed in PHPMyAdmin.

There is clearly something very wrong with the online database and I need someone at Stirling to investigate.  It almost looks like there are two separate versions of the database, but for the fact that running an ‘empty’ query in PHPMyAdmin definitely affected the data my scripts access.  I can’t really proceed with the rollout of the new system until this is resolved, which will hopefully be next week.  It’s all very frustrating as I just want to get the new system finished.

Also this week I spent some time fixing an old resource with Luca.  I’d received an email saying that the Working Class Marriage website (https://workingclassmarriage.gla.ac.uk/) was not working.  This wasn’t a website I had worked on, but it contains a bunch of interactive maps and a lot of data and Luca and I wanted to get it back online again, which we managed to do after some tweaking of the code.

I also spent a bit of time working for the Anglo-Norman Dictionary, fixing the ‘add source’ feature of the management system, which was unfortunately broken due to a database update not being reflected in the code.