I spent the majority of this week continuing to work on the interactive map of Burns correspondence, most of it working on implementing the filter options. I started developing the filters form within the map menu, but I came to the conclusion that the menu was just too small for the large number of filter options that we want to offer. The following screenshot shows my initial attempt, with hardly any of the filter form visible in the menu.
Instead I decided to set the filters tab of the menu to display a summary of the currently selected filters and a count of the matching correspondence. I then added in a button that opens up the filter options in a popup similar to the one used to display the biographical information about correspondents, as the following screenshot demonstrates:
The filter options are split into two columns, with the right column containing the categories in a scrollable area. Both hierarchical levels are represented, and you can select / deselect all, or select / deselect an entire category group by pressing on the checkbox next to its parent. On the left there are filters for correspondent and correspondence. For correspondent you can search for an individual person. This is an ‘autocomplete’ and if you start typing (e.g. ‘rob’) then a list of possible matches appear, from which you can select one. I figured it wasn’t really necessary to allow users to search for multiple people as this would increase the complexity and correspondence only ever involve two people anyway. In addition to selecting a person you can choose to filter by gender (e.g. all correspondence involving women). You can also specify the role of the correspondent, e.g. limiting the map to showing correspondence that were sent by women, or received by ‘Clarinda’. Leaving this option blank means the filter matches both sender and recipient.
For correspondence you can limit to a single date or a range. I’ve added tooltips to give information about what exactly is possible, but you can search for a year, a year and a month or an exact day in each field. You can also limit by location. This isn’t an autocomplete but instead matches whatever you enter against the full location field. I figured this would be more useful as (for example) you can enter ‘Edinburgh’ and find all locations in Edinburgh, or you can enter a specific place in Edinburgh.
Any combination of these filters can be chosen, although not all combinations will make sense or give results. Pressing on the ‘Filter’ button at the bottom of the popup closes the popup and performs the search, limiting the data on the map accordingly. When you press on a map marker the correspondence listed are similarly filtered. After performing a search and closing the filter popup a summary of your filters is displayed in the tab (as mentioned earlier). If you re-open the filter popup your selected filters remain in the form and you can either edit your selection or press the ‘Reset filters’ at the top of the popup to start afresh.
I had to do a lot of behind the scenes work to get all of this operational – lots of updates to the API to process the filter selection for both the map data and the list of correspondence. I think it’s working pretty well and it’s much nicer to use the map when it’s filtered rather than showing all of the data. Next week I’ll continue with the outstanding tasks mentioned last week: the timeline, citation and bookmarking etc.
Other than working on the correspondence map I spent some further time on the article for the Books and Borrowers project, adding in new sections on the four-tier book structure, authors, genre and borrower occupations. I also spotted an issue with the online resource’s advanced search, whereby the library registers were not getting added to the multi-select area. I realised this was an unintended consequence of an update to the API I’d made a couple of weeks back to cut down on complex queries. The call to return registers was running several subqueries to work out counts of things like associated borrowers and borrowings. I had thought the call was only ever used when viewing registers for a specific library, and in this use case the query was pretty rapid as only a few registers at a time were ever involved. I therefore blocked a call to the API that didn’t specify a library, not realising this call was actually used to generate the list of registers for the advanced search form.
Once I realised that the call was actually needed I decided to create static cached fields in the register table for the counts, rather than conducting the counts on the fly in subqueries. With the new fields in place I then created a script that would populate the fields and updated the API to reference these fields rather than perform the additional queries. This resulted in a significantly faster query time and the advanced search form now includes registers again.
Also this week I had an email conversation with Matthew Creasy about an interactive map we would like me to build. I can’t say much more about it at this stage, but we’ve arranged to meet the week after next to talk about it in more detail. I also spoke to Thomas Clancy about a possible funding application and made a few minor tweaks to the Speak For Yersel region resources, which will hopefully be going live fairly soon.
I had my PDR session this week, which was all very positive. I also spoke to Jane Stuart-Smith about adding data to the VARICS project website, which I’ll take forward next month, and made a few further updates to the Anthology of 16th and Early 17th Century Scots Poetry. I also updated the layout of the sparklines in entry pages on our test interface of the Dictionary of the Scots Language as someone had spotted a slight issue when viewing entries on Mobile devices.
The bulk of my week was split between writing the article on the development of the Books and Borrowing project and implementing the interactive map of Burns correspondence. I completed my sections for the article and sent it on to Katie for feedback on Wednesday. Katie then got back to me with some further suggestions on Friday and I’ll hopefully finish a complete first draft early next week. I spent at least two full days on this task this week, but there’s not really much more I can say about it here.
I devoted most of Thursday and Friday to the Burns correspondence map, which I think is coming along very nicely now. Previously I’d suggested that the correspondence category system was rather unwieldy (featuring around 50 categories) and that it might be better to amalgamate some categories and add in groupings to these categories. I’ve now implemented the two-tier category system, assigning a ‘parent’ category to all categories in the system. This required me to update the database structure to add a new parent category ID. I also updated the CMS to take parent categories into consideration: when adding or editing a category the parents now appear in a drop-down list and the ‘browse categories’ page now features the parent category as a column in the table.
However, I spent most of my time developing the front-end. A couple of minor things to start with: Firstly, the ‘attribution’ text in the top right was a bit long on mobile devices so I’ve replaced this with a shorter link that opens a popup with more complete information. Secondly, the ‘reset map’ option on the ‘Home’ tab now works, and does what you might expect.
When a user selects a location on the map the information in the ‘Correspondence’ tab is now more complete, as the screenshot below demonstrates. Previously only the date was displayed but now each correspondence features a ‘From’ and ‘To’ section with a little envelope and arrow icon in between. The name, dates and location of the sender and recipient feature in these sections. I also added a ‘Highlight’ button to the top-right of the correspondence area. Pressing on this gives the selected record a yellow background and more importantly highlights the locations and the joining line in yellow on the map. It also zooms and pans the map so both locations are visible, but unfortunately this doesn’t take the presence of the menu into consideration so you’ll either have to manually move the map or minimise the menu to see things properly. The yellow is also a little indistinct and I might see about changing this. It can also get lost in the mass of red lines too, even though I’ve set the code to bring the yellow line to the front. You can highlight any number of records at a time in this way. I’ll need to do some further testing, though, as I’ve spotted some instances where the map data isn’t highlighting when it should.
I also added a count of the number of correspondence at the selected location that appears above the list of records as I figured this might be helpful. Previously I had been considering adding all information about the correspondents to the records (with an option to switch from a ‘compact’ to ‘full’ view) but I’ve realised that this would mean repeating a huge amount of data (e.g. having all of the biographical data for Burns listed over and over again). I’ve decided instead to have the correspondents’ names as links. Pressing on a name opens up a popup with the full biographical information in it, as you can see in the following screenshot:
I think this works much better. All available correspondent information is listed (e.g. names, dates, gender, biography, image, links) plus a count of the number of times the person appears as a sender and a recipient as I figured this would be useful.
The last thing I did this week was to update the API to ensure category data is published, and I added this information to the ‘Correspondence’ tab for each record, underneath the ‘From’ and ‘To’ sections. This lists the parent and child categories. If multiple child categories are included for one parent the parent only appears once rather than each time.
Next week I’ll move onto the implementing the filters. This will allow users to limit the data on the map (and in the correspondence tab) to various filters in combination, e.g. people, genders, dates, categories. Once this is in place I’ll still need to develop the interactive timeline feature and also add options to cite / bookmark specific views and options to download data as CSV. So lots still to do.
I spent a fair amount of time this week continuing to work on the Burns Correspondence map. Map markers are now currently all in grey by default, and I’ve added in grey dotted lines showing the connections between places. This results in a very cluttered map with all data visible, but of course once filters are applied (not yet developed) the map will contain less data. It is also now possible to press on a marker to highlight the place, the connections and the places connected to. This highlights the pressed on place with a red border, turns the connecting lines red and highlights the connected locations with a black border. I had previously allowed the user to select multiple locations but I decided this just made things messy so now if you select a different location your previous selection is unselected and replaced with the new selection.
I also worked on a map menu, which now has four tabs: Home (which will include an introduction to the resource and a ‘reset map’ button (not currently operational); Correspondence, which will contain all information pertaining to the location the user presses on; Filters which will contain the filter options and Timeline, which will include the facility to animate the correspondence over time.
I’m still in the middle of working on the Correspondence tab. When you press on a location marker this tab gets selected (if the menu is currently hidden it opens) and all of the correspondence sent from/to this location will be listed, in date order. For now only the dates are displayed, with correspondence items split into sections by year. There is going to be rather a lot of data contained in each of these sections, and I might add in an option to further expand the menu to help people view the data. We might want to give the option of switching from a ‘compact’ to an ‘expanded’ view of the data, with things like the bibliographies only appearing in the latter. I did this for the Books and Borrowing project and it worked pretty well – see https://borrowing.stir.ac.uk/search/0/0/0/advanced/bhids|9803 with the ‘switch to…’ option in the menu bar to the right of the ‘Search Results’ text. I might also add in a further highlighting option whereby people can press on the correspondence item in the menu and see its corresponding line on the map. Below is a screenshot of the ‘full screen’ view with ‘Lochlie’ selected so you can see how things currently look. There’s clearly still a lot to do, but I feel that I’ve made some pretty good progress this week.
I participated in two meeting this week with researchers who are putting together proposals that I’ll be involved with. The first was with Garrick Allen in Theology and the second was with Emily Pickard in Scottish Literature. I needed to spend some time prior to each meeting reading through the proposal documentation and preparing some notes and questions, and then some further time after each meeting writing some materials to be included and responding to questions raised. Both meetings went very well and hopefully both researcher will secure their respective funding.
I also had a further email conversation with Gerry Carruthers and members of his team about their new proposal and fixed a small bug in the new DSL sparklines that I’d introduced last week. I also removed a video that needed to be taken down from the Speech Star website, made a few further tweaks to the contents of the Irish and Northern Irish surveys for Speak For Yersel and make a few minor changes to the Anthology of 16th and Early 17th Century Scots Poetry.
I spent the remainder of the week working on the article about the development of the Books and Borrowing resource that I’m writing with Katie and Matt. I made good progress on my sections and I hope to get a first draft finished next week.
I was off work sick on Monday and Tuesday this week, but thankfully I was feeling well again by Wednesday. On Wednesday I had a meeting with Katie Halsey and Matt Sangster about an article we’re going to write about the Books and Borrowing online resource. We now have an overview of what we’re going to write about and will try to have a first draft written by the end of the month. I also had to spend a bit of time investigating some speed issues with the Books and Borrowing website. One thing that may have been contributing to this was a call to the API that isn’t used in the front-end but was being submitted by a web crawler. The call was returning details for all library registers in the system, resulting in a query that was taking around 13 seconds to execute. The endpoint is only ever used in the front-end with a library specified, which is a much quicker query and I therefore blocked the endpoint from running if a library isn’t specified.
On Thursday I had a meeting with the Rangar’s Islands project (https://blogs.nottingham.ac.uk/ragnasislands/). The project is based at Nottingham and they’re putting together a place-name database for three of the Orkney Islands. They’d like to use the system I’ve created for other place-name projects such as Iona and we met to discuss how this might come about. The project will be applying for funding to develop the front-end and all being well I’ll be involved in setting this up.
Also this week I made some further tweaks to the new date search facilities of the Dictionaries of the Scots Language. The team wanted the date range that is visible to the left of the sparklines in the search results to default to the limited range reflected in the sparkline text rather than the actual first and last dates of attestation. So for example a DOST entry with dates from 1175-1780 would not show these dates but would instead display ‘<1375-1700+’. Implementing this wasn’t as straightforward as I’d hoped it might be. The Solr index contains the first and last dates of attestation, and it’s these that are used to display the dates. However, I couldn’t update them to something like ‘<1375’ or ‘1700+’ as these dates are used in the filter options and need to be valid integers. I did consider updating the Solr index to store the amended first and last dates as new fields, but this would have meant updating the structure of the Solr indexes, plus another regeneration of the Solr data. Instead the results script picks out the first and last dates from the sparkline text on the fly, taking into consideration any ‘<’ in the first date, ‘+’ in the last date and ensuring only one date is displayed if ‘from’ and ‘to’ are the same.
I’d also spotted a bug in the search when a phrase was searched for that wasn’t surrounded by double quotes. This was bringing back some very garbled results. To fix this I needed to update the search results page to ensure such searches were analysed and surrounded by double quotes if these were needed prior to sending the request to the API. Implementing this was also rather trickier to fix than I’d hoped. I couldn’t just add quotes if the query had a space as I also needed to check for Booleans. If these are present then we definitely don’t want quotes to be added. Also the query string includes all of the other bits that appear after the phrase (i.e. /quotes/full/both/full) so I couldn’t just surround the entire thing in quotes but instead needed to extract just the phrase, add quotes then add it back to the whole string. However, I managed to get it working and the results returned now make more sense.
Also this week I had email conversations with Joanna Kopaczyk and Gerry Carruthers about new (separate) proposals they are putting together that I will be a part of and arranged to meet Garrick Allen to discuss a proposal he is developing. I also replaced a couple of videos on the various Speech Star websites at the request of Eleanor Lawson.
I spent the remainder of the week working on the Burns Correspondence map, mostly on the API and queries and planning on how the map interface will function, specifically the lines joining locations. Displaying all of the lines swamps the map and is just too much information to display at once, so I’m going to have to think about how best to proceed. The screenshot below demonstrates just how busy the map gets with all of the lines on:
I’ll continue to think about this next week.
This week I returned to working for the Books and Borrowing project, spending about a day creating a new version of the public data that incorporated some large additions to the Orkney dataset that one of the researchers has been working on over the past few months. It takes quite a bit of time to prepare the data for publication, which results in a new version of the site’s Solr index plus the generation of several other cached files. I also had a chat with project PI Katie about a potential publication about the project’s online resource. We’re probably going to speak more about this next week.
Also this week the updates I’d made to the DSL’s Solr index last week went live and I also fixed an issue with the sparkline tooltip text in the search results on touchscreens, which one of the DSL team had spotted. This issue was that the search results are block-level links – pressing on anywhere within the area of a result triggers the loading of the entry. As the sparkline is within this block pressing on it was also triggering the load. As there is no ‘hover-over’ on touchscreens the sparkline tooltip can only be triggered by pressing on it, which also triggers the page load.
Thankfully there was a quick fix for this: a small amount of JavaScript that ensured that pressing on the sparkline no longer triggers the page load. You can still press anywhere else in the result block to load the entry, while pressing on the sparkline now makes the sparkline tooltip appear and nothing more. I tested this out on my iPad and Android phone and it’s working fine for me.
I also spent about a day making updates to the new Speak For Yersel regions (these regions are not yet publicly available). Following pilot testing of the regions I’d been given rather lengthy lists of things to change for Ireland and Northern Ireland. These ranged from changing the question and answer option text to replacing or deleting entire questions or swapping the sound files for new versions.
One of the researchers also wanted the order for the Ireland survey questions to be randomised each time the survey loads and in order to implement this I had to update the database structure and the survey setup script to add in an ‘random question order’ option when creating each survey. I then needed to update the script that pulls questions into the survey so that the order was randomised when this option has been set. And as the question number that is displayed in the top-right of the question pane was dependent on the fixed ‘question order’ field stored in the database I had to ensure this data was reset after the question data was randomised and before it was used in the front-end. So there was quite a lot to update, but it’s all working now.
Also this week I responded to questions about WordPress plugins from Petra Poncarova about her Erskine site (https://erskine.glasgow.ac.uk/) and Amy Wilcockson about the Robert Fergusson site (https://robert-fergusson.glasgow.ac.uk/).
I spent the remainder of the week beginning work on the front-end for the Robert Burns correspondence map. I’d completed work on the CMS for this project several months ago and since then a researcher has populated it with data about locations, people and correspondence and it’s now time to see how all of this will work on an interactive map. Some of my time was devoted to planning how the resource will function and some further time was spent setting up the basic infrastructure and an initial map.
For now the map uses the six-inch Ordnance Survey maps (1843-1882) from the NLS and the same base-map as I used on Gerry McKeever’s Regional Romanticism map (https://regionalromanticism.glasgow.ac.uk/paul-jones/), although this may change. Currently the map appears within the layout of the site, with a ‘full screen’ option in the top left, but I may change this to make the default view take up the entire browser window, as with the Paul Jones map linked to above.
I decided that I wanted to try and have a map menu that appeared across the full width of the bottom of the map, but was unable to find a Leaflet plugin that offered this. It’s likely that this is because Leaflet plugins have to be added to a specific container that is associated with either the top left or right or bottom left or right of the Leaflet map – there doesn’t appear to be anything that expands across the full width of the map. I tried to implement a layout that would break out of these confines but didn’t manage to find a CSS solution within the Leaflet framework so instead I created my own pane using jQuery. This currently takes up the bottom third of the map, has a semi-transparent background and has a button that slides the panel down and hides it, and another button that slides the panel up and reinstates it, as you can see in the following screenshot:
The menu is currently empty but it’s here that I will place the timeslider to limit the data to specific dates, plus options to filter the map by categories and people. Currently every location in the system is displayed as a red dot on the map, but as of yet there is no information about correspondents and correspondence. I will add lines between locations to show correspondence and I still need to figure out how I’m going to add correspondents and display the information about each correspondence. I’ll continue with this next week.