I spent most of this week working on the new timeline features for the Historical Thesaurus. Marc, Fraser and I had a useful meeting on Wednesday where we discussed some final tweaks to the mini-timelines and the category page in general, and also discussed some future updates to the sparklines.
I made the mini-timelines slightly smaller than they were previously, and Marc changed the colours used for them. I also updated the script that generates the category page content via an AJAX call so that an additional ‘sort by’ option could be passed to it. I then implemented sorting options that matched up with those available through the full Timeline feature, namely sorting by first attested date, alphabetically, and length of use. I also updated this script to allow users to control whether the mini-timelines appear on the page or not. With these options available via the back-end script I then set up the choices to be stored as a session variable, meaning the user’s choices are ‘remembered’ as they navigate throughout the site and can be applied automatically to the data.
While working on the sorting options I noticed that the alphabetical ordering of the main timeline didn’t properly order ashes and thorns – e.g. words beginning with these were appearing at the end of the list when ordered alphabetically. I fixed this so that for ordering purposes an ash is considered ‘ae’ and a thorn ‘th’. This doesn’t affect how words are displayed, just how they are ordered.
We also decided at the meeting that we would move the thesaurus sites that were on a dedicated (but old) server (namely HT, Mapping Metaphor, Thesaurus of Old English and a few others) to a more centrally hosted server that is more up to date. This switch would allow these sites to be made available via HTTPS as opposed to HTTP and will free up the old server for us to use for other things, such as some potential corpus based resources. Chris migrated the content over and after we’d sorted a couple of initial issues with the databases all of the sites appear to be working well. It is also a really good thing to have the sites available via HTTPS. We are also now considering setting up a top-level ‘.ac.uk’ address for the HT and spent some time making a case for this.
A fairly major feature I added to the HT this week was a ‘menu’ section for main categories, which contains some additional options, such as the options to change the sorting of the category pages and turn the mini-timelines on and off. For the button to open the section I decided to use the ‘hamburger’ icon, which Marc favoured, rather than a cog, which I was initially thinking of using, because a cog suggests managing options whereas this section contains both options and additional features. I initially tried adding the drop-down section as near to the icon as possible, but I didn’t like the way it split up the category information, so instead I set it to appear beneath the part of speech selection. I think this will be ok as it’s not hugely far away from the icon. I did wonder whether instead I should have a section that ‘slides up’ above the category heading, but decided this was a bad idea as if the user has the heading at the very top of the screen it might not be obvious that anything has happened.
The new section contains buttons to open the ‘timeline’ and ‘cite’ options. I’ve expanded the text to read ‘Timeline visualization’ and ‘Cite this category’ respectively. Below these buttons there are the options to sort the words. Selecting a sort option reloads the content of the category pane (maincat and subcats), while keeping the drop-down area open. Your choice is ‘remembered’ for the duration of your session, so you don’t have to keep changing the ordering as you navigate about. Changing to another part of speech or to a different category closes the drop-down section. I also updated the ‘There are xx words’ text to make it clearer how the words are ordered if the drop-down section is not open.
Below the sorting option is a further option that allows you to turn on or off the mini-timelines. As with the sorting option, your choice is ‘remembered’. I also added some tooltip text to the ‘hamburger’ icon, as I thought it was useful to have some detail about what the button does.
I then updated the main timeline so that the default sorting option aligns itself with the choice you made on the category page. E.g. If you’ve ordered the category by ‘length of use’ then the main timeline will be ordered this way too when you open it. I also set things up so that if you change the ordering via the main timeline pop-up then the ordering of the category will be updated to reflect your choice when you close the popup, although Fraser didn’t like this so I’ll probably remove this feature next week. Here’s how the new category page looks with the options menu opened:
I spent some more time on the REELS project this week, as Eila had got back to me with some feedback about the front-end. This included changing the ‘Other’ icon, which Eila didn’t like. I wasn’t too keen on it either, was I was happy to change it. I now use a sort of archway instead of the tall, thin monument, which I think works better. I also removed non-Berwickshire parishes from the Advanced Search page, tweaked some of the site text and also fixed the search for element language, which I had inadvertently broken when changing the way date searches worked last week.
Also this week I fixed an issue with the SCOTS corpus, which was giving 403 errors instead of playing the audio and video files, and was giving no results on the Advanced Search page. It turned out that this was being caused by a security patch that had been installed on the server recently, which was blocking legitimate requests for data. I was also in touch with Scot Spurlock about his crowdsourcing project, that looks to be going ahead in some capacity, although not with the funding that was initially hoped for.
Finally, I had received some feedback from Faye Hammill and her project partners about the data management plan I’d written for her project. I responded to some queries and finalised some other parts of the plan, sending off a rather extensive list of comments to her on Friday.
I again split my time mostly between REELS and Linguistic DNA and the Historical Thesaurus this week. For REELS, Carole had sent an email with lots of feedback and suggestions, so I spent some time addressing these. This included replacing the icon I’d chosen for settlements, and updating the default map zoom level to be a bit further out, so that the entire county fits on screen initially. I also updated the elements glossary ordering so that Old English “æ” and “þ” appear as if they were ‘ae’ and ‘th’ rather than at the end of the lists, and set the ordering to ignore diacritics, which were messing up the ordering a little. I also took the opportunity to update the display of the glossary so that the whole entry box for each item isn’t a link. This is because I’ve realised that some entries (e.g. St Leonard) have their own ‘find out more’ link and having a link within a link is never a good idea. Instead, there is now a ‘Search’ button at the bottom right of the entry, and if the ‘find out more’ button is present this appears next to it. I’ve changed the styling of the number of place-names and historical forms in the top right to make them look less like buttons too.
I also updated the default view of the map so that the ‘unselected’ data doesn’t appear on the map by default. You now have to manually tick the checkbox in the legend to add these in if you want them. When they are added in they appear ‘behind’ the other map markers rather than appearing on top of them, which was previously happening if you turned off the grey dots then turned them on again.
Leaflet has a method called ‘bringToBack’, which can be used to change the ordering of markers. Unfortunately you can’t apply this to an entire layer group (i.e. apply it to all grey dots in my grey dots group with one call). It took me a bit of time to figure out why this wasn’t working, but eventually I figured out I needed to call the ‘eachLayer’ method on my layer group to iterate over the contents and apply the ‘bringToBack’ method to each individual grey dot.
In addition to this update, I also set it so that changing marker categorisation in the ‘Display Options’ section now keeps the ‘Unselected’ dots off unless you choose to turn them on. I think this will be better for most users. I know when testing the map and changing categorisation the first thing I always then did was turn off the grey dots to reduce the clutter.
Carole had also pointed out an issue with the browse for sources, in that one source was appearing out of its alphabetical order and with more associated place-names than it should have. It turned out that this was a bug introduced when I’d previously added a new field for the browse list that strips out all tags (e.g. italics) from the title. This field gets populated when the source record is created or edited in the CMS. Unfortunately, I’d forgotten that sources can be added and edited directly through the add / edit historical forms page too, and I hadn’t added in the code to populate the field in these places. This meant that the field was being left blank, resulting in strange ordering and place-name numbering in the browse source page.
The biggest change that Carole had suggested was to the way in which date searches work. Rather than having the search and browse options allow the user to find place-names that have historical forms with a start / end date within the selected date or date range, Carole reckoned that identifying the earliest date for a place-name would be more useful. This was actually a pretty significant change, requiring a rewrite of large parts of the API, but I managed to get it all working. End dates have now been removed from the search and browse. The ‘browse start date’ looks for the earliest recorded start date rather than bringing back a count of place-names that have any historical form with the specified year, which I agree is much more useful. The advanced search now allows you to specify a single year, a range of years, or you can use ‘<’ and ‘>’ to search for place-names whose earliest historical form has a start date before or after a particular date.
I also finally got round to replacing the base maps with free alternatives this week. I was previously using MapBox maps for all but one of our base maps, but as MapBox only allows 50,000 map views in a month, and I’d managed almost 10,000 myself, we agreed that we couldn’t rely so heavily on the service, as the project has no ongoing funds. Thanks to some very useful advice from Chris Fleet at the NLS, I managed to switch to some free alternatives, including three that are hosted by the NLS Maps people themselves. The Default view is now Esri Topomap, the satellite view is now Esri WorldImagery (both free). Satellite with labels is still MapBox (the only one now). I’ve also included modern OS maps, courtesy of NLS, OS maps 1840-1880 from NLS and OS maps 1920-1933 as before. We now have six base maps to choose from, and I think the resource is looking pretty good. Here’s an example with OS Maps from the 1840s to 1880s selected:
For Linguistic DNA this week I continued to monitor my script that I’d set running last week to extract frequency data about the usage of Thematic Headings per decade in all of the EEBO data I have access to. I had hoped that the process would have completed by Monday, and it probably would have done, were it not for the script running out of memory as it tried to tackle the category ‘AP:04 Number’. This category is something of an outlier, and contains significantly more data than the other categories. It contains more than 2,600,000 rows, of which almost 200,000 are unique. My script stores all unique words in an associative array, with frequencies for each decade then added to it. The more unique words the larger the array and the more memory required. I skipped over the category and my script successfully dealt with the remaining categories, finishing the processing on Wednesday. I then temporarily updated the PHP settings to remove memory restrictions and set my script to deal with ‘AP:04’, which took a while but completed successfully, resulting in a horribly large spreadsheet containing almost 200,000 rows. I zipped the resulting 2,077 CSV files up and sent them on to the DHI people in Sheffield, who are going to incorporate this data into the LDNA resource.
For the Historical Thesaurus I continued to work on the new Timeline feature, this time adding in mini-timelines that will appear beside each word on the category page. Marc suggested using the ‘Bullet Chart’ option that’s available in the jQuery Sparkline library found here: https://omnipotent.net/jquery.sparkline/#s-about and I’ve been looking into this.
Initially I ran into some difficulty with the limited number of options available. E.g. you can’t specify a start value for the chart, only an end value (although I later discovered that there is an undocumented setting for this in the source code), and individual blocks also don’t have start and end points but instead are single points that take their start value from the previous block. Also, data needs to be added in reverse order or things don’t display properly.
I must admit that trying to figure out how to hack about with our data to fit it in as the library required gave me a splitting headache and I eventually abandoned the library and wondered whether I could just make a ‘mini’ version using the D3 timeline plugin I was already using. After all there are lots of example of single bar timelines in the documentation: https://github.com/denisemauldin/d3-timeline. However, after more playing around with this library I realised that it just wasn’t very well suited to being shrunk to an inline size. Things started to break in weird ways when the dimensions were made very small and I didn’t want to have to furtle about with the library’s source code too much, after already having had to do so for the main timeline.
So, after taking some ibuprofen I returned to the ‘Bullet Chart’ and finally managed to figure out how to make our data work and get it all added in reverse order. As the start has to be zero, I made the end of the chart 1000, and all data has 1000 years taken off it. If I hadn’t done this then OE would have started midway through the chart. Individual years were not displaying due to being too narrow so I’ve added a range of 50 years on to them, which I later reduced to 20 years after feedback from Fraser. I also managed to figure out how to reduce the thickness of the bar running along the middle of the visualisation. This wasn’t entirely straightforward as the library uses HTML Canvas rather than SVG. This means you can’t just view the source of the visualisation using the browser’s ‘select element’ feature and tinker with it. Instead I had to hack about with the library’s source code to change the coordinates of the rectangle that gets created. Here’s an example of where I’d got to during the week:
On Friday I returned to working on the mini-timelines, this time figuring out how to incorporate them into the test version of the Thesarus’s category page, for both main and subcats (which are handled differently in the code). It took quite some time to figure out how the load the required data into the page so that the library could access it, but I figured it out (data is added to a hidden element in each word’s section and my JavaScript file can then access this as required).
I positioned the timelines to the right of each word’s section, next to the magnifying glass. There’s a tooltip that displays the fulldate field on hover. I figured out how to position the ‘line’ at the bottom of the timeline, rather in the middle, and I’ve disabled highlighting of sections on mouse over and have made the background look transparent. It’s not, actually. I tried this but the ‘white’ blocks actually cover up unwanted sections of the other colour so setting things to transparent messed up the timeline. Instead the code works out if the row is odd or even and grabs the row colour based on this. I had to remove the shades of grey from the subcat backgrounds to make this work. But actually I think the page looks better without the subcats being in grey. So, here is an example of the mini timelines in the category test page:
I think it’s looking pretty good. The only downside is these mini-timelines sort of make my original full timeline a little obsolete.
I worked on a few other projects this week as well. I sorted out access to the ‘Editing Burns’ website for a new administrator who has started, and I investigated some strange errors with the ‘Seeing Speech’ website whereby the video files were being blocked. It turned out to be down to a new security patch that had been installed on the server and after Chris updated this things started working again.
I also met with Megan Coyer to discuss her ‘Hogg in Fraser’s Magazine’ project. She had received XML files containing OCR text and metadata for all of the Fraser’s Magazine issues and wanted me to process the files to convert them to a format that she and her RA could more easily use. Basically she wanted the full OCR text, plus the Record ID, title, volume, issue, publication date and contributor information to be added to one Word file.
There were 17,072 XML files and initially I wrote a script that grabbed the required data and generated a single HTML file, that I was then going to convert into DOCX format. However, the resulting file was over 600Mb in size, which was too big to work with. I decided therefore to generate individual documents for each volume in the data. This results in 81 files (including one for all of the XML files that don’t seem to include a volume). The files are a more manageable size, but are still thousands of pages long in Word. This seemed to suit Megan’s needs and I moved the Word files to her shared folder for her to work with.
Monday this week was the May Day holiday, so it was a four-day week for me. I divided my time primarily between REELS and Linguistic DNA and updates to the Historical Thesaurus timeline interface. For REELS I contacted Chris Fleet at the NLS about using one of their base maps in our map interface. I’d found one that was apparently free to use and I wanted to check we had the details and attribution right. Thankfully we did, and Chris very helpfully suggested another base map of theirs that we might be able to incorporate too. He also pointed me towards an amazing crowdsourced resource that they had set up that has gathered more than 2 million map labels from the OS six-inch to the mile, 1888-1913 maps (see http://geo.nls.uk/maps/gb1900/). It’s very impressive.
I also tackled the issue of adding icons to the map for classification codes rather than just having coloured spots. This is something I’d had in mind from the very start of the project, but I wasn’t sure how feasible it would be to incorporate. I started off by trying to add in Font Awesome icons, which is pretty easy to do with a Leaflet plugin. However, I soon realised that Font Awesome just didn’t have the range of icons that I required for things like ‘coastal’, ‘antiquity’ ‘ecclesiastical’ and the like. Instead I found some more useful icons: https://mapicons.mapsmarker.com/category/markers/. The icons are released under a Creative Commons license and are free to use. Unfortunately they are PNG rather than SVG icons, so they won’t scale quite as nicely, but they don’t look too bad on an iPad’s ‘retina’ display, so I think they’ll do. I created custom markers for each icon and gave them additional styling with CSS. I updated the map legend to incorporate them as well, and I think they’re looking pretty good. It’s certainly easier to tell at a glance what each marker represents. Here’s a screenshot of how things currently look (but this of course still might change):
I also slightly changed all of the regular coloured dots on the map to give them a dark grey border, which helps them stand out a bit more on the maps, and I have updated the way map marker colours are used for the ‘start date’ and ‘altitude’ maps. If you categorise the map by start date the marker colours now have a fixed gradient, ranging from dark blue for 1000-1099 to red for after 1900 (the idea being things that are in the distant past are ‘cold’ and more recent things are still ‘hot’). Hopefully this will make it easier to tell at a glance which names are older and which are more recent. Here’s an example:
For the ‘categorised by altitude’ view I made the fixed gradient use the standard way of representing altitude on maps – ranging from dark green for low altitude, through browns and dark reds for high altitude, as this screenshot shows:
From the above screenshots you can see that I’ve also updated the map legend so that the coloured areas match the map markers, and I also added a scale to the map, with both metric and imperial units shown, which is what the team wanted. There are still some further changes to be made, such as updating the base maps, and I’ll continue with this next week.
For Linguistic DNA and the Historical Thesaurus I met with Marc and Fraser on Wednesday morning to discuss updates. We agreed that I would return to working on the sparklines in the next few weeks and I received a few further suggestions regarding the Historical Thesaurus timeline feature. Marc has noticed that if your cursor was over the timeline then it wasn’t possible to scroll the page, even though a long timeline might go off the bottom of the screen. If you moved your cursor to the sides of the timeline graphic scrolling worked normally, though. It turned out that the SVG image was grabbing all of the pointer events so the HTML in the background never knew the scroll event was happening. By setting the SVG to ‘pointer-events: none’ in the CSS the scroll events cascade down to the HTML and scrolling can take place. However, this then stops the SVG being able to process click events, meaning the tooltips break. Thankfully adding in ‘pointer-events: all’ to the bars, spots and OE label fixes this, apart from one oddity: if your cursor is positioned over a bar, spot or the OE label and you try to scroll then nothing happens. This is a relatively minor thing, though. I also updated the timeline font so that it uses the font we use elsewhere on the site.
I also made the part of speech in the timeline heading lower-case to match the rest of the site, and I also realised that the timeline wasn’t using the newer versions of the abbreviations we’d decided upon (e.g. ‘adj.’ rather than ‘aj.’) so I updated this, and also added in the tooltip. Finally, I addressed another bug whereby very short timelines were getting cut off. I added extra height to the timeline when there are only a few rows, which stops this happening.
I had a Skype meeting with Mike Pidd had his team at DHI about the EEBO frequency data for Linguistic DNA on Wednesday afternoon. We agreed that I would write a script that would output the frequency data for each Thematic Heading per decade as a series of CSV files that I would then send on to the team. We also discussed the Sparkline interface and the HT’s API a bit more, and I gave some further explanation as to how the sparklines work. After the meeting I started work on the export script, which does the following:
I started my script running on Thursday, and left my PC on over night to try and get the processing complete. I left it running when I went home at 5pm, expecting to find several hundred CSV files had been outputted. Instead, Windows had automatically installed an update and restarted my PC at 5:30, thus cancelling the script, which was seriously annoying. It does not seem to be possible to stop Windows doing such things, as although there are plenty of Google results about how to stop Windows automatically restarting when installing updates Microsoft changes Windows so often that all the listed ways I’ve looked at no longer work. It’s absolutely ridiculous as it means running batch processes that might take a few days is basically impossible to do with any reliability on a Windows machine.
Moving on to other tasks I undertook this week: I sorted out payment for the annual Apple Developer subscription, which is necessary for our apps to continue to be listed on the App Store. I also responded to a couple of app related queries from an external developer who is making an app for the University. I also sorted out the retention period for user statistics for Google Analytics for all of the sites we host, after Marc asked me to look at this.
I continued to work on the REELS website for a lot of this week, and attended a team meeting for the project on Wednesday afternoon. In the run-up to the meeting I worked towards finalising the interface for the map. Previously I’d just been using colour schemes and layouts I’d taken from previous projects I’d worked on, but I needed to develop an interface that was right for the current project. I played around with some different colour schemes before settling on one that’s sort of green and blue, with red as a hover-over. I also updated the layout of the textual list of records to make the buttons display a bit more nicely, and updated the layout of the record page to place the description text above the map. Navigation links and buttons also now appear as buttons across the top of pages, whereas previously they were all over the place. Here’s an example of the record page:
The team meeting was really useful, as Simon had some useful feedback on the CMS and we all went through the front-end and discussed some of the outstanding issues. By the end of the meeting I had accumulated quite a number of items to add to my ‘to do’ list, and I worked my way through these during the rest of the week. These included:
I also investigated some issues with the Export script that Daibhidh had reported. It turned out that these were being caused by Excel. The output file is a comma separated value file encoded in UTF-8. I’d included instructions on how to import the file into Excel to allow UTF-8 characters to display properly, but for some reason this method was causing some of the description fields to be incorrectly split up. If instead of importing the file following the instructions it was opened directly into Excel the fields get split up into their proper columns correctly, but you end up with a bunch of garbled UTF-8 characters.
After a bit of research I figured out a way for the CSV file to be directly opened in Excel with the UTF-8 characters intact (and with the columns not getting split up where they shouldn’t). By setting my script to include ‘Byte Order Marking’ at the top of the file, Excel magically knows to render the UTF-8 characters properly.
In addition to the REELS project, I attended an IT Services meeting on Wednesday morning. It was billed as a ‘Review of IT Support for Researchers’ meeting but in reality the focus of pretty much the whole meeting was on the proposal for the high performance compute cluster, with most of the discussions being about the sorts of hardware setup it should feature. This is obviously very important for researchers dealing with petabytes and exabytes of data and there were heated debates about whether there were too many GPUs when CPUs would be more useful (and vice versa) but really this isn’t particularly important for anything I’m involved with. The other sections of the agenda (training, staff support etc) were also entirely focussed on HPC and running intensive computing jobs, not on things like web servers and online resources. I’m afraid there wasn’t really anything I could contribute to the discussions.
I did learn a few interesting things, though, namely: IT Services are going to start offering a training course in R, which might be useful. Also, Machine Learning is very much considered the next big thing and is already being used quite heavily in other parts of the University. Machine Learning works better with GPUs rather than CPUs and there are apparently some quite easy to use Machine Learning packages out there now. Google has an online tool called Colaboratory (https://colab.research.google.com) for Machine Learning education and research, which might be useful to investigate. Also, IT Services offer Unix tutorials here: http://nyx.cent.gla.ac.uk/unix/ and other help documentation about HPC, R and other software here: http://nyx.cent.gla.ac.uk/unix/ These don’t seem to be publicised anywhere, but might be useful.
I also worked on a number of other projects this week, including creating a timeline feature based on data about the Burns song ‘Afton Water’ that Brianna had sent me for the RNSN project. I created this using the timeline.js library (https://timeline.knightlab.com/), which is a great library and really easy to use. I also responded to a query about some maps of the Ramsay ARHC project, which is now underway. Also, Jane and Eleanor got back to me with some feedback on my mock-up designs for the new Seeing Speech website. They have decided on a version that is very similar in layout to the old site, and they had suggested several further tweaks. I created a new mock-up with these tweaks in place, which they both seem happy with. Once they have worked a bit more on the content of the site I will then be able to begin the full migration to the new design.