This was another week of many projects, the first being the Burns Supper project.  This week I finished implementing all of the data visualisations on the ‘facts and figures’ pop-up, with a mixture of bar, column and pie charts depending on how many data types there are.  All charts are ordered by number of suppers, other than the pie charts.  I also added in a statement above each visualisation about the number of suppers that supplied the data and a note when a supper may have more than one type of data.  The screenshot below shows part of the popup with three visualisation types (mostly) visible:

I haven’t updated the narrative text to update the percentages to be based around the number of suppers that supplied the data rather than the total number of suppers and to include additional explanatory text, but I’ll do this next week.

For the HiMuJe Malabar project I have now incorporated the updated JSON file that contains cats and subcats into the map.  I have also incorporated the GeoJSON polygons that project RA Renu has created so far.  Where a place has a polygon this is used instead of a marker, and polygons feature a tooltip on hover-over in the same way as markers.  I added the type and subtype to the tooltip as well.  I also made a start on the categorisation of places based on cat and subcat, although this still needs some work.  I added in a ‘legend’ box on the right of the map and have split the places into layers based on their cat and subcat.  Each of these appears in the legend, and you can turn each layer on or off.  You can also select or deselect all layers, which I find quite useful (e.g. deselect all then only add in the layers you’re interested in).  There is also a count of the number of places in each layer as part of the layer label.  Note that the cats and subcats only appear if there is at least one place that has latitude and longitude assigned to the subcat, which is why (for example) ‘Mosque’ is not currently listed.  Below is a screenshot showing the categorisation options that are currently in place:

There’s still quite a lot to do and some issues that need to be addressed.  At the moment only places that have a subcat appear.  I still need to implement the top-level cats, so for example ‘Sri Lanka’ that has cat ‘Region’ and no subcat is not currently appearing on the map.  I’m hoping to update the legend to give it a two-level hierarchy (as mentioned in my specification document) but I didn’t have time to implement this.  Also, marker and layer colours are currently arbitrarily assigned.  I think we will eventually have icons as well as colours, but in addition to this I will update the colours, probably to have different shades for each subcat in a cat.  I also need to ensure that the polygons always appear behind the markers as this is not currently the case.  E.g. if you turn the ‘Region: Kingdom’ layer off and then on again the layer is then added to the front, meaning it’s no longer possible to press on any markers located within the polygons.  Also, there will be several other categorisation options in addition to cat and subcat (as discussed in the specification document) and these still need to be implemented.

Also this week, I returned to working on the Playbills project for the first time since before Easter.  This included writing an executing a script to merge genre classifications, which reduced the number of genres from 188 to 69 and reassigned any plays that were assigned to a deleted genre.  I also executed my scripts that process performers to extract individual names and titles from the free-text ‘full name’ column and assigned gender based on the title.  My scripts process individual performers and also split multiple performers into individual records, resulting in the number of performers going up from 49498 to 52146.  In these cases the newly extracted performers have also been assigned to the same role as the existing row.  Of the 52146 performers we now have a Male or Female gender assigned to 51724, which I think is pretty good.  I also updated the playbill and play pages so that the extracted names and genders now appear, and multiple performers for a role all now appear.  I still need to deal with the ‘unprocessed’ performer spreadsheet, which is probably going to need some manual work as this contains the ‘edge cases’ that my processing scripts were unable to tackle.  I hope be able to continue with this next week.

My fourth project of the week was the DOST Auld Laws project, for which I continued to work on the search facilities.  This has involved setting up an Exist-db XML database and learning how to run full-text queries on the documents contained in it and how to generate KWIC (keyword in context) snippets for the results.  There’s still a lot to do, but an initial version of the quick search is now operational, as the screenshot below demonstrates:

You can also use an asterisk wildcard to represent any number of characters, for example ‘wyn*g’ finds all terms beginning ‘wyn’ and ending in ‘g’.  A question mark wildcard can also be used to represent a single character, e.g. ‘r?cht’ matches ‘richt’ and ‘rycht’.  I haven’t added in pagination of results yet, so they currently all appear on one page.  Each result features the document name and the page number where the result is found, plus the snippet with the term highlighted.  Pressing on the snippet loads the page with the line where the term is found highlighted in both the image and the text.

It’s definitely something of a milestone to get the quick search working, but there is still a lot to be done.  The highlighting of the term in the snippet is not currently working properly when the term includes characters in <expan>.  The search finds the term, but the highlighting is only getting applied to the part of the term before the <expan> tag.  Also, Exist-db uses the Lucene full-text engine and this does not support wildcards at the beginning of terms, so while a search for ‘grant*’ finds all terms beginning with ‘grant’, it’s not possible to find all words ending in ‘*ting’, for example.  I’m not sure how big an issue this is, but if such functionality is required I’ll have to investigate an alternative.  Also, phrase searching is not currently operational.  I have a query that should be able to handle this, but I haven’t had time to add this in yet.  It’s been quite tricky to implement as phase searches need to cross line and page boundaries whilst still returning the line and page IDs where the start of the phrase is found.

In addition, Boolean searches are not yet working (e.g. term1 AND term2, or term1 OR term2).  However, we might want to consider how useful these would be and how they would work.  Currently the search looks for occurrences of the term in entire documents and returns snippets showing the context of the term.  If a Boolean search is using an entire document as its source then how useful would it be if (for example) term1 is found on page one and term2 is found on page 29?  I’m just not sure how helpful this would be.  I guess an OR search would be useful in finding variant spellings?

I also still need to add in results traversal to the document page, so you can navigate directly through results when looking at one specific result, and I need to add in a link back to the results page from the document page.  There’s also still a lot of work to do on the user interface, which is still not finalised and I’ll work on this (e.g. colours, fonts, layout) once I’ve finished with the search facilities.

My fifth and final project of the week was the redevelopment of Mapping Metaphor.  I didn’t have much time left to spend on this, but I did manage to tidy up the loose ends from last week. Where a combined metaphor has both OE data and E data and the E data does not begin in the OE period the timeline in the visualisation card popup now includes OE selected and highlights the period when the E data begins.  This second highlight currently uses the green used for the visualisation background but can be changed.  We should also maybe include some explanatory text so users understand why two periods are selected.  The following screenshot shows the timeline with two periods highlighted:

I also updated the card popups so that both E and OE IDs appear in the header of the card (where applicable) now, as you can also see in the above screenshot.  Also, where a metaphor connection exists in E but not OE (or vice-versa) counts of lexemes in each joining category in the other period are now displayed.

Also this week I had an email conversation with Craig Lamont about an online resource for his new research project and arranged for a subdomain to be set up for it.  I also updated the DSL survey so that it only appears on mobile devices and responded to a request from Kirsteen McCue regarding stats for song downloads on the Editing Burns and Burns Choral websites.