I continued with the new developments for the Dictionaries of the Scots Language for the first three days of this week, during which time I managed to complete a first version of the new search and results filtering facilities for parts of speech and regions, which is a great milestone to reach.

I reordered the parts of speech in the Advanced Search form so that the most common parts (e.g. noun, verb) are listed first.  I updated the map so that search icons appear beside regions in the right-hand box whenever a region is selected, with the search icon disappearing when a region is deselected.  It took quite some time to get this working in all cases, as regions can be selected in several different ways (e.g. clicking on the map, selecting from the drop-down list, searching), but I think I’ve got it all sorted now.  When you press on one of these search icons the Advanced Search form loads with ‘Quotations’ selected and the relevant region / dialect area selected in the list.  I did wonder about the map automatically representing any regions entered into the advanced search form when the map button in the form is pressed, but unfortunately this isn’t going to be possible (at least not without a lot of reworking), as any number of disparate regions may be entered into the search form and the map is set up to only display one region and its intersects at a time.

I then started to work on the search results filter for POS in a headword search.  This has required some further major reworking of the API both the implement the filtering and to deal with POS passed in the URLs, but an initial version is now working.  I can’t share any screenshots for now, but the POS filter works by listing all of the parts of speech found in the results, each with a count of the number of results that match.  As entries may have multiple POS these counts in total can be higher than the total number of results, and the filters may also therefore include POS that are beyond any filters applied via the POS selector in the Advanced Search form.

What I wasn’t sure about as I worked on this is what should happen to the display of POS filters and their counts when another filter has been selected.  When a date filter is applied I’m fairly certain what should happen:  the POS filters and counts update to reflect the filtered data.  E.g. if the headwords are filtered to show only those that were first attested between 1700 and 1800 then the POS filter options should be updated to reflect these results.

But what should happen when one or more POS filters are selected?  In my initial tests what happened was that the filter is executed and the new POS options reflect the currently applied filter.  E.g. filtering the results to first attested between 1700 and 1800, and also filtered to only display only verbs.  The POS filter and counts are also updated based on this further filter to the data so only showed POS and counts that were contained within the verb filter.  However, this didn’t seem all that useful.

I reckoned we had two possible options: Firstly, when a POS filter is applied, the other POS filter options and counts are updated to reflect the currently filtered results (my initial approach); secondly, when a POS filter is applied the other POS filter options and counts are taken from the results without a POS filter applied.

I decided that the second option made more sense, as the first does wasn’t actually working as you might expect anyway, due to the POS filters being joined by OR.  For example, a results set lists 76 nouns and 37 verbs.  If you select ‘verb’ and press ‘update’ the results are limited to the 37 entries that are verbs.  The noun filter count is updated to 21, as 21 of the 37 verb entries are also nouns.  BUT, if you then select ‘noun’ in addition to ‘verb’ and press ‘update’ your results are currently not limited to 21.  What happens is the filter is re-run to find all entries that are either verbs OR nouns (or both) and you end up with 76 nouns and 37 verbs selected in the filters and 92 results, rather more than the 21 you might have expected.

If we went with option 2, after selecting ‘verb’ and limiting the results to 37, the ‘noun’ count would still be 76 and not 21, and when pressed on would give the expected results, which seemed more like the expected behaviour to me.  I therefore redeveloped the POS filter to go with option two, and whilst doing so I also slightly tweaked the layout of the filter box to divide the sections up.

With this in place I then moved onto implementing the region filters.  Previously we didn’t have a ‘filter results’ option for quotation searches and I added this in. giving the option to filter the results by dates of quotation and /or region.  This required further major updates to the API to incorporate these new options, but all is in place now.  When I came to displaying the various regions in the results filter box I decided to use the same dropdown list widget as in the advanced search form, as there can be lots of listed regions.  I also decided to place all region filters in one list rather than separate ones for geographical and dialect areas (as we have in the Advanced Search form) as this distinction isn’t evident in the listed quotes, so I figured just having them altogether would make more sense.

If a search brings back more results than we allow, the numbers for each region reflect the full results to enable the user to tell which region filters might help to being the number down to something more manageable.  However, one thing that initially confused me is that the counts in the region filter would appear to be well below the total number of results when searching both dictionaries.  This is because regions only apply to SND, so and DOST entries in the results have no regions.  Note that if you only search DOST the region filter does not appear on the results page.

I realised that if you combine a date and region search it’s possible that your selected region will not be present in the quotations for the dates specified.  E.g. ‘dre*’ limited to 1900-1950 with ‘Western Islands’ selected finds nothing as there are no ‘Western Islands’ quotes in the chosen period.  In such cases the selected region will be ignored and the result will default to the date filter.  I also updated the headword POS and date combination to work in the same way.

I also updated the search results header so that POS and region selections entered into the Advanced Search form are now reflected in the search results heder.  E.g. “Quotation search, 1700 – 1800, Glasgow.”  or “Headword search, noun, verb.”.

The quotation search (as you’d expect) searches quotations.  The API then parcels these up by entry and the results are displayed by entry.  But the underlying data is individual quotations.  The filter counts currently display the number of matching quotations NOT the number of matching entries, as the filter counts come directly from the underlying data.  So for example, a regions search may feature Edinburgh in the filters with a count of 36 next to it when there are only 32 results.  This is because some of the entries in the results have more than one ‘Edinburgh’ quotation.  I wasn’t sure whether this would be confusing, or whether the counts should instead relate to entries not quotes.  However, the DSL team think this is ok as it is.

 

So I think that’s the new DSL search and filter facilities just about complete now and my gigantic ‘to do’ list now has a pleasing number of ‘[DONE]’ items in it.  Next week I’ll start on the site structure and interface updates, of which there are many.

I spent most of Thursday this week preparing for this year’s PDR session, which I’ll be having next Tuesday, and I spent most of Friday developing the CMS for Craig Lamont’s Scottish Life Writing project.  I’m about half-way through implementing this, and I hope to finish it next Monday.

Also this week, I gave some further feedback to Sara Pons-Sanz about her AHRC application, responded to some queries for the HiMuJe Malabar project, discussed migrating the Iona place-names site to our third-party host with Faidon, had a bit of an email catchup with the Place-names of Armagh people and discussed Tom Bartlett’s site migration with Luca, who has very kindly arranged to take over this task as he thinks he’ll be able to get it sorted before the November deadline.