On Monday this week I addressed a couple of issues with the Books and Borrowing search that had been identified last Friday.  Multi-word searches were not working as intended and were returning far too many results.  The reason being (as mentioned last week) a search for ‘Lord Byron’ (without quotes) was searching the specified field for ‘Lord’ and then all fields for ‘Byron’.  It was rather tricky to think through this issue as multi-word searches surrounded by quotes need to be treated differently, as do multi-word searches that contain a Boolean.  We don’t actually mention Booleans in the search help, but AND, OR and NOT (which must be upper-case) can be used in the search fields.

I wrote a new function that hopefully sorts out the search strings as required, but note that search strings containing multiple sets of quotes are not supported as this would be much more complicated to sort out and it seemed like a bit of an edge case.  This new function has been applied to all free-text search fields other than the quick search, which is set to search all fields anyway.  After running several tests I made the update live, and now searching author surnames for ‘Lord Byron’ finds no results, which is as it should be.

Here are some examples that do return content.

 

1. If you search book titles for ‘rome’ you currently find 2046 records:

https://borrowing.stir.ac.uk/search/0/0/0/simple/bookname|rome

2. If you search book titles for ‘rome popes’ you currently find 100 records (as this is the equivalent of searching book titles for ‘rome’ AND ‘popes’:

https://borrowing.stir.ac.uk/search/0/0/0/simple/bookname|rome%20popes

3. Using Boolean ‘AND’ gives the same results:

https://borrowing.stir.ac.uk/search/0/0/0/simple/bookname|rome%20AND%20popes

4. A search for ‘rome OR popes’ currently returns 2046 records, presumably because all book titles containing ‘popes’ also contain ‘rome’ (at least I hope that’s the case):

https://borrowing.stir.ac.uk/search/0/0/0/simple/bookname|rome%20OR%20popes

5. A search for ‘rome NOT popes’ currently brings back 1946 records:

https://borrowing.stir.ac.uk/search/0/0/0/simple/bookname|rome%20NOT%20popes

6. And searches for a full string also work as intended, for example a search for “see of rome”:

https://borrowing.stir.ac.uk/search/0/0/0/simple/bookname|%22see%20of%20rome%22

With this update in place I then slightly changed the structure of the Solr index to add a new ‘copy’ field that stores publication place as a string, rather than text.  This is then used in the facts, ensuring the full text of the place is displayed rather than being split into tokens.  I then regenerated the cache and asked the helpful IT people in Stirling to update this on the project’s server.  Once the update had been made everything then worked as it should.

Also this week I exported all of the dictionary entries beginning with ‘Y’ for the Anglo-Norman Dictionary as these are now being overhauled.  I also fixed an issue with the Saints Places website – a site I didn’t develop but I’m responsible for now.  A broken query was causing various errors to appear.  The strange thing is the broken query must have been present for years, but presumably the query was previously failing silently while the server on which the site now reside must be more strict.

I spent the rest of the week developing the tool for publishing linguistic surveys for the Speak For Yersel project.  I’d spent a lot of time previously working with the data for the three new linguistic areas, ensuring it was consistently stored so that a tool could be generated to publish the data for all three areas (and more in future).  This week I began developing the tool.  I spent the week developing a ‘setup’ script that would run in a web browser and allow someone to create a new survey website – specifying one or more survey types (e.g. Phonology, Morphology) and uploading sets of questions and answer options for each survey type.  The setup script then provides facilities to integrate the places that the survey area will use to ascertain where a respondent is from and where a marker corresponding to their answer should be located.  This data includes both GoeJSON data and CSV data, both of which need to be analysed by the script and brought together in a relational database.  It took most of the week to create all of the logic for processing all of the above, as you can see a screenshot of one of the stages below:

As I developed the script I realised further tweaks needed to be made to the data, including the addition of a ‘map title’ field that will appear on the map page.  Other fields such as the full question were too verbose to be used here.  I therefore had to update the spreadsheets for all three survey types in all three areas to add this field in.  Similarly, one survey allowed more than one answer to be selected for a few questions, with differing numbers of answers being permitted.  I therefore had to update the spreadsheets to add in a new ‘max answers’ column that the system will then use to ascertain how the answer options should be processed.

I also needed to generate new regions for the Republic of Ireland survey.  Last week I’d created regions using QGIS based on the ‘county’ stored in the GEOJSON data.  This resulted in 26 regions, which is rather more than we used for Scotland.  The team reckoned this was too many and they decided to use a different grouping called NUTS regions (see https://en.wikipedia.org/wiki/NUTS_statistical_regions_of_Ireland).  A member of the team updated the spreadsheet to include these regions for all locations and I was then able to generate GeoJSON data for these new regions using QGIS following the same method as I documented last week.

When processing the data for the Republic of Ireland (which I’m using as my first test area) my setup script spotted some inconsistencies with the data as found in the GeoJSON files and the spreadsheets.  I’ve passed these on to the team who are going to investigate next week.  I’ll also continue to develop the tool next week, moving onto the development of the front-end now that the script for importing the data is now more-or-less complete.