I continued to develop the front-end for the Books and Borrowing project for most of this week, completing work on an initial version of the advanced search facility.  Last week I decided to change the way the API is referenced for the search.  Previously there was going to be one endpoint for the quick search, which would accept one search parameter, and another for the advanced search, which would accept multiple parameters.  I decided instead to amalgamate the two into one single search endpoint as in reality both search facilities will need to do the same things:  format the search options for Solr, work out the pagination, deal with ordering options and work out which filters need to be applied.

In order to amalgamate the endpoints I needed to rework the quick search facility that I had already created, and this meant breaking the quick search for a while.  Thankfully I managed to put it all back together again with the quick search working once more, but with slightly different URLs and a differently structured API call.  With this in place I began to add the advanced search data types to the API so as to construct the query that will be passed to Solr to return the advanced search results.  This basically allows specific fields in the Solr data (e.g. author names, library names, dates) to be queried rather than querying all fields, which the quick search does.

As I worked on this I ran into a spot of bother with author years of birth and death that were negative (i.e. BC).  They just weren’t working as they should have done and a bit of investigation revealed that this was because I was storing the years as strings rather than integers.  I regenerated the data on my laptop, saving the years as integers, and after that negative dates worked.  However, I soon realised why I hadn’t been saving the years as integers:  some author dates are not integers but are things like ‘1650?’ or ‘16__’.  When I tried ingesting these into Solr the records gave errors and failed to get added.  I therefore had to add a further check to avoid any non-integer dates getting added to Solr.  This means the associated records now get added but don’t have the offending dates.  This isn’t a huge issue as the dates would never have been searchable anyway.  For now this update is not present on the live site as I will wait until the next data export to add this, so in the meantime negative author dates will not work but the issue has been sorted.

I also ran into another issue with how I was structuring the URLs for the advanced search.  Short URLs as I’ve previously used work fine, but the advanced search is going to potentially result in some very long URLs, with advanced search fields and values stored in a specific section of the URL between slashes, for example:

/search/advanced||register|270||register|287||register|273||transcription|transcription||bdate|1789_1791||btitles|Miss||bfnames|forename||bsnames|surname||bothernames|othername||boccs|Church%20of%20Scotland||boccs|Presbyterian%20Seceder||bettlements|Selkirk/

However, such URLs were resulting in a 403 forbidden error on the server.  I contacted Stirling IT Services to enquire about this and discovered that the issue wasn’t the length of the URL but the length of the text between slashes in the URL.  The file system only allows filenames to be 255 characters in length and even though the above URL isn’t actually referencing filenames but is split up into variables by my script, the server first has to treat the URL as if it contained filenames (well, folder names) and it’s the server at a very fundamental level that is preventing things from working.

Unfortunately this meant I had to go back to the drawing board regarding how the search URLs would work.  Previously (as shown above) the search variables appeared first, with variable names and values separated by a bar and each pairing separated by a double bar.  After that things like filter queries, pagination and ordering options are included in the URL.  I needed to split the variable pairings up with a slash instead to avoid the lengthy text between slashes, but this would mean I could no longer be certain where in the URL things like pagination would appear.  Instead I needed to switch around the order of things in the URL, ensuring pagination, filter queries and sorting options appear first and then all of the search criteria follow, as many as are required.  This took quite some time to implement and does unfortunately mean that none of the existing links I’ve sent the team will work any more, but we are now in a better place and the search’s lengthy URLs will now work.

This week I fully updated the API to enable any and all combinations of the advanced search fields to be queried.  I have updated the JavaScript that picks out the search options that have been selected in the search forms and processes the search URLs and I have connected the front-end to the API to enable the advanced search to function and return data.  For example a ‘simple’ search can be limited to Edinburgh and Glasgow University libraries for borrowing records containing authors with forename ‘William’ and surname ‘Shakespeare’.  Above the search results there is a grey box that lists what search options have been selected and gives users the option of refining their search or starting a new one.  I also worked on the formatting of the search options that get displayed (e.g. libraries show their full name rather than the ‘slug’ used in the search and registers show their name rather than their ID).  The ‘refine’ option works, and displays either the ‘simple’ or ‘advanced’ search form as applicable with all search terms ‘remembered’ in the form.  Users also have the option to clear the search forms and start again if required too.

So for example an advanced search is for borrowing records in two Edinburgh registers (Da.2.10 and Da.2.11) where the borrower surname is ‘smith’, the book edition language is ‘english’ and the format is ‘8v0’ gives results and if you press ‘refine search’ the ‘advanced’ tab is displayed with these options already filled in, allowing you to update them as required.

I also investigated and fixed an issue with selecting / deselecting the third level religion occupations and I began to make some of the information in the search results searchable.  I’ve currently added in ‘click to search’ options for the borrowed date and the borrower name.  These now appear with a dotted line under them and if you press on one you will immediately see all of the associated borrowings  The click through for borrower name still needs a bit of work as it is actually a new search option not present in the search form – a search for borrower ID.  At the moment it’s only the ID that appears in the ‘you searched for’ section but I will fix this.

There are still some things that are not yet working properly.  Search filters that feature slashes or bars or ampersands currently cause things to break.  Also Solr is sometimes being too clever for its own good in bringing back records that are of relevance but don’t actually match the search criteria.  For example a search for the transcription field containing ‘betsy thoughtless’ currently brings back records that don’t include this text in the transcription field but in other fields, meaning Solr returns the records because it thinks they might be of interest.  This can be avoided by using quotes but I need to investigate whether there is a better way to deal with this.

On Friday I dealt with the normalisation of data that Matt and the team had been working on.  This rationalises the data in fields such as borrower title and book edition language so that the same form is always used for the same thing.  For example there is just one form for ‘captain’ rather than there being ‘capt’, ‘capt.’, ‘captain’ etc.  I wrote scripts to process all of these updates and further scripts to pick out forms that need additional checking.  I also regenerated the distinct forms for things like borrower title so Matt could check that no further unwanted forms had been added since I last exported the forms.

Also this week I made some further updates to the Edinburgh’s Enlightenment map and investigated an issue Ann was having with entries that featured slashes in the Dictionaries of the Scots Language.