I spent much of the week working on the Books and Borrowing project, working with the new Solr instance that the Stirling IT people set up for me last week.  I spent some time creating a new version of the API that connects to Solr and then setting up the Solr queries necessary to search all fields of the Solr index for a regular quick search and the ‘borrowed’ fields for date searches.  This included returning the data necessary to provide the facetted search options on the search results page (i.e. filters for the search results).  I also set up a new development version of the front-end, leaving my existing pre-Solr version in place in case things go wrong and I need to revert to it.

As with the previous version of the site, you can perform a quick search, which searches the numerous fields that are specified in the requirements document.  Dates can either be a single date or a range.  Text searches can use wildcards * to match any characters (e.g. tes* will match all words beginning ‘tes’) and ? to match a single character (e.g. h?ll matching ‘hill’ and ‘hell’).

Currently the results still display the full records with 100 records per page.  I did consider changing this to a more compact view with a link to open the full record, but I haven’t implemented this as of yet.  I might add an option to switch between a ‘compact’ and ‘full’ view of the records instead, as I think having to click through to the full record each time you’re interested would get a bit annoying.

There have been a lot of changes to the back end, even if the front-end doesn’t look that different.  Behind the scenes the API now connects to the Solr instance and queries are formatted for and passed to Solr, which then returns the data.  Solr is very fast, but the loading of 100 results which are full records does still take some time.  Queries to Solr currently return the 100 relevant borrowing IDs and then another API call retrieves all of the data for these 100 IDs from the regular database.  A compact view could potentially rely solely on data stored in Solr, which would be a lot quicker to load, if we want to pursue that option.

In addition to returning the IDs for the 100 borrowing records that are to be displayed on any one results page, Solr also returns the total number of matching borrowing records plus the facetted search information.  For the moment the following information is included in the facetted data: Borrowing year, library name, borrower gender and occupation, author name and book language, place of publication and format.  These appear as ‘Filter’ options down the left-hand side of the results page, currently as a series of checkboxes, the name of the item in question and the number of results in the overall results that the item is found in.  Pressing on a checkbox filters the results in question and the results page reloads as soon as you press a checkbox.  This causes both the results and the filters to narrow, displaying only those that continue to match.  You can then click on other checkboxes to narrow things further, or deselect a checkbox to return to the non-filtered view.

I think the filters are going to be hugely useful, but they’re not perfect yet.  There are issues with the data for occupations and authors.  This is because the data has been stemmed by Solr for search purposes, meaning the field is broken down into individual word stems (e.g. ‘educ’ for ‘education’).  I will fix this but it will require me to regenerate the data and get the Stirling IT people to replace the existing data.  I’ve also noticed that data from test libraries is in Solr too and I’ll need to ensure this gets removed.

With all of this in place I then moved on to providing different sorting options for the search results, for example ordering the results by borrowed date, library or author name.  This required some tweaking of the Solr queries and the API and then some updates to the front-end to ensure the selected sorting option is dealt with and remembered.  However, I did come across a limitation in Solr, in that it is not possible for Solr to order data by fields that contain multiple values.  This means that for now sorting by things like author name and borrower occupation won’t work as each of these can contain multiple values per record.  I’ll therefore have to make concatenated versions of these fields for sort purposes and will do this when I regenerate the data.

This initial version of the facetted search results page displayed years in the same way as other search filters:  as a series of checkboxes, year labels and counts of results in each year.  What I really wanted to do was to display this as a bar chart instead, using the HighCharts library that I use for other visualisations in the front-end.  I wanted to group years into decades where the range of years is greater than a decade and enable the user to then press on a decade bar to view the results for individual years within the decade, with the bar chart then displaying the individual years.  I managed to get the ‘by decade’ bar chart working this week.  You can hover over a bar to view the exact total for the decade.  You can also click on a decade to filter the search to that decade.  This is the bit I’m still working on.  Currently no bar chart is displayed and you need to use your browser’s ‘back’ button to return, but the filter does actually work.  Eventually a bar chart with borrowings for each year in the decade will be displayed, together with a button for returning back.  In this view you will be able to further click on a year bar to filter the results to the selected year.  I’ll continue with this next week and the ‘Year borrowed’ checkboxes will be removed once the bar chart is fully working.  It’s took quite a while to get the bar chart working as there was a lot of logic that needed to be worked out in order for borrowings to be grouped into decades and to accommodate gaps in the data (e.g. if there is no data for a decade we still need this decade to get displayed otherwise the graph looks odd).  Below is a screenshot of the new front-end with facetted searching and ‘year borrowed’ bar chart:

Also for the Books and Borrowing project this week I had a Zoom call with Katie and Matt to discuss genre classification (which has now been decided upon) and batch editing the book edition records to fix duplicates and to auto-generate book work records for any editions that need them.  I also sent on some data exports containing the distinct book formats and places of publication that are in the system as they will need some editorial work as well.  I also responded to a few queries from one of the project RAs who wanted some queries run on the data for a library he has worked on.

Also this week I created an initial WordPress site for the VARICS project after the domain was set up by Russell McInnes, an IT guy from the College of Engineering who is helping out with Arts IT Support due to their staffing issues.  Russell has been hugely helpful and it’s such a relief to have someone to work with again.  Also this week I spoke to Marc Alexander about some financial issues relating to a number of projects I’m involved with and spoke with him about equipment that I might need in the coming year and conferences I’d like to attend.  I also made a tweak to the ‘email entry’ feature I’d changed o the DSL website last week.  Next week I’ll be continuing to work on the B&B front-end.