I continued to make updates to the Books and Borrowing website this week after the soft launch last week. I had noticed last week that the figures appearing on the ‘Facts’ page didn’t correlate with the number of results returned through the search facilities. I reckoned this was because the ‘Facts’ page, which queries the database directly was not necessarily including the full chain of interrelated tables and their individual ‘isactive’ flags when returning figures whereas the search facilities use the Solr index, and the data stored within this were generated using the full chain. So for example, counts of borrowing records for books may not incorporate the register page record, but if the register page is set to ‘inactive’ then it is important to factor in the register page.
Another issue was the ‘in stats’ flag. We have this flag for borrowing records to decide whether the record should appear in the stats or not, so for example a duplicate record could be omitted from the stats, but would still be findable when viewing the register page. The search results were finding records with ‘instats’ set to ‘no’ but these were not included in the ‘Facts’ page, meaning the numbers would never match up.
To try and make the figures more consistent I decided to update the ‘Facts’ page to use the Solr index for calculating borrowing records. This slightly changed the figures that appear in the summary section on this page, but as a borrowing record can involve multiple borrowers the total borrowings broken down by gender may not equal the total borrowings. If a borrowing has two borrowers, one male and one female then it will count as one borrowing in the total borrowings and one each in the totals per gender.
Further investigation into the discrepancies between figures and on the ‘Facts’ page and the number of records returned in a related search did seem to suggest that these are caused by the ‘instats’ flag. For example, for Chambers library, Fiction is the most popular genre with 4417 borrowings listed. This is the figure for borrowings with ‘instats’ set to ‘Y’. But following the link through to the search results listing borrowing records involving the genre ‘Fiction’ at Chambers displays 4420 borrowings. This is because the search does not take into consideration the ‘instats’ field. I tried running the queries (and others) through Solr and this does appear to be the reason for the discrepancies. After discussion with the team we decided therefore to update the search page to avoid returning any ‘instats’ results. These records will still be findable when viewing register pages, but will not now appear in any search results. With this update in place the figures on the ‘Facts’ page began to match up with the search results.
However, after further testing I noticed that the stats for authors and genres in the site-wide facts page were still not aligned with the number of search results returned and I spent most of Wednesday investigating this. It was a very long and complex process but I finally managed to sort it. I updated the ‘top ten’ cache files for each library to use data from the Solr index to calculate the number of borrowings, but this alone wasn’t giving the correct figures. The reason was that each library had its own ‘top ten’ for authors and genres, both overall and for each gender. Then when the facts page was being generated for multiple libraries these lists were then brought together for each library to formulate overall top tens. If an author or genre was not in the top ten for a library the data for this item was not included in the calculations as only the top ten were being stored. For example, if ‘Fiction’ was the eleventh most borrowed genre at a library then its borrowings were not found and were therefore not getting added to the total. What I’ve had to do instead is store all genres and authors for each library in the cache rather than just the top tens, thus ensuring that all items are joined and their number of borrowings are compared. Unfortunately this does mean the cache files are now a lot bigger and more processing needs to be done. But at least the figures do now match up. Also this week I updated the Chambers Library Map to add a link to view the relevant borrowing records to the borrower popups (see https://borrowing.stir.ac.uk/chambers-library-map).
I then moved onto making further updates to the Place-names of Iona map. Sofia had spotted two minor issues with the map and I fixed them. Cross references were only set to be included in the data if both records had ‘landranger’ set to ‘Y’. I’m not sure why this was – it must have been something decided in an earlier project. I removed this requirement and the cross references now display in the full map record. However, pressing on the links wasn’t actually doing anything so I also fixed this. Pressing on a cross reference now works in the same way as the links in the tabular view of the data – the popup is closed, the map focusses on the location of the cross reference and the marker pop-up opens. The other issue was that the glossary pop-up could end up longer than the page and wasn’t using the updated in-popup scrollbar that I’d developed for the tabular view, so I updated this.
I also continued working through my ‘to do’ list for the project, which is now almost entirely ticked off. I updated the searches and browses so that they now shift the map to accommodate all of the returned markers. I wasn’t sure about doing this as I worried it might be annoying, but I think it works pretty well. For example, if you browse for place-names beginning with ‘U’ the map now zooms into the part of the island where all of these names are found rather than leaving you looking at a section of the island that may not feature any of the markers.
I also updated the citations to use shorter links. Whenever you press on the ‘cite’ button or open a record the system grabs the hash part of the URL (the part after ‘#’) and generates a random string of characters as a citation ID. This random string is then checked against a new citation table I’ve created to ensure it’s not already in use (a new string is generated if so) and then the hash plus the citation ID are stored in the database. This citation ID is then displayed in the citation links. Loading a URL with a citation ID then queries the database to find the relevant hash and then redirects the browser to this URL. I’ll need to keep an eye on how this works in practice as we may end up with huge numbers of citation URLs in the database.
I also updated the maximum zoom of the map layers to be whatever each layer supports. The satellite with labels view supports the highest zoom of the layers (20) and if you’re on this then switch to a different layer (e.g. Modern OS with a max zoom of 16.25) then the map will automatically zoom out to this level.
The only major thing left to tackle is the bilingual view. My first step in implementing this will be to go through the site and document each and every label and bit of text. I’ll create a spreadsheet containing these and then someone will have to fill in the corresponding Gaelic text for each. I’ll then have to update the site so that the code references a language file for every bit of text rather than having English hard-coded. I’ll begin investigating this next week.
Also this week I spent a bit of time working for the DSL. They wanted the Twitter logo that is displayed on the site to be replaced with the ‘Artist formerly known as Twitter’ logo. Unfortunately this was a bigger job than expected as the logos are part of a font package called ‘Font Awesome’ that provides icons across the site (e.g. the magnifying glass for the search button). The version of the font used on the website was a couple of years old so obviously didn’t include the new ‘X’ logo. I therefore had to replace the package with a newer version, but the structure of the package and the way in which icons are called and included had changed, which meant I needed to make several updates across the site. I also replaced the Twitter logo on the entry page in the ‘Share’ section of the right-hand panel.
I spent most of Thursday working on the migration of the Anthology of 16th and Early 17th Century Scots Poetry. I’ve now migrated more than 25 poems to TEI XML and I’ve completed the interface for displaying these. I still need to make some tweaks to the XSLT (e.g. to decide how line indents should be handled) but I have now managed to sort other issues (e.g. poems with line numbers that start at a number other than 1). I should now be able to continue manually migrating the poems to XML when I have a few spare minutes between other commitments and hopefully in a couple of months at most I’ll be able to launch the new site.
On Friday I began looking at developing the Speak For Yersel tool (https://speakforyersel.ac.uk/) for new geographical areas. We now have data for Wales, the Republic of Ireland and Northern Ireland and I began going through this. I had hoped I’d just be able to use the spreadsheets I’d been given access to but after looking at them I realised I would not be able to use them as they currently are. I’m hoping to reshape the code into a tool that it will be possible to easily apply to other areas. What I want to do is create the tool for one of our three areas and once it’s done it should then be possible to simply ‘plug in’ the data from the other two areas and everything will just work. Unfortunately the spreadsheets for the three areas are all differently structured and for this approach to work the data needs to be structured absolutely identically in each region. I.e. every column in every tab of every region must be labelled exactly the same and all columns must be in exactly the same location in each spreadsheet.
For example, the ’Lexis’ tab is vastly different in each of the three areas. ROI has an ID field, part of speech, a ‘direction of change’ column, 6 ‘variant’ columns, a ‘notes’ column and an ‘asking in NI’ column. It has a total of 14 columns. NI doesn’t have ID, PoS or direction of change but has 7 ‘Choice’ columns, a ‘potential photos’ column, no ‘notes’ column and then two columns for cross references to Scotland and ROI. It has a total of 12 columns. Wales has ‘Feature type’ and ‘collection method’ columns that aren’t found in the other two areas, it has no ‘Variable’ column, but does have something called ‘Feature’ (although this is much longer than ‘Variable’ in the other two areas). It has an ‘if oral – who’ column that is entirely empty, six ‘Option’ columns plus a further unlabelled column that only contains the word ‘mush’ in it and four unlabelled columns at the end before a final ‘Picture’ column. It has a total of 17 columns, plus the data for the feature ‘butt, boyo, mun, mate, dude, boi, lad, la, bruh, bro, fella’ in ‘Option F’ states ‘Lad ++ la/bruh/bro/fella (these should be separate options)’. As you can probably appreciate that there’s no way that any of this can be automatically extracted.
I therefore worked on reformatting the spreadsheets to be consistent, starting with the ‘lexis’ spreadsheets for the three areas, which took most of the day to complete. Hopefully the other tabs won’t take as long. The spreadsheet has the following column structure:
Next week I’ll work on the other two survey question types and once the data is in order I’ll be able to start developing the tool itself. I’ve taken Monday next week off, though, so I won’t be starting on this until Tuesday, all being well.