The project I spent the most time working on this week was the DOST Auld Laws project, which I hadn’t worked on for a couple of weeks.  The last time I worked on the project I’d managed to get an initial version of the quick search working, but there were some issues with it.  A phrase search wasn’t working, wildcards at the start of a search term were not working, the full term was not getting highlighted in the results where an <expan> is present in the term, I hadn’t implemented the pagination of search results and I still needed to update the page view to enable results traversal directly from the page when accessed via the search results.

I was struggling somewhat with phrase searching and full term highlighting and ended up going round in circles with ChatGPT for several hours and getting nowhere.  I eventually asked my colleague Luca, who has considerably more experience and knowledge of eXist-db and XML document querying, for some advice.  Thankfully he was able to come up with a query script that did exactly what I needed it to do, which was a massive help as I was really struggling.  This is definitely an example of where a conversing with actual person is much more effective than AI and  I really must give Luca credit for the help he gave.

By the end of the week my updates meant that it was possible to use wildcards at the start of a search term, as you can see from the following screenshot that shows the results for ‘*other’:

With Luca’s help, phrase searches now work, and the following screenshot shows the results of a search for “the landis of”:

Again with Luca’s help, the search term in each result is now fully highlighted even when an <expan> is present in the word, so for example a search for ‘pebillis’ previously found two results but only highlighted ‘pebill’ in the results as the term was recorded as ‘pebill<expan>is</expan>’.  But now the full term is highlighted.

I also implemented results pagination, with is currently set to display a maximum of 20 results per page.  The following screenshot shows a search for ‘r?cht’ with the pagination in place:

And now when you reach a document page that is in the search results (e.g. by selecting it from the search results) a search results navigation bar appears above the document navigation bar, allowing you to navigate to the next or previous search result or return to the full search results, as the following screenshot demonstrates:

I’ll probably add in a ‘clear search results’ button here too, and I still need to fix an issue with the results navigation when there are multiple results on one line.  At the moment there is no way for the code to differentiate these so the ‘next’ and ‘previous’ links get stuck.  I also need to look into speed issues with the search too.  But some good progress has been made this week and I feel much more confident using eXist-db now.

Also this week I spent a bit more time on the Burns Supper Map, adding in videos and images for the recently imported supper records.  I also had an email conversation with the Dictionaries of the Scots Language people about parts of speech and new data imports, which will probably be taking place in the next few weeks.  I also attended a ‘coffee and catch-up’ with the other developers in the College, which was really valuable as always.

The rest of my week was divided between the Playbills project and the HiMuJe Malabar project.  For Playbills I processed a spreadsheet containing around 400 performers that Deven had manually processed last week.  I ended up doing a bit more manual tweaking after investigating the appearance of some of the performers in the playbill images, and when I ran the spreadsheet through my import script we then had 52,178 performers in the system, and of these some 52,048 of these have a gender assigned, which I pretty amazing.

I then moved onto processing the plays based on the spreadsheet Deven had worked on before Easter that notes which performances actually feature the same play, even if it is not referred to in exactly the same way.  Using this I created canonical records for plays, I set some marked plays as ‘special attractions’  and I deleted certain plays that had been marked for deletion.  Before I did this I updated the database so that all tables include an ‘isactive’ field, and updated the API so that it only includes data where ‘isactive’ is set to ‘Y’.  Then when it came to deleting plays I didn’t actually delete them but set them (and all associated data such as performers and roles) to ‘isactive = N’.  This means if we realise something has been ‘deleted’ that needs to be reinstated I’ll just need to update the relevant fields back to ‘isactive = Y’ rather than having to find and re-insert all associated data.

For the most part the script was successful and result in 261 plays being deleted and 1098 converted to special attractions. It then created 2039 canonical records and 4134 other plays were then assigned to these.  There were a few issues with some plays referencing canonical records for other plays that were set to be deleted (so canonical records weren’t created for them), and I passed these on to Deven so she could look into them.

I then created an API endpoint and front-end pages for listing the canonical plays and the details for a selected canonical play.  Below is a screenshot of part of the list of canonical plays, ordered by number of plays:

It’s just an initial version that lists the titles, the record type (either play or ‘special attraction’) and a count of the number of associated plays.  You can press on column headings to order the table by the column and if you press on a canonical play name you can access a list of plays that are associated with it, as you can see below:

This lists each associated play’s name, date, playbill, venue, location and genres, and you can click on each linked item to reach the relevant page (e.g. the details for a play or the associated playbill page).  This is just a work in progress and we’ll probably want to update it, for example to include a genre filter on the canonical plays list, or including access to lists of associated roles and performers.  I also updated the playbill page to add links through from play titles to the relevant play page, and I’ve added a ‘Type’ field for each play that either displays ‘Play’ or ‘Special Attraction’.

For the Malabar project I sorted out the legend for source texts, alphabetising the list and ensuring the pane has a maximum width.  I also added in the two other categorisation options that I’d included in my specification document:  Number of references and placename languages.  Number of references categorises the markers and polygons based on the number of times each place is referenced in the source texts.  I set up the categories to match the available data (0, 1-5, 6-10, 11-20, 21+) but these can very easily be altered as the data and number of references grow.  I think it will prove quite useful to be able to quickly identify the places that are referenced the most in the texts.  Below is a screenshot showing the currently available data categorised by number of references:

The other new categorisation option was ‘Placename languages’  and this categorises the places based on the languages of the placename variants included in each record.  This will likely need some further work, such as adding in full language names rather than the codes, and possibly filtering out ‘en’ as I’m guessing these would not have been found in the original sources, but it’s still interesting to use the categorisation – for example finding all placenames that have a Hebrew form.  Here is a screenshot showing placename languages:

The data itself is still being compiled and I still need to work on the marker colours (and icons) and also to ensure the polygons always appear behind the markers and don’t make the markers unclickable, as is sometimes the case at the moment.  Also this week I added the selection of base map, menu section and categorisation type to the URL, meaning it’s now possible to bookmark / share / cite specific views of the map.  You can also link directly to a specific record as well.