I spent most of this week continuing to implement the new cognate reference search facility for the Anglo-Norman Dictionary.  When I left off last week I was midway through implementing a script to generate search form variants for all of the cognate reference forms found in the dictionary entries.  These are the words found in the grey box towards the top of an entry page such as https://anglo-norman.net/entry/asseger.  My script splits up multiple forms and bracketed sub-headwords, where brackets are found within a form the script generates variants with and without the bracketed text, multi-word forms have each word stored separately (in addition to the full string) to enable each word to be searched for, and where a dash is used variants are generated with the dash, with a space in place of the dash and with no space at all.

After fully testing the script and its output I then needed to work on the scripts that update dictionary entries to ensure that they also generate or regenerate the cogref search forms whenever they execute.  This included the batch upload process I use when handling large-scale updates to the database and the entry publication script found in the dictionary’s content management system, as used by the editorial team when updating individual entries.  I worked with the editor Geert to test out these updates and with a bit of tweaking I got everything working.

I then updated the ‘View entry’ page in the dictionary’s content management system so that all of the cogref search terms are displayed, so the team can check what can be searched for.  As mentioned previously, cogrefs are split into individual search terms on the space character, allowing individual words to be searched and this does mean that parts of speech and numerals also get stored as individual search terms, but I don’t think this will be a problem – no-one will see these search terms, only the listed ‘Display form’ will be shown in the results and the ‘search forms’ just show the variants that are stored behind the scenes.  I did consider stripping out all numerals and parts of speech but there are just so many possible permutations that it would be very tricky to catch them all, and storing them is not a big issue.  So for example for the entry ‘asseger’ linked to above, the following forms are generated:

Where there are multiple forms (e.g. Gdf) these are stored as separate search forms.  Where there are subheadwords (e.g. DEAF) these are similarly split into separate search forms.  Where there are forms with spaces (e.g. OED) the full string is saved as a search form, but so too are each individual word.  Where there are brackets within a word (e.g. MED) then forms are generated both with and without the bracketed text.  This also works where there’s only an open bracket, as happens quite often with MED headwords.  There were also a handful of MED cogref forms that had multiple brackets (around 13, I think).  In such cases the generated forms only include versions with all of the bracketed text and none of the bracketed text, as the generated forms for the entry ‘Oxherd’ demonstrate:

He above also demonstrates the variants that are created when a dash is used, and any forms with slashes are treated in the same way.

With all of the cogref search data in place and facilities established to ensure the data will be updated when entries are edited I then moved onto the creation of the new search facility itself.  The first step was to create a new endpoint in the API through which the search requests will be processed.  It took a while to implement and test this, ensuring that wildcards can be used in the search and enabling the search to be limited to one or more cognate dictionaries.  The search results initially brought back rows for each cogref form, so if (for example) the search was for ‘siege’ as found in the above screenshot there would be three individual DEAF search results, each with information about the associated entry.  However, what the search needed to return was entries and then within each entry a further section listing the individual search results and I therefore needed to switch things around slightly.

With the API updated and the new search endpoint working I then needed to update the front-end to add the new search form to the advanced search page, connect the form to the API and ensure that any returned results were processed in the same way as the other advanced search options (for example with pagination, links to entries and a way to return to the form with previously entered options remembered).  I created all of this in a test version of the page so the editors and I could test things out without making the update publicly available.

The new Cogref search appears as a new tab in the Advanced Search after ‘Languages’.  It consists of some introductory text, a search box where a cognate form can be entered (with or without wildcards), and a section where the cognate dictionaries you want to search can be selected.  These appear as checkboxes which are all checked by default.  There’s an ‘All’ checkbox that allows you to select / deselect all of the other checkboxes, so to limit your search to just DEAF, for example, press ‘All’ to deselect everything then press ‘DEAF’.  The cognate dictionaries have tooltips that display the full names, as the following screenshot demonstrates:

As with the headword search, by default any text you enter into the ‘Cognate Form’ box matches any corresponding string in the search forms – so for example ‘noble’ is actually treated as ‘*noble*’.  To perform an exact search use double quotes.  As previously mentioned, search forms have been created for many different variants and where a form has multiple words or is split with dashes each individual word is treated as a separate search form.  So, for example, if you perform an exact search for “herd” you will find things like ‘nout-herd’, ‘shep-herd’ etc.

The search results display details of what you’ve searched for, the entries that feature a cogref form that matches and the specific cogrefs that feature a form that matches your search criteria, as the following screenshot demonstrates:

The editors are now testing out the new search and hopefully we’ll be able to go live with it in the next week or so.

Also for the AND this week, the editor Karen spotted an issue with the cross references between entries, which are generated whenever an entry is updated.  Some cross references weren’t being found and it turns out that this was because the homonym number is separated from the lemma with a hash in the XML, but with an underscore in the ‘slug’ used in the website (this is correct and how things should be). What this meant was that the cross reference generator was taking the slug (e.g. Partein_1) and looking for that in the XML files and obviously not finding any matches as it’s recorded as ‘Partein#1’ in the XML.  I therefore updated the code so that the xref search turns ‘_’ into ‘#’ and the xrefs are now being found and generated.

This week I also did a bit of work for the VARICS project, adding in a new batch of short descriptions for measurements in the lookup feature I’m developing and discussing the ancillary text with members of the project team.  I also had a chat with Jennifer Smith about our Speak For Yersel paper for DH2025 and gave some feedback to Matthew Creasy about his Mallarmé Chancellor’s Fund application, which he submitted on Friday.

I also responded to a Query from William Ashford of the DSL about Google Analytics, mobile versus desktop users and search engine optimisation and responded to a query from Ann Ferguson regarding a couple of issues relating to the bibliography search.