It was Easter Monday this week, and I’d taken the rest of the week except Friday off as holiday.  I’ll be off for some of next week as well.  After writing my blog post and catching up with emails on Friday I spent the rest of the day working for the Anglo-Norman Dictionary, as the editor was ready to launch a major update to the content – replacing every entry for the letter ‘T’, plus a number of updates to entries in other letters.  I have a documented process that I created for such updates, so thankfully I could follow these instructions (which of course include making backups before changing anything), but it’s still a little scary when thousands of entries are getting deleted and replacement data is generated.

After backing everything up and making all necessary preparations I set the update scripts in motion and thankfully all went smoothly.   The update replaced or created 4472 dictionary entries, of which 1805 were ‘main’ entries, with the rest being cross references.  Of the ‘main’ entries these included 3310 main senses, 1286 subsenses, 2958 locutions, 2060 locution senses and 328 locution subsenses.  Overall the entries included some 15380 citations.

However, during the afternoon one of the editors encountered a problem when updating one of the new entries via the Dictionary’s online management system.  The script that publishes updates quit midway through with an SQL error, the entry could no longer be found on the live site by searching for the headword or variants and duplicates of the entry appearing in the ‘Browse’ pane.

This was clearly rather concerning so all work on the dictionary stopped whilst I investigated.  Thankfully I managed to figure out what was going on.  When subsequently editing the entry some UTF-8 characters had ended up in the attestation dates, but the corresponding columns in the database weren’t set up to store UTF-8.  When the system attempted to insert the characters the database complained and the publication script stopped.  What this meant was the remaining parts of the script didn’t get a chance to execute, and it was in these parts that older versions of the entry were archived, search terms were generated and everything was tidied up.  As these parts didn’t run this meant duplicate entries crept in and search terms weren’t working properly.  I therefore updated the database so all columns could handle UTF-8 data and removed the duplicate entries and thankfully all was working fine again afterwards.