Week beginning 18th November 2013

With the big relaunches of the Historical Thesaurus and SCOTS out of the way last week, I returned to some of the other outstanding items on my long-term ‘to do’ list and decided specifically to focus on updating the Grammar app.  A few weeks ago Marc’s Course 20 students tested out the three STELLA apps that I have created and posted their feedback on the course’s Moodle page.  There was some very useful feedback, and a recurring theme was that navigation between the exercises and the ‘book’ pages could be better.  I was aware myself that this was something of a weak spot in the app – if you are midway through answering an exercise but need help and go to the book pages then your progress in the exercise (and all previous exercises) is lost when you navigate away from the exercise page.

As I mentioned in a previous post, I have been investigating HTML5’s ‘Local Storage’ facilities – basically the next generation of client-side storage that replaces cookies.  Local Storage allows a web page to store any number of ‘key/value’ pairs on the user’s computer (or phone) that can then be reloaded whenever the user returns to the page.  This very simple mechanism greatly enhances client-side applications by essentially allowing client-side scripts to have a client-side database.

My plan was to use HTML5 Local Storage to store a user’s exercise answers so that these could be reloaded and used to repopulate the exercise whenever the user returned to the exercise in question – either after navigating to the ‘book’ part of the app or after closing the app and opening it another time.  Firstly I added a very simple local storage check – storing the name of the exercise page the user is currently viewing and then checking for this when a ‘book’ page is loaded, adding a link to the exercise page if the variable is set and thus providing a better navigation pathway between exercises and book pages.

After that was in place I started tackling the first exercise – ‘Unit 2: Parts of Speech 1’.  I had hoped to be able to store a user’s answers as an associative array (an array where the ‘key’ is text) in local storage, but I ran into some problems with this approach.  Firstly, unbeknownst to me, Javascript doesn’t actually support associative arrays as such – e.g. answers[‘answer-id-1’] = ‘Aj’ doesn’t actually give you an array that you can use array functions on.  You have to use objects instead – e.g. answers{“answer-id-1”:”Aj”}.

Secondly, HTML5 Local Storage only allows you to store key/value pairs and the value can only be a string – i.e. plain text rather than a Javascript array or object.  However, once I discovered this it turned out to not be such a problem – my Javascript objects (by their very nature) were stored in the JSON format and the JSON functions of Javascript include a handy function called ‘stringify’ which turns any JSON object into a string, which can then be stored using local storage.  This can then be retrieved using the ‘JSON.parse’ function, converting the string to proper Javascript objects for reuse in a script again.

I managed to get the first exercise working with local storage as a proof of concept.  Whenever a user selected a part of speech this saved the selected value.  Whenever the page was reloaded the script then checked for any saved values and used these to populate the answer boxes.  It worked pretty nicely.  However, I began to realise just what a mess the code for the exercises was.  When previously developing the exercises I was really just getting to grips with using Javascript for such a relatively complicated task, and I approached each exercise individually, tailoring the code for the specific exercise and saving it as a separate file.  This meant that there was a lot of code repetition, which is never a good thing.  I was also storing a lot of HTML snippets for the answer boxes etc within Javascript variables, which was also not so good as these could instead be generated.  The data for each exercise was also spread across a variety of different variables and structures – sentences here, answers there, notes somewhere else etc.  I decided I should really rationalise things so I went back to the drawing board and started designing the exercise Javascript afresh.

Each exercise would be stored as a Javascript object and loaded from a JSON file.  Each object would contain all of the required data about the exercise – its words, the type of answer boxes required, their positions, the answers, notes, multiple stage information etc.  There would also be only one Javascript file that would process every exercise rather than individual files for each exercise and a single set of functions that would handle the common tasks of every exercise – e.g. load sentence, show answers, clear, load next sentence etc.

I spent the rest of the week developing the JSON structure and the Javascript code for handling the exercises, including the local storage discussed earlier.  By the end of the week I had the structures in place for handling all but the more complex exercises – specifically those that are not just multiple sentences but multiple stages within each sentence.  I hope to be able to finish off the redevelopment of all of the exercises early next week.

Week beginning 11th November 2013

This was an important week, as the new version of the Historical Thesaurus went live!  I spent most of Monday moving the new site across to its proper URL, testing things, updating links, validating the HTML, putting in redirects from the old site and ensuring everything was working smoothly and the final result was unveiled at the Samuels lecture on Tuesday evening.  You can find the new version here:
It was also the week that the new version of the SCOTS corpus and CMSW went live too, and I spent a lot of time on Tuesday and Wednesday working on setting up these new versions, undertaking the same tasks as I did for the HT.  The new version of SCOTS and CMSW can be accessed here:
I had to spend some extra time following the relaunch of SCOTS updating the ‘Thoms Crawford’s diary’ microsite.  This is a WordPress powered site and I updated the header and footer template files so as to give each page of the microsite the same header and footer as the rest of the CMSW site, which looks a lot better than the old version which didn’t have any link back to the main CMSW site.
I had a couple of meetings this week, the first with a PhD student who is wanting to OCR some 19th century Scottish courts records.  I received some very helpful pointers on this from Jenny Bann, who was previously involved with the massive amounts of OCR work that went on in the CMSW project and was able to give some good advice on how to improve the success rate of OCR on historical documents thanks to Jenny’s input.
My second meeting was with a member of staff in English Literature who is putting together an AHRC bid.  Her project will involve the use of Google Books and the Ngram interface, plus developing some visualisations of links between themes in novels.  We had a good meeting and should hopefully proceed further with the writing of the bid in future weeks.  Having not used Ngrams much I spent a bit of time researching it and playing around with it.  It’s a pretty amazing system and has lots of potential for research due to the size of the corpus and the also the sophisticated query tools that are on offer.

Week beginning 4th November 2013

This week was a bit of a short one as I was off on Thursday to attend a funeral and I took the Friday off as well.  On Monday I created another new version of the DSL mock-up interface, based on further feedback from the SLD team.  It’s a combination of elements from versions 6 and 7 and it can be found here:


It’s possible that further changes will be required, even though ‘final’ appears in the URL.

A couple of weeks ago the Course 20 students did some usability testing on the three STELLA apps I have so far made, and I spent a bit of time this week looking into some of the issues that were raised.  Most concerned navigation paths between the exercises and the ‘book’ – it’s a bit of a pain to follow a link from an exercise to the relevant book page as you then have no way of returning to the exercise page with your answers already filled in.  Luckily in HTML5 there’s a handy mechanism for storing user content locally for the browser to access either on subsequent page loads within the same session or during future sessions.  These mechanisms are called ‘localStorage’ and ‘sessionStorage’ and they replace (and greatly improve upon) cookies that were previously used by web sites to store user information at the client side.  I’ve been playing about with the mechanisms this week and they are really very easy to use – just set some variables in JavaScript and they’re accessible across pages.  Very nice!  I’ve so far been investigating how this can be used to ensure the user can return to the relevant exercise page after viewing a book page, but I will also need to expand the exercises as well so that a user’s answers are also stored and can be used to repopulate an exercise when the page is reloaded.  This will be rather tricky to implement but should be doable.

I also spent some time working on the Burns project this week as well, including writing a WordPress ‘cheatsheet’ for the new RA staff and preparing a tutorial session to get them up to speed with using the WordPress interface.  I also made a few tweaks to the website and helped get some video clips published.

There’s not really much else to report this week due to my time off.

Week beginning 28th October 2013

I completed all of the outstanding work on the redevelopment of the CMSW website this week, including replacing the green colour scheme I’d previously chosen with a snazzy purple one.  The new version of the site can be found here:


But I will be replacing the old SCOTS and CMSW sites with the new version in the next week or so, at which point the ‘new-design’ URLs will stop working.

The biggest challenge this week was creating a script that could process the digitised images of the texts, including displaying them in page, displaying a thumbnail index view and generating ‘next’ and ‘previous’ navigation paths.  These all used the JSON files I created last week that extracted image file IDs and directory names from the file structure on the server.  As I hoped I would be able to do, I managed to get one single (and pretty simple) PHP script to process all of the image management stuff, where previously there existed an individual HTML page for each of the thousands of images in the site.  It works very well, although as I’m embedding the full-size images in the page that’s generated (in order to make the standard view bigger than the more limited view the old site gave) it can take a while for the page images to load.  If this is an issue I might have to revert to using the smaller images.

Also this week I completed work on the ‘microsites’ – ‘Life in old letters’ and the ‘Burns Kilmarnock Edition’.  This mainly required tweaking the image navigation code I had previously created and wasn’t too tricky a task to complete.  I’ve had to leave the update of the Thomas Crawford diary microsite until we go live with the new design as it’s a WordPress powered site and I don’t want to update the templates until everything is ready to go.  I also updated the search input form to make the multi-select options (Genre and year group) look nicer – HTML multi-select boxes look pretty awful as it’s not possible to apply styles to the selected options so they just stay as the browser’s default colours (e.g. a garish blue in Firefox).  Instead of this I’ve implemented checkboxes in a scrollable div, which looks a lot nicer.

There was a strike this week so I lost a day’s work so there is less to report than normal.  Other than the CMSW stuff I wrote a blog post for the Mapping Metaphor project blog and I began looking into the development of the fourth STELLA App – ‘Essentials of Old English’.  I’m just at the planning stage with this for the moment though – going through the existing site and noting any possible tricky parts.  I also read through the feedback the Course 20 students had given for the three existing STELLA Apps, which was all very positive apart for some suggestions about navigation paths in the Grammar app, something that I definitely need to look into.