Week Beginning 6th June 2016

Last week I started to redevelop the old STELLA resource ‘The Basics of English Metre’ and I spent much of this week continuing with it.  The resource is split into three sections, each of which feature a variety of interactive exercises throughout.  Last week I made a start on the first exercise, and this week I made quite a bit of progress with the content, completing the first 12 out of 13 pages of the first section.  As with the previous STELLA resources I redeveloped, I’ve been using the jQueryMobile framework to handle the interface and jQuery itself to handle the logic of the exercises.  The contents of each page are stored in a JSON file, with the relevant content pulled in and processed when a page loads.  The first exercise I completed required the user to note the syllable boundaries within words.  I was thankfully able to reuse a lot of the code from the ARIES app for this.  The second exercise type required the user to choose whether the syllables in a word were strongly or weakly stressed.  For this I repurposed the ‘part of speech’ selector exercise type I had created for the Essentials of English Grammar app.  The third type of exercise was a multi-stage exercise requiring syllable identification for stage 1 and then stress identification for stage 2.  Rather than just copying the existing code from the other apps I also refined it as I know a lot more about the workings of jQueryMobile than I did when I put these other apps together.  For example, with the ‘part of speech’ selector the different parts of speech appeared in a popup that appeared when the user pressed on a dotted box.  After a part of speech was selected it then appeared in the dotted box and the popup closed.  However, I had previously set things up so that a separate popup was generated for each of the dotted boxes, which is hugely inefficient as the content of each popup is identical.  With the new app there is only one popup and the ID of the dotted box is passed to it when the user presses on it.  This is a much better approach.  As most of the remaining interactive exercise are variations on the exercises I’ve already tackled I’m hoping that I’ll be able to make fairly rapid progress with the rest of the resource.

Other than working on the ‘Metre’ resource I communicated with Jane Stuart Smith, who is currently putting a new proposal together.  I’m not going to be massively involved in it, but will contribute a little so I read through all of the materials and gave some feedback.  Bill Kretzschmar is also working on a new proposal that I will have a small role in too, so I had an email chat with him about this too.  I also completed a second version of the Technical Plan for the proposal Murray Pittock is currently writing.  The initial version required some quite major revisions due to changes in how a lot of the materials will be handled, but I think we are now getting close to a final version.

I also spent a little bit of time working with some of the Burns materials for the new section and gave a little bit of advice to a colleague who was looking into incorporating historical maps into a project.  I also fixed a couple of bugs with the SciFiMedHums ‘suggest a new bibliographical item’ page and then made it live for Gavin Miller.  You can suggest a new item here: http://scifimedhums.glasgow.ac.uk/suggest-new-item/ (but you’ll need to register first).  Finally, I continued processing the Hansard data using the ScotGrid infrastructure.  By the end of the week 200 of the 1200 SQL files had been processed, resulting in more than 150,000,000 rows.  I’ll just keep these scripts running over the next few weeks until all of the data is done.  I’m not sure than an SQL database is going to be quick enough to actually process this amount of data, however.  I did a simple ‘count rows’ query and it took over two minutes to return an answer, which is a little worrying.  It’s possible that after all of the data is inserted I may then have to look for another solution.  But we’ll see.


Week Beginning 16th May 2016

I had rather a lot of different things to get through this week, which made the week fly past alarmingly quickly.  On Monday I attended an all-day meeting for the Scots Syntax Atlas project.  The meeting was primarily focussed on the online resources that the project intends to produce, so it was obviously quite important for me to attend, seen as I’ll be creating the resources.  We discussed some existing online linguistic atlas projects, looking at some in detail and figuring out aspects of each that we liked or didn’t like.  We also tried out a few online language quizzes too as the project is hoping to incorporate such a feature.  We also discussed some of the requirements of our intended online resource, and how the ‘general public’ interface that the project intends to create should differ from the ‘academic’ interface.  I have a much clearer idea now of what it is the project would like to develop, although we still need to have a more detailed discussion about the underlying data itself and how this will be presented on the map-based interface.  Things are beginning to take shape, however.

Gareth Roy from Physics and Astronomy got back to me this week to say that one of his colleagues had set up a database on a server for me to use for the Hansard data, which was excellent news.  I didn’t have much time to test this out this week, but did manage to confirm that I can access it from both my desktop machine and the little test server located in my office.  Hopefully I will find the time next week to migrate some of the existing data to this database and start to write a script that can process the thousands of SQL files that contain the actual data.

I also had to spend a bit of time this week doing AHRC review duties, and I attended a Digital Humanities talk on Friday afternoon, which was interesting but perhaps lacking a bit in detail.  I spend some further time on the Mapping Metaphor project, as Wendy and Carole had prepared new batches of both OE and non-OE data that I needed to upload to the database.  This all seemed to go smoothly enough, although as a result of the upload Wendy noticed that the data from certain OE categories appears to have been overlooked.  It would appear that several of the categories that were split off from category H27 were omitted from the spreadsheets at some earlier stage in the data processing and I had to spend some time trying to figure out when and how this happened, which was not easy as this must have happened more than a year ago and it may have been because of one of my scripts or it may have been as a result of some of the processes Ellen and Flora conducted.  We still hadn’t managed to get to the bottom of the issue by the end of the week and I’ll have to return to this next week.

I also spent some time this week helping out Vivien Williams of the Burns project, who is putting the finishing touches to lots of new content for the website.  She needed my help to get some parts of her pages laid out as she required them, and in converting some WAV files and other such tasks.  I also had a detailed email conversation with Alison Wiggins about the proposal she’s putting together, and I think we now have a much clearer idea of how the technical aspects of the proposal will fit together.  I also set up a ‘bare bones’ WordPress installation for Christine Ferguson’s new project and sent some code on to some researchers that were interested in how the SciFiMedHums bibliographic database was put together.  Oh, I also published the Android version of ‘Readings in Early English’, which can be found here: https://play.google.com/store/apps/details?id=com.gla.stella.readings


Week Beginning 15th February 2016

I was on holiday on Monday so it was a four-day week for me, which was nice but it meant I had rather a lot to try to squeeze in. I mainly worked on four projects this week, which I will run through in no particular order. On Wednesday I met with Michael Shaw, the RA on the ‘People’s Voice’ project. The project is getting to the stage now where they really need their research database in place so Michael and I met to talk about the sorts of data they need to record about the poems they are researching and what sort of collection method would be suitable. We agreed that having an online content management system and database would work well in most situations, but there would be times when researchers would not be able to guaranteed internet access and would need to work offline. I suggested that I could create a CSV based template file that could be filled in using Excel when an internet connection was unavailable. A simple ‘drag and drop’ script would then allow rows from this file to be integrated with the main database once a connection could be established. We talked through the structure of the data and following on from our meeting I created a first version of a database specification document, a CSV template file and a set of guidelines for filling in the template. Michael has already started conducting research at the Mitchell so will be able to use the template until the content management system is ready for use. My documents were sent on to the rest of the team for feedback. There will no doubt be a bit of toing and froing over the structure of the data, but once that has been sorted out I will be able to start work on the database.

My second project this week was REELS. The project met last week to finalise the database specification, and following on from this I had created the underlying database for the project. This week I began to create the content management system that will sit on top of this database. As I’ve already created many such systems before it was pretty straightforward to set up the basic structure for the CMS, such as logins, layout templates etc. After that I focussed on the place-name element side of the system. We need facilities to create, list, edit and delete place-name elements and I managed to get all of these facilities set up with little difficulty. Carole had also suggested that the project reuse the elements from the Fife place-names project so I created a script that would migrate and clean-up the element data from this project. The script stripped out any tags and timed white spaces and ensured that each distinct element was only recorded once. A total of 1752 elements have been transferred, with 4 languages represented, plus existing and proper names. REELS wants to record data for elements that was not present in the Fife data, such as part of speech, a URL and a description, so these fields are currently blank and will need to be filled in as the project proceeds.

My third project was SciFiMedHums, for which I completed work on the facilities to allow users of the site to suggest new bibliographic items and for administrators to manage these submissions. Users need to register and log into the site in order to post suggested items and the submission form allows the user to enter an item title, select a type and medium and select one or more themes from the list of existing themes. There is also an additional ‘comments’ box where users can supply additional information, such as suggestions for new themes. I’ve updated the item database to include new columns for whether an item was submitted by a user of the site, what the status of the item is (approved, not approved or deleted), who submitted the item and their comments.

Upon submission the details are as a new item, but with ‘user submit’ set to ‘Y’ and ‘status’ set to ‘not approved’ and an email is sent to the project email address. The admin interface now has a new page called ‘List Pending Contributions’ which lists tems that have been submitted by users but have not yet been approved. From this list an Admin user can view the submitted data and decide whether to add it to the main item list (set ‘status’ to ‘approved’) or to delete it (set ‘status’ to ‘deleted’).


If status for an item is set to ‘deleted’ the item is removed from the list and won’t appear anywhere else in the system. If the status for an item is set to ‘approved’ the item will be removed from the list and will instead appear in the main ‘SFMH Bibliography’ list. This also generates a WordPress ‘post’ for the item, thus fully integrating the item with our custom post type. The feature is not ‘live’ yet, as I am awaiting feedback from Gavin before I do that.

My fourth project of the week was Metaphor in the Curriculum, the Mapping Metaphor follow-on project. The original Mapping Metaphor project has been nominated for ‘Best DH Data Visualisation’ in the DH Awards 2015. Voting is open to everyone so if you would like to vote for the project you can do so here: http://dhawards.org/dhawards2015/voting/

I continued this week with the app version of the Mapping Metaphor visualisations, migrating all of the existing server-side parts of the code to Javascript so that everything will work in a self-contained app. I lost a couple of hours of progress to investigating a rather annoying bug that was caused by the different way that source data is processed. Previously the data was stored in an SQL database and relevant parts were then pulled out and assigned to variables in the code for processing. With the app version the data is now stored in a JSON file which is loaded into the Javascript code. As with before, relevant parts are then pulled out and assigned to variables for processing. What I didn’t quite remember is that assigning a Javascript variable to an object loaded from the JSON file doesn’t actually create a new object, all it does is create a new pointer to the existing object. So updating the data in what I had considered to be a new copy of the data was actually updating the source data too, which was causing major problems when other parts of the code then tried to grab the same data. Thankfully I figured out what was causing the weirdness and jQuery has a nice little function for copying an object to a new variable rather than just creating a new pointer to the existing object. Basically instead of doing var newObject = metaphorData[i] you do var newObject = jQuery.extend({},metaphorData[i]) and the source object remains untampered with.

I managed to get the visualisation view of the data completed this week, including both aggregate and drilldown views and the metaphor cards for each. I also started work on the tabular view of the data, although I haven’t managed to get this working yet. There is still so much to do, but hopefully I will continue to make good progress with this next week.

Week Beginning 1st February 2016

I worked on several projects this week. I continued to refine the database and content management system specification document for the REELS project. Last week I had sent an initial version out to the members of the team, who each responded with useful comments. I spent some time considering their comments and replying to each in turn. The structure of the database is shaping up pretty nicely now and I should have a mostly completed version of the specification document written next week before our next project meeting.

I also met with Gary Thoms of the SCOSYA project to discuss some unusual behaviour he had encountered with the data upload form I had created. Using the form Gary is able to drag and drop CSV files containing survey data, which then pass through some error checking and are uploaded. Rather strangely, some files were passing through the error checks but were uploading blank data, even though the files themselves appeared to be in the correct format and well structured. Even more strangely, when Gary emailed one of the files to me and I tried to upload it (without even opening the file) it uploaded successfully. We also worked out that if Gary opened the file and then saved it on his computer (without changing anything) the file also uploaded successfully. Helpfully, the offending CSV files don’t display with the correct CSV icon on Gary’s Macbook so it’s easy to identify them. There must be some kind of file encoding issue here, possibly caused by passing the file from Windows to Mac. We haven’t exactly got to the bottom of this, but at least we’ve figured out how to avoid it happening in future.

On Friday I had a final project meeting for the Medical Humanities Network project. The meeting was really just to go over who will be responsible for what after the project officially ends, in order to ensure new content can continue to be added to the site. There shouldn’t really be too much for me to do, but I will help out when required. I also continued with some outstanding tasks for the SciFiMedHums project on Friday too. Gavin wants visitors to the site to be able to suggest new bibliographical items for the database and we’ve decided that asking them to fill out the entire form would be too cumbersome. Instead we will provide a slimmed down form (item title, medium, themes and comments) and upon submission an Editor will then be able to decide if the item should be added to the main system and if so manage this through the facilities I’ll develop. On Friday I figured out how the system will function and began implementing things on a test server I have access to. So far I’ve updated the database with the new fields that are required, added in the facilities to enable visitors to the site to log in and register and I’ve created the form that users will fill in. I still need to write the logic that will process the form and all of the scripts the editor will use to process things, which hopefully I’ll find time to tackle next week.

I continued to work with the Hansard data for the SAMUELS project this week as well. I managed to finish the shell script for processing one text file, which I had started work on last week. I managed to figure out how to process the base64 decoded chunk of data that featured line breaks, allowing me to extract and process an individual code / frequency pairing. I then figured out a way to write each line of data to an output text file. The script now takes one of the input text files that contain 5000 lines of base64 encoded code / frequency data and for each code / frequency pair it writes an SQL statement to a text file. I tested the script out and ensured that the resulting SQL statements worked with my database and after that I contacted Gareth Roy in Physics, who has been helping to guide me through the workings of the Grid. Gareth provided a great deal of invaluable help here, including setting up space on the Grid, writing a script that would send jobs for each text file to the nodes for processing, updating my shell script so that the output text file location could be specified and testing things out for me. I really couldn’t have got this done without his help. On Friday Gareth submitted an initial test batch of 5 jobs, and these were all processed successfully. As all was looking good I then submitted a further batch of jobs for scripts 6 to 200. These all completed successfully by late on Friday afternoon. Gareth then suggested I submit the remaining files to be processed over the weekend so I did. It’s all looking very promising indeed. The only possible downside is that as things currently stand we have no server on which to store the database for all of this data. This is why we’re outputting SQL statements in text files rather than writing directly to a database. As there will likely be more than 100 million SQL insert statements to process we are probably going to face another bottleneck when we do actually have a database in which to house the data. I need to meet with Marc to discuss this issue.

The rest of my week was spent on the Mapping Metaphor follow-on project. I am developing an app version of the metaphor map for this project and I started looking into the task this week. The existing map is a mixture of client-side and server-side code, with most server-side stuff processing AJAX calls and generating JSON data from our database. The app will need to move all of the data storage to the client-side as we want the app to work without an internet connection – there will be no ‘server-side’ for the app to connect to. This means I will need to migrate all of the server-side PHP and MySQL code to client-side Javascript and static JSON files. Once I got stuck into this task I realised just how horrible it’s going to be. So much of the existing code needs to be rewritten and it’s very difficult to know where to begin as everything is so interconnected. It’s difficult to test the functionality of one migrated bit of code as it relies on many other bits that still need to be migrated. However, after a few mild panic attacks I set to work and I have managed to migrate a few parts of the map to a ‘pure Javascript’ solution so far. I started with the ‘infobox’ and the pop-ups (e.g. change view, the key) and got all of this working. I then tackled the map view, starting with the aggregate map. I have now managed to get the aggregate map working with a local JSON file supplying the data. It’s possible to click on a node to view its connections and rotate the map. I still haven’t got the differently sized circles working yet, though. Or the facility to change metaphor strength. There is a lot left to do, and considering the project is hoping to have the app completed some time next month I’m going to have a busy few weeks. As always, my progress will be posted here.

Week Beginning 4th January 2016

I returned to work on Wednesday this week, after very enjoyable but seemingly all too brief Christmas holiday. I spent a bit of time on Wednesday sorting through my emails, replying to things and checking that there was nothing lurking in my inbox from last year that I’d failed to deal with. I also spent a few hours helping Simon Taylor out with some problems a user of his Fife Placenames website had encountered. I had previously helped Simon to extract the placename data from Word files and created scripts that would split these up and generate a relational database for the resulting data, but it would appear that in a few cases the splitting script hadn’t quite worked properly and some places were being appended to the entries for the previous place. I think I managed to get this all sorted now, though.

I also spent a fair amount of my time this week on the Medical Humanities Network website and database. The project is going to be launching this month and there have been a number of further tweaks and updates that Megan and Hannah have requested, such as allowing Twitter feeds to be associated with collections, changing the way teaching materials are organised and updating the list of icons for projects. Everything is coming together quite nicely, though.

Gavin Miller also contacted me about his SciFiMedHums project. I’d previously created a WordPress plugin that allows him and his RA to create and manage bibliographical data and now he is hoping to get the public involved in submitting data too. We had an email discussion where I described a few possible ways in which this could be handled and I’ll just need to wait and see how Gavin wants to proceed with this.

I also worked on the Metaphor in the Curriculum project, greatly expanding the amount of content that is currently available in the prototype app that I’ve created. Previously we had one quiz available (metaphor and war) but before Christmas Rachael had uploaded some new quizzes to the project’s shared drive (in Word format) so I created interactive versions of these. There are now four quizzes available, each with their own nice background image. Things are shaping up quite nicely, I think.

So, a bit of a short report this week. Next week I’m going to have lots to do, such as starting work on the People’s Voice project, continuing with the SCOSYA project and working with the Grid to try and get the Hansard data processed.

Week Beginning 23rd November 2015

These weeks seem to be zipping by at an alarming rate! I split most of my time this week between three projects and tackled a few bits and bobs for other projects along the way too. First up in Metaphor in the Curriculum. Last week I created a fully functioning mockup of a metaphor quiz and I’d created three basic interface designs. This week I created a fourth design that is an adaptation of the third design, but incorporates some fairly significant changes. The biggest change is the introduction of a background image – a stock image from the very handy free resource http://www.freeimages.com/. The use of a background image really brightens up the interface and some transparency features on some of the interface elements helps to make the interface look appealing without making it difficult to read the actual content. I also reworked the ‘MetaphorIC’ header text so that the ‘IC’ is in a different, more cursive font and added a ‘home’ button to the header. I think it’s coming together quite nicely. We have another project meeting next week so I’ll probably have a better idea about where to focus next on this project after that.

My next project was the Burns project. Last month Pauline sent round a document listing some fairly major changes to the project website – restructuring sections, changing navigation, layout and page content etc. I set up a test version of the live site and set about implementing all of the changes that I could make without further input from project people. After getting it all working pretty well I contacted Pauline and we arranged to meet on Monday next week to go through everything and (hopefully) make all of the changes live.

The third project I worked on this week was the SCOSYA project and this took up the bulk of my time. Last week Gary had sent me a template of the spreadsheet that the project fieldworkers will fill in and email to Gary. Gary will then need to upload these spreadsheets to an online database through a content management system that I need to create. This week I began working on the database structure and the content management system. The project also wants the usual sort of project website and blog, so first of all I set up WordPress on the project’s domain. I toyed with the idea of making the content management system a WordPress ‘plugin’, but as I want the eventual front-end to be non-Wordpress I decided against this. I also looked into using Drupal for the content management system as Drupal is a tool I feel I ought to learn more about. However, the content management system is going to be very straightforward – just file upload plus data browse, edit and delete and using Drupal or other such tools seemed like overkill to me. I was also reluctant to use a system such as Drupal because they seem to change so rapidly. SCOSYA is a 5 year project (I think!) and my worry is that by the end of the project the version of Drupal that I use would have been superseded, no longer supported and seen as a bad thing to have running on a server. So I decided just to create the CMS myself.

I decided that rather than write all of the user authentication and management stuff myself I would tie this in with the WordPress system that I’d set up to power the project website and blog. After a bit of research I figured out that it is remarkably easy for non-Wordpress scripts to access the WordPress authentication methods so I set up the CMS to use these, following the instructions I found here: http://skookum.com/blog/using-wordpress-as-a-user-and-authentication-database. With this in place SCOSYA staff can manage their user accounts via WordPress and use the same details to access the CMS, which seems very neat.

Gary will be uploading batches of CSV files and we met again this week to discuss some questions I came up with whilst thinking about the upload script. We tweaked the template a little and I created a database structure that will be able to store the data in a usable format. I also looked into how best to handle the batch upload of the data. I quite like WordPress’s media upload facility, whereby users can just drag and drop files into the browser window and these are then processed. After a bit of Googling I found a nice little Javascript library that allows similar functionality: Dropzone.js (http://www.dropzonejs.com/). This works really nicely as it can process each ‘dropped’ file as a separate Ajax request in the background and then display content from the server based on the output of the server-side upload script. It also features client-side error checking (e.g. file type checking), can display thumbnails and shows file sizes, which are all nice features too (although of course you can’t rely solely on client-side error checking and must implement lots of checking on the server side too).

By the end of the week I had created an upload script that allows you to drag and drop multiple files into the upload pane, for these to be checked both on the client and server side and for a log of uploads to be built up dynamically as each file is processed in a scrolling section beneath the upload pane. I still need to do quite a lot of work on the server-side script in order to extract the actual data from the uploaded files and to insert this data into the relevant tables, but I feel that I have made very good progress with the system so far. I’ll continue with this next week.

Other than these three main projects I was involved with some others. I fixed a few bugs that had crept into the SciFiMedHums bibliographic search facility when I’d updated the functionality of it last week. I slightly tweaked the Medical Humanities Network system to give a user feedback if they try to log in with incorrect details (previously no feedback was given at all). I also contacted John Watt at NeSC to see whether he might be able to help with the extraction of the Hansard data and he suggested I try to do this on the University’s High Performance Compute Cluster. I’ll need to speak with the HPCC people to see how I might be able to use their facilities for the task that needs performed.

Week Beginning 16th November 2015

I met with Gary Thoms this week to discuss the content management system I’m going to build for the SCOSYA project. Gary has prepared a spreadsheet template that the fieldworkers are going to be using when conducting their interviews and we talked through how this was structured. After that I began to think about the database structure that will be used to store the spreadsheet uploads.   I also had a meeting with the Burns people to discuss the new bid that they are putting together, which is getting very close to completion now. I also talked to Pauline about the restructuring of the Burns online resource and we agreed that I would begin work on this at a temporary URL before moving everything across to replace the existing site. I’ll need to start work on this in the next week or so. I also updated the search functionality of the SciFiMedHums bibliography system to enable users to search for multiple themes and mediums (once the system goes live). I also made a few tweaks to the Medical Humanities Network website, mainly adding in the site text and helping out with some video uploads. I made a couple of small tweaks to the new Thesaurus of Old English content management system and set up some user accounts for people too.

My major focus of the week was the Metaphor in the Curriculum project. At our last project meeting Ellen had given me some sample questions to show how the metaphor quizzes that will be found in the apps and the website will be structured. I spent a lot of this week creating working digital prototypes of these, using a similar approach to the one I’d taken for the interactive STELLA apps I’d previously produced: there is a question, some possible answers, a ‘check answer’ button, a ‘restart’ button and next and previous question buttons (where applicable). The question content itself is pulled in from a JSON file and there are three question types, although really types 1 and 3 are handled in the same way in the current version.  Types 1 and 3 (questions 1-3 and 6-7) present possible answers, the user can click on one and then press the ‘check answer’ button.  A tick will be placed beside their answer if it was correct, a cross if incorrect.  No other feedback is currently offered and there is no tracking of right / wrong answers (this is something that might be changed).

Question type 2 (questions 4-5) allows the user to ‘drag and drop’ an answer into the dotted box in the question.  Once an answer is dropped into place the user can press the ‘check answer’ button and a tick or cross will be displayed beside the sentence and also beside the item they dragged. I’ve tested this drag and drop functionality out on iOS and Android devices and it works fine, so hopefully we’ll be able to include such functionality in the final version of the quizzes.

The prototypes I’d created focused entirely on the functionality of the quiz itself and not on the overall design of the interface, and once I’d completed the prototypes I set to work on some possible interface designs. So far I have created three, but I need to work on these some more before I let anyone see them. Having said that, I think the technical side of the project is currently on schedule, which is encouraging.

On Friday there was a corpus linguistics workshop at the University, which was being organised by Wendy. I had been invited to be part of the panel for a roundtable discussion session at the end of the event so I spent some time this week preparing for this. Unfortunately my son was ill on Friday and I had to leave work to pick him up from school, which meant I had to miss the event which was a real shame.


Week Beginning 26th October 2015

I returned to a more normal working week this week, after having spent the previous one at a conference and the one before that on holiday. I probably spent about a day catching up with emails, submitting my expenses claim and writing last week’s rather extensive conference report / blog post. I also decided it was about time that I gathered all of my outstanding tasks together into one long ‘to do’ list as I seem to have a lot going on at the moment. The list currently has 47 items on it split across more than 12 different projects, not including other projects that will be starting up in the next month or two. There’s rather a lot going on at the moment and it is good to have everything written down in one place so I don’t forget anything. I also had some AHRC review duties to perform this week as well, which took up some further time.

With these tasks out of the way I could get stuck into working on some of my outstanding projects again. I met with Hannah Tweed on Tuesday to go through the Medical Humanities Network website with her. She had begun to populate the content management system with projects and people now and had encountered a few bugs and areas of confusion so we went through the system and I made a note of things that needed fixed. These were all thankfully small issues and all easily fixable, such as supressing the display of fields when the information isn’t available and it was good to get things working properly. I also returned to the SciFiMedHums bibliographical database. I updated the layout of the ‘associated information’ section of the ‘view item’ page to make it look nicer and I created the ‘advanced search’ form, that enables users to search for things like themes, mediums, dates, people and places. I also reworked the search results page to add in pagination, with results currently getting split over multiple pages when more than 10 items are returned. I’ve pretty much finished all I can do on this project now until I get some feedback from Gavin. I also helped Zanne to get some videos reformatted and uploaded to the Academic Publishing website, which will probably be my final task for this project.

Wendy contacted me this week to say that she’d spotted some slightly odd behaviour with the Scots Corpus website. The advanced search was saying that there were 1317 documents in the system but a search returning all of them was saying that it matched 99.92% of the corpus. The regular search stated that there were 1316 documents. We figured out that this was being caused by a request we had earlier this year to remove a document from the corpus. I had figured out a way to delete it but evidently there was some data somewhere that hadn’t been successfully updated. I managed to track this down: it turned out that the number of documents and the total number of words was being stored statically in a database table, and the advanced search was referencing this. Having discovered this I updated the static table and everything was sorted. Wendy also asked me about further updates to the Corpus that she would like to see in place before a new edition of a book goes to the printers in January. We agreed that it would be good to rework the advanced search criteria selection as the options are just too confusing as they stand. There is also a slight issue with the concordance ordering that I need to get sorted too.

At the conference last week Marc, Fraser and I met with Terttu Nevalainen and Matti Rissanen to discuss Glasgow hosting the Helsinki Corpus, which is currently only available on CD. This week I spent some time looking through the source code and getting a bit of server space set aside for hosting the resource. The scripts that power the corpus are Python based and I’ve not had a massive amount of experience with Python, but looking through the source code it all seemed fairly easy to understand. I managed to get the necessary scripts and the data (mostly XML and some plain text) uploaded to the server and the scripts executing. The only change I have so far made to the code is to remove the ‘Exit’ tab as this is no longer applicable. We will need to update some of the site text and also add in a ‘hosted by Glasgow’ link somewhere. The corpus all seems to work online in the same way as it does on the CD now, which is great. The only problem is the speed of the search facilities. The search is very slow, and can take up to 30 seconds to run. Without delving into the code I can’t say why this is the case, but I would suspect it is because the script has to run through every XML file in the system each time the search runs. There doesn’t appear to be any caching or indexing of the data (e.g. using an XML database) and I would imagine that without using such facilities we won’t be able to do much to improve the speed. The test site isn’t publicly accessible yet as I need to speak to Marc about it before we take things further.



Week Beginning 4th October 2015

I worked on quite a number of different projects this week. My first task was to set up a discussion forum for Sean Adams’ Academic Publishing conference website. The website is another WordPress powered site and I hadn’t worked with any forum plugins before so it was interesting to learn about this. I settled for the widely adopted ‘bbpress’ plugin, which turned out to be very straightforward to set up and integrates nicely with WordPress. I had to tweak the University theme I’d created a little so that the various sections displayed properly, but after that all appeared to be working well. I also spent some time continuing to contribute to the new Burns bid for Gerry Carruthers. I’d received some feedback on my first version of the Technical Plan and based on this and some updated bid documentation I created a second version. I also participated in some email discussions about other parts of the bid too. It seems to be shaping up very well. Ann Fergusson from SND contacted me this week as someone had spotted a missing section of text in one of the explanatory pages. I swiftly integrated the missing text and all is now well again.

On Tuesday I had a meeting with Susan and Magda about the Scots Thesaurus. We went through some of the outstanding tasks and figured out how and when these would be implemented. The biggest one is the creation of a search variants table, which will allow any number of different spellings to be associated with a lexeme, enabling it to be found by the search option. However, Magda is going to rework a lot of the lexemes over the coming weeks so I’m going to hold off on implementing this feature until this work has been completed.

I spent a fair amount of time this week working on the ‘Edinburgh and the Enlightenment’ map that Craig Lamont is preparing for Murray Pittock. We’re creating this within the University’s T4 system, which has complicated matters somewhat, but things are now shaping up nicely. We have a historical map with a variety of differently coloured pins on it and the key that allows different colours to be turned on or off. The biggest issue I addressed this week was the size of the map. When viewed within the University’s page layout the map was rather small and I suggested that I could make a full-screen version in a JavaScript based popup. This proved to be slightly trickier to implement than I had expected. Initially I had intended to use a jQuery UI dialog box, as I did for the Scots Thesaurus visualisations. The University site already incorporates the jQuery UI JavaScript code so I started to use that. However, for some reason the jQuery UI styles are not incorporated into the University site, meaning the dialog failed to appear anywhere. After adding the necessary styles to our page the popup appeared and I managed to get the map to display in it. The next problem was that you can’t share markers between multiple maps in the same page when using the Leaflet library. Well, you can but when you move one map around the marker positions all get messed up on the other map. I had to completely change the way we store out marker data so that the code could automatically add them to two different maps. The next problem I encountered was with the Leaflet layer controls – the options that allow differently coloured pins to be turned on or off. These simply wouldn’t work properly within the jQuery dialog box. I spent quite a bit of time trying to figure out what was causing this and eventually tried moving the map out of the dialog box. This immediately fixed the problem so there would appear to be some sort of conflict between the jQuery UI dialog code and the Leaflet layer control code. In the end I had to make my own simple overlay for displaying the full-screen map, and thankfully this appears to be working nicely. Craig now has to do a lot of data entry to convert our marker data into the new format, but once that has been completed the map should be ready to launch.

I also had a Mapping Metaphor task to do this week: updating the database with new data. Wendy has been continuing to work with the data, adding in directionality, dates and sample lexemes and Ellen sent me a new batch. It has been a while since I’d last uploaded new data and it took me a while to remember how my upload script worked, but after I’d figured that out everything went smoothly. We now have information about 16,378 metaphorical connections in the system and 12,845 sample lexemes linked into the Historical Thesaurus.

My final task of the week was to continue working on my WordPress plugin for Gavin Miller’s Science Fiction and the Medical Humanities website. Last week I started working on a browse facility that will let users view things like years, themes, mediums and people and to then click through to search results showing the bibliographical items that have been associated with these things. I had included jQuery in this to power some tabs and unfortunately this had conflicted with the jQuery version that was already included within WordPress. I have now figured out how to include jQuery in a plugin and to reference WordPress’s version rather than my own version. Basically at the top of the plugin you have to include ‘wp_enqueue_script(‘jquery’);’ and then in the JavaScript you need to wrap the code with ‘jQuery( function ( $ ) {});’ rather than ‘$(function() {});’ that I’m used to. With that conflict resolved I continued to develop the browse feature and the search results page that this passes data to. It’s now possible to browse for things (e.g. the theme ‘surgery’), see all of the corresponding bibliographic items and then click through to an actual item. I also updated the item page so that whenever browsable things appear in the page (e.g. the associated themes) these link in to the search results page too. It seems to be working pretty well. The next task will be to create advanced and quick search features.

I’m going to be on holiday next week and the following week I’m going to be at a conference so there will be no more from me until after I return.

Week Beginning 28th September 2015

This week was a return to something like normality after the somewhat hectic time I had in the run-up to the launch of the Scots Thesaurus website last week. I spent a bit of further time on the Scots Thesaurus project, making some tweaks to things that were noticed last week and adding in some functionality that I didn’t have time to implement before the launch. This include differentiating between regular SND entries and supplemental entries in the ‘source’ links and updating the advanced search functionality to enable users to limit their search by source. I also spent the best part of a day working on the Technical Plan for the Burns people, submitting a first draft and a long list of questions to them on Monday. Gerry and Pauline got back to me with some replies by the end of the week and I’ll be writing a second version of the plan next week.

I also had a few meetings this week. I met with Luca Guariento from Music to help him with a JavaScript problem he was having. He’s using the Diva.js library as an image viewer but he was wanting to be able to pass specific image page IDs to the library via links in a search results page. We got something worked out using hashes to pass IDs between pages, but encountered some problems with the initialisation of Diva.js. Thankfully Luca has been in contact with the Diva developers and they told him how he could hook into the Diva initialisation event and all is working nicely now. I must remember to use Diva on a future project as it’s a very nice interface. I also met with Iain Edmonds, who is developing some quizzes for the Sounds of the City project. He was wanting some advice on being able to link to specific parts of a web page and I gave him some pointers.

On Friday we had a team meeting for the Metaphor in the Curriculum project. We spent a couple of hours going over the intended outputs of the project and getting some more concrete ideas about how they might be structured and interconnected, and also about timescales for development. It’s looking like I will be creating some mockups of possible exercise interfaces in early November, based on content that Ellen is going to send to me this month. I will then start to develop the app and the website in December with testing and refinement in January, or there abouts.

I also spent some time this week working on the Medical Humanities Network website for Megan Coyer. I have now completed the keywords page, the ‘add and edit keywords’ facilities and I’ve added in options to add and edit organisations and units. I think that means all the development work is now complete! I’ll still need to add in any site text when this has been prepared and I’ll need to remove the ‘log in’ pop-up when the site is ready to go live. Other than that my development work on this project is now complete.

Continuing on a Medical Humanities them, I spent a few hours this week working on some of the front end features for the ScifFiMedHums website, specifically features that will allow users to browse the bibliographical items for things like years and themes. There’s still a lot to implement but it’s coming along quite nicely. I also helped Alison Wiggins out with a new website she’s wanting to set up. It’s another WordPress based site and the bare-bones site is now up and running and ready for her to work with when she has the time available.

On Friday afternoon I received my new desktop PC for my office and I spent quite a bit of the afternoon getting it set up, installing software, copying files across from my old PC and things like that. It’s going to be so good to have a PC that doesn’t crash if you tell it to open Excel in the afternoons!