This week was a pretty busy one, with two major AHRC funded projects that I’m involved with heading towards their conclusions. Much of the week was spent on Mapping Metaphor duties, ahead of the project launch on Wednesday. On Monday we had our final team meeting and in addition to this I completed a number of final website tasks before the launch. This included uploading more data to the online database. All of the connections between metaphors and their strengths (about 16,000 records) had already been added to the online database, but some aspects of the connections, such as their dates of inception, directionality and sample lexemes were mostly still to be added. I had created sample, randomly generated data in order to test the system but I have now stripped all of this out in preparation for the launch. Ellen also passed on a spreadsheet containing more real data for the dates, directionality and sample lexemes and I uploaded this. We now have 651 metaphorical connections that have real data, up from 108. We also now have 1886 real sample lexemes in the system, up from 348. Lots more data will continue to be uploaded over the next few months, either through the batch upload script that I’ve already created or through the content management system that I created for editing individual metaphor connection records.
In addition to sorting out the data I also made some last minute tweaks to the interface. The biggest of these was to add a ‘cite this page’ feature to every data view. This works in the same way as the Historical Thesaurus ‘cite’ facility, presenting users with a pop-up that lists four different citation styles that they can choose from. I also added in the structure for the ancillary pages that are required (e.g. the help pages) and created some empty placeholder pages that I will populate once Ellen sends me on the required content.
The launch of the project on Wednesday went very well. It was great to see the website up on a big screen and Ellen did a great job of talking everyone through the major features of the interface. Everyone in the audience seemed suitably impressed and we all got to enjoy some very nice Prosecco and nibbles. Although this was a launch event we’re not actually going live with the website at this stage. There will be an official media launch of the site in a couple of months and until then I’m afraid I can’t link to the site, which is a bit of a shame. However, there is still quite a lot of work that people will be doing with the data of the project in the interim, plus we hope to have a dedicated Old English section of the site set up before then too. My day to day involvement with the project is now at an end though. It has been a wonderful project to have been a part of.
The other AHRC funded project that I soon to come to an end is the SAMUELS project. My involvement with this project has been mainly linked to the refactoring of the Historical Thesaurus dataset so that the semantic tagger can make full use of it. I will also be developing some data visualisations of the tagged Hansard data, which I managed to get hold of towards the end of last week. I transferred this from Lancaster as a 10Gb tar.gz file, and Steve at Lancaster had created a Ruby gem script that could be run to split the data up into individual files. I ran into a bit of a problem as I only had 30Gb of hard drive space left on my desktop PC and extracting the tar.gz file resulted in a bunch of files that were larger than this. Chris managed to find some server space where I could extract the files to, but unfortunately I am unable to run Ruby gems on the servers so I couldn’t then split these files up using Steve’s script. I spoke to Marc about this and he said that Fraser had an external hard drive that was purchased for the project that I could use temporarily so I managed to get the data onto this. Unfortunately I didn’t have time to run the gem script on the data. Well, I had time to start it off, but after several hours it was still running and Fraser needed the hard drive back so I had to terminate the process. I’ll need to meet with Fraser and possibly Marc next week to see exactly what I need to do with the data.
The main SAMUELS task of the week was to attend the end of project meeting that was taking place on Thursday afternoon and Friday morning. It was great to participate in this meeting as it gave me an opportunity to find out what had been happening in the parts of the project that I’ve not been directly involved with (which is most of it). I am amazed at how much work the project has managed to achieve in such a short period of time and there is much potential for future research using the foundations established by the project.
Other than these two projects, I had a chat with Megan Coyer about her Medical Humanities Network project, which I’m delighted to say received funding. We’ll be starting to work on this over the summer. I had a further chat with Jane about the proposal she’s putting together, I also spent a few hours on AHRC reviewer duties and I helped Chris out with a couple of problems with an older website.
I was on holiday on Monday this week but returned to spend most of the rest of the week putting the finishing touches to the Mapping Metaphor website in advance of next week’s launch. I had managed to fix all the previously identified bugs last week so this week I worked on the final remaining addition to the website, was to be some sort of ‘breadcrumb’ or ‘history’ feature that would allow people to easily navigate back to visualisations that had previously looked at. I quickly ruled out having a ‘breadcrumb’ trail at the top of the page as this sort of hierarchically structured set of links would provide a quite limited trail, plus it’s already represented to a certain extent in the left-hand info box. Instead I decided a ‘history’ feature would be more useful. This would log each data view a user opens, including visualisations, timelines, cards and tables, plus each time a category is clicked on or a section expanded.
I implemented this as an HTML5 sessionStorage variable within the user’s browser. No history information is stored on the MM server and the information is deleted as soon as the user closes their browser. The variable stores a chunk of HTML, consisting of table rows of links to data views plus the time when the view was loaded. Each time a view is loaded (e.g. by clicking to highlight a category) the code adds a link to this view (plus the timestamp) to the top of the chunk. I then added a new ‘history’ button to the left-hand info box and pressing on this opens a dialog box where the history variable is loaded and displayed. I also added a ‘clear’ button so people can empty their history if required. I think the feature works rather well and I’m finding it to be surprisingly useful. It’s nice to know you can browse ‘off piste’ and still get back to the things you were really supposed to be looking at.
Ellen had also sent me a list of other questions and requests, such as tooltip help text and revised text for the homepage so I spent a bit of time adding these. I also noticed a bug with the modal dialog boxes as used for the metaphor card and other pop-ups. A grey overlay is supposed to appear on top of all other page elements when such dialog boxes are shown so no other page elements can be clicked on. However, Ellen had wanted the ‘Key’ box to appear behind the left-hand info box and never on top of it and I’d updated the z-index of the info-box so it was on top. Unfortunately this applied to all other dialog boxes too, which was a mistake. I’ve therefore reverted to the earlier layout, meaning the key box once again appears on top of the left-hand box. However, I don’t think this is such an issue now because the key is now fixed in position on screen and no longer scrolls with the page. Plus the key is now in the bottom right rather than the left. So really the key only ever appears on top of the left-hand box if the user manually drags it over there.
Next week I’ll need to replace the existing data with the updated data and also remove all of the random data that I’d created for test purposes (dates, lexemes, directionality). I should hopefully be doing this on Monday, which will give me a day or so to deal with any problems that this might introduce before the launch on Wednesday.
Other than Mapping Metaphor stuff I had a few meetings with people. The first was with Jane Stuart-Smith to discuss the ESRC bid she is putting together. I can’t really go into too much detail here at the moment, but it was a useful meeting and we sorted out a few issues relating to the Data Management Plan and also how the various web-based components of the project will fit together. There’s probably more that needs to be discussed on the technical side of things but Jane didn’t have time to talk further this week. I also met with Johanna Green to discuss a Chancellor’s Fund bid she’s hoping to submit in collaboration with Special Collections. Again, I can’t go into details here but it was a useful meeting and I hope I can be a part of the project in some capacity. My final meeting of the week was with Marc Alexander to discuss some new potential arrangements for the roles of developers within the School. I can’t say more than that for the time being but we had a good discussion and hopefully some good things will emerge from it.
There were a few server issues that cropped up this week, which seemed to have been caused by some electrical work that was going on over the weekend. This affected a number of project websites, including the Scots Corpus and the DSL website. The latter took a little while to get fixed as the site had come back online perfectly except for the aspects of the Advanced Search that used the Solr functionality. Solr on the server was working, but for some reason the API couldn’t connect to it. Thankfully Chris McGlashan got to the bottom of the problem, which had been caused by the order in which services had come back online after the power outage.
I spent a few hours this week on AHRC reviewer duties and I also spent a little bit of time this week on SAMUELS duties, liaising with Steven Wattam at Lancaster about the Hansard data that he has been compiling for the project. This was completed this week and he made the data available to me – a tar.gz file that is almost 10Gb in size. I didn’t have time to look at the data this week but I’ll hopefully be able to look into it next week.
With the launch of the Mapping Metaphor project creeping ever closer I spent the bulk of this week going through my list of bugs and fixing them. I’m happy to report that I have now ticked off every item from my list, plus I’ve fixed a handful of other bugs that I’ve come across too. As with last week, the bugs are a mixed bag of relatively trivial blighters and some rather more tricky beasts.
First of all some updates to the way category info buttons work. These buttons bring up the descriptors for the category when clicked on, but when viewing a level 3 category these buttons were not being displayed unless the category in question was clicked on in the visualization. They are now available when no category is selected by default, whether the category in question is selected or not. These info buttons also now work in the tabular and card (‘cardular’?) views when the user clicks on any of the listed categories too. Category info pop-up titles now say ‘Category Descriptors’ where previously they either said nothing or ‘Metaphor’.
As the number of metaphors that were first recorded in Old English are generally more numerous than those first recorded in other time periods this was unbalancing the timeline view. Marc and Ellen wanted the Old English column to be wider than the others and to have multiple connections listed per row. Implementing this has proved to be very tricky. I’m afraid I couldn’t find a way to make one column of the graph wider than the rest as the columns are generated programmatically by the d3 library. It would mean hacking about with the source code for the library and I didn’t think that was such a good idea. However, as the columns are pretty wide as they are what I’ve managed to do is squeeze three dots per row into the Old English column, much in the same way as the ‘top level’ timeline has five squares per row. I think this works pretty well at evening out the size of the OE column. I’ve also added a tooltip and an asterisk to the ‘Old English’ column label. Hover over the label to view information about what we mean by OE.
I’ve updated the key in a couple of ways. In the top level timeline the key now displays squares rather than circles. And on all pages I’ve updated they so that it keeps it position at the bottom of the browser when the page is scrolled. It’s still possible to move the key somewhere else too and wherever you leave it, it will also remain fixed and visible when the page scrolls. I think this works a lot better than having it scroll off the page when the user scrolls, but there is a slight downside: Adding this feature is using up more of the browser’s memory and it can slow down the operation of the visualisation in some browsers, especially on older hardware. On my work PC in Firefox with the key visible scrolling the browser on the visualisation page is slightly ‘laggy’ and the visualisation rotation is a bit jerky when it contains a lot of data. For some reason my PC has no such problems when using Chrome or IE though. I think stopping the key scrolling off the screen is much better but the increased jerkiness might be a big issue.
In the top-level timeline some pink highlighting was remaining when the user moved their mouse over some squares. I’ve figured out what was causing this and have fixed it. I’ve also ensured that when viewing the top-level timeline and clicking ‘view connections’ from a card the drilldown view is now the timeline view rather than it reverting to the visualisation view. I’ve also added a ‘Select category to highlight’ drop-down list underneath the top-level timeline. Simply select a category to highlight all occurrences of that category in the timeline, or select ‘none’ to remove the highlighting. I also noticed a bug whereby navigating from the timeline or table view to the card view at top level and then selecting a category to highlight in a card would result in a ‘page not found’ error and I’ve fixed this too.
I’ve updated the search page to split it into two tabs, one for ‘quick’, the other for ‘advanced’. The ‘advanced’ tab loads by default if the user navigates to the search page from a ‘refine results’ link or when selecting to clear the search boxes. In the ‘browse’ page I’ve fixed the CSS to ensure that moving the mouse over a level 2 heading the text goes white and is easier to view. I’ve also made it so that the ‘plus’ icon turns white as well instead of staying grey. Links in general have now been given a colour rather than being the browser default. They are now green by default and magenta when hovered over. Updating this led to a lot of unwanted messing up of other buttons and links in the site so it took quite a while to track all of these down and fix them.
There were some issues with the timeline SVG download files. They were getting cut off when downloaded because the height and width properties were not being properly set. This has been fixed now. SVG downloads of the timeline now appear in full colour in Inkscape rather than just monochrome. There were some stylesheet errors that stopped the colours working, although IE managed to cope with the errors just fine. All validates now (except for elements having titles when they’re not allowed them in the SVG standard, but we need titles in order to get the hover-over tooltips
I’ve also fixed the top-level timeline metaphor card heading, which was displaying ‘-2’. It now displays ‘Metaphor Connections’ as a title.
I also realised this week that the quick search facility for searching historical thesaurus words had broken. It was never returning any results. This happened due to a change I made to the HT database a few weeks ago – Marc wanted to ‘MMCat’ column moved out of the HT category table and put in its own linking table instead. I thought I’d found all the places in the MM site where this update required sorting but missed the HT search. It’s working now.
Category names in the metaphor cards are now clickable and as requested by Ellen and the team, I’ve removed the buttons that lead to views showing all other categories linked to the two in the card. It’s a bit of a shame to lose these views but the options were a potential source of confusion for users.
Finally, I created a ‘favicon’ for the site this week that appears in the browser tabs. It’s a simple green circle like the visualisation with a yellow ‘M’ in it. The yellow is the same as that used when metaphor connection lines are highlighted in the visualisation and the idea was to make the ‘M’ look like lines connecting up different parts of the circle.
So that’s everything ticked off my list. I still need to implement some sort of breadcrumb or history feature, which I’ll take a look at next week, but other than that everything has really come together rather well.
I did a few non-metaphor related jobs this week, including giving some advice to Johanna Green about a project she’s putting together involving Special Collections, communicating a little with Jane Stuart-Smith about the bid she’s working on and doing some AHRC related duties. I also had a very brief meeting with Susan Rennie about her Scots Thesaurus project. At this meeting Susan gave me an old CR-ROM version of Johnson’s dictionary. It was made in the early 90s and won’t work on modern computers and Susan wondered what could be done to access the content. The software it uses is something called ‘DynaText’, which is proprietary software made by a company that went out of business in 2002. As far as I can tell, it’s not possible to extract the underlying SGML files from the CD-ROM as they seem to be stored in a proprietary, binary format rather than plain text. This means it’s probably not going to be possible to do anything with the data other than get access to it through the old interface. To repurpose the data you’d have to get the source files from whoever produced the CD-ROM.
I tried to get the ‘install’ command to work on my Windows 7 PC using the available compatibility options by right clicking on the ‘install.exe’ filename then selecting ‘properties’ then choosing the ‘Compatibility’ tab and selecting the mode. But even with the mode set to Windows 95 the installer just gave an error.
I then looked into using a virtual machine. This is a piece of software that emulates a computer, through which you can install older operating systems and access them through a window on your PC. I followed the instructions here: http://www.makeuseof.com/tag/forget-end-life-woes-windows-8-xp-mode/ and managed to get a Windows XP operating system running using the ‘VirtualBox’ software. Through this I managed to run the CD-ROM installer and get access to the dictionary as you can see below:
I’ve taken Monday off next week so I’ll be back in the office on Tuesday.
I had fully recovered from my horrible cold by the start of this week, which was quite a relief after suffering with it for the past couple of weeks. I spent the majority of this week getting stuck into the Mapping Metaphor bug fixes. I had previously created a document with 30 bugs that had been identified either by me, the project team or through the user testing sessions. They’re quite a mixed bunch of bugs, some of which are pretty simple (e.g. link colours needing changed) while others have the potential to be rather complex (e.g. the endlessly rotating visualisation bug). I’m happy to report that I managed to tick off 16 of the 30 items this week, including a few that were particularly tricky to get to the bottom of.
The one I’m most pleased about fixing is the aforementioned endlessly rotating visualisation bug. This problem cropped up months ago within a few days of me creating a ‘rotatable’ visualisation. There are two control buttons for rotating the visualisation clockwise or anti-clockwise and the user had to press and hold a button to make the visualisation rotate in the required direction. Letting go of the button caused the rotation to stop. Except that in some browsers the rotation didn’t stop, but went on endlessly twirling until the page was reloaded. It turns out that this was being caused by my code not adequately stopping the call to the rotation function. On mouse up my code sent a command to stop the visualisation ‘transitioning’, but what it didn’t do was stop the rotation function from running. In some browsers and operating systems my code successfully stopped the visualisation from rotating (but the ‘rotate’ code would still be running in the background) whereas in other browsers / operating system the ‘rotate’ code just kept on doing what it was supposed to do. Having figured this out I rewrote my code so that when a user invokes a command to stop the rotation the code actually breaks out of the perpetual loop. I also updated the rotate buttons to make the rotation feature a bit easier to use on touch-screens: rather than requiring the user to press and hold a button for the rotate to work I’ve changed it so that one press of the button starts the rotation and a second press stops the rotation, with the button’s icon changing to a square ‘stop’ icon whilst the visualisation twirls round. It all seems to be working well now on a variety of devices, which is quite a relief.
A request that was made during the recent user testing sessions was to have the ‘key’ visible by default. Previously a user had to press a button for the key to appear, but users thought it was unclear what the various colours and lines meant without being able to see the key. I’ve updated the key so that it is always visible until a user closes it. After that the system remembers it’s closed (using HTML5 session storage) and doesn’t show it again on any pages until the user clicks to open it by pressing the ‘key’ button, at which point the key will remain open on all pages until the user closes it again. I think this works quite well.
I also realised this week that the system was missing a required connection type between metaphors. The system previously allowed for three types of connection for demonstrating directionality: A->B, A<-B and A<->B. However, there will be cases where directionality is not known, which requires a connection as follows: A-B. I updated the database, the content management system and the front end to incorporate this type of connection.
Some of the more minor things I fixed this week included ensuring the Metaphor ID appears in the timeline card pop-up and adding the metaphor ID to the CMS table of metaphor connections . I’ve updated the stylesheet so that highlighted rows in the tabular view of the data have a yellow border rather than a yellow background, which makes things a little easier on the eyes. I’ve also made the highlight border in the card view more prominent. I’ve also updated the card view so that it scrolls to the appropriate part of the page when you click on a card’s category to highlight it. It’s also now possible to ‘unhighlight’ a category in the tabular view by clicking on it when it’s highlighted.
If the user had selected a category in a view other than the visualisation and then switched to the visualisation view this selection was not being remembered. I managed to get to the bottom of this issue, which took more time and effort to fix than that one sentence might suggest! Similarly, strength is now remembered when following all links within the visualisation and other data views. This took a lot of work, but during the course of it I also fixed a few other issues too. I’ve also added a popup stating that there is no matching data when this is appropriate. e.g if looking at connections between two categories with strength set to ‘weak’ and then changing strength to ‘strong’ there may not be any connections. This was previously giving a Javascript error but now the error is caught and a warning pop-up is displayed.
I’ve ensured a ‘loading’ icon is now displayed when card and table views are opened as these can take a few seconds to load up. Metaphor strength now appears in the card view when looking at cards with strength set to ‘both’ and random sample lexemes now appear as links to a search for the work in the HT. The erroneous table heading that was appearing in the tabular download file has been removed and time periods now appear in this file rather than exact dates. I’ve added in some arrow images for representing direction in the tabular view as opposed to just using ‘<‘, ‘>’ etc. and sample lexemes now appear as a new column in the tabular view. They also appear in the download file.
I’ve also fixed a number of other bugs that I became aware of whilst addressing the ones above, such as a Javascript error that cropped up when expanding a L2 category when a L3 category was already selected. So all in all it was a particularly productive week of bug fixing. I should be able to fix the remaining bugs before the project launches on the 25th.
Other than Metaphor duties, I spent the best part of a day this week going through Jane Stuart-Smith’s ESRC bid documentation and making additions and commenting on her data management plan. I sent this off to her with some further feedback on Tuesday morning but I’ve not heard anything back since. I also gave some further advice to Bryony Randall about her bid and spent a little time working on the DSL tasks. Ann is going to be busy for the next few months on another project which means I’m not really going to be able to proceed with the majority of tasks as they require input from her so we have agreed that other than the two tasks that I’ve more or less completed already we’ll wait to tackle the other tasks until she is available again.
A bit of a short report this week as I was still suffering rather badly from a cold that I was struck down with last week. I had hoped I would be free from it by Monday but unfortunately it got worse not better and I really struggled to get through Monday and was then off on Tuesday and Wednesday. I managed to come back to work on Thursday but I really probably shouldn’t have as I was still full of the cold and unable to really get much in the way of meaningful work done. By Friday I was mostly ok, but was still struggling by mid-afternoon.
Anyway, despite all of that I did manage to get a few things done. On Monday we had what has turned out to be the penultimate project meeting for Mapping Metaphor before the project officially comes to an end at the end of March. I spent some time before the meeting looking at different possible screencast software that we might use to make some video tutorials for the website. The free software I’d used for a project when I was still working for HATII was Camstudio (http://camstudio.org/). It’s free, but it’s one of those free pieces of software that tries to make you install a load of unnecessary (and potentially dangerous) ‘bloatware’ at every opportunity, including dodgy ‘download’ links in the page and several ‘click yes to continue’ prompts during installation that actually install other software unless you press ‘cancel’. It’s all really quite unpleasant. I managed to get it installed and working (which required the installation of further codecs plus a Visual C++ package from Microsoft) but at the meeting Marc mentioned that the Learning and Teaching people have a license for Camtasia (http://www.techsmith.com/camtasia.html) which is supposedly a better piece of software (but isn’t free) so we will probably just try to use this.
I helped out with a few requests from the Burns people this week, including Pauline who needed a few tweaks to her Bawdy Chapbooks website and Gerard McKeever, who has recently started working for the ‘Editing Burns’ project. I also had email conversations with Susan Rennie and Bryony Randall about their respective projects. I spent a few hours working with Fraser Dallachy to fix a few quirks in the Historical Thesaurus’s ‘version converter’ feature. The converter wasn’t supplying results for some level 2 categories when converting from version 1 to version 4. We went through the reasons for this and eventually came up with a solution which involved adding in a small number of over-ride checks to the converter to present conversions that were not actually possible with the data we have in the database. It seems to work quite well.
I also continued with DSL redevelopment work for a little while this week. I managed to get predictive searching working with the advanced search for headwords. This is something that I’d been meaning to get working since shortly after the launch. Now when a user selects a headword search in the advanced search and then begins typing a word into the ‘word’ box the predictive search queries the API and displays a drop-down list if possible matches that the user can then select. Selecting one then automatically runs the search. This takes into consideration the ‘source dictionary’ the user has selected and the ‘match type’ too. It all appears to work fine, although for now it is only available through the development version of the site rather than the live version.
My final task of the week was to begin to look over the materials for Jane Stuart-Smith’s new ESRC bid. I can’t go into any detail about it here, but I will be helping Jane out with some of the technical aspects of the bid over the next few weeks.