This week I continued to get back into the development duties for Mapping Metaphor and I spent a sizable portion of the week working on the project. On Monday I met with Ellen to discuss the metaphor data and the plans she has for completing a variety of data related tasks and getting this data to me. It was a very useful meeting and we agreed how we should proceed to record and share certain aspects of the data such as the sample lexemes and directionality. For example, it has now been agreed that there may be two different sets of sample lexemes for each metaphor connection if the connection between the two categories is bidirectional: one set for each direction. I also completed my mammoth ‘to do’ list of outstanding technical related tasks and disseminated it to the team. There are around 50 items on it and some of them are rather large (e.g. ‘develop a timeline of metaphor inception’) so I’ll certainly be kept busy for the next few months!
I managed to return to the actual code of the Metaphor website and visualisations this week and re-familiarised myself with the workings of d3.js. I began looking into how transitions (i.e. animations between states) are handled in d3 with a view to updating the ‘rotate’ buttons on the website. Currently users have to click on a button to rotate the visualisation 10 degrees left or right. To rotate the visualisation more than that the user must click multiple times, which is a bit clunky. I figured it would be much more appealing if the visualisation rotated smoothly for as long as the user held down the rotate button. I managed to get such a behaviour working this week, which was very satisfying. However, although it worked perfectly in Firefox on my home PC, when I tried it in Firefox on my work PC the visualisation often just kept on rotating endlessly, even when you stopped pressing the rotate button. Unfortunately this behaviour also happened in iOS and Android browsers so I’ll definitely need to do some further work on this. Watching the visualisation spin round endlessly is rather hypnotic, but not really a whole lot of use.
We had the first Metaphor team meeting since May this week too, and it was good to catch up with everyone again. Almost all of the data has been coded now, which is great. I got a few comments on my ‘to do’ list and some possible timeline approaches I’d been working on too. I continued to work on updates to the metaphor visualisations after the meeting, this time focussing on getting circles included in the nodes. In addition to giving nice ends to the connecting lines, these circles will hopefully server a more practical purpose too as I’m intending to get their size to update to reflect the number of connections contained within an L2 category. I managed to get the circles included and have them change colour when a node is clicked on, but I haven’t quite managed to get the size updating dynamically yet. I hope to get this working next week though.
Other than Metaphor duties I made some further tweaks to the DSL website as a result of feedback from users. I also created a development version of the site that connects to Peter’s version of the API so he can test out updates to the API without running the risk of breaking the public version. I responded to a couple of digital humanities related queries that came in and tweaked the STELLA Apps a little for Marc, who was hoping to use them for teaching this week. I also helped provide feedback on Stevie Barrett’s DASG website, which will be launching next month. It has come on a long way since our early experiments with the Open Corpus Workbench software and is looking really great.
Last week saw the launch of the Dictionary of the Scots Language website, which all went very well. Work on the DSL has taken up quite a lot of my time over the past few months and now that it is out of the way I’ll be returning to the Mapping Metaphor project, reworking the ‘Beta’ version of the resource that I completed for the colloquium all the way back in March. However, although DSL has launched there are always further tweaks that need to be made following the launch, and I spent a little further time this week doing just that. The print CSS file was printing out too small for some people so I updated that. There was also a small bug that was encountered when a user prints an ‘about scots’ page when they are already scrolled down the page and I’ve fixed this too. I fixed some other minor bugs that people had encountered since the launch too, such as the positioning of the ‘I’ image in the advanced search on medium width screens, the searches submitting with their default text (something I’d previously fixed but had inadvertently broken again when I added a facility to deal with accented characters), updating the font colour of the ‘geo’ element which was a little too pale for some people to see easily and making the in-entry search results default to showing 10 rather than 20 results when the browser height is less than a certain value, thereby fixing the bizarre screen jumping issue encountered with short entries that I thought I had fixed by making the ‘browse’ feature shorted on less tall screens.
With these tweaks out of the way I still didn’t quite manage to get on with Mapping Metaphor tasks as I had some further updates to make to the Historical Thesaurus database – a category had been given the wrong name and some other category names needed updating too. Fairly small updates, but it did mean I had to re-export the updated tables as new CSV files and also re-generate the ‘all tables joined as one’ Access database that I had made for offline research. I also completed a first draft of the Technical Plan for Katie’s project and engaged in some email discussion about the technologies that would be employed and how the project workflow would fit together.
After all this was done I could begin to get back into the Metaphor project. It’s been quite a while since I was involved with the project so I spent some time this week getting to grips with the website and visualisations I had produced and familiarising myself with the project and its data again. Months ago I had promised to write a further blog post about the visualisations for the Mapping Metaphor project blog and I figured a good way to get back up to speed with the project would be to write such a post. It was a good idea as it was really useful to revisit my earlier experiments with the d3.js library and to try and put into words why (and indeed how) certain approaches were taken. I ended up writing two blog posts rather than one – the first focussing on my early experiments with d3 (which can be read here) and the second looking at the ‘Beta’ version of the website and visualisations that I completed for the colloquium (not published yet). Next week I will go through all of my notes and the feedback from the colloquium and write a gigantic ‘to do’ list and will hopefully get back into the actual development work.
There was quite a lot going on this week, with the big thing being the relaunch of the DSL website (www.dsl.ac.uk). I spent quite a bit of time making final tweaks to the site – updating content and fixing small bugs that I’d noticed at the last minute. We updated the DNS to point the domain at the new website on Wednesday and once the update propagated everything went well. I added in the Google Analytics code and removed the password protection on the site and there it was – available for all to use. I was giving a demo of the new website at the launch on Friday so spent some time writing and preparing that, plus I helped Jean get the rooms ready for the launch too. The launch itself went very smoothly – no problems with the website or the API and the audience seemed both interested in the new site and pleased with how it worked. Many people came up to me afterwards to offer words of congratulations and after the launch I was given a rather splendid bouquet of flowers. I can honestly say that that is the only time I’ve ever been given flowers. Getting them home was a bit tricky but they do look very nice.
Other than DSL work I did some further Historical thesaurus work. I attempted to upload the SAMUELS thematic headings again but uncovered some further problems with the spreadsheet I’d been given, namely category numbers in it not matching up with anything in the online database. Thankfully Fraser got to the bottom of the problem and sent me a fixed version of the spreadsheet and I managed to get all of the headings uploaded. After a couple of further tweaks I managed to get the database ready for export as CSV files for the people at Lancaster to use. I also created a new Access version of the data with all of the tables joined together again like in the old HT database. This should be of use to researchers who want direct access to the data in a flat table.
Other tasks this week included giving some advice to Maria Dick on a bid she’s putting together and starting to write the technical plan for a bid Katie Lowe is putting together. I can’t really say too much about this but I hope to have a first version of the plan sent to her next Monday. Oh, and I also reviewed another technical plan for the AHRC, which takes more time than you might think.
This week marks my second anniversary as the School of Critical Studies’ Digital Humanities Research Officer. I have to say that is has been a really excellent year and I’ve been involved in some amazing projects over the course of it. Redesigning the Historical Thesaurus was an amazing opportunity, as was creating a new interface for the Scottish Corpus. Working towards the relaunch of the Dictionary of the Scots Language (which takes place next Friday) has been really rewarding too. It has been wonderful to gain experience developing apps, and seeing the three STELLA apps I created available through the Apple App store is very rewarding. I have also really got to grips with technical plans and AHRC bids in the past year, having created plans for numerous projects, and becoming a technical reviewer for the AHRC is something I’ve been hoping to do for a long time. Developing visualisations for the Mapping Metaphor project has also been a tremendous opportunity to learn more about data visualisation techniques, an area that is increasingly important. Tons more than this has happened over the course of the year, but I would say the above are the highlights. I’ve probably forgotten to mention some other amazing things that happened this year, but that’s what these weekly reports are for after all. Here’s hoping the coming year has lots more exciting projects in store for me!
Anyway, back to the current week. This was mostly split between two projects: DSL and the Historical Thesaurus. For the HT I had to return once again to the problem of duplicate categories. I spent a large amount of time trying to figure out how best to programmatically fix the issue, including writing numerous little scripts that listed some of the troublesome rows, but I just couldn’t find the key that would let me fix things and it looked for a long time like manual intervention would be required to fix the errors relating to the several hundred erroneous categories. Marc, Fraser and I met on Monday afternoon and had a bit of a brainstorming session and we finally figured out how to pick out the erroneous rows. Many of the problematical categories were empty and incorrect main categories that had the same heading and number as a valid subcategory, but a different part of speech. I created a little script that would pick out all main categories that had the same number and heading as a subcategory but a different part of speech and was able to identify and delete several hundred dodgy main categories (checking first that they were empty). What a relief! I then wrote another script to identify any subcategories that didn’t have a main category, with the bulk of these getting fixed by the removal of a duplicate t7 number.
For future reference here is a summary of what caused the errors we have been dealing with over the past week or so:
1. At some point during the initial redevelopment of the HT database I imported a bunch of empty categories to fill the gaps in the hierarchy. During this process something went wrong and some subcategories ended up appearing as main categories with erroneous parts of speech. These new main categories were all empty and were duplicates of valid subcategories. We have hopefully now tracked all of these down (there are certainly no duplicate category numbers in the database any more).
2. During the recent renumbering process some categories were shifted up a number and any that had a t7 number should have had the t7 column cleared of data. Unfortunately I overlooked this, leading to the t7 column containing the same value as the t6 column. As far as I can tell these have now all been sorted.
After getting the data in order I proceeded to integrate the thematic headings that Fraser has been preparing for the SAMUELS project. Rather than add these as new columns to the category table, I have created a new table for this data. This is to avoid duplication of data: There are just over 4000 thematic headings and these are applied to almost a quarter of a million categories and representing the 7 new thematic heading columns would therefore have resulted in a lot of duplication. I managed to get the 4000 headings uploaded and add in the links to the categories, but as always there were some issues. I’ve emailed Fraser about these and I should be able to get the data all finalised next week.
On to DSL now. This week I managed to get through pretty much all of the outstanding items on my ‘to do’ list and the site is just about ready for its launch next week. I had another sizable chunk of text to migrate from the old site to the new one this week – ‘The Scots Language’. Thankfully the markup of this section of the old site was nowhere nearly as bad as the ‘History of Scots’ section and didn’t require a lengthy and tedious complete retagging, just some tweaking with regards to paragraphs, headings and footnotes. It still took some time but it’s all done now. I also added in maps of Scotland and England from the original printed dictionaries as pannable, scrollable OpenLayers.js map images, which I think is rather nice. I also added in the content of the ‘cite’ boxes too. The last bits of content were also added, for example the abbreviations page.
I also completed some more interesting, more ‘developery’ tasks this week too, such as updating the ‘add yogh’ feature so that it adds the yogh character at the cursor position in the text box rather than at the end of the text and adding in the new ‘supplemental’ information that now gets returned for certain entries via the API. I also added in some error messages and warnings too. If a search reaches the maximum number of results permissible by the API, the API now returns an HTTP ‘partial content’ error code. My scripts now recognise this and display a warning. Similarly, if the API is offline it will send a 503 server error and in such an event the front end now displays a warning message (here’s hoping this one won’t be seen very often!). I think we’re pretty much all ready to switch the DNS to point to the new site, which will take place some time next week.
Some other tasks I carried out this week included finally getting round to completing my PDR form and sending it to Jennifer, speaking to Pauline Mackay about another couple of projects she’s hoping to get funding for, agreeing to review yet another AHRC bid and fixing the maps on the Burns website, which had stopped working due to OpenLayers changing the location of their scripts (I’m using local versions now so that won’t happen again).