Seeing speech
The resource provides ultrasound tongue imaging (UTI) and lip video of speech, magnetic resonance imaging (MRI) video of speech, and 2D midsagittal head animations based on MRI and UTI data. Find out more
Month: January 2015
Week Beginning 26th January 2015
This was another week that was mostly split between Mapping Metaphor and the Burns projects. The launch of the Burns Highland Tour map for Burns Night last week went down very well and there was quiet a sizable spike in visitors to the website. I started to investigate always visible labels on the map last week but ran into a few problems getting them to actually display the pop-ups when clicked on. I spent a few hours this week trying to get to the bottom of this problem.
The static label in the leaflet.label example page wasn’t working as a clickable label because the leaflet.label CSS file specifies labels with ‘pointer-events: none;’. The plugin has an option ‘clickable: true’ (the default is false) but setting this to ‘true’ doesn’t seem to override the pointer-events setting so I had to manually remove it from the CSS file.
Setting ‘clickable: true’ doesn’t seem to do anything anyway, even when pointer events are allowed. I had to add in an ‘onclick’ event within the label contents to trigger anything. Even though this event was firing and the leaflet openPopup function was being triggered (causing the map to reposition to make room for the popup) no popup was being displayed.
What I realised is that the onclick event was executing before the popup contents could be processed. I added in a ‘setTimeout’ before executing openPopup and now the pop-ups appear (at least they do on my PC – it’s possibly not so good relying on the timeout as a slower PC may still not have processed the popup even after the timeout).
The end result uses markers rather than circles (using circles would require another hack of leaflet.marker as documented here http://stackoverflow.com/questions/15543141/label-for-circle-marker-in-leaflet) . Labels are visible at all zoom levels, which does make things rather cluttered when zoomed out. I investigated whether it would be easy to specify a zoom level beyond which the labels are visible, but the author of the plugin advises against this (see here: https://github.com/Leaflet/Leaflet.label/issues/83). My test map with static labels and markers can be viewed here: http://burnsc21.glasgow.ac.uk/nls-maps/highland-labels.html (note that this is a temporary URL and may stop working at some point. Note also that not all pop-ups contain much in the way of content in this version of the map). I still need to speak to Pauline to see whether she actually wants this version of the map to replace the current version.
I also started working on the interface for Pauline’s new ‘Bawdry in Scottish Chapbooks’ project. She had emailed me a page image featuring a thistle that I’ve managed to build an interface around. It’s only a first draft and it didn’t take long to make. The website isn’t live yet and I will include the URL in one of these posts once it’s publicly available.
The rest of my week was spent on Mapping Metaphor duties. There was a project meeting on Monday at which I demonstrated the new timeline interface. People were generally quite positive about it and gave some good feedback and suggestions, which I acted upon in the days following the meeting. It was agreed that the timeline columns should have some sort of count visible so I added a total above each column, which is a really useful addition. People were getting confused by the use of 6 different colours for the timeline dots (e.g. cyan was used for connections between a category in the external world (green) and the mental world (blue). It was decided that just the already established three colours (red, green and blue) should be used, with each marker given a colour based on the target category only. It was also agreed that the aggregate view data should not appear in the same timeline as individual metaphorical connections as this was just confusing things.
I reached the conclusion that the aggregate timeline view was misleading and really quite useless. What each square represents is the fact that there is one or more metaphorical connection from a level 3 category within the level 2 category that the user is currently looking at (e.g. 2D Emotion) and a level 3 category within another level 2 category that appears in the label (e.g. that 2D14 Pride has 5 strong connections to categories within 1N Movement). The position of the square in a time period means that the earliest of the connections between that specific level 3 category and the other categories takes place within that time period. So with the above example, the square for 2D14 and 5 connections within 1N appears in the OE column but this doesn’t mean there are 5 connections from 2D14 that were first noted in the OE period, just that that’s when the first was noted. Clicking on the square brings up the aggregate metaphor card view with the option to expand 1N movement. This then presents the timeline with all of the connections from 2D and 1N plotted on it, with the 5 for 2D14 appearing within this across the different periods without any highlighting or anything. This is all very confusing and Ellen and I spent some time trying to think how this could be handled better.
On Thursday Ellen, Marc and I had a meeting to discuss the aggregate timeline view, but unfortunately it didn’t go very well and instead went off on a tangent about the validity of the timeline view in general. After the meeting Ellen and I agreed that we would drop the aggregate timeline view entirely (for the time being at least) and only present a timeline view when Level 3 categories are being viewed. I do still want to try and come up with a top-level timeline view that can maybe represent all of the level three connections over time, but that is still going to need a lot of further consideration.
Also this week I updated the metaphor card pop-up and the metaphor card view to replace the in-card timeline with a version that uses the same time periods as the timeline view. I’ve had to update the layout as the text was too long to fit in the old layout (previously the points were individual years e.g. 1250 whereas now we have a range e.g. 1250-1299). The new layout also has ‘Old English’ covering everything pre 1150. It’s possible that the new version is too prominent or it may look like you should be able to click on or drag the highlighted period so it may need further work.
I also noticed that the tabular view of the data was still displaying specific years rather than a period so I updated this too. It is still possible to order the table by the date column and have ‘Old English’ appear first, even though alphabetically / numerically it no longer is (getting that working took a little time).
I also spent a bit of time working with the metaphor data. At Monday’s meeting Marc had suggested that the SAMUELS ‘thematic headings’ for each metaphor category could be appended to the ‘keywords’ column for each category so I wrote a little script that extracted these headings, checked the keywords field to see if the heading was present or not and then appended the heading if it wasn’t already there. I encountered some problems with the categories that were previously classed as ‘H27’, which have now been split into several categories. The thematic headings were all being added to one of the new categories instead of being split across them all. Thankfully Ellen noted which heading should belong to which category and I managed to address the issue. I also made a start with replacing all of the metaphor connection data with the updated data that I’d created for Flora and Ellen just before Christmas, but again ran into some ‘H27’ issues. Ellen is still working through these but hopefully next week I’ll get the new data.
Week Beginning 19th January 2015
In the run-up to Burns Night there was quite a lot of Burns related activity going on. Over the course of the week Pauline sent me a bunch of texts and images for the ‘interactive highland tour map’ that we were producing. This map plots the rout of Burns’s highland tour on a map from around the time the tour took place, with various location pinned to the map linking to information about the significance of these places within the context of the tour. The base map was provided by NLS and their digital maps specialists also plotted the route on the map using the Leaflet.js library. I then tweaked the interface for this and added in the pop-ups containing Pauline’s text and selected images.
The main tasks I had to do were to update the map visuals (mainly just changing the colours of the route line and the location circles). I also switched the order in which the line and the circles were added to the map so that the line is added to the map first and the circles are then overlaid as having the line on top was interfering with mouse interaction with the circles. I then added in the leaflet.label plugin so that labels appear when the user hovers over a map point. I then removed the map boundary as pop-ups at the top of the map were extending beyond the boundary and getting cut off and I added in two classes of popups, a standard one for shorter entries and a longer one with a fixed height and a scrollbar for longer entries. You can see the end result here: http://burnsc21.glasgow.ac.uk/highland-tour-interactive/
I’ve also been experimenting with always visible labels to replicate the look of the original maps created for Nigel’s book and I’ve almost got these working but getting the labels to be clickable and opening up the relevant popup is proving to be a bit tricky. The opening of the popup on clicking on the label is rather intermittent. Most of the time it repositions the map but fails to open the popup but occasionally it does open it. I’m going to continue to look into this next week.
Also this week I had another request from Fraser for some updated Historical Thesaurus category codes for the SAMUELS people in Lancaster. They need versions of the codes that match the ones used by the OUP people, which I created before Christmas. However, Fraser had uncovered some further changes that were made and I needed to create and run a few scripts that could replicate these. I managed to get it all done and hopefully Lancaster have all of the data they need now.
I spent Friday morning this week attending the recruitment and selection training course, meaning I can now sit on an interview panel if the need ever arises.
The rest of my week was devoted to Mapping Metaphor duties once more, continuing with the timeline interface. I managed to fully integrate the timeline view with the other views available, including providing options to download the timeline visualisation as an SVG file, and adding in a key. I’ve also integrated the metaphor cards with the visualisation too, allowing a user to click on a point to view the metaphor card or the aggregated metaphor card. This also involved updating the way the links within the cards worked to ensure that clicking on them (e.g. to expand a category or to view different connections) performed the actions in the timeline view rather than reverting to the visualisation view. I also looked into dynamically updating the height of the timeline depending on the tallest column in the visualisation. I got this mostly working now, but I would still like to split the line of metaphor connections into two lines within the same date column when there are more than a certain number of them, primarily to avoid the Old English section being so much longer than other sections in a lot of cases. I’ll need to spend a bit more time looking into this. I did begin to implement the timeline view of the ‘top level’ data but I haven’t quite figured out how best to go about this. If we just take the earliest metaphorical connection within each aggregate category to be the point that gets plotted then we end up with every point being in the OE section, which isn’t much use. I spoke to Ellen about this and she has suggested some alternatives, which I’m going to investigate next week.
Week Beginning 12th January 2015
I completed the migration of the audio and video files for the Scottish Corpus website this week. The Quicktime plugin is no longer used and instead audio uses the standard HTML5 Audio tag while video uses the HTML5 Video tag. It took a while to get all of the media files migrated to newer formats and to get all of the Javascript updated so things like synchronisation and ‘jump to’ continued to work, but that’s it all done now. It now means that the media files can be heard in all modern browsers on all devices (including smartphones and tablets), plus the server no longer needs to run the Darwin Streaming Server, which simplifies the server side of things and should make migrating the system to a new operating system (which Chris intends to do later this year) a lot easier. You can see an example of the new interface here: http://www.scottishcorpus.ac.uk/document/?documentid=805
I had thought I would be spending some of this week working on Burns related things, but unfortunately Pauline is still not well so this has been pushed back to next week. We should hopefully still be able to get the timeline and the tour maps available in time for Burns’ Night though.
The rest of my week was spent on Mapping Metaphor duties. I completed a first draft of the advanced search last week and I received some feedback from Ellen. She suggested that rather than jumping directly from the search form to the visualisation it might instead be better to present users with a list of categories that match the user’s search criteria to enable them to select which of these they’re interested in. This would get around the problem of someone searching for a concept such as ‘light’ being faced with hundreds of categories when the majority of them might not be relevant. I added in such an intermediary page, and it works a lot better. Matching categories are listed with a checkbox beside them and by default all categories are ticked. An option is available enabling a user to select or deselect all, in addition to allowing individual categories to be ticked / unticked and after pressing the ‘continue’ button the search runs.
After completing this task I began to investigate how the timeline might operate. I decided against using an off the peg timeline solution (such as Timeglider, which I had previously used for a mock-up). This was because all the points plotted on the timeline will be rounded into 50 year chunks so will be stacks of points, plus we don’t need to be able to zoom in on specific periods – 50 year chunks is a granular as we are going to get. Instead I decided to implement something using d3.js, based on some code from a scatterplot chart (http://bl.ocks.org/mbostock/3887118). Our timeline would have time as 50 year chunks across the bottom, points of two sizes (representing strong and weak metaphors) and 6 colours representing which top level categories the connections belong to: 1 (External World) is green, 2 (Mental World) is blue and 3 (Social World) is red, so 1-1 is green, 2-2 is blue, 3-3 is red, 1-2 is cyan, 1-3 is yellow, 2-3 is magenta.
For my first attempt I didn’t round the dates to 50 year chunks and made the position of the dots on the vertical axis random, as you can see from the following screenshot:
I then made two further tests where the dates were rounded and the points were ‘stacked’, one with dots, and the other with rectangles, as you can see below:
Ellen preferred the dots to the rectangles so that’s what I focussed on. The next stage was to make the diagram dynamic (i.e. working with any data based on the user’s selection) and to integrate it with the other data view options. Ellen also informed me that there would be no 50 year chunks before 1150 and everything before this point should just be grouped into an Old English ‘bucket’. I also realised that in order to deal with the datasets as used in the other data views I would have to be able to plot aggregated points as well as individual metaphorical connections. This meant updating the data output files so that aggregated data (e.g. that there are connections from 1B03 Age and the level 2 category 2A Mental Capacity) also returned a start date (e.g. the earliest date when 1B03 had a connection to any level 3 category within 2A).
I decided that aggregated points should be displayed as rectangles on the visualisation. Having two different shapes actually proved to be rather tricky to implement as previously the entire dataset was cycled through from within the ‘add circle’ function and the way the code was set up meant that adding a little ‘if’ statement to position a different shape instead was not particularly straightforward. However, I managed to figure out a way to do it and by the end of the week the ‘Change View’ option for the timeline was operational for the ‘drilldown’ data, producing something like the following:
There is still a lot to be done, though. For example, sometimes there is too much data for a column and it gets cut off the top of the visualisation. I need a way to manage this. I also need to implement the key, facilities to open the metaphor card when a data point is clicked on and the download option and the top-level view as well. I’ll continue with this next week.
Other than the above I also had a meeting with Marc this week, which was mainly an opportunity to talk about projects and my work levels and things like that.
Week beginning 5th January 2015
This was my first week back after my Christmas holiday. The holiday was lovely but despite being two whole weeks it did seem to zip by rather too speedily. This week I split my time primarily over three projects – Mapping Metaphor, Burns and the Scottish Corpus. For Mapping Metaphor I returned to the task I had begun before Christmas but hadn’t quite managed to complete – the updating of all of the database queries in the site from dedicated MySQL functions to PDO style functions. This was a rather tedious task, but I managed to get it completed and thoroughly tested this week. Using PDO will make the site more future-proof and should also help boost security too.
The next task I completed this week was the Advanced Search functionality. The Quick Search performs a search for categories, while the advanced search is intended to search for metaphor connections. A document stating what search facilities were required had been prepared a while ago and I worked through this, implementing all of the desired search options. I also implemented wildcard searching for both the quick and advanced searches. This facility is the same as is offered on the Historical Thesaurus website, enabling a user to place an asterisk at the start and/or end of their search term in order to run a partial match search for the beginning, middle or end of words. The advanced search options that are now available enable a user to search for category names or keywords, select one or more level 2 categories in which to search (with an option to state whether the search should be limited to connections purely within these categories or just involving one of the categories), specifying a date of metaphor inception, metaphor strength and directionality. I think this covers all of the search options that the project requires, and will users to perform searches such as ‘show all of the bi-directional, strong metaphors that began in the 19th century’ or ‘show all strong metaphor connections within 1E Animals and 2D Emotion that began before 1200’. The advanced search results are displayed using the visualisation interface, and as with all of the metaphor browse options, users can also then choose to display the results as a table or cards. I think it’s working rather well at the moment, but I’m still awaiting feedback from the team.
After completing the advanced search I made another few tweaks to the site, for example I added in a print CSS file so the pages now print a bit more nicely. I also made the interface ‘responsive’ – i.e. it resizes to fit different screen widths. This works ok, but the visualisation itself still doesn’t resize. I’m not sure I am going to make it resize, as squashing it down to fit on a small screen it going to impede the usability of the visualisation. I think it might be preferable to just have the visualisation scroll off the user’s screen and make them move their view. The only major tasks I have left to develop for the project now are the timeline view of the data and the content management system.
For Burns I began to work with the leaflet.js based historical maps of Burns’s tours that the NLS people had very kindly prepared for us last year. Just before Christmas Pauline had sent me a document containing a few of the items that we will ‘pin’ to the map so I added these to see how they will look. I’ve made two styles of ‘pop-up’ for the map – a default one that is used for shorter items and a bigger one that is used for longer items. The latter featured scrollbars so that long items can be read without the pop-up spilling out over the map. It’s all looking promising but I need to get more content before I can proceed further. Pauline was off sick this week so couldn’t get this to me, but hopefully all can be completed next week. I also tweaked the timeline as well, removing the image bar at the top of the timeline as this was taking up too much space and didn’t really add much to the look of the timeline (images of old pages tend to look the same as thumbnails – boring beige squares). We hope to launch both the timeline and the interactive tour maps next week.
My final task of the week was to being phase two of the redevelopment of the Scottish Corpus website. This phase involves migrating the audio and video of the site away from using the proprietary Quicktime plugin and instead using standard HTML5 audio and video. Currently audio and video files open up in a new browser window and the files only work if the user’s computer has the Quicktime plugin installed. This is causing problems for some users, plus it makes accessing the files on tablets and smartphones rather tricky. In migrating the files to HTML5 I had a few tasks to consider:
- Migrating the audio files to MP3
- Replacing the ‘new window’ with a jQuery UI in-page dialog box
- Adding in the HTML5 audio / video player
- Updating the site’s Javascript so that the synchronisation of audio and the highlighting of transcript sections continues to function.
I wasn’t sure how long these tasks might take, or even if I’d be able to complete them, but thankfully my experiments were rather successful. I set up a test version of the ‘document’ page for one of the documents that had audio and a synchronised transcription and then added the necessary jQuery UI code to enable a dialog box to appear when the ‘play audio’ link is pressed. Within this I added the HTML audio tag, linking to my newly migrated MP3 file. I did have to do some fairly major reworking of the Javascript that synchronises the audio to the text, but I managed to get this working. I’m hoping to be able to replace the old Quicktime version with the new HTML5 version next week, all being well.