For Sound and Vision, in the context of the Dutch Open Data initiative “Nederland opent Data” (The Netherlands Opens Data), I created the basis for the demo that is described in this post. The demo shows how you can play a video in an enriched context, by linking open data sources to terms that are found in speech transcripts rendered from videos. For the Code Camping event, organized by Open Cultuur Data (Open Cultural Data) I extended the demo with newly linked data sets.
Basics
The starting point for this demo application was the reuse and linking of data sets to the Open Images collection, which contains more than 1,500 freely (re)usable videos containing mostly old news items from the ‘20 throughout the ‘80. All of these videos are published using Creative Commons licences.
The basis for the application lies in the use of the speech transcripts, which were generated by using automatic speech recognition (ASR) software (from X-MI) on these videos.
The main idea for the demonstration is to contextualise videos while they’re being watched, in order to provide the user with fun, interesting and unexpected background information about the things that are spoken in the video.
For example: when Philip Bloemendal (the presenter of the news items) – in a video titled: ‘Large parts of Holland completely snowed in’ – talks about: ‘(…) but on several places in Drenthe there (…)’, next to the video, several blocks of information about Drenthe (a province in The Netherlands) are shown. Each of these information blocks gets its data from a specific open data source. For the first prototype the data sources used were (amongst others): Google Maps and Wikipedia. To illustrate this some more: in the example where ‘Drenthe’ was recognized as a concept, the Wikipedia block shows an article about Drenthe; in the Google Maps block the map is zoomed in on the province of Drenthe in The Netherlands.
For the Code Camping event, organized by ‘Hack de Overheid’ (Hack the government), I added two new data sets to the demo: the collections from the Rijksmuseum and the Amsterdam Museum.
How it all works
As mentioned, the main building blocks for this demo are the Open Images videos and the corresponding speech transcripts that are used to link the words that are spoken (in the video) to an exact time code. (Note: Automatic speech recognition software is not perfect, which means that not every word in a speech transcript will exactly match the actual words that were spoken).
Step 1
Because not every word in a sentence is particularly interesting, the first step is to filter out stop words from the speech transcript, such as: articles, prepositions and verb modifiers.
Step 2
In the second step, a script is run on the remaining words to sort them by ‘importance’. Importance in this matter is calculated by combining a preset word score (coming from a special lexicon) with the frequency the word is spoken. In this way, words with a high score and a high frequency will end up high in the list.
Step 3
After sorting, the words are used, in order of importance, as query input for the GTAA thesaurus (used by Sound and Vision) and also for Freebase. The latter is a Google service and offers a big collection of interrelated concepts, containing descriptions from a large variety of domains. Freebase can be seen as an extensive thesaurus containing information from a large number of areas of expertise.
When, after querying, the GTAA or Freebase webservice yields a concept, it is put in a list of candidates. After processing all the words, this list is filtered using a very simple disambiguation algorithm (i.e. whenever the yielded concept is comprised of more than one word, it is taken out of the list).
Step 4
In step 4, each of the GTAA and Freebase concepts from the list of candidates is used for querying the open data webservices, which are:
Google Maps (only queried for location type concepts)
Each result returned, will be linked to the time code of the (spoken) word from the speech transcript that was used to find the eventual information.
(For those interested: the collection from the Amsterdam Museum has three different end-points: Adlib, OAI-PMH and SPARQL. For this demo, I used the latter, because, unlike OAI-PMH, it does not require to be harvested and indexed before it can be queried. In any case I thought it was a good idea to play around again with the Semantic Web and refresh my SPARQL skills. For the Rijksmuseum, I first harvested the collection from OAI-PMH and then indexed it with SOLR. This way the collection can be searched using Lucene queries.
Step 5
The last step was to send back the time-coded contextdata back to the browser. I do this by using a JSON object, which in turn I use as input for Popcorn.js to generate events. These events are linked to an HTML5 video player and make sure the right (context) information is shown in the different blocks/panels in the user interface.
Because the processing of these five steps takes around 15-20 seconds per video, I store all of the results in .json files. When opening the demo these files are loaded instead of fetching the data live from the web.
There is still a lot to do
The demo shows what can be done by using concept detection (a.k.a. Named Entity Recognition) in combination with open data sources. For several aspects however (significant) improvements can be made:
Better concept detection
The concept detection as described in this demo could be improved much more. For instance, concepts that comprise of more than one word are not recognized, e.g.: ‘Amsterdam Museum’ now yields two concepts, ‘Amsterdam’ and ‘Museum’, but the actual concept ‘Amsterdam Museum’ is not found.
Moreover, specific Named Entity Recognition (NER) services like DBpedia Spotlight should be investigated (having good results for English) in order to improve results. For Dutch however, it seems it’s an ongoing search for a decent (open source) solution.
Selection of relevant sources for the user
Concerning the relevance of the ‘context information’ that is currently shown to the user, there is still much to think about how to make the best selection of data sources. For instance: why somebody who is watching a video about ‘Holland’s oldest steam-powered pumping station’ would be interested in ‘Hens chalice from the Company of Nine’ (found on the basis of the word ‘Gorinchem’, which is a town in The Netherlands) is something to think about.
Optimizing Popcorn.js usage
The demo was made with an older version of Popcorn.js (v0.7) and therefore doesn’t make full usage of all of the latest features and plugins Popcorn.js has to offer. Future releases of the demo will incorporate the newest version (currently v1.1.1).
In any case the demo does show how speech transcripts of videos can be combined with open data sources and how this can enable (mutual) contextualisation of these sources. For the ‘Nederland opent Data project’ this demo will be further enhanced. Any progress of this will be reported here!
Access to the audiovisual content on Open Images is provided under Creative Commons licences. These licenses facilitate the reuse of content in different ways. One of the possible ways media from Open Images can be reused is on Wikipedia. For this purpose the videos on Open Images are transferred to Wikimedia Commons, the online repository where freely licensed media files used for Wikimedia projects like Wikipedia are stored. In the beginning this was done manually, but in the meantime this process has been automated through the Open Images API. Currently, there are more than 1500 media items from Open Images available on Wikimedia Commons. This means that Open Images is responsible for about 15% of the total amount of videos, which makes Open Images the largest supplier of videos on Wikimedia Commons.
The Wikipedia community uses the videos from Open Images to enrich the entries on the Wikipedia. For instance, the English article on the ‘Elfstedentocht‘ has a video of the Elfstedentocht of 1954:
Besides the reuse of complete videos, derivative works (such as screenshots) are also used. These are then for example employed in articles on famous people, for instance in this article on Dutch politician Pieter Oud:
3 million views
The reach of Open Images content on Wikipedia turns out to be substantial. In May 2011 the Wikipedia articles with media items from Open Images were viewed more than 3 million times. This is almost three times as much as the number of views in December 2010. Noteworthy is that the majority of the views are not on the Dutch Wikipedia, even though most of the videos on Open Images have Dutch subjects and are in Dutch. Of the 3 million views a mere 880,000 were on the Dutch language Wikipedia. The remaining 2.2 million views were on Wikipedias in different languages. The five Wikipedias where articles with Open Images content got the most views in May 2011 were:
More than 850 articles on the different Wikipedias make use of content from Open Images.
The article with the most views in May 2011 was Mother’s Day on the English Wikipedia, which was viewed almost 1.5 million views. The video used in this article is used on several Wikipedia sites. Besides the English and the Dutch Wikipedia, it is also used on for example the Tibetan and Persian Wikipedia. The Wikipedia articles containing Open Images media with the most views in May 2011 were:
Conclusion
These statistics demonstrate that offering their material under a free license certainly has an added value for cultural heritage institutions. For the cultural heritage field it is a sound strategyfor opening up their collections to a large audience. It also gives the (internet) community a chance to enrich their projects with historic images. This reuse is of course not restricted to Wikipedia. By offering collections under a free license they turn into a rich source for (re)use fora large number of cultural, educational and creative purposes.
At the moment Wikipedia articles don’t contain a lot of videos (less than 0,1% of all files on Wikimedia Commons are video files). Open Images would like to change this. Therefore, most videos from Open Images are already automatically mirrored to Wikimedia Commons. To stimulate users to use more video on Wikipedia, Open Images will be handing out a special video prize. The maker of the best video uploaded as part of Wiki Loves Monuments will be awarded a 2 year Premium subscription to Spotify, or alternatively an Amazon gift voucher.
Wiki Loves Monuments is a contest organised by Wikimedia, the movement behind Wikipedia. To be eligible for the video prize participants have to upload a video of one or more monuments to Wikipedia in September. The rules are:
The emancipation of the Dutch woman has made a lot of progression during the last century. Women are now actively engaged in the labour market, in sports and in many other places. Polygoon newsreels made a number of reports on women in a man’s world throughout the years. Special in this case is footage of the female aviation pioneer ms. Beppie Versluys. She was the first woman to receive a Dutch pilot license in 1930. In an article in Het Vaderland she talks in detail about her exam and her preparation for her exam. She understands that people might be a little nervous about a female pilot: “It must be an unsafe feeling for many people that their Dutch sky is traversed by not only a woman, but also one of a reckless age.” It wasn’t always easy being a woman during her exam. For example, when she had to fly for an hour at a height of 2000m she had a hard time: “It was very cold and I cried because of the pain in my fingers. (My colleagues probably never did that).” But nevertheless she passed all the tests and qualified as a pilot:
Ms. Versluys never had the intention to make a profession out of flying. But there were women in the 1930s who started working in professions that were seen as male professions. For example, Dutch women were trained to be a fire fighter. With the impending war they had to be able to take over the work from the men. Even though women weren’t thought to be suitable for the job, this didn’t really matter according to Het Vaderland: “If women are suitable for this job – fighting fire is men’s work – should be disregarded, because there will be no other choice.” Polygoon newsreels filmed a demonstration of the female firefighters of the The Hague fire brigade in 1939. Dressed in firegighter suits the women show among other things how to extuingish a fire:
In 1953 the police in Heerlen also had something new, for the first time they started recruiting women to patrol the streets in uniform. According to the Nieuwe Leidsche Courant there tasks would be “monitoring the youth, teaching the youth crossing guards and supervising movie theatres, bars and other amusements.” In the same year Polygoon newsreels went to Heerlen to see the women at work. Besides their normal tasks the report also shows them taking part in lessons in judo to become more resistant:
A report at the steel company Hoogovens in IJmuiden shows how things have changed in the 1980s. In an industry that was dominated by men for a long time, more and more women were employed. After a training the women could work as crane operator with the same pay and career opportunities as the men in the company:
The 1500th video that was added to Open Images is a report from the Polygoon newsreels on the 60 years existence of Philips in 1951. The electronics company Philips was founded in 1891 by Gerard Philips and his father Frederik Philips. In 1951 Eindhoven was decorated with thousands of light bulbs to celebrate the 60 years existence. There was also an unveiling of a statue of Anton Philips. This week it is 120 years ago that Philips was founded:
The former Netherlands Antilles celebrated a special day on October 10, 2010. On that day Curaçao and Sint Maarten became constituent countries within the Kingdom of the Netherlands, a status Aruba had since 1986. The other islands – Saba, Saint Eustatius and Bonaire – became a direct part of the Netherlands as special municipalities. The status of the former colonies is laid down in the Charter for the Kingdom of the Netherlands, which dates from 1954. The alteration made in 2010 wasn’t the first one. In 1974 Polygoon newsreels did a report on a conference on the Charter for the Kingdom of the Netherlands in The Hague. Subject of discussion was Suriname’s desire to become independent. This would become a reality a year later, when the charter was altered for the first time:
In the countryside of Groningen they had other things to worry about in 1975. There the village of Weiwerd had to be demolished to make room for the expanding industry of the neighbouring Delfzijl. Polygoon newsreels made a report on the demolition of the village. The first houses had already been taken down by then and many inhabitants had left the village. The school in the village only had 14 students and 2 teachers left and the church in the middle of Weiwerd hadn’t had a service in months. Polygoon interviewed a woman who hadn’t moved yet:
More than 20 years earlier workers were doing the exact opposite in the IJsselmeer. There they were busy building a new city, Lelystad. In 1950 they started working on the construction of a ring-dike for the future Flevopolder from the working island Lelystad. In the first months of 1954 they hadn’t reached the mainland yet and the winter caused problems. Because of the ice they couldn’t reach the mainland with a boat anymore. And after the thaw had set in, the ice wasn’t thick enough to provide the inhabitants with food and mail via that route. Thus the sixty inhabitants of the island were provided with the necessary goods via a helicopter:
After the 1500th video the collection of videos on Open Images will keep expanding gradually. Later this year we hope to make the 2000th video available for reuse. Amongst the themes that will be added in the future is a theme on women with subjects like female football, female police officers and the feminist movement.
With this blog post we look back on the past year. How did Open Images contribute to an open collection of audiovisual material and stimulate the reuse of it?
In September Open Images launched its open API. Items published on the platform and their descriptions (metadata) and are accessible through an Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). This enables third parties to retrieve the stored metadata and media files in a structured way, making it easy to reuse material from the platform in their own applications (for example to create a mashup).
Video on Wikipedia
Since the start of the project, Open Images has contributed its audiovisual content to Wikimedia Commons to enable reuse of video on Wikipedia, for instance to ‘illustrate’ an article.
At first the ‘donation’ to Wikimedia Commons was a manual process, but in 2010 – in collaboration with Wikimedia Netherlands – we were able to fully automate this process, thanks to the Open Images API. As a resultOpen Images is now responsible for almost 12% of the video content available on Wikimedia Commons, hence being one of the biggest contributors of video that is reusable on Wikipedia.
We are getting more and more insight in the impact of the availability of Open Images material through Wikimedia Commons. We’ve learned that a large proportion is used to enrich over 550 entries on Wikipedia with related audiovisual content. In December 2010, these entries were viewed nearly 1.2 million times. This shows the great potential for the cultural heritage sector to collaborate with the Wikimedia Foundation to reach new and greater audiences within a meaningful context.
New Projects Reusing Open Images
When Open Images was launched in 2009 the material was almost immediately reused within several projects, including the OPEN CITY audiovisual archive of urban life from the Dutch public broadcaster VPRO and the ArtTube video platform about art and design from the Boijmans Van Beuningen Museum in Rotterdam.
In 2010, tens of projects, small and large, were added to the list. Among them Picture War Monuments, a location-aware iPhone app that enriches the on-site visit to war monuments with audiovisual heritage, including newsreel footage and oral history video material on the Second World War available through Open Images. Another notable initiative was Image on a Map (‘Beeld in kaart’), a Google Maps mashup for the educational sector in the Netherlands combining several (educational) video sources – including Open Images – within a map interface. With this interface users are able to filter results based on subject (geography or history), location and time period.
What’s Next?
In 2011 the Open Images platform will receive a major update, with both functional and visual improvements. Part of this update is the realisation of portal functionality, allowing third party content providers to build and manage their own entrance to the platform (think: http://partner.openimages.eu). This will, for example, allow them to highlight their own contributions to the platform and to design their presence on the platform according to their own wishes and branding.
The platform functionality is part of a larger campaign we are organising to attract more third party content keepers to contribute to an even larger and more diverse offering of open audiovisual content through Open Images. This campaign will focus on public broadcasters, regional and local archives and broadcasters, institutional archives and business archives.
Workshop 1 – Remixing and Re-Use of Open Video Collections
In the context of the Open Images project, participants of the Remixing and Re-use of Open Video Collections workshop will get creative with material from the historical newsreel collection of the Netherlands Institute for Sound and Vision, to make their own short videos. Led by Maarten Brinkerink of the Netherlands Institute for Sound and Vision, and facilitated by artists Emile Zile and José Miguel Biscaya, creative students, media producers and video amateurs are invited to start reusing and remixing the growing wealth of open video collections that are made available online for use in inspired and inspiring video creations. During the workshop, participants will first become familiar with the Open Images collection, seeing what is available for re-use, and are then, through the guidance of workshop facilitators, offered a creative toolkit of possible styles and techniques they can use with the material to create and tell their own story using this found footage. The workshop takes place in collaboration with MediaLAB Amsterdam.
Participants should come with their own laptop and lots of creative energy.
Workshop 2 – Animated Gif Mashup Studio
Led by artist Evan Roth, Animated Gif Mashup Studio invites participants to work collaboratively to create a single music video composed of their favorite animated gifs. Animated gifs individually tell the story of one meme but, when archived and mashed together, they can tell the story of the entire Internet at that particular moment. During the workshop, Roth will teach participants about the open source animated gif mash-up software he built, how to search for animated gifs and how to put together their own animated gif compilations. By the end of the workshop, the group will create a music video that will be screened during the symposium and on a browser near you (for reference, see http://evan-roth.com/cream.php).
Participants should come with their laptops as well as any songs and/or animated gifs they might like to include.
The rush on Dutch roads is off all time. Indications of the first blockages find their origin in the twenties from the 20th century. They are the heralds of the well-known frustrations in the morning and evening: traffic jams.
Over the years, the demand for motor vehicles grew, resulting in headaches for many civil servants. This led in 1938 to the establishment of the “Legion of Benevolent Road Users”. An initiative of the Royal Dutch Automobile Club to bring the politeness back in traffic, so that the safety on the street could be guaranteed. Members of the Legion ‘operated’ under the slogan “prepared for politeness on the street.”
Traffic can not just trust in the benevolence of the road users, so in 1965 a number of new traffic regulations got introduced where safety was of paramount importance. Pedestrians in the built-up area should not be hindered during their cross-over by inattentive motorists and mopeds as swift as an arrow. The Polygoon Dutch News shows in a report that the changes of the traffic regulations also restrict the various functions of emergency lanes.
Nearly 30 years after the establishment of the Legion of Benevolent Road Users, the situation on the roads are often still an awful affair. The undesirable behavior of many road users was not yet nipped in the bud. Especially in the busy capitol city of Amsterdam, motorists are a major problem, their pride possessions are anyway. Cars are double-parked at the strangest spots. However, it is not only the citizens where the finger should be pointed to. There is not enough parking space available in the city. The picturesque bridges that connect the canals do briefly serve as parking lots in 1967. It soon becomes apparent that this solution is unnerving to other road users, so the Amsterdam city council sets foot on the street armed with yellow paint.
In addition to addressing anti-social driving and parking, a car-free city centre of Amsterdam is becoming a more prominent issue. In reply to this, the industrial designer Luud Schimmelpennink in the early 70s came with an innovative concept for environmentally friendly transport: the Witkar. A three-wheel electric vehicle for two persons that could go up to 30 kilometers per hour. This electric tricycle is designed for the collective: subscribers don’t have to worry about a parking lot because the Witkar could be ‘hovered’ in a charger at five locations in Amsterdam.
Fourty videos on sports have now been added to Open Images. These items from the Polygoon newsreels give an overview of different sports in the Netherlands. Besides well known sports like football, hockey and cycling, there are also videos on lesser known sports. One of these is a sport that orginated in the USA, pushball. In a game of pushball two teams try to push an enormous ball across the line of the side of the opposing team. A large crowd of spectators in traditional attire watches a pushball game in Volendam in 1927. A report of this match shows how falling players frequently get run over by the enormous ball:
Falling down and getting up again also seems to be the motto during a cross-country of the Royal Military Sports Club. In a report by Polygoon newsreels we can see how the mounted soldiers have to ride a course with ditches, fences and slopes. This doesn’t always go smoothly: horses refuse, go through instead of over the fences or fall down with their rider:
The participants of the 4th Amstel Gold Race in 1969 also had a hard time. The weather conditions were far from ideal. The participants of this cycling classic in the hills of Limburg had to deal with cold and wet weather. Many cyclists had to give up. Only 34 of the 170 cyclists finished the race. Eddy Merckx was the favourite for the win, but his fellow countryman Guido Reybroeck eventually won the sprint:
The water skiers on the Bosbaan in 1962 had better weather. The participants of the Northern European and also Open Dutch Championships Water Skiing particpated in two categories: trick skiing and ski jumping. The 14-year-old Dutch Conny Dane won the Dutch as well as the Nothern European title in the category trick skiing. The jumps of the Danish waterskier Glasner didn’t go as smoothly. He came 9m short of the eventual winner in his first jump. During his second attempt he fell, but luckily he was “not discouraged” according to commentator Philip Bloemendal, but “he was wet”:
As small as the Netherlands may be, as big the freedom has been for the performing artists, most of the time. Throughout the years a free rein has been given to many artists within different art disciplines to unfold their talents in front of a large audience.
As early as in 1958, the 14-year-old violinist Dick Bor was given the opportunity to perform as soloist within the Dutch Youth Orchestra during a concert at the Kurhaus in Scheveningen. This concert was held within the scope of the Holland Festival, an event that is held annually in Amsterdam and acts as a platform for the Dutch, as well as the international performing arts.
The Dutch Youth Orchestra has not been the only breeding ground for performing artists. Since the establishment in 1888, the notable Royal Concertgebouw Orchestra (RCO) has brought forth many talented musicians. The RCO is known throughout the world for being a celebrated symphonic orchestra, so it’s not surprising that international performing artists throughout the years have been eager to cooperate with the orchestra. As early as 1946 the well known German choirmaster Bruno Walter took off to Amsterdam at the age of 71 to rehearse with the Royal Concertgebouw Orchestra. Having been the confidant of Gustav Mahler for years, Walters’ choice of repertoire for the cooperation with the RCO was easy: Mahlers Fourth Symphony.
The harmonious teamwork between different artists inspired not only the sector of the performing arts. The government also understood the significance of the fine arts and by the end of the forties the Raad voor Cultuur (Arts Council) is established with the intention to advise the government on the field of ‘art management’ in the sectors of film art, theatre, music, expressive arts and literacy. In 1956 the council is installed in a celebratory way in The Hague by Jozelf Cals, the minister of Education, Arts and Science. The positions within the board are being occupied by prominent people from the five sectors of the fine arts. Hence ballet dancer Sonia Gaskell became head of the sector of theatre.
At the moment Wikipedia articles don’t contain a lot of videos (less than 0,1% of all files on Wikimedia Commons are video files). Open Images would like to change this. Therefore, most videos from Open Images are already automatically mirrored to 