Basics
The starting point for this demo application was the reuse and linking of data sets to the Open Images collection, which contains more than 1,500 freely (re)usable videos containing mostly old news items from the ‘20 throughout the ‘80. All of these videos are published using Creative Commons licences.

The basis for the application lies in the use of the speech transcripts, which were generated by using automatic speech recognition (ASR) software (from X-MI) on these videos.

The main idea for the demonstration is to contextualise videos while they’re being watched, in order to provide the user with fun, interesting and unexpected background information about the things that are spoken in the video.

For example: when Philip Bloemendal (the presenter of the news items) – in a video titled: ‘Large parts of Holland completely snowed in’ – talks about: ‘(…) but on several places in Drenthe there (…)’, next to the video, several blocks of information about Drenthe (a province in The Netherlands) are shown. Each of these information blocks gets its data from a specific open data source. For the first prototype the data sources used were (amongst others): Google Maps and Wikipedia. To illustrate this some more: in the example where ‘Drenthe’ was recognized as a concept, the Wikipedia block shows an article about Drenthe; in the Google Maps block the map is zoomed in on the province of Drenthe in The Netherlands.

For the Code Camping event, organized by ‘Hack de Overheid’ (Hack the government), I added two new data sets to the demo: the collections from the Rijksmuseum and the Amsterdam Museum.

How it all works
As mentioned, the main building blocks for this demo are the Open Images videos and the corresponding speech transcripts that are used to link the words that are spoken (in the video) to an exact time code. (Note: Automatic speech recognition software is not perfect, which means that not every word in a speech transcript will exactly match the actual words that were spoken).

Step 1
Because not every word in a sentence is particularly interesting, the first step is to filter out stop words from the speech transcript, such as: articles, prepositions and verb modifiers.

Step 2
In the second step, a script is run on the remaining words to sort them by ‘importance’. Importance in this matter is calculated by combining a preset word score (coming from a special lexicon) with the frequency the word is spoken. In this way, words with a high score and a high frequency will end up high in the list.

Step 3
After sorting, the words are used, in order of importance, as query input for the GTAA thesaurus (used by Sound and Vision) and also for Freebase. The latter is a Google service and offers a big collection of interrelated concepts, containing descriptions from a large variety of domains. Freebase can be seen as an extensive thesaurus containing information from a large number of areas of expertise.

When, after querying, the GTAA or Freebase webservice yields a concept, it is put in a list of candidates. After processing all the words, this list is filtered using a very simple disambiguation algorithm (i.e. whenever the yielded concept is comprised of more than one word, it is taken out of the list).

Step 4
In step 4, each of the GTAA and Freebase concepts from the list of candidates is used for querying the open data webservices, which are:

1. Google Maps (only queried for location type concepts)
2. Wikipedia
3. Amsterdam Museum
4. Rijksmuseum

Each result returned, will be linked to the time code of the (spoken) word from the speech transcript that was used to find the eventual information.

(For those interested: the collection from the Amsterdam Museum has three different end-points: Adlib, OAI-PMH and SPARQL. For this demo, I used the latter, because, unlike OAI-PMH, it does not require to be harvested and indexed before it can be queried. In any case I thought it was a good idea to play around again with the Semantic Web and refresh my SPARQL skills. For the Rijksmuseum, I first harvested the collection from OAI-PMH and then indexed it with SOLR. This way the collection can be searched using Lucene queries.

Step 5
The last step was to send back the time-coded contextdata back to the browser. I do this by using a JSON object, which in turn I use as input for Popcorn.js to generate events. These events are linked to an HTML5 video player and make sure the right (context) information is shown in the different blocks/panels in the user interface.

Because the processing of these five steps takes around 15-20 seconds per video, I store all of the results in .json files. When opening the demo these files are loaded instead of fetching the data live from the web.

There is still a lot to do
The demo shows what can be done by using concept detection (a.k.a. Named Entity Recognition) in combination with open data sources. For several aspects however (significant) improvements can be made:

Better concept detection
The concept detection as described in this demo could be improved much more. For instance, concepts that comprise of more than one word are not recognized, e.g.: ‘Amsterdam Museum’ now yields two concepts, ‘Amsterdam’ and ‘Museum’, but the actual concept ‘Amsterdam Museum’ is not found.
Moreover, specific Named Entity Recognition (NER) services like DBpedia Spotlight should be investigated (having good results for English) in order to improve results. For Dutch however, it seems it’s an ongoing search for a decent (open source) solution.

Selection of relevant sources for the user
Concerning the relevance of the ‘context information’ that is currently shown to the user, there is still much to think about how to make the best selection of data sources. For instance: why somebody who is watching a video about ‘Holland’s oldest steam-powered pumping station’ would be interested in ‘Hens chalice from the Company of Nine’ (found on the basis of the word ‘Gorinchem’, which is a town in The Netherlands) is something to think about.

Optimizing Popcorn.js usage
The demo was made with an older version of Popcorn.js (v0.7) and therefore doesn’t make full usage of all of the latest features and plugins Popcorn.js has to offer. Future releases of the demo will incorporate the newest version (currently v1.1.1).

In any case the demo does show how speech transcripts of videos can be combined with open data sources and how this can enable (mutual) contextualisation of these sources. For the ‘Nederland opent Data project’ this demo will be further enhanced. Any progress of this will be reported here!

Jaap Blom | Software engineer | R&D department, Netherlands Institute for Sound and Vision

The Wikipedia community uses the videos from Open Images to enrich the entries on the Wikipedia. For instance, the English article on the ‘Elfstedentocht‘ has a video of the Elfstedentocht of 1954:

Besides the reuse of complete videos, derivative works (such as screenshots) are also used. These are then for example employed in articles on famous people, for instance in this article on Dutch politician Pieter Oud:

3 million views
The reach of Open Images content on Wikipedia turns out to be substantial. In May 2011 the Wikipedia articles with media items from Open Images were viewed more than 3 million times. This is almost three times as much as the number of views in December 2010. Noteworthy is that the majority of the views are not on the Dutch Wikipedia, even though most of the videos on Open Images have Dutch subjects and are in Dutch. Of the 3 million views a mere 880,000 were on the Dutch language Wikipedia. The remaining 2.2 million views were on Wikipedias in different languages. The five Wikipedias where articles with Open Images content got the most views in May 2011 were:

1. the English Wikipedia
2. the Dutch Wikipedia
3. the French Wikipedia
4. the Portuegese Wikipedia
5. the Japanese Wikipedia

More than 850 articles on the different Wikipedias make use of content from Open Images.

The article with the most views in May 2011 was Mother’s Day on the English Wikipedia, which was viewed almost 1.5 million views. The video used in this article is used on several Wikipedia sites. Besides the English and the Dutch Wikipedia, it is also used on for example the Tibetan and Persian Wikipedia. The Wikipedia articles containing Open Images media with the most views in May 2011 were:

1. Mother’s Day (EN) 1,445,756 views
2. AFC Ajax (EN) 121,322 views
3. AFC Ajax (NL) 111,190 views
4. Billy Graham (EN) 94,485 views
5. Giro d’Italia (EN) 73,055 views

Conclusion
These statistics demonstrate that offering their material under a free license certainly has an added value for cultural heritage institutions. For the cultural heritage field it is a sound strategyfor opening up their collections to a large audience. It also gives the (internet) community a chance to enrich their projects with historic images.  This reuse is of course not restricted to Wikipedia. By offering collections under a free license they turn into a rich source for (re)use fora large number of cultural, educational and creative purposes.

Wiki Loves Monuments is a contest organised by Wikimedia, the movement behind Wikipedia. To be eligible for the video prize participants have to upload a video of one or more monuments to Wikipedia in September. The rules are:

• Self made and self uploaded
• Uploaded in September 2011
• Freely licensed
• Feature one or more monuments

So be creative and enter the contest! The people of Video on Wikipedia have a howto explaining how to post a video to Wikipedia. More information on Wiki Loves Monuments can be found on their website.

Hundreds of Items Added to the Platform

In 2010 we have uploaded hundreds of interesting items to the platform from the historical newsreel collection of the Netherlands Institute for Sound and Vision, reaching the milestone of a thousand items available on the platform on the UNESCO World Day for Audio Visual Heritage in October. In our selection procedure some themes received special attention; sports, performing arts, winter, technology, and Indonesia.

This year the Sound and Vision was not the only contributor of content to the platform. Other wonderful additions to Open Images were done by the EYE Film Institute Netherlands, the Institute for Network Cultures and the Dutch National Committee May 4th and 5th.

API Launched

In September Open Images launched its open API. Items published on the platform and their descriptions (metadata) and are accessible through an Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). This enables third parties to retrieve the stored metadata and media files in a structured way, making it easy to reuse material from the platform in their own applications (for example to create a mashup).

Video on Wikipedia

Since the start of the project, Open Images has contributed its audiovisual content to Wikimedia Commons to enable reuse of video on Wikipedia, for instance to ‘illustrate’ an article.

At first the ‘donation’ to Wikimedia Commons was a manual process, but in 2010 – in collaboration with Wikimedia Netherlands – we were able to fully automate this process, thanks to the Open Images API. As a result Open Images is now responsible for almost 12% of the video content available on Wikimedia Commons, hence being one of the biggest contributors of video that is reusable on Wikipedia.

We are getting more and more insight in the impact of the availability of Open Images material through Wikimedia Commons. We’ve learned that a large proportion is used to enrich over 550 entries on Wikipedia with related audiovisual content. In December 2010, these entries were viewed nearly 1.2 million times. This shows the great potential for the cultural heritage sector to collaborate with the Wikimedia Foundation to reach new and greater audiences within a meaningful context.

New Projects Reusing Open Images

When Open Images was launched in 2009 the material was almost immediately reused within several projects, including the OPEN CITY audiovisual archive of urban life from the Dutch public broadcaster VPRO and the ArtTube video platform about art and design from the Boijmans Van Beuningen Museum in Rotterdam.

In 2010, tens of projects, small and large, were added to the list. Among them Picture War Monuments, a location-aware iPhone app that enriches the on-site visit to war monuments with audiovisual heritage, including newsreel footage and oral history video material on the Second World War available through Open Images. Another notable initiative was Image on a Map (‘Beeld in kaart’), a Google Maps mashup for the educational sector in the Netherlands combining several (educational) video sources – including Open Images – within a map interface. With this interface users are able to filter results based on subject (geography or history), location and time period.

What’s Next?

In 2011 the Open Images platform will receive a major update, with both functional and visual improvements. Part of this update is the realisation of portal functionality, allowing third party content providers to build and manage their own entrance to the platform (think: http://partner.openimages.eu). This will, for example, allow them to highlight their own contributions to the platform and to design their presence on the platform according to their own wishes and branding.

The platform functionality is part of a larger campaign we are organising to attract more third party content keepers to contribute to an even larger and more diverse offering of open audiovisual content through Open Images. This campaign will focus on public broadcasters, regional and local archives and broadcasters, institutional archives and business archives.

Finally, we would proudly like to mention our nomination for the Museums and the Web – Best of the Web Award 2011 in the category ‘Innovative / Experimental’.

EUscreen started in October 2009 as a three-year project funded by the European Commission’s eContentplus programme. Over the project’s duration more than 30,000 items representing Europe’s television heritage (videos, photographs, articles) will be made available online through a freely accessible multilingual portal. As part of the project Open Images will function as a platform for European broadcasters to experiment with open content distribution of television heritage.

The portal will be launched in 2011 and will be directly connected to Europeana. The EUscreen consortium is co-ordinated by University of Utrecht and consists of 28 partners (comprising audiovisual archives, research institutions, technology providers and Europeana) from 19 different European countries. In October the project will organize its first international conference:

Date: 7-8 October 2010.
Location: Casa del Cinema. Largo Marcello Mastroianni 1, Rome, Italy.

EUscreen has organized a two-day conference on content selection policies and contextualisation in the audiovisual domain, to be held in Rome on October 7 and 8 2010. The conference will focus on contextualisation of audiovisual material, especially in the academic field. The conference programme is still under construction, but the first day includes a plenary session focussing on contextualisation of audiovisual material with keynotes and presentations of use cases. The second day comprises two workshops: one on European IPR legislations in the audiovisual sector and the impact on the exploitation of audiovisual and television archives, and one on best practices and guidelines for digitising audiovisual heritage. Attendance at the conference is free but online registration is required.

See www.euscreen.eu for more information on the final programme and for registration.

Confirmed speakers

• Prof. Andrew Hoskins, Professor of Cultural Studies at Nottingham University on media, digitization and memory.
• Dr. Lilian Landes, scientific co-ordinator of the recensio.net project at Bavaria State Library on creating a European Open Access infrastructure for historical reviews.
• Dr. Alec Badenoch, from Utrecht University on Making Europe, virtual exhibits on European cultural heritage.
• Johan Söderberg, lecturer and filmmaker from Sweden on using and reusing archival material in his works, like the series “Read my lips”.
• Dr. Tibor Hirsch, from Film Studies at ELTE University on using digitized material in a creative way to help students understanding the language of film and television.
• Dr. Andreas Fickers, from the Art and Social Sciences at Maastricht University on audiovisual source critique in the age of the web 2.0.
• Peter B. Kaufman, President and executive producer of Intelligent Television. He is also the author of “Marketing Culture in the Digital Age: A Report on New Business Collaborations between Libraries, Museums, Archives, and Commercial Companies”.
• Prof. John Ellis, Professor of Media at Royal Holloway – University of London.

The first Open Video Conference was held at NYU Law School on June 19-20. Eminent speakers and practitioners shared their thoughts on the emerging open video movement. The impressive line-up included: Matt Mason (author of The Pirate’s Dilemma), Yochai Benkler and Jonathan Zittrain (both Harvard Law School), Xeni Jardin (Boing Boing), Peter Kaufman (Intelligent Television), Mike Hudack (blip.tv) and Christopher Blizzard (Mozilla Corporation). The conference was put on by Kaltura, Yale Internet Society Project, Participatory Culture Foundation, iCommons and the Open Video Alliance, in partnership with Mozilla, Red Hat, Creative Commons, Level 3, Akamai and many more. Open Images was also actively involved, as Sound and Vision and Kennisland hosted a session “Audiovisual Archives” that investigated how memory institutions could provide access their holdings in a way that enables creative reuse.
Open Video is a broad-based movement of video creators, content distributors, technologists, academics, filmmakers, entrepreneurs, activists, remixers, and many others. From the conference announcement:

When most folks think of “open,” they think of open source and open codecs. They’re right—but there’s more to Open Video than open codecs. Open Video is the growing movement for transparency, interoperability, and further decentralization in online video. These qualities provide more fertile ground for independent producers, bottom-up innovation, and greater protection for free speech online.

Yochai Benkler, Professor of Harvard Law School and Faculty Co-Director, Berkman Center for Internet and Society, delivered the opening keynote. He mentioned how “the smartest, most creative people never work for the same companies; open innovation platforms allow innovation to speed up.” His talk evidenced how distributed action and innovation is key to both business innovation (and technologic and/or legal restrictions aren’t) as well as a more participatory and democratic public sphere. It provided a good backdrop for discussions in the sessions that followed.

Some conference highlights

[1] Later this year Wikipedia will release new functionality that allows the easy inclusion of video in Wikipedia later this year. The Mediawiki software will feature a new button labelled Add Media. Users can search through open repositories (initially Internet Archive, Wikimedia Commons and Metavid) containing and drag chosen portions into Wikipedia articles. Further down the road, Wikipedia will include more repositories, and also provide tools to edit the clips within the Wikipedia website, comparable with the way it is currently done with text. Considering the influential position of Wikipedia (the only non-profit website in the top-10 of most-visited sites), it is safe to predict this will mean a great push in strengthening the role of video within the collaborative production and sharing of knowledge online. This is one important platform where an open video infrastructure and an active user base will come together.

[2] Chris Blizzard from Mozilla was applauded as he showcased some innovative features on Firefox 3.5. The release of Firefox 3.5 will come with HTML 5 support, which allows video to be embedded into the Web page without needing to install plug-ins. When asked what type of content Open Video will enable, Blizzard answered, “The killer app for Open Video will come once the environment is created. Mozilla won’t create it.” 300 million people are using Firefox, so this is indeed a big step forward towards open video. Although this is indeed an important breakthrough, the open video ecosystem is not quite there yet. On the second day of the conference the HTML5 and <video> Working Group described some of the current issues with the actual implementation and adoption of the HTML5 specification. Because there is no universal approach yet – Chrome, Firefox and Safari all differ in their approach and Internet Explorer seems reluctant to even adopt HTML5 any time soon – this severely hinders HTML5 <video> being an actual standard for video on the web.

[3] Developers from the P2P Next consortium showcased the Swarmplayer, which is capable of streaming live video in true 4th generation P2P style using a zero-server approach. This could be a real breakthrough in the way broadcasting is delivered. With Swarmplayer technology viewers help serving videos in a peer-to-peer fashion, hence reducing or even removing the need for a central server solution.

[4] In a fascinating talk, Matt Mason, author of The Pirate’s Dilemma described how industries could innovate when they not only compete with pirates, but also think about novel ways to work with them. He made a convincing argument by putting the concept of ‘piracy’ in a historical context uncovering the importance of piracy for the establishment of many powerful industries (Hollywood) and even countries (USA). He concluded with the advice that “one of the best ways to grow your business is to give pirates the space to do things you can’t do or don’t think of.” His book (including many examples) can be downloaded from: http://thepiratesdilemma.com/

[5] During his talk How to Make a Political Remix Video Jonathan McIntosh premiered his short film Buffy vs. Edward (Twilight Remixed). Apart from being an enjoyable, his video is primarily a statement about what constitutes ‘fair-use’ of copyrighted works. This important wake-up-call for filmmakers shows that open video doesn’t necessarily conflict with “all rights reserved”, but that it is also a matter of actively claiming the freedoms that the law already grants citizens (of the United States). Since its premiere the movie has gone ‘viral’ and claims over a million views.

Session on “Audiovisual Archives”

The break out session focused on a couple of key questions that we found influence how successful the networked archive will be in establishing themselves as a key node in media consumption; and how memory institutions will continue to serve as care keepers and storytellers of our mediated past.
Audiovisual archives across the globe are engaged in large-scale migration programmes. An important driver behind the investments related to these programmes is the physical state of the analogue carriers; the films, the tapes, the optical discs and so on. Migration is a way to preserve the information on these physical carriers and securing access for future generations, a key mission of these institutions. However, migration also opens the door to the establishment of the networked archive; where material can be made available online to an infinitely large audience. Different services can be built with this ever-growing resource, such as specialized services for education, video on demand, and access through portals such as YouTube and Blip.tv. Also, as viewing has shifted away from television and onto the Internet, the public interest in access to archive resources online has exploded. Some collection owners go a step further and allow their material to be downloaded so everyone can truly engage with the material and use it as building blocks for new productions. Back in 2003, the BBC coined the term “the creative archive” and entities across the globe are bringing this concept to life. Archive.org is another one of the leading examples.

The session addressed two topics in particular:

[1] The first is related to access and licenses. In the case of Archive.org, many videos are in the public domain. But in many cases, holdings are still copyrighted and their exploitations rights belong to their respective owners (archives rarely own the material they hold). Creative Commons is a suitable model to distribute in-copyright heritage content and allowing certain freedoms (“some rights reserved”). But what are the incentives for memory institutions and public broadcasters to adopt open licenses? And how can archives support such a transition? Is there a moral obligation for memory institutions and public broadcasters to provide open access?

[2] The second is related to business models. In providing different access routes, we might ask how free access and traditional revenue streams (i.e. footage sales) can coincide in a new economic eco-system for broadcasters? Can access fund preservation? Should it? When do producers have an interest in ensuring long term access to the materials they create? (e.g. PBS in the US has an educational mission, the more people see it, the better PBS is doing its job)

Cases and discussion

Moeed Ahmad from Al Jazeera, the first independent Arabic news channel in the world talked about the Al Jazeera’s Creative Commons Repository launched earlier this year. This repository provides access to broadcast quality Gaza conflicts, released under the ‘Creative Commons 3.0 Attribution’ license that allows for commercial and non-commercial use. This means that news outlets, filmmakers and bloggers will be able to easily share, remix, subtitle or reuse our footage. The only obligation that comes with this ‘free’ reuse of the material is that Al Jazeera gets attributed. This has been an enormous success for Al Jazeera in terms of reputation, getting more exposure for their material and even attracting commercial leads.

Maarten Brinkerink (Netherlands Institute for Sound and Vision) talked about the Open Images project that will make a corpus of Dutch audiovisual heritage available under a Creative Commons license. Open Images (www.openimages.eu, to launch in September) is an open media platform to stimulate (creative) reuse of archive material by adopting an open infrastructure and Creative Commons licensing. Open Images combines open technology with a curated selection of freely licensed audiovisual heritage and – hopefully – community engagement, to fully explore the possibilities of open video.

Nan Rubin project director of the Preserving Digital Public Television project at CHANNEL 13 was the third speaker. One of the aims of this project is to secure investments in digital preservation in order exploit public broadcasting well into the future. She also talked about the American Archive project and how US-based broadcasters are trying to offer access to their holdings online.

Sara Chapman is Executive Director of the Media Burn Independent Video Archive, an online repository for four decades of nonfiction video work by independent producers. She focussed on the ways in which her organization has broadened their strategies for making online video accessible to a wide audience. Media Burn uses other platforms (including YouTube and many others) to promote their institutional website.

Summary

As mentioned in the introduction, the Open Video Conference covered a wide number of topics, from the nitty-gritty world of metadata interoperability, democratization and journalism, interests of commercial players, to novel ways of production and the role the ‘pirates’ are playing. Although many industrial players where present in the programme, some notable absentees included Microsoft and agencies such as MPAA. If open video is to driven by a movement (cf. Benkler speech), than this movement will operated from the ‘bottom up’; showing the advantage of for instance HTML5, open video codecs, Creative Commons, fair use and so on to a wide audience on the web. Wide, omnipresent adaptation of these essential building blocks by practitioners (consumers, web developers) will eventually force large industrial entities that are still clinging on to closed and proprietary systems to change their current practices and eventually embrace the concept of open video. This might sound overly positive but the main message the Open Video Conference send out to the word is the fact that open video is maturing in a stunning pace. In a short time a great variety of initiatives across the globe have been working to offer alternatives for the closed practices. The chain from production to distribution no longer depends on proprietary software, more and more content is being offered under open licenses. It is the combination of ‘brands’ like Creative Commons, Firefox, Linux, EFF, Wikipedia, OGG, VLC, P2P and so on that manages to offer an alternative ecosystem in which innovation and creativity (but also business opportunities) will be able to flourish.

For further reading the conference website provides a good overview of press coverage of the event.

Johan Oomen and Maarten Brinkerink

Click here for the pictures on Flickr The Commons

Parts of the special collection ‘Labour Inspectorate’, digitized in the Images for the Future framework, are placed onto the Flickr website. Users are invited to add tags and comments to the photos. As a result of the new collaboration between the National Archive and Spaarnestad Photo, photographs of this archive have been added to the Flickr collection as well.

On the 4th November, there is a seminar about the value of social tagging, with among others, delegates from Flickr and the National Maritime Museum. In the first two days, the photo’s have been viewed over 300.000 times and more than 400 comments have been added.

The Nationaal Archive is proud to be a member of The Commons on Flickr. Photographs of the Nationaal Archive that are part of the Commons on Flickr have “no known copyright restrictons”, this means that there are no copyright restrictions on the works designated, either because the Nationaal Archief owns the copyright of the photographs and authorizes others to use the work without restrictions, or because the copyright may have expired.