lod2.okfn.org

Last week we started a spreadsheet to compile examples of EU companies using open data. There are currently 46 examples from 11 EU Member States. You can view the spreadsheet here.

In the first instance we want the list to be illustrative rather than comprehensive – highlighting interesting examples of reuse and reuse in different European countries, rather than striving to capture every example of how companies have used open data.

If you have an example which you think should be added, please feel free to edit the spreadsheet! If you want to discuss these examples further, you can join the euopendata and/or the open-government mailing lists.

Here are the slides from our presentation on Work Package 9 (PublicData.eu) from our first year Review Meeting in Luxembourg in September:

Last week Neelie Kroes handed out prizes to the 1st Prize winners of the Open Data Challenge at the first European Digital Agenda Assembly in Brussels. You can find a full list of the winners at OpenDataChallenge.org.

I just published a piece in the Guardian about the competition. You can also find more coverage and commentary at:

• ABC (Spain)
• Artesi (France)
• Buongiorno Slovacchia (Slovakia)
• ChangeNet (Slovakia)
• Deutsche Welle (Germany) – both here and here
• digibusiness.fi (Finland)
• ePSIplatform (EU)
• Heise Online (Germany)
• Marketer.sk (Slovakia)
• Mitteldeutsche Zeitung (Germany)
• My News Desk (Sweden)
• OpenSource.com
• Örebro kommun (Sweden)
• Portal.de (Germany)
• Svet Komunikacie (Slovakia), Euroalert (Spain)
• Talis (UK)
• Vox Publica (Norway)
• WTM News (Greece)
• Zeit Online (Germany)
• Zive (Slovakia)
• If you know about other press coverage of the Open Data Challenge please do let us know and we’ll add a link!

You can find photos of the event here, here and here. We’ll add links to further footage as soon as it comes online!

Today we’re happy to release a first beta of publicdata.eu, the Open Knowledge Foundation’s European-level data registry. After releasing an experimental data catalogue federation and scraping frontend earlier this year, this is the first iteration based on CKAN, our data management system. While the basic functionality is still that of a read-only dataset search, a lot has changed behind the scenes.

The site now uses CKANs new harvesting capabilities, originally developed for the UK’s location programme. Using this framework, we were able to pull a large number of data catalogues into this joint index – including all instances of CKAN (such as data.gov.uk), France’s Data Publica, Swedens OpenGov.se, CSI Piemonte’s Dati Piemonte and several municipal catalogues, including those of London, Paris and Vienna. In the near future, we hope to also include some geodata directories, such as the EU’s national INSPIRE registries.

Another major story in the current development was RDF support. While CKAN has had batch export to RDF for a while and the semantic.ckan.net subdomain is offering those exports for download, publicdata.eu is stepping up support: We’re now offering a live RDF API for DCat export, a SPARQL endpoint based on a background triple store that is updated whenever data changes as well as some support for DCat RDF imports in our harvesters. This means CKAN now potentially has round-trip support for DCat and that we can go ahead in implementing the proposed standard for DCat data catalogue federation.

As we started to gather increasing numbers of data packages, we decided to try out a few normalization techniques to the data we had gathered. Starting in the messiest place, the first aspect to tackle was file formats. While there is no hope for datasets with “paper” as the mime type, “shapefiles” and “commasheets” can be easily translated into their proper types via a simple script.

Another piece of information that we were easily able to generate was the member state (and in some cases NUTS classification) of the affected region. This allowed us to create a map-based overview of data availability thoughout Europe. Besides being a nice way to facet the data, this also helps to show which countries are leading in their effort to open up government information.

We then did the same thing to categorizations: several of the catalogues we harvested contain their own small taxonomies. Looking at the similarities, it was easy to extract a set of 14 common categories – most of which roughly align with first-level Eurovoc items. Still, the larger number of source categorizations remains untranslated and highlights the need for a proper taxonomy management to be integrated with the catalogue in LOD2.

Finally, comes the most visible aspect: CKAN received both a face lift and an integrated apps catalogue. Realizing the need to give some of the fabulous contestants for the Open Data Challenge a permanent home, we decided to integrate a gallery of the shortlisted entries right into the core of publicdata.eu.

The Open Data Challenge, Europe’s biggest open data competition, is now over! From the website:

There were a total of 430 entries from 24 EU Member States. Our amazing panel of judges are currently scouring through the entries to select the winners, which will be announced at the European Digital Agenda Assembly in Brussels on the 16th June. All winners will be listed on the website as soon as they are announced.

If you’d like to keep in touch with other people interested in open data, you can join the open-government and euopendata mailing lists or follow the #opendata hashtag on Twitter.

Background

Anyone who follows the #opendata hashtag on Twitter, or who hangs out on the Open Knowledge Foundation’s open-government mailing list will know that nearly every week there is a new local, regional, or national data catalogue being announced somewhere in the world. People interested in using data from different sources may want to search across these different catalogues to find datasets of interest to them (e.g. all the openly licensed spending datasets, or all of the legislative corpora in formats X, Y or Z, from anywhere in the world). We are currently working on things like PublicData.eu and OpenDataSearch.org to do this. However in order to make services like this work, we need up to date lists of data catalogues.

A few weeks ago we discussed exactly this at an extremely useful meeting in Edinburgh on data catalogue interoperability. One of the outcomes of this meeting was an agreement between the Open Knowledge Foundation, DCAT, CTIC, and RPI to collaborate on creating a shared, collaboratively curated, comprehensive list of data catalogues on a new website called datacatalogs.org. This would include a source list of local, regional and national catalogues, catalogues created by public bodies and catalogues created by citizens and NGOs, and so on.

OpenDataSearch.org

Where are we now?

Today we had a brief call to discuss how to take the forward. The call included:

• James Gardner, CKAN Project Lead
• John Glover, CKAN Developer
• Kendra Levine, Librarian in Berkeley
• Jonathan Gray, OKF Community Coordinator (me)

First we went over the plan we made in Edinburgh, which is:

• to define a basic set of metadata about data catalogues that we want to collect, taking into account work that has been done on DCAT, by RPI, by CTIC, and so on
• to amalgamate existing lists into one big list, collecting all relevant metadata
• to start a new customised instance of CKAN on datacatalogs.org – with features like moderation to allow a group of administrators to curate the list of data catalogues, with a custom ‘catalogue metadata’ plugin to show the fields we’re interested in displaying, and so on
• to import the big amalgamated list into the new CKAN instance
• to brand the new CKAN instance with the logos of other organisations who are supporting/updating it
• to invite key stakeholders (e.g. government representatives, policy makers, researchers, open data advocates, and others) to curate the list

Existing lists of catalogues

RPI International Dataset Catalog Search prototype

Here is a list of various kinds of lists of catalogues that we know about (if you know any more – please ping us a comment below and we’ll add them here for future reference!):

• http://ckan.net/package?q=&tags=package-type.catalog
• http://lod2.okfn.org/eu-data-catalogues/
• http://opengovernmentdata.org/data/catalogues/
• http://datos.fundacionctic.org/sandbox/catalog/faceted/
• http://logd.tw.rpi.edu/node/9903
• http://www.guardian.co.uk/world-government-data
• http://open.mflask.com/
• http://publicdata.eu/
• http://opendatasearch.org/
• http://semantic.ckan.net/
• http://nationaldatacatalog.com/

CTIC’s Public Dataset Catalogs Faceted Browser

Towards a basic metadata standard

Kendra is having a shot at developing a basic data catalogue metadata standard – on the basis of existing work. To start with, she will be using this comparison of metadata from DCAT, CTIC, RPI that we created at the Edinburgh meeting.

Amalgamating and importing the list of data catalogues

After we have the metadata standard, we are going to start amalgamating the lists on a spreadsheet, which will subsequently be imported into CKAN, using one of our spreadsheet importer scripts:

• http://bit.ly/datacatalogs_org

Customising CKAN

In addition to having a single resource list which is updated by key organisations and stakeholders, we want to create an easy mechanism for enabling datacatalogs.org to be administered. At the Edinburgh meeting there was a strong feeling that this should be curated – and all new suggested catalogues should undergo some sort of review and approval process.

The CKAN team have been busy developing a simple but surprisingly sophisticated moderation mechanism for managing suggested updates and revisions to information about data catalogues.

Here are a few sneak previews of the functionality:

Next steps?

Here’s a rough schedule of how we’d like to proceed over the next few weeks:

• From 7th June – start work
• On 13th June – metadata standard ready (based on DCAT and existing lists) and start populating spreadsheet based on metadata standard
• On 20th June (or before if possible) – first deployment on datacatalogs.org
• On 27th June – final import of data from the spreadsheet
• On 28th June – polishing at the pre-OKCon 2011 CKAN workshop
• On 30th June – launch at OKCon 2011

A few weeks ago we had a small workshop on “Open Government Data in Europe” in Budapest. The meeting brought together representatives from the European Commission, the Hungarian government, and other EU member states to discuss the current state of open government data across Europe. Discussions included legal, technical and economic aspects of running an open government data initiative.

We started out with a brief introduction from myself and David Kitzinger, co-founder of Szabad Adat, a new open data organisation in Hungary. Then we went onto presentations to introduce the idea of open data, to give an overview of the state of play in Europe, and to look in more depth at open data in Poland:

• Open Data: What, Why and How – Dr. Rufus Pollock, Open Knowledge Foundation (UK)
• Open Data in Europe – Richard Swetenham, Head of Unit, European Commission (EU)
• Open Data in Poland – Igor Ostrowski, Advisor to the Prime Minister, Chancellery of the Prime Minister (Poland)

Then we had a discussion about “Challenges and Opportunities for Open Government Data in Europe” with the speakers, representatives from Hungarian government and other participants at the meeting.

Then we went onto several presentations focusing on local open data – particularly at city level.

• Open Data in Amsterdam – Katalin Gallyas, City of Amsterdam (Netherlands)
• Open Data in European Cities – Esteve Almirall, Project Coordinator, EU Open Cities Project (Spain)

Finally we had a closing discussion on what kinds of data could be opened up at city level, how to get started, and how to engage with developers and reusers of the data. We discussed how to set up a data catalogue, how to put data into PublicData.eu and encouraged public bodies to enter datasets into the Open Data Challenge.

The event is organised by the Open Knowledge Foundation and supported by EVPSI, HUNAGI, and LAPSI. EurActiv.hu was the official media partner for the event (see their post about the event here).

Some photos from the workshop are available here:

If you’re interested in keeping in touch with other people interested in open government data in Hungary, you might like to join our okfn-hu mailing list.

With just over a week left to enter the Open Data Challenge, we’re busy organising our fantastic panel of judges to help them to select the best entries to Europe’s biggest open data competition.

The prizes will be handed out in a plenary session at the first European Digital Agenda Assembly by Vice-President of the European Commission Neelie Kroes (who recently blogged about the competition here).

Following is a list of the judges for the competition who are confirmed so far (we will continue to add to this in the coming days!):

• Dániel Antal, Euractiv.hu (Hungary)
• Sören Auer, University of Leipzig (Germany)
• Brian Behlendorf, World Economic Forum (US)
• Omar Benjelloun, Google (France)
• Lorenzo Benussi, TOPIX (Italy)
• Sir Tim Berners-Lee, W3C (UK/US)
• Adam Bly, Seed Media Group (US)
• Jacob Bøtter, We Mind (Denmark)
• Victoria Anderica Caffarena, Access Info Europe (Spain)
• Laura Creighton, Investor (Sweden)
• Bastiaan Deblieck, TenForce (Belgium)
• Juan Carlos De Martin, NEXA Centre (Italy)
• Anke Domscheit-Berg, Government 2.0 Netzwerk Deutschland (Germany)
• Herve Dupuy, European Commission (EU)
• David Eaves, Advisor to the Mayor of Vancouver (Canada)
• David Kitzinger, Szabad Adat Alapítvány (Hungary)
• Peter Krantz, Department of Commerce (Sweden)
• Henri Laupmaa, Open Data Estonia (Estonia)
• Tom Lee, Sunlight Foundation (US)
• David McCandless, Information is Beautiful (UK)
• Nataša Pirc Musar, Slovenian Information Commissioner (Slovenia)
• Séverin Naudet, Etalab (France)
• Kaisa Olkkonen, Nokia (Belgium)
• Olav Anders Øvrebø, University of Bergen (Norway)
• Antti Poikola, ePSIplatform (Finland)
• Rufus Pollock, Open Knowledge Foundation (UK)
• Thomas Roessler, W3C (Luxembourg)
• Simon Rogers, Guardian (UK)
• Juliana Rotich, Ushahidi (US)
• Marietje Schaake, MEP (Netherlands)
• Alek Tarkowski, Centrum Cyfrowe Projekt: Polska (Poland)
• Julian Todd, ScraperWiki (UK)
• Andy Updegrove, Open Forum Academy (UK)
• Andrew Vande Moere, Infosthetics (Belgium)
• Sascha Venohr, Zeit Online (Germany)
• Richard Wallis, Talis (UK)
• Anthony Williams, Author of Wikinomics (UK)
• Ton Zijlstra, ePSIplatform (Netherlands)

There are now 30 days left to enter the Open Data Challenge – Europe’s biggest open data competition. To help to publicise the competition in the final few weeks before it closes, we’ve produced a variety of promotional materials, including:

• Banner (728 x 90)
• Banner (468 x 60)
• Bubble (1500 x 1500)
• Bubble (500 x 500)
• Bubble (240 x 240)

Other bits and pieces are available on the OKF’s Flickr account, tagged opendatachallenge. If you want to help out by creating more promotional material please add it to Flickr with the opendatachallenge tag.

We’d be very grateful for any help in publicising the competition. If you use one of the banners, or if you write/blog about the Challenge – please leave a comment below with the link so we and others can see it!

Update 2011-05-09: the banner is running on the Guardian Technology website from now until the 5th June.

The Data Catalogue Interoperability Workshop was a two day meeting at the Informatics Forum in Edinburgh on 3-4th May 2011, for those who administer open data catalogues (particularly those located within Europe).

The meeting included discussion, design and development work aimed at:

• improving interoperability between different types of data catalogues (including but not limited to those powered by CKAN);
• allowing them to be federated more easily into projects like opendatasearch.org or publicdata.eu.

Key outcomes include:

• A set of best practises for data catalogues operators
• A proposed mechanism for harvesting metadata from catalogues
• Agreement to collaborate on datacatalogs.org – a canonical list of data catalogues maintained by key organisations and individuals

If you’re interested in any of these things – please come and say hello on the ckan-discuss list!

Detailed notes from the meeting are now available at:

• http://wiki.okfn.org/OpenDataCatalogues/2

Some photos from the workshop are available here: