On Wednesday 18 February 2009 the State Services Commission hosted a discussion on aspects of data re-use. Sponsored by GOVIS, with five expert panellists (Nat Torkington, Fiona Romeo, Toby Segaran, Adrian Holovaty and David Recordon) supplied by Webstock, this was a wide-ranging discussion covering policy questions and technical issues. Staff of the Strategy and Innovation team, GCIO made short presentations, the panel commented, and the audience was invited to ask questions. The session was recorded and is summarised below.

Welcome

Vikram Kumar is Manager of the Strategy and Innovation Team, GCIO, State Services Commission.

Photo by David Recordon

Information and data re-use is one of the more important areas for government going forward. “The degree to which government makes data available freely is the extent of progress in e-government going forward”.

Take the recent example of the Victorian bush fires. Information was available from the private sector, but hard to get from state government. There’s 40% of GDP in the public sector. Closed data means major barriers.

Discussions like this help us exchange views on what’s required. Government working under constraints: we want to open up, but must be realistic, practical. We should identify high priority steps, not just expressing good intentions. Move discussion from a higher view down to practical steps.

Introduction to the Panel

Nat Torkington

Runs KiwiFoo camp, an “un-conference” mixing up artists, business people, programmers, people from the political scene etc. Spent 10 years in the US having run New Zealand’s first web server. Consultant on open source and the web.

Fiona Romeo

Head of Digital Media at National Maritime Museum in Greenwich, UK. Looking at data mining, data visualisation, syndicating content more widely (platforms like Commons on Flickr), crowd-sourcing science information (“citizen science”).

Toby Segaran

Author of books on collective intelligence and data mining and practical use of semantic web technologies. Works for MetaWeb on their Freebase product, taking and integrating datasets from US agencies such as the Securities Exchange Commission and the Food and Drug Administration.

Adrian Holovaty

Journalist, computer programmer based in Chicago. Works with news organisations, bringing government data online, creating works of journalism around it. Runs everyblock.com: local news at the address level (city municipal level) tapping into “a ton of public records”.

David Recordon

Employed by SixApart working on technologies, products and policy to enable people to control their data: use existing accounts more widely, take profile information, relationships etc. with them.

Keitha Booth manages the work programme in the GCIO, State Services Commission looking at stimulating re-use of government data for economic and social gain. The project spans strategic and technical areas.

Vast repositories of non-personal information stored across public sector, e.g. water information, scientific, climate, health, education etc. What of that may be opened up for use?

Information created for legitimate purposes funded by taxpayer purposes. Generally managed to high standards. There are opportunities to liberate this data, develop policies and strategies to support this across government. Already happening in limited uncoordinated way. We intend to improve coordination between suppliers and users, working together to understand how to open up.

Data re-use is a new area. Overseas experience helps to inform us. A 2006/7 Australian study indicated potential gains of $6 to 12 million through the opening up of geo-spatial information. Areas the GCIO are looking at include:

• Clarity about copyright and crown copyright. Clearer copyright statements on websites.
• Clarity about licensing arrangements.
• Economic modelling.
• Updating 1997 best practice guide, making it more broadly applicable.

What policy related re-use issues do you face?

NT: Big fan of open govdata site. What constitutes “open”? Qualities are completeness, primary, timely, accessible, machine-processable, non-proprietary. [see Wikipedia]

FR: We often hold content we are not the copyright holder for. We hold the physical item but sometimes there can be restrictions on making copies even for preservation. The British Government is lobbying for custodians to make copies for preservation. No progress yet on adoption. There are various licensing agencies to engage with. Each needs to feel rewarded. Some material is copyright of museum, some is crown copyright, in other cases the museum has been granted permission for use. Sites like Flickr have “baked in” good methods for sifting material licensed under Creative Commons from other material. Multiple licenses are a usability issue.

TS: Crown copyright is “a little confusing”. In the US, all government data is public domain. However, the Library of Congress is a “semi-government agency”. Figuring out which parts of government are real agencies and which are semi-private with specific licensing is difficult. If something is funded by the government then it should be public domain. It would be great if there were standard ways of expressing how things can be used, but these are all disparate groups. The Securities Exchange Commission had an exclusive deal with reseller Lexus Nexus. Carl Malamud forced this data to be made generally available online.

AH: As a data consumer the issues preventing re-use are often political: government officials may not want to release certain data because it makes them look bad; there may be jealousy of non-government sites; privacy issues; accuracy concerns.

DR: Set management expectations. How do you deal with someone making use of data in a new and unexpected manner?

NT: What does it mean to be a government department producing data? You shouldn’t expect to be the originator of every awesome application that uses that data. Government is not the place where innovation happens. Manage expectations: innovation on the “outside” can only be possible due to the work opening up data from the “inside”.

Audience: Often there is tension between the party commissioning a report (e.g. an agency or Crown Entity), and the party carrying out the actual research (e.g. a university), over who owns the data.

NT: Sometimes this is solved by a six month abeyance before raw data may be published. However, funding agencies (e.g. Bill and Melinda Gates Foundation) are sometimes unhappy with this as it slows down progress of research.

Audience: Where data is collected using a variety of differing methodologies re-use of that data may result in “wildly strange conclusions”. How is this mitigated?

NT: Here data is highly dependant on methodology, i.e. itself complex metadata not expressible in a standard format. Just mandating that the data itself must be open breaks down the biggest barrier. It makes it harder for the research party to reject requests for information about methodology.

FR: Where you release data others may use that data in ways you may not agree with. Be clear where you are the publisher of the raw data, publish your own reports. Even raw data can be misunderstood. You have to accept what happens when data is released. As uncomfortable as that makes us all, that’s the truth of it.

NT: Research is already misused!

TS: No one has ever managed to come up with a metadata format that accurately described a procedure that was machine-readable. You just have to trust that people will read the description of how the experiment was done and use it appropriately.

NT: We’re not setting back science by releasing data.

Are Creative Commons licenses the most obvious candidates for all-of-goverment adoption?

NT: No copyright at all is the easiest, most straightforward way to go. If you choose a Creative Commons license you’re involving a lawyer: it’s a license. Government is both the provider of the first resort, and the provider of the last resort. Popular information will be added to and improved. If someone wants the original, pure data they will return to Government. Creative Commons licenses, like all licenses, attempt to specify what people may do with data.

FR: Creative Commons licenses are increasingly well understood, widely used and expressed clearly for the layman. However, they do place restrictions on what people may do with your data. Government must decide if controlling the end-use is important.

TS: In the United States we will never trust processed data without checking it against the original source. If the government releases data into the public domain it gives up control of how the data is used, but it doesn’t give up its authority.

AH: Government data must be placed into public domain. Otherwise government is admitting that it is not comfortable making the data open. Think about your motivations for making the data open.

DR: Make it clear what people may do with data. Public domain makes it easy. The worst thing you could do is have five different licenses, or, write your own copyright license. Don’t make people think.

NT: The lesson from open source software is that it’s an almost impossible to use two licenses together in same project. Lawyers spend a lot of time attempting to make the licenses place nicely with each other.

Audience: If your license doesn’t require people to credit the source, how do people know where the authoritative source is?

FR: Even if you don’t require attribution in the license, you can still encourage best practice. You can handle this through guidelines. Services like YouTube and Flickr make it easy to embed their content with a wrapper to say where the data has come from. Make it easy to do the right thing.

TS: Anyone re-using data has to say where their data comes from to ensure the credibility of the results.

NT: Distinguish between the naïve users of a final product (e.g. a chart in a newspaper) and those who want to re-use data (who want guaranteed, known-good data).

Audience: There are six Creative Commons license flavours. Mixing them up might be a “bit of a nightmare”.

Audience: You need a granular licensing structure: data is not black or white. With Privacy Commons a person may grant an organisation limited rights to personal information and perhaps later revoke those rights. [NT observes that here we are talking about non-personal data]. Material is sometimes donated on the condition that it may only be used for research or non-commercial purposes.

NT: Most data doesn’t fall under donor rights. Public domain is the best baseline.

DR: Make it easy for people to embed data and link to its source.

Can you point us to economic modelling or other evidence to show the benefit of opening up non-personal government data for re-use?

NT: No. However, there is anecdotal evidence. The US Census Bureau released its Tiger database into the public domain. Tiger maps street names to geographical coordinates. NAVTEQ sells its improvements to the Tiger database. The original Tiger database is sufficient for many GIS applications and has itself spawned an open source equivalent to NAVTEQ.

AH: Government ought to be providing services, not visualisations of data. Government should focus on things that only government can do. It is relatively cheap to make the data available. Once data is available, anyone in the world can create an interface.

DR: All San Francisco buses carry tracking devices. The San Francisco government hired a company to show buses’ locations on some bus routes in real time. A third party obtained access to the data and made a Google Maps mashup for the remaining routes. The San Francisco government then shut down access to the data.

FR: The UK Freedom of Information Act has driven data re-use in the UK. Rather than respond individually to requests, agencies have realised it is more efficient to make data sets freely available.

AH: Where agencies in the US have made data sets available the number of Freedom of Information requests have dropped.

TS: In the US, thousands of people are employed processing data from Census Bureau, Securities Exchange Commission, and the Food and Drug Authority.

FR: The UK government published a paper called Creative Economy.

Audience: With a population of just four million are we going to see any significant economic change as a result of opening up government information and data in New Zealand?

NT: We are in early days. There are no whole-of-government case studies looking at cost/benefit yet.

Audience: Do we need to see economic benefit, or is it something we should just be doing anyway?

Mark Leicester is a Senior Technical Analyst in the GCIO, State Services Commission looking at opening up job vacancies for re-use.

Broadly speaking the GCIO is looking at a range of opening a range of information types for re-use, including calendaring information, news stories and alerts. The New Zealand Defence Force and Government Jobs Online asked GCIO to simplify the process of publishing a job vacancy to the agency’s own site and to Government Jobs Online. To find quality candidates the NZDF must key a job vacancy into its own site, re-key it into Government Jobs Online, and again re-key it into commercial third parties such as Seek or TradeMe. Ideally the NZDF would be able to publish a job vacancy once such that it would be published everywhere.

• Do you adopt, adapt or create your semantic model? Who should own and maintain the schema?
• How do manage translation between schemas? How do you handle translations between controlled value lists?
• How do you make the right choice of representational format? Do you emphasise human readable or machine readable?
• What sort of latency is acceptable? How do you make processes fault tolerant? How do you manage deletes; instruct consumers to delete information?

DR: First you try to adopt a schema, you may need to adapt and failing that you create. It’s always better to find a few other people who are going to make use of your schema. Write code and create applications that will make use of your schema. The end result will be so much better.

TS: Don’t worry too much about on-the-wire formats. Others can build endpoints as necessary. Writing guidelines and code to demonstrate a commitment to stability.

AH: You have the incentive to get the data in front of as many eyeballs as possible, but in most cases a consumer will take anything they can get. Consistency matters. A consumer will do the work.

TS: More important are the naming of things, reconciliation of entities.

NT: Locations are straightforward enough, but job titles are more complicated. Don’t worry about achieving a perfect representation. Release coarse-grained and you will see others make refinements.

FR: Start rough. Iterate in response to demand. The Maritime Museum began crudely, but later were able to join Europeana at a week’s notice.

NT: Create version one. Say what it does. Improve it. Offer version two. Retire version one over time. Iterate. People will complain less if you don’t attempt to create the “one true feed”.

Audience: In government you get funding, deliver and then start again in five years time. Iteration may not be possible.

NT: Don’t build prototypes, they may not be taken seriously. Find a compromise between building a prototype that does very little, and taking five years to get it right.

FR: A government opening up data for re-use will need to have an operational team.

Audience: Data quality is always a problem. Is it better to make sure a dataset is cleaned up before it is released?

Audience: Data can be corrected by consumers. It’s not always clear where feedback should be directed but an open dataset will be subjected to more scrutiny.

FR: My Society made video content publicly available and encouraged the public to help describe it.

NT: With job vacancies government is publishing information, but in general you have a choice: is it a publication, or a project? Perhaps you may get suggestions and advice from other people. Like open source software projects you need to vet the contributions and figure out who you trust. How engaged does the government want to get?

NT: Think about the differences between a publication, a feed, and a database. With a publication you get a snapshot of a dataset. With a feed you get the updates to the dataset over time, typically just the “new stuff”. With a database you are replicating the current version of the database in as near to realtime as possible.

AH: In some cases – restaurant inspections for example – everyblock.com requests that government agencies supply only the records that have changed. This requires greater technical sophistication.

NT: Think about how data might be re-used as the data is collected.

Audience: There’s no single answer for data quality. Recruitment information may not need to be as accurate as geographical information.

DR: In some circumstances it might be appropriate for an agency to offer a “bleeding edge” dataset as well as a “cleaned up” dataset.

AH: Human readable is always derivable from machine readable. Perhaps there are circumstances where you might offer both a machine readable and human readable feed.

Audience: Perhaps different audiences might prefer different degrees of interaction, from RESTful APIs to downloading of raw data sets.

AH: Everyblock.com focuses on data with a date and a geographical location.

NT: Hybrid compromises both human- and machine-readable forms.

AH: PDF does not equal machine-readable!

TS: JSON is ideal.

AH: XML where the schema is well defined is ideal.

Audience: Common Alerting Protocol is an Atom based format with extensions. It is human readable where Atom extensions contain machine readable data.

DR: It depends on the definition of “human readable”.

NT: That it doesn’t double as a consumer format.

NT: No one mentioned RDF. Why?

AH: We don’t get any.

TS: RDF gives you standardised namespaces like Dublin Core, but there aren’t enough of those yet. I don’t like mixing machine- and human-readable content as in RDFa. Just separate the files.

DR: With some sites it may be easier to add machine-readable markup to existing human-readable pages. If you’re starting from the beginning it’s better to create content optimised for machines.

Audience: Finland has a project for a national ontology. How can we use ontologies to make data machine readable?

NT: Look at what is already out there. Build your own only if you really have to.

TS: The best ontologies are built from existing data.

DR: It’s less important what you call something, and more that the format is consistent.

AH: Governments should also publish the details of controlled value lists to explain data.

Audience: We have nationally significant databases, metadata, solid methodologies etc., but we also have “grey data”: data that may be incomplete and lacking documentation.

NT: Release a piece of code that uses the data. Publish the e-mail address of someone that a consumer might talk with, especially if documentation is absent.

DR: Create a mailing list. Consumers can help each other.

Audience: How far should government go in encouraging re-use?

TS: The World Bank opened up an API and didn’t know how to get people to use it. Working code will get people started much more quickly than any amount of documentation or a mailing list. People learn from examples.

AH: Remove all barriers, such as human-approved signups to APIs, that prevent developers from engaging with your data quickly and easily.

Audience: How much do contests help drive the opening up of government data?

AH: Contests are good.

FR: Show Us a Better Way attempted to discover which datasets people wanted. The BBC introduced an annual “hack weekend”. Events have also worked well for the museum and may be a good approach for government.

NT: Holding a contest will not guarantee that people to use your APIs or data.

Audience: All of the examples have come from the US or UK. Are things different in a nation of four million people?

Audience: Is consolidation important?

NT: I’d rather have open APIs than worry about standardisation.

TS: It’s easy to underestimate what people are willing to do for free.

Audience: To what extent are the various mashup platforms, toolsets etc. enable programming without being a programmer. Might we see rich, easy to use environments where programming knowledge is not so important?

TS: Tools such as IBM’s Many Eyes enable mixing and matching and visualisation of datasets.

Vikram Kumar is manager of the Strategy and Innovation Team, State Services Commission.

Government data is taxpayer funded. People expect data to be made available free of cost. Agencies aren’t funded to make data available. What is your view?

NT: I think it’s reasonable to charge fees for the copying of information (but not the preparation). Open source software projects may be created using existing infrastructure at little or no cost. Ideally this will be possible with open data too such that it is more efficient to release data.

FR: I work in an institution with a large collection. The business model relies on exposure. For museums, libraries and archives the biggest risk is obscurity. Charging might work for small, local uses, but perhaps not for large scale services. Whatever model you choose shouldn’t limit usage.

NT: Charge for a snapshot of a dataset, but not for ongoing access. That would punish popular applications.

TS: Don’t get paralysed by the cost. Instead, do the cheapest [NT: useful] thing you can do right now.

AH: Everyblock.com pays for data where it’s reasonable. The American Freedom of Information Act permits charging for data. Make it easy to pay for it.

DR: If you going to release data across government try to share infrastructure and code across government.

Matthew Ross is a Senior Technical Analyst in the GCIO, State Services Commission looking at government information that is not necessarily found in a formal data set.

Agencies autonomously and independently publish information (such as job vacancies) that ‘ought’ to exist in a single cohesive dataset. What can government do to improve users’ all-of-government experience and make disaggregated information more accessible?

Portals such as Government Jobs Online attempt to create aggregated datasets from disparate sources. Should government aggregate disparate data sources? Should government create mash-ups? Or, should government publish to the cloud and leave mash-ups to third parties?

FR: Government should deliver transactional sites, e.g. to allow users to claim benefits or pay tax. Government should release data, and not spend time implementing visualisations.

NT: It’s very difficult to create a single point of entry for all government sites. Instead spend time on Search Engine Optimisation.

NT: Matapihi, the precursor to Digital NZ, attempted to index metadata from all New Zealand images. It’s hard to retrofit order on to chaos. However, it’s harder to mandate uniformity.

Audience: There have been efforts to create an “whole-of-border” portal for Customs, Bio-security, Food Safety Authority.

NT: A lack of common naming conventions can be a barrier to “post hoc” mashups. Agencies must have previously agreed on common identifiers. However, avoid creating a “New Zealand” ontology.

DR: The Smithsonian consists of over forty different museums. There has been tension between the central brand, and innovation from the individual museums. Our advice was stop worrying about the branding problem, but also to create portals across museums.

NT: Government could try to control the process at the schema level. Define the schema and require others to justify departures.

FR: Alternatively, agree a minimum agreed standard, e.g. 2-3 common fields.

Audience: Please use common identifiers.

Audience: There may be economic value for aggregation, but whose job is it to aggregate?

NT: Is aggregation useful? Yes, it sounds like there should be a single feed for job vacancies. It seems like there’s value in aggregation. Is the challenge that no one is funding it?

FR: There were about 20 quantitative measures of value from the UK’s Department of Culture, Media and Sport. However this measure is moving to a more qualitative model. People want comprehensive views of a dataset but it depends on use case.

NT: Please open up consultations.

Audience: Agencies determine their own way of reaching their audience.

FR: MySociety runs a planning application service where you can enter your postcode and see every planning application that affects your area.

NT: Users want to know that a dataset exists. Users want to understand the schema.

Audience: You need common identifiers such as geocode or postcode.

Audience: It’s enough to know that every agency is exposing their data in some form.

Audience: I want to know which datasets are available.

NT: Government should decide best practices and minimum standards and then advertise the existence of the datasets in some way.

Audience: There can be unintended consequences both positive and negative. For example, the third-party publication of league tables is discouraged, so should government not publish school performance data?

TS: If government doesn’t publish school performance data people will make league tables anyway, but the data will be unverifiable.

FR: The solution is to work with under-performing schools.

Audience: There are many examples of poor use and bad interpretation of statistics whether datasets are released or not.

NT: The important thing is to describe how the data was collected so that government can show data doesn’t support a faulty conclusion.

Summary

Derek Rayner is a Senior Technical Analyst in the GCIO, State Services Commission.

What should government do to open up a particular data type (of your choice) for re-use?

• How would you find it?
• What kind of formats and standards should be used?
• Funding and pricing?
• Licensing?
• Goodness?

AH: I’d like to see street layers of the entire country opened up: street coordinates, boundaries of parks, hydrology etc. Encourage people to contribute back to the dataset. Integrate with the OpenStreetmap project. There should be a two-way conversation. Citizens have a lot of valuable things to add to the collective.

TS: I’d like to see financial information made open, where the banking system sits, where companies have too much debt on the books.

FR: Open up transport data for re-use: timetabling, realtime information.

DR: Open up traffic data.

NT: Open up consultations. Open up property information. In the US sites like Zillow and Trulia allow you to find out past sales, current value, rates assessments etc. In New Zealand users must pay QV for this information.

Audience: Open up NIWA data such as average temperatures, rainfall etc.

Audience: Local government has a cost recovery policy. Central government information is largely free.

NT: Central government makes its money by making the economy bigger. Local government makes its money by raising rates.

Thanks to the panel, the audience, GOVIS and Webstock.

Technorati Tags: data, government, information, re-use