open data tutorial at icegov

Open Data Tutorial

ICEGOV

Jim Hendler, @jahendlerJeanne Holm, @JeanneHolm

22 October 2012

Co-author: Hadley Beeman, @HadleyBeeman

ICEGOV Open Data Tutorial 22012 INTERNATIONAL OPEN GOVERNMENT DATA CONFERENCE—OPEN GOV

DATA TUTORIAL

Introductions!

• Please introduce yourself– Name– Organization– Three (3) words that explain either why you are

here or what you hope to learn

9 July 2012


DATA TUTORIAL

Understanding the Foundations of Open Data

• Why do countries and people share data?• What will citizens, businesses, scientists, and

journalists do with the data?• How can we manage it?

9 July 2012


DATA TUTORIAL

Why Countries Share Data

• Meet regulatory compliance• Provide transparency into government

operations

9 July 2012


DATA TUTORIAL

Why Countries Share Data

• Anticipate economic development• Initiate innovation

9 July 2012


DATA TUTORIAL 9 July 2012


DATA TUTORIAL

Why People Want Open Data

Swati Ramanathan9 July 2012

2012 INTERNATIONAL OPEN GOVERNMENT DATA CONFERENCE—OPEN GOV DATA TUTORIAL

Real Outcomes = Better Lives

• In health care– Data empowers communities to make changes

that improve the quality of life of citizens• In California, ReLeaf plants trees in areas identified as

danger areas for asthma sufferers

– Companies use government data to innovate and create high-value jobs

– Civic Commons has a great collection of good open use cases: http://civiccommons.org/

9 July 2012 ICEGOV Open Data Tutorial 8

http://civiccommons.org/

http://civiccommons.org/


Energy Drives Innovation

• Communities like Energy.Data.gov connect innovators, industry, academia, and government at federal, state, and local levels



Challenges Spark Ideas

• Energy.Data.gov connects works with challenges across the nation to integrate federal data and bring government personnel to code-a-thons



Data Drives Decisions

• Apps transform data in understandable ways to help people make decisions



DATA TUTORIAL

Changing Economic Equations

Study from Malaysian government: http://www.transknowformance.com/article.cfm?id=53

9 July 2012

http://www.transknowformance.com/article.cfm?id=53

http://www.transknowformance.com/article.cfm?id=53


DATA TUTORIAL

Why People Want Open Data

9 July 2012


DATA TUTORIAL

What Makes Data Open

• Open Format– The US Government through the Open

Government Directive defines an open format as “one that is platform independent, machine readable, and made available to the public without restrictions that would impede the re-use of that information.”

• http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-06.pdf

9 July 2012

http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-06.pdf





DATA TUTORIAL


• Example Open Formats– PDF for documents (but not data)– CSV for data– Web standards for publishing, sharing or linking

• HTML, XML, RDF

– Web standards for syndication• RSS, Atom, JSON

9 July 2012


DATA TUTORIAL


• Metadata– The information about the data being shared

• Who produced it• Where• When• Use restrictions• Etc.

– Use standards such as ADMS or Dublin Core

9 July 2012


DATA TUTORIAL

Dataset extension to Schema.org (pending): Google, MS (Bing), Yahoo!

• Improve SEO• Improve international

search and federation• Unique opportunity

for public/private partnership

9 July 2012

Express your support at: http://blog.schema.org/2012/07/describing-datasets-with-schemaorg.html

http://blog.schema.org/2012/07/describing-datasets-with-schemaorg.html

http://blog.schema.org/2012/07/describing-datasets-with-schemaorg.html


DATA TUTORIAL

What Topics of Data Are Published

• Analytics based on over 1,000,000 datasets from around the world can be seen at – http://logd.tw.rpi.edu/iogds_data_analytics

• The examples that follow are from that page

9 July 2012

http://logd.tw.rpi.edu/iogds_data_analytics

http://logd.tw.rpi.edu/iogds_data_analytics


DATA TUTORIAL

Countries Sharing Data

Important note:quantity is not really the most important issue

9 July 2012


DATA TUTORIAL

Example: U.S.

Data.gov

9 July 2012


DATA TUTORIAL

Example: UK

Data.gov.uk

9 July 2012


DATA TUTORIAL

Example: Spain

9 July 2012


DATA TUTORIAL

Topics (Across All Catalogs)

9 July 2012


DATA TUTORIAL

Data “Mashups” of Many Kinds

More than 50 at http://logd.tw.rpi.edu

9 July 2012

http://logd.tw.rpi.edu/


DATA TUTORIAL

Making Data Open, Accessible, and Discoverable

• Architecture for systems and technology• Processes for publishing data• Policies for ensuring data is open, accessible,

and obtainable

9 July 2012


DATA TUTORIAL

Creating an Open Data Architecture

• Key components– Workflow for release approval (often overlooked)– Dataset storage

• Can be centralized or via linking

– Data cataloging• Metadata critical to a good open

data site

– Data API• Can be via download or via access• Technical issues with syndication, usage rules, etc.

9 July 2012


DATA TUTORIAL

Processes

• Publication (and cleaning)• Data reuse and integration• Community input

9 July 2012


DATA TUTORIAL

Policies Become Essential

• Policies help drive the ecosystem and “motivate” departments to continue to share data openly

• Build the policies based around issues that are universal • Licensing, provenance http

://creativecommons.org/licenses/

9 July 2012

Open data on food, security,

transportation, and transparency

http://creativecommons.org/licenses/




DATA TUTORIAL

Semantic Web and Linked DataCounty Council

Ordnance Survey

Royal Mail

9 July 2012


DATA TUTORIAL

Linking Data Via Common Naming (Usually URLs)

9 July 2012


DATA TUTORIAL

Example: Agency Names

9 July 2012


DATA TUTORIAL

Can Be Lots of Things

9 July 2012


DATA TUTORIAL

“Linking” Data

http://linkeddata.org/

Government data is currently over half the cloud in size (~17B triples), 10s of thousands of links to other data (within and without)

9 July 2012


DATA TUTORIAL

5 Star Data

9 July 2012


DATA TUTORIAL

Creating the Open Data Community

9 July 2012


DATA TUTORIAL

Creating Community

• Communities are public-facing spaces that present data, information, and subject matter knowledge about a single topic from many organizations in one place– The topics for communities can be

chosen based on priorities from the public, departments based on their mission, or issues of national importance

9 July 2012


DATA TUTORIAL

Community Vision

• These questions help to guide early discussions1. Vision: What will the community connection and collaboration look

like in the future?2. Leaders: Who will help to lead the community?3. Participants: Who will participate?4. Outcome: What are the expected outcomes, metrics, and

measurements that will show success? How will this community work to improve the lives of citizens?

5. Functionality: What types of activities will be conducted on the site (forums, blogs, wikis, ranking, rating, challenges, or apps)?

6. Content: What content should be displayed7. Interactivity: What ways will the community interact with the

leaders, with each other, and with the public?

9 July 2012

402012 INTERNATIONAL OPEN GOVERNMENT DATA CONFERENCE—OPEN GOV

DATA TUTORIAL

Open Communities

ICEGOV Open Data Tutorial

Community

Developers ✓

Open Data ✓

Semantic Web ✓

Health ✓

Law ✓

Energy ✓

Education ✓

Ocean ✓

Safety ✓

Manufacturing ✓

Business ✓

Ethics ✓

Smart Disclosure ✓

Sustainable Supply Chain ✓

Cities ✓

+ many more…

9 July 2012


Supporting Global EventsJapanese tsunami, earthquake,

and radiation monitoring

Restore the Gulf: Deepwater Horizon

Response

ICEGOV Open Data Tutorial 419 July 2012


Health.Data.gov

42

Champion: Todd ParkU.S. Chief Technology Officer

Apps Forums

Challenges

Blogs

ICEGOV Open Data Tutorial9 July 2012


Publicizing Data to Innovators

43

• Challenges and code‐a-thons (health2challenge.org)

• Many innovator “meetups” and conferences

• Annual health data-paloozas• Over 139 applications• 50 new businesses• Thousands of lives improved

each day• 1700 attendees at the Health

Data Palooza in 2012



Creating Apps That Improve Lives: Asthmapolis

44ICEGOV Open Data Tutorial9 July 2012

452012 INTERNATIONAL OPEN GOVERNMENT DATA CONFERENCE—OPEN GOV

DATA TUTORIAL

Creating Apps That Save Lives: iTriage and Hospital Compare



DATA TUTORIAL

Additional Topics

• Licensing, provenance, languages• Metadata design (international)• Trust – government data is controversial, who

controls it?• Scaling – over 1M datasets and growing fast

– How to search, store, link, translate, and archive• Versioning and updating• Visualization beyond the single dataset• Boundaries of open data9 July 2012


DATA TUTORIAL

Summary• More and more government agencies throughout the world are sharing “raw

data” with their citizens– Enhance transparency– Increase Innovation– “Crowd source” government services

• Open data in open formats allows governments, agencies, and third parties to develop analyses, information graphics, and other ways to share information

• Development of correct processes and policies are an important aspect of Open Government Data sharing– Need to support, not squelch, information sharing– Need to find appropriate balance of data release with privacy, security, and other

citizen/government mandates

• Community mechanisms are an important aspect of an open data ecosystem

9 July 2012


DATA TUTORIAL

Questions

9 July 2012


DATA TUTORIAL

Summary and Next Steps

• Join a community– W3C eGovernment Interest Group

• http://www.w3.org/egov/wiki/Main_Page

– Open Data Innovation Network on LinkedIn• http://bit.ly/ODNetwork

9 July 2012

http://www.w3.org/egov/wiki/Main_Page

http://www.w3.org/egov/wiki/Main_Page

http://bit.ly/ODNetwork



open data tutorial at icegov

Documents