open data presentation 2014 v1.3 - nov 2014

1

The Shift to {Open|Big|Linked} DataOr, how I learned to stop worrying and love the data

Pia WaughDirector of Coordination and Gov 2.0

Technology and Procurement Division

Department of Finance

22

Great expectationsPublic expectations and problem solving

http://www.flickr.com/photos/zebble/8212264/

33

[Open|Big|Government] Data

Credit: Joel Gurin (with permission)

44

http://xkcd.com/898/

The emergence of the technocracyOr... Remember to engage your geeks!

55

Key Benefits to Community/Industry in Opening Data

Economic

• Creates opportunities for industry to value-add to government data

• New services, systems and industries

• New opportunities and innovation in industry, research, civil society

Accountability

• Visibility to government spending, projects, effectiveness, etc

• Increases incentive to follow evidence based approach

Better policy and programs

• Enables greater participation in policy planning and implementation

• More informed public → better decision making

• Improvements to data → better policy and decisions

66

Key Benefits to the Public Service in Opening Data

• Cheaper and more modular services delivery

• Efficiency gains in publishing and answering requests

• Reduced regulatory burden through machine readable data supporting industry and agency compliance

• Better policy outcomes by leveraging cross-agency data

• More consistency across government(s)

• Improved opportunities to leverage innovation and collaboration (citizens, industry, other depts)

• Opportunities to improve data quality through verifiable public contributions

77

Key Case Studies

• Publishing Budget 2014 Data Report

• Open data – Transforming the Provider / Stakeholder Paradigm

• On the Value of Open Roof Prints

• 100 years of patent and IP data released on data.gov.au

More available soon at http://toolkit.data.gov.au

Other Australian case studies/documentation

• SA Open Data Toolkit

• QLD Government Case Studies

• Victorian Government Showcase

• NSW Apps Showcase

• ACT examples

88

The APS eGov and Open Data Policy Landscape

Others:

• Publishing Public Sector Information & National Standards Framework

• Declaration of Open Government

• Gov 2.0 Taskforce Report

• Statement of IP Principles for Government (CC-BY)

• Ahead of the Game

• Digital Transition Policy (Archives) & Accessibility Policy

• Emerging Open Research Policies

• Open Public Sector Information: From Principles to Practice Report

99

State and Territory Policies

1010

Policies Components

APS policies in aggregate:

• Permissive copyright – CC-BY as the default

• Open by default, machine readable accessible data

• Support reuse and innovation

• More public engagement

• Better use of data for government policy and service development

States/Territories add:

• Procurement – open by design

• Reporting – dashboards

• Departmental strategies

• Declaration of Open Data

1111

Policy and Implementation in the Commonwealth

Department of Finance

(CTO & AGIMO)

Department of Communications(Data Policy Branch)

Data Efficiency Working Group

(Project 4)

Project 4 Implementation

Sub-groupOpen Data

Delivery Network(CTO & Spatial Policy Branch)

• Data.gov.au• FIND• NationalMap• NEII• ABS data delivery • Other gov data gateways

• Identify datasets• Release datasets• Identify shared services• Develop private/public

partnerships

• Spatial Policy• Open/Big/Spatialstrategic planning

• Open & Big data policy• Open/Big/Spatial strategic planning

ImplementationPolicy and Planning

Finance

Communications

Joint

Open Data Community Forum

(CTO coordinated)

• Cross-jurisdiction collaboration• Data.gov.au publishing community• Bi-monthly meetings & training• Online forum

1212

Simplified Guidance, Policy and Case Studies

http://toolkit.data.gov.au

1313

History of open data in Australia

Gov 2.0 Taskforce Report 2009 (based on PoIT UK). Led to:

1) Declaration of Open Government

2) CC-BY as default

3) Information Commissioner

4) data.gov.au and social media support/policy

5) Cloud/shared services

Myriad supporting tech and copyright policies over time

States/Territories, Federal and Local now largely aligned on open data

1414

Data Portals

Local Portals:• City of Melbourne

1515

Open Data Discovery Model

National Map

Datavis Applicationdevelopment

Analysis& Policy

Value Creation

Discovery

Data

Full Discovery

New Services

FINDNational

spatial index (gov, private,

research)

1616

data.gov.au

Free, cloud, scalable API enabled platform for hosting government data.

Staged approach

1. Publishing (2013 – mid 2014)Improving the functionality and ease of

publishing for agencies with training and

documentation

2. Value realisation (Late 2014)Providing useful front end tools for data.gov.au

including data visualisation and analysis tools.

Publishing quality data a pre-requisite.

3. Data quality (Late 2014)Looking at ways to provide agencies the ability

to accept iterative data improvements in a

verifiable way

Features

•Manual and automated publishing options

• API access to government data

• Easy to publish, download & interact

• Use cases and site|data|org analytics

• Data Request Site

• Metadata harvesting from gov data gateways

• National Map integration

• Data model registry

In Planning

• 5 star quality plugin

• Selective crowdsourcing for updates

• Federated search for discoverability

• League Table

1717

NationalMap

1818

1919

DSS and ATO Statistics

2020

Spatial Case Study

http://www.finance.gov.au/blog/2014/02/24/guest-post-on-the-value-of-open-roof-prints/

1) Reduced work

2) Value adding

3) Quality improvements

4) Zero cost API access to their own data

2121

The Government Data Landscape (latest version online)

2222

Other Data Projects

• Administrative

• Spatial

• Research data

• Imagery

• Sensor

• Realtime (eg Transport)

• Census/Statistics

• Cultural

• Data about government

• Fiscal

2323

Data Integration

• Challenging but great potential for improved policy/services.

• Unit record sharing is complex, raises privacy concerns.

• Unit record data is mostly useful to researchers, who have appropriate mechanisms with legal, technical, ethical constraints to access such data.

• Data aggregated by common spatial boundaries is comparative across datasets and over time.

• Unfortunately, data owners traditionally aggregate to boundaries that constantly change (electorates, postcodes, etc).

• The Australian Statistical Geography Standard (ASGS) provides a consistent set of spatial boundaries that can be mapped to other needs.

• Anonymisation on the fly APIs also provide mechanism for appropriate public/agency access to unit record level data (e.g. ABS.Stat)

http://statistical-data-integration.govspace.gov.au/

https://toolkit.data.gov.au/index.php?title=Definitions#Types_of_data

2424

Open by Design – drawing a line in the sand

Building proactive publishing into:

• Systems

• Processes

• Procurement

• Planning

• Records management

Leveraging open data through:

• Public APIs

• Analysis tools and datavis

• Internal processes looking for external sources

• Existing databases!

2525

Some Challenges

• Education

• Legislative

• Culture

• Systems

• Privacy and anonymisation

• Reactive vs proactive

• Metadata/semantic context

• Too much data

• Real time vs historic

• Definitions and common references

• Limited skills and over specialisation

2626

New and Old Skills Required

• Publishing and Automation

• Project management, reporting

• Metadata/linked data

• API development and serving

• Plumbing between systems

• Analysis and statistics

• Policy development

• Data, info and policy visualisation

• Public consultation and engagement

• Online community management

2727

Hypothetical: Government as an API

Care of fedAPI.gov

2828

The future is here....

And it is already widely distributed

http://www.flickr.com/photos/mr_matt/3568892622/

Challenge #1: Collaborate

Challenge #2: Share

Challenge #3: Measure

Challenge #4: Play

Questions?@piawaugh

@datagovau

data.gov.au

toolkit.data.gov.au