1

Government Challenges With Big Data:A Semantic Web Strategy for Big Data

Dr. Brand NiemannDirector and Senior Enterprise Architect – Data Scientist

Semantic Communityhttp://semanticommunity.info/

AOL Government Bloggerhttp://gov.aol.com/bloggers/brand-niemann/

January 24, 2013Link

2

Outline• Data Science Team• Semantic Community 2013 Mission Statement• Why We Are Here• NIST Cloud Computing AND Big Data Forum and Workshop• Spotfire for Big Data Analytics and Data Science Analytics Library• From the Year of Big Data to the Year of the Data Scientist Working With Big Data• Cross-Walk Table (in progress)• The Practice of Data Science• Current US Government Semantic Web Strategy• International Linked Open Data Strategy• Our Semantic Web Strategy for Big Data• My 5-Step Method To Get to 5-Stars With Open Data• System of Systems Architecture• Data Federation in Spotfire• 15th SOA, Shared Services, and Big Data Analytics Conference (DRAFT)• Comments: Semantic Medline, Noblis, Cray, and ORBIS Technologies• Q & A

3

Data Science Team

• Dr. Brand Niemann, Director and Senior Data Scientist, Semantic Community

• Dr. Tom Rindflesch, Research Group Lead for Semantic Medline, National Library of Medicine

• Dr. Victor Pollara, Senior Principal Scientist, Noblis• Dr. Eric Little, Director of Information Management,

Orbis Technologies• Mark Guiton, Director, Government Relations, Cray Inc.– Cray-YarcData announced semifinalists last month

• http://www.yarcdata.com/press-release-12-3-12.html

4

Semantic Community:Mission Statement for 2013

• Help the Data Transparency Coalition help the 113th Congress with the re-introduction of the Data Act by Building the Federal Financial Information Network in the Cloud for the 113th Congress, January 4, Slides.

• Continue to work with Big Data Analytics (e.g. Recorded Future, Spotfire, etc.), Content Analytics and Knowledge Management (e.g. MindTouch), and Semantic Technologies (e.g. Be Informed, Semantic Insights, etc.) for data science and data journalism. Slides.

• Help start Open Government Data for Japan (and the US and Europe) with the Right Data (Statistical) with the Right People (Data Scientists) Working on the Right Business Problems (Return on Investment): January 21, Slides.

• Help the Federal Big Data Senior Steering Group with A Semantic Web Strategy for Big Data and to move From the Year of Big Data to the Year of the Data Scientist Working With Big Data, January 24, Slides.

• Help the ACT-IAC AMWG, C&T SIG, and ET-SIG with Big Data on Mobile Devices, Collaboration and Transformation, & Government Challenges With Big Data , January 16 and February 23, Slides.

http://semanticommunity.info/#Welcome_to_Semantic_Community.info:_Community_Infrastructure_Sandbox_for_2013

5

Why We Are Here• Killer Semantic Web Application:

– George Strawn and Tom Rindflesch– Data Architecture (5 Steps to Get to 5 Stars and Systems of Systems)

• Audit Function:– Brand Niemann, Data Science Products at the US EPA and then for AOL Gov, etc.– Todd Park, Data.gov and Communities (Health, Energy, Safety, and Education)– Chris Greer, New Cloud AND Big Data Web Sites

• Using Science (and Statistics) as Evidence in Public Policy:– Kenneth Prewitt, NRC Committee Chair, and Former Census Bureau Director– Connie Citro, NAS National Committee on Statistics Director (CNSTAT)– Niall Brennan, Director of the Office of Information Products and Data Analytics at the Centers for Medicare and

Medicaid Services, and the Institute of Medicine (IOM) Study on Geographic Variation in Health Care Spending and Promotion of High-value Care

• Workforce Education:– Professor Peter Fox, RPI, Individual and Group Projects– Professor Jeffrey Hammerbacher, UC Berkley and Cloudera Chief Scientist, Exploratory Data

Analysis/Visualizations and Modeling– Professor Kirk Borne, GMU, Presentation to BDSSG Last September– Professor Michael Joseph, Big Data Pilot for First Year Engineering and Computer Science Students Last Semester

6

NIST Cloud Computing AND Big Data Forum and Workshop

• Microscopes and Telescopes for Big Data, Professor Alexander Szalay– My Note: Spotfire 5 Application

• Scalable Parallel Interoperable Data Analytics Library (SPIDAL), Professor Geoffrey Fox Large scale data analytics on clouds Keynote in CloudDB '12 Proceedings of the Fourth International Workshop on Cloud Data Management Pages 21-24, October 29, 2012.– My Note: Spotfire 5 Application

• Interoperability Between Clouds, Alan Yoder– My Note: MindTouch and Spotfire 5 Application

http://semanticommunity.info/Emerging_Technology_SIG_Big_Data_Committee/Cloud_Computing_AND_Big_Data_Forum_and_Workshop_January_15-17_2013

7

Spotfire for Big Data Analytics:Microscope

http://semanticommunity.info/Emerging_Technology_SIG_Big_Data_Committee/Government_Challenges_With_Big_Data#Spotfire_Dashboard

NASA GCMD: Gateway to Big DataNSF Big Data Awards: Follow the Work

OSTP Harnessing The Power of Digital Data Report: Well-Defined URLsPCAST Designing a Digital Future Report: Interoperability InterfaceNITRD Dashboards: Live DemonstrationsFour Clicks: See, Sort/Search, Download, & Share (iPad)

8

Data Science Analytics Library:Telescope & Library

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public

Live Links to Outside Data SourcesLive Information Links Between Analytics

9

From the Year of Big Data to the Year of the Data Scientist Working With Big Data

http://semanticommunity.info/Emerging_Technology_SIG_Big_Data_Committee/Government_Challenges_With_Big_Data#Story

Administration AnnouncementNSF Leads Federal Efforts

TechAmerica Report Case StudiesMy Big Data at the Hill ReportMy Cross-Walk Table

NASA Big Data Challenge Series:“How to make heterogeneous dataseem more homogeneous”.The Practice of Data Science

10

Cross-Walk Table (in progress)

http://semanticommunity.info/Emerging_Technology_SIG_Big_Data_Committee/Government_Challenges_With_Big_Data#Cross-Walk_Table

11

The Practice of Data Science

• Josh Wills, Data Scientist @Cloudera, writing on The Practice of Data Science said:– Key trait of all data scientists. Understanding “that the heavy lifting of [data] cleanup

and preparation isn’t something that gets in the way of solving the problem: it is the problem.” (DJ Patil)

– Inverse problems. Not every data scientist is a statistician, but all data scientists are interested in extracting information about complex systems from observed data, and so we can say that data science is related to the study of inverse problems. Real-world inverse problems are often ill-posed or ill-conditioned, which means that scientists need substantive expertise in the field in order to apply reasonable regularization conditions in order to solve the problem.

– Data sets that have a rich set of relationships between observations. We might think of this as a kind of Metcalfe’s Law for data sets, where the value of a data set increases nonlinearly with each additional observation. For example, a single web page doesn’t have very much value, but 128 billion web pages can be used to build a search engine.

– Open-source software tools with an emphasis on data visualization. One indicator that a research area is full of data scientists is an active community of open source developers.

12

Current US GovernmentSemantic Web Strategy

• Data.gov Advocates RDFa 1.1 Lite for Semantic Web Strategy.– See Comment From Owen Ambur on Next Slide.

• I believe there is a better way to handle this that I showed the W3C eGov Special Interest Group on January 21st and have recommended for the reintroduction of the Data Act to the 113th Congress.– Create a Semantic Index of Strong Relationships (SR) in RDF Format in a

Spreadsheet.• See next slide for example (spreadsheet and words)

– Integrate That With Other Spreadsheets and Relational Databases in An Interoperability Interface (e.g. Dashboard) That Can Searched.

• Essentially:– Computer Scientists Use RD2RDF (James Hendler)– Data Scientists Use SR2Excel2RDF (Brand Niemann)

13

Comment From Owen Ambur• OMB's official guidance to agencies on implementation of section 10 of

the GPRA Modernization Act (GPRAMA) says they may use XML, JSON, spreadsheets or CSVs in order to meet the requirement to publish their strategic and performance plans and reports in machine-readable format... but not PDF or HTML -- at least not without "enhanced structural elements".[1]

• I couldn't help but chuckle at how [1] is a PDF. I get your pointhowever, which I think reinforces mine, that there is no US federalpolicy that prefers RDFa 1.1 over HTML Microdata for publishingmetadata in HTML.– [1] RDFa Lite 1.1, W3C Recommendation, June 7, 2012, Manu Sporny, editor, see

http://www.w3.org/TR/rdfa-lite/• Source: Owen Ambur, December 18, 2012, W3C eGov Mailing List.

14

International Linked Open Data Strategy:Linked Open Data Cloud Data

http://semanticommunity.info/@api/deki/files/8824/=VIVO.xlsx

My Question: Is it easy to add columns for who links to who?Answer: Not in a single table. SPARQL can't do cross-tabulation (Richard Cyganiak).

15

International Linked Open Data:Comments to David Wood

• The Linked Open Data Cloud is not actually “linked data”.– RDF at Data.gov is not linked data.

• The analytical and statistical communities view Data.gov and Linked Open Data as “IT projects”.– Former Census Bureau Director Robert Groves.

• Conventional tools can do linked data and data integration.– Spotfire Information Designer, Informatica, Information

Builders, etc.http://manning.com/dwood/LinkedData_MEAP_ch1.pdfhttp://semanticommunity.info/AOL_Government/Exploiting_Linked_Data_with_BI_Tools

16

International Linked Open Data:My EPA Green App Data App Example

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?EPAGreenAppsDataApp-Spotfire

17

Our Semantic Web Strategy for Big Data:Previous Presentations

http://semanticommunity.info/Emerging_Technology_SIG_Big_Data_Committee#A_Semantic_Web_Strategy_for_Big_Data

MITRE: October 2, 2012GSA: November 29, 2012

18

Our Semantic Web Strategy for Data:Simple Explanation

• One Table:– Two Columns

• Example: Column 1: Section and Column 2: URL• Note: A Column 3: Description could be in the URL• Example: See Next Slide

– Three Columns:• Example: Column 1: Subject, Column 2: Object, and Column 3: Predicate• Note: This is the Semantic Web’s Linked Open Data Cloud as Linked Open Data for

Network Analytics!• Example: See Semantic Medline

– Four Columns:• Examples: Column 1: Subject, Column 2: Attribute, Column 3: From, and Column 4: To,

or Column 1: City, Column 2: Country, Column 3: Longitude, and Column 4: Latitude• Note: This is the format for Spotfire’s Network Analytics Module developed for the CIA• Example: See Semantic Medline

19

Our Semantic Web Strategy for Data:NASA Big Data Example

http://semanticommunity.info/@api/deki/files/20313/NASABigData.xls

20

Our Semantic Web Strategy for Data:Spotfire Network Analytics

http://semanticommunity.info/AOL_Government/Social_Media_-_Six_Degrees_of_Separation_and_Now_Even_Less

21

My 5-Step Method• So what I like to do to illustrate (data science) and explain (data

journalism) is the following (like a recipe):– Put the Best Content into a Knowledge Base (e.g. MindTouch*)

• NASA Big Data– Put the Knowledge Base into a Spreadsheet (Excel*)

• Linked Data to Subparts of the Knowledge Base– Put the Spreadsheet into a Dashboard (Spotfire*)

• Data Integration and Interoperability Interface– Put the Dashboard into a Semantic Model (Excel*)

• Data Dictionaries and Models– Put the Semantic Model into Dynamic Case Management (Be

Informed*)• Structured Process for Updating Data in the Dashboard

* Examples of tools used.

22

To Get to 5-Stars With Open DataStar Definition Example / Tool*

Make your stuff available on the Web (whatever format) under an open license

This Story / MindTouch

Make it available as structured data (e.g., Excel instead of image scan of a table)

Spreadsheet / Excel

Use non-proprietary formats (e.g., CSV instead of Excel)

Table / MindTouch and Spotfire

Use URIs to identify things, so that people can point at your stuff

Table of Contents / MindTouch and Spotfire

Link your data to other data to provide context

Table / MindTouch and Spotfire

* Examples of tools used.Source of Star and Definition: http://www.w3.org/DesignIssues/LinkedData.html

23

System of Systems Architecture

SSemantic Index ofLinked Data(e.g. Excel)

Dynamic Case Management (e.g. Be Informed)

Data Science Library (e.g. Spotfire)

Data Science Products (e.g. Spotfire)

24

Data Federation in Spotfire:In-Memory and In-Database Data

• In-Memory Data– When you are working with in-memory data tables (text files, Excel files,

information links, etc.) you have access to all the functionality of Spotfire. You have the opportunity to use all columns as filters and perform any number of calculations. You can also use any of the tools within Spotfire to cluster data, calculate new columns, bin columns, make predictions etc. See Working With Large Data Volumes for some tips on how to improve the performance of an analysis with lots of data.

• In-Database Data– When a connection to an external source is set up, all calculations are done

using the external system and not with the Spotfire data engine. This will allow you to work with data volumes too large to fit into primary memory and take advantage of the power of the external system. When working with external data connections, you access only the current selection of data and all aggregations and calculations are made in-database (in-db).

25

Data Federation in Spotfire:Database Connections, Information Links, & Analytics Library

• A database connection dialog is used to set up a connection to say a Teradata database, where you can analyze data from the database without bringing it into your analysis.

• An information link is a structured request for data which can be sent to the database. These specifications include one or more columns, and may include one or more filters.– Stated in plain English, an information link could

be: "Fetch the Name, Address and Phone_number for employees that pass the filter High_Income."

– Information links can also be used to limit what data to open in an analysis in a number of different ways.

• The library provides publishing capabilities for all of your analysis materials, so you can share data with your colleagues. The library can be used directly from Spotfire by anyone who has at least read privileges.

26

Data Federation in Spotfire:Data Panel

• The Data panel is used to get an overview of the columns in all data tables, in-memory as well as in-database (in-db). When working with in-database data the Data panel is the starting point for configuring both visualizations and the filters panel, since no filters are created automatically for external data.

https://silverspotfire.tibco.com/us/library#/users/bniemann/Public?NASABigData-Spotfire

27

Data Federation in Spotfire:Information Designer

• The Information Designer is a tool for setting up data sources and creating and opening information links. An information link is a database query specifying the columns to be loaded and any filters needed to narrow down the data table prior to creating visualizations in Spotfire. In Information Designer, information links are created from building blocks such as columns and filters using joins, calculations and aggregations.

http://semanticommunity.info/Build_DoD_in_the_Cloud/Enterprise_Information_Web_for_Semantic_Interoperability_at_DoD/Spotfire_Data_Federationhttp://semanticommunity.info/Build_DoD_in_the_Cloud/Enterprise_Information_Web_for_Semantic_Interoperability_at_DoD/Spotfire_Information_Designer

28

15th SOA, Shared Services, and Big Data Analytics Conference (DRAFT)

• Jason Bloomberg, President, ZapThink, a Dovel Technologies: New Book and Poster.– The Agile Architecture Revolution: How Cloud Computing, REST-Based SOA, and Mobile Computing Are Changing

Enterprise IT• Steven Woodward, CEO and Founder of Cloud Perspectives, Member of the Shared Services Canada Advisory Committee,

Leading Contributor to the NIST FedRamp, and Canadian Cloud Council Government CTO.– Shared Services Canada

• Geoffrey Fox, Professor of Informatics and Computing, Physics, Indiana University– Large Scale Data Analytics on Clouds

• Dr. Christopher L. Greer, Acting Senior Advisor for Cloud Computing and Associate Director, Program Implementation Office, ITL, NIST.– NIST Cloud Computing AND Big Data Forum and Workshop

• NITRD Interagency Big Data Senior Steering Group (BDSSG) Representative (s)– What Does the Interagency Big Data Senior Steering Group Do?

• Peter Lyster, Program Director, Division of Biomedical Technology, Bioinformatics, and Computational Biology, National Institute of General Medical Sciences, National Institutes of Health, (NIH/NIGMS).– NIH Chief Data Scientist.

• Adam Fuchs, Chief Technology Officer, Sqrrl– Accumulo for the NSA and Sqrrl for Open Source

• Will Cukierski, Data Scientist, Kaggle– What is a Data Scientist?

• Big Data Use-Cases and Pilots– Data Science Teams working on Semantic Medline and Other Government Big Data Projects

http://semanticommunity.info/Federal_SOA

29

Comments: Semantic Medline, Noblis, Cray, and ORBIS Technologies

• Semantic Medline, Tom Rindflesch• Noblis, Victor Pollara• Cray, Mark Guiton• ORBIS Technologies, Eric Little

30

Q & A

• Contact Information:– Brand Niemann

• bniemann@cox.net– Tom Rindflesch

• trindflesch@mail.nih.gov– Victor Pollara

• victor.pollara@noblis.org– Mark Guiton

• mguiton@cray.com– Eric Little

• elittle@orbistechnologies.com