november 7th, 2012 mike daniels eol data services update: legacy data, earthcube and the earth...
TRANSCRIPT
November 7th, 2012
Mike Daniels
EOL Data Services update: Legacy Data, EarthCube and the Earth Science Information
Partners (ESIP)
Legacy Data
From the 2011 EOL Science Review Team Report
“The EOL effort to rescue and digitally archive historical field data that are presently stored in older paper, chart, and tape formats will be an extremely valuable information asset for research requiring long-term records.”
“Continue progress toward and maintain a priority for making current and historical field data easily accessible”
(2003!) Data Lessons Learned
Climate Science 1.0:Observational data is very
similar to model data
Time
Val
ue
Obs data
Model data
Climate Science 2.0: Observational data is very different from model data
IntegratedParameter
EngineeringUnits
Parameter
DerivedParameter
Metadata
Metadata
Metadata
QC
QC
QC
Legacy Data Activities, 1st proposed in Spring 2010 (FY11 Annual Budget Review)
• Data and Metadata Rescue
• Project Documentation - Collect, digitize and webify- Improve documentation
• Digitize Aircraft Videos
• Media and Format Migration (e.g. GENPro to NetCDF)
• NCAR Iron Mountain Archives
• Embarked on two pilot projects:
• Sounding Legacy Project (CSU)
• T-28 Facility Archive (SDSMT)
1974
Protecting the legacy of current and future datasets: the EOL Data Audit
• What is the data flow (from sensor to archive) of all EOL datasets, including individual instruments and HAIS? Who is responsible for key aspects of DM for these datasets? Do we have backup copies of recently acquired data?
• Does each dataset have the necessary metadata in a recognized format? Are these metadata sufficient?
• Are all EOL data secured in a common archive that is linked to other portals?
• Are the data formats and access methods we use for these datasets today adequate for future communities?
New Programmer/Technician Position
One-time funds of .40 FTE allocated in FY13:
C. Brooks Snyder, a student assistant in EOL, has accepted the position!
EarthCube
earthcube.ning.com
EarthCube: What is it?
Community, not NSF, driven
Multi-disciplinary across the geo- bio- cyber- sciences and “from sun to the center of the earth”
Proposes to reduce efforts spent on data issues to free up time for research
Strongly supported by the GEO Directorate
Addresses data issues from point observations to peta-scale models
A Data andKnowledgeManagement system for the next decade and beyond
• “Charette” #1 November 2011, 106 whitepapers received which created a list of common E3 capabilities (Mike Daniels participated virtually)
• EarthCube Stakeholder Alignment Survey (sent to EOL Scientists)• Early 2012: “Community Groups” formed to gather user requirements• “Concept Award Groups” formed to develop prototypes• Less formal “Special Interest Groups” address EarthCube-related issues
not covered by the existing Community and Concept Groups.• Charette #2 June 2012 (KateYoung and Mike Daniels in attendance,
other EOL staff participated virtually)• Defined roadmaps of next steps for moving groups forward• Vignette’s (use cases) from EOL submitted to EarthCube’s Workflow
Community Group• E^3 moving toward domain-specific workshops and community events• Governance Group has plans to be in place by Spring 2013
What has happened thus far
EarthCube Groups and Awards
Concept Awards – Develop Prototypes Brokering
Layered Architecture
Web Services
Earth Systems Modeling
Interoperability
Special Interest Groups Education and Workforce
Development
Software Engineering Community of Practice
Website and Collaboration Environment
Early Career
Data integration for Biology and Earth Science
Physical Samples Cyberinfrastructure
Data Discovery, Access and Mining
Workflow
Governance
Semantics and Ontology
Working Groups – Form Requirements
• Set out concrete example cases, showcasing how real-time data can be leveraged and addressed within EarthCube.
• Bring together those in the real-time data community.• Focus on modeling efforts that require data quickly to aid in emergency
management, forecasting, decision-making, risk assessment, deployment of resources, etc.
• Determine how the needs of real-time data researchers, developers, data providers and stakeholders affect the overall EarthCube system architecture.
• Discuss framework architectures and ideas relating to the distributed, online, highly interactive and time critical real-time operations.
• Discussions of adaptive sensing networks, un-tethered sensing systems and distributed real-time observations occurring around the globe.
• Explore concepts related to real-time quality control and monitoring. • PI: M. Daniels, Co-PIs: Dr. V. Chandrasekar (CSU), Dr. S. Graves (University
of Alabama), Dr. B. Kerkez (University of Michigan), Dr. F. Vernon (UCSD)
Real-time data E3 workshop
www.esipfed.org
• Established to bring NASA groups together as an outcome of a National Academy report in 1998, UNIDATA was a Charter Member
• Globe, NOAA, NSIDC accepted as members in 2004 but no NCAR(!)• I led an organization-wide effort for NCAR to become a member, we were
accepted in January 2012• ESIP Vision Pillars
• Provides the Earth science informatics intellectual commons to drive innovation
• Is the trusted community authority that supports the integration of science and data into mainstream use
• Leads the development of the science data information profession• Achieves sustainability through diversification, global partnerships and
partner recognition• Members are recognized as leaders in Informatics at AGU and other venues• Represents a forum to allow EOL and NCAR to develop and expand our
data services tools and technologies
History and Purpose
Organic Growth of Topics
Cluster
Working Group
Standing Committee
(10) Very Informal, Monthly Telecons: e.g. Cluster on Climate Forecasting Conventions
(6) Fixed Duration, Task Oriented: e.g. Working Group on Data Management Training
(4) Formal, Chair is member of ESIP Assembly: e.g. Committee on Data Preservation and Stewardship
Annual Meetings: Winter Business and Summer Technical
Semantic Web:Deborah McGuinness
Open Source Summit
Comparison of EarthCube and ESIP
FUNding Friday!
This year, all proposers were winners of a $5K award!
