cyberinfrastructure status july, 2011. 2 nsf reverse site visit refactoring and cleanup after review...
TRANSCRIPT
![Page 1: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/1.jpg)
Cyberinfrastructure Status
July, 2011
![Page 2: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/2.jpg)
2
• NSF reverse site visit• Refactoring and cleanup after review preparations• Coordinating Node technology changes (reducing
latency, generalizing)• Authentication• Identity management• Access control• TeraGrid Compute Node design
2011 – First half
![Page 3: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/3.jpg)
3
• Public release of cyberinfrastructure• Synchronization, replication, authentication,
authorization, resolution, discovery, retrieval
• Coordinating Nodes• UNM, UCSB, ORC
• Member Nodes• KNB, Dryad, ORNL-DAAC, AKN, NBII, CUAHSI, replication
targets, Merritt, other Metacat, other Mercury
• Investigator Toolkit• Libraries, Web interface, Zotero, R, Morpho, DataONE
Drive
Goals for 2011
![Page 4: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/4.jpg)
4
The Major Components
Investigator Toolkit
Web Interface Analysis, Visualization Data Management
Client LibrariesJava Python Command Line
Member Nodes
Service Interfaces
Bridge to non-DataONE Services
Data Repository
Coordinating NodesService Interfaces
Object Store Index
Resolution
RegistrationReplication
Discovery
Identifiers
MonitorPreservation
Catalog
Identity Authen
![Page 5: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/5.jpg)
5
Investigator Toolkit SoftwareSearchPortal R Client Morpho
Client LibrariesJava Python Command Line
Member Node Software
Metacat
Coordinating Node SoftwareService Interfaces
Object Store Index
Software Delivered at Public Release
Zotero DataONE FS (Excel)
Dryad
GMN (CUASHI)
(Merritt) Preservation MonitorCatalogIdentifiers
Replication DiscoveryResolution Registration
Mendeley…
DataONE Service Programming Interface (SPI)
![Page 6: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/6.jpg)
6
Tier 1: Public read, synchronization, search, resolveTier 2: Read with access controlTier 3: Write using client toolsTier 4: Able to operate as a replication target
• Some Member Nodes with Tier 4 support and corresponding Coordinating Node functionality required for 2011
Member Node Tiers
![Page 7: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/7.jpg)
7
Node Tier Status Priority ReleaseKNB 4 2 1 2011ORNL DAAC 2 1 1 2011Dryad 2 ? 2 1 2011CUAHSI 2 1 1 2011AKN 4 1 1 2011Merritt 2 0 2 2011 ?Fedora ? 1 3 2012Repl. Targets (GMN) 4 2 1 2011TeraGrid 4 1 2 2012Mercury based 2 1 2 2011 ?Metacat based 4 1 3 2011?
Member Nodes
Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low
![Page 8: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/8.jpg)
8
Coordinating Node ServicesService Tier Status PriorityNode registry 1 2 1Object storage 1 3 2Identifier resolution 1 3 2Synchronization 1 1 1Discovery 1 1 1Log retrieval 1 0 2Identity mapping 2 2 2Session state 2 2 2Identifier reservation 3 3 3Replication 4 0 2
Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low
![Page 9: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/9.jpg)
9
Coordinating Node Components
cn_service
cn_rest cn_rest_proxy
Metacatidentity_manager
nodes_registry
synchro-nization
replic-ationindexer
SOLR Postgres
JMS
LDAP
MercurySearch
![Page 10: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/10.jpg)
10
Component Tier Status Priority ReleaseLibclient_python 4 2 1 2011Libclient_java 4 2 1 2011R plugin 2 (3) 1 1 2011D1Drive - FUSE 2 1 1 2011D1Drive - Dokan 2 1 1 2011D1Drive - WebDAV 2 0 3 2012Mercury UI 2 2 1 2011Morpho 3 1 2 2012Excel 3 1 3 2012Zotero, Mendeley 2 1 2 2011
Investigator Toolkit
Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low
![Page 11: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/11.jpg)
11
Collect
Assure
Describe
Deposit
Preserve
Discover
Integrate
Analyze
Data Lifecycle
11
Morpho
![Page 12: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/12.jpg)
12
Other PiecesComponent Status Priority ReleaseDocumentation 2 1 2011
Member Node test service 1 1 2011Integration testing 2 1 2011Monitoring 1 2 2011Hardware 2 1 2011
Continuous build 2Issue tracking 3Repository 3Conferencing 2
Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low
![Page 13: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/13.jpg)
13
![Page 14: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/14.jpg)
14
Infrastructure will change over time. • Message between services• Behavior of services, components• Message semantics
How to accommodate change while maintaining operations?
Versions
![Page 15: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/15.jpg)
15
Infrastructure Version: Tag identifying the overall functional capabilities, types, messages supported by the DataONE infrastructure
Component Version:Tag of a specific component indicating the current revision of the component
Example: Version XXX of Metacat supports version YYY of the DataONE infrastructure
Infrastructure + Component Versions
![Page 16: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/16.jpg)
16
1. Changes to Types Schema2. Update test data3. Build libcommon Pyxb / JIBX 4. Propagate changes through libclient5. Propagate changes through remainder of components6. Deploy CN test7. Update test Member Nodes8. Deploy staging9. Update / synchronize content to match production10. Switch staging and production
Where to update all Member Node stacks?Where to update all Investigator Tools
DataONE Infrastructure Release
![Page 17: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/17.jpg)
17
Release Coordinating Nodes• Touches everything deployed – MNs, ITK, monitor, ... • Likely to require content migration, synchronization• Expensive
Update Member Nodes• Vendor driven updates (e.g. core changes to Metacat)• DataONE driven updates (e.g. access policy fix)• Known contact points for updates
Update client tools• Automated version checks?• How to notify users?• How to notify third party developers?
Component Updates
![Page 18: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/18.jpg)
18
For an infrastructure upgrade:• What is the acceptable latency between change
announcement and infrastructure back in sync?• Should DataONE maintain old versions of service
interfaces? If so how many revisions?• Some changes may not be backward compatible• Option to make old version CNs readonly while content
migrated to new systems?
Version Latency and Acceptable Delta
![Page 19: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/19.jpg)
19
New package available with clear indication of affected components, signed off by component developersAffect CNs:
• Install on staging, process metadata updates, lock production, switch.
Affect MNs:• Notify instance administrators of update• Install updates and verify operation• Registry picks up updated services
Affect ITK:• Post update information• Notify active clients of update• Notify third party developers of changes
Administrative Process for Updates
![Page 20: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/20.jpg)
20
![Page 21: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/21.jpg)
21
July • New hardware deployed at UNM, UCSB, ORC• CN development with focus on Auth and synchronization• Integration test services
August • CN development focus on indexing, replication• Deploy CN staging to new hardware• ITK libraries updated• Full Tier 1 functionality deployed
September • Identity mapping, authentication, authorization• Evaluation of Tier 1 functionality• Support Tier 2 operations
October • Support Tier 3 and 4 operations on specific Member Nodes• CNs support replication, load testing of CNs• Evaluation of Tier 2 and initial tier 3 support
November • Final evaluation of functionality• Incorporate or schedule suggestions• Final stress and usability testing
December • Final documentation, testing• Announcement of initial release
General Timeline
![Page 22: Cyberinfrastructure Status July, 2011. 2 NSF reverse site visit Refactoring and cleanup after review preparations Coordinating Node technology changes](https://reader036.vdocuments.net/reader036/viewer/2022070406/56649dff5503460f94ae7c63/html5/thumbnails/22.jpg)
22