qcloud queensland cloud data storage and services 27mar2012 qcloud1
TRANSCRIPT
QCloud 2
NATIONAL RDSI
• 6 primary, 4 additional nodes• 150-200 PB by 2014• 100Gb/sec node interconnect
• Storage for large national collections• Shared access by research communities• Research data management• On-shore data cloud• Low-cost research production quality of service27Mar2012
QCloud 3
NATIONAL NeCTAR RESEARCH CLOUD
• 6 primary nodes (maybe 7)• 24,000 cores by 2013
• On-demand compute capacity for research• Hosted shared services for research communities• Virtual laboratories• Research tools, workflows
Uni of Melbourne QCIF/UQ ANU
Monash Uni27Mar2012
QCloud 4
QCLOUD COMPONENTS
RDSIDATASTORE
NeCTARRESEARCH
CLOUD
QERN CLOUD
GENOMICS CLOUD
5x
>15x
27Mar2012
QCloud 5
QUEENSLAND RDSI NODES
• ~ 30 Petabytes by 2014• Primary node hosted at UQ (embargoed)• Additional Node at JCU (awaiting decision)• National focus: ecosystems, genomics
75% Merit Allocated Collections 25% “CollectionDevelopment”
Can add RDSI+ commercial research storage
27Mar2012
QCloud 6
COLLECTIONS
Market Data CollectionsActive research projects
Collaborations across multiple locationsFast disk, flash memory – immediate access
Vault Data CollectionsRarely accessed data archival storage
Broad availabilityTape, slow disk
27Mar2012
QCloud 7
RESEARCH COMMUNITY
RDSI NODE SERVICESOWNERQualification
CaptureIngest
Bulk Update Curation
Metadata (with ANDS)Resilience (multiple copies)
ManagementEntitlements to Access
MonitoringSupport, Help Desk, Training
USEREntitlements to Access
DiscoveryRead
UpdateDuplicate
MonitoringSupport, Help Desk, Training
27Mar2012
QCloud 8
TARGET RESEARCH USERS/SERVICESPOWER USERSPB-scale data collectionsLow-level access servicesSpecialised treatment
LARGE DATA USERSTB-scale data collectionsDirect data access, FTPAutomated allocations
LONG-TAIL USERSGB-scale data collectionsCloud accessDropbox style on PCLittle allocation surveillance
DATA
CO
LLEC
TIO
N S
IZE
NUMBER OF USERS27Mar2012
QCloud 9
QCIF/UQ NeCTAR NODE
• 4,000 cores, up to 4,000 virtual machines• Research service quaility• 70% for NeCTAR Allocated Services• 30% for Queensland Allocation
27Mar2012
QCloud 10
SERVICES
HOSTED SERVICESPermanent web sites
Research communitiesVirtual laboratories
Research toolsCollaboration
COMPUTATIONOn-demand virtual
machinesOff-load “small” jobs from
HPC/clusters
27Mar2012
QCloud 11
COMPUTE USERS/SERVICES
LONG-TAIL USERSHOSTED SERVICES1-4 coresSmall HPC jobsN
UM
BER
OF
CORE
S RE
QU
IRED
NUMBER OF USERS
Tier 1,2,3 HPCMore than 16 coresSpecialised treatment
MEDIUM COMPUTE USERS4-16 coresLarge memory VMs (1TB+)Automated allocationsSmaller HPC jobs
27Mar2012
QCloud 12
EARLY NODE (QERN)• pQERN– Operational October 2011– Pre-RDSI/NeCTAR– Genomics++
• QERN– Delivered February– Operational ~April– 330 cores – 0.5 PB disk– RDSI/NeCTAR services– Early adopter researchers– Operational experience27Mar2012
QCloud 13
PROJECTED NODE SCALE-UP
27Mar2012
Cores4000
3000
2000
1000
0
2011 2012 2013 2014
NeCTAR Research Cloud
RDSI Data Storage
Today, QERN
QCloud 14
PARTICIPANT ROLES
27Mar2012
QCIF Governance, Management, Research Support
RDSI Lead Agent
NeCTARLead Agent
UQ RCCManagement,
SysAdmin
JCUManagement,
Operations
UQ ITSAcquisition, Operations
PartnersResearch Support
Collection & Service Owners & Users
QCloud 16
QERN Progress
27Mar2012
Familiarisation• Hardware installation• Software systems
Service Definition and
Delivery
• COO and Solution Architect start• Service catalogue• Software stack• Registration processes• Trial usage
Production• Early adopters• Basic services• Continuing service introduction
Primary Node(Jul2012)
Mid Feb
QCloud 17
QCLOUD PARTNERS
• QCIF• UQ• JCU• CQU• Griffith• QUT• USQ• USC• Bond
• CSIRO• NICTA• TERN• DERM• DTMR• DEEDI• Ergon Energy
27Mar2012
QCloud 18
FUNDING FOR QCLOUD
27Mar2012
Cash $15M
RDSI Primary NoDE $1.5M (contracting)
NeCTAR Research Cloud $2M (contracting)
Smart Futures Co-investment Fund $3.6M (signed)
RDSI Additional NoDE $266K (awaiting decision)
RDSI ReDS, DaSh, NRN funds $6.5M+ (projected)
DERM, DTMR co-investments $600K (offered)
QCIF QERN $200K
In-kind $5M
UQ Data centre facilities, operations and power consumption plus researcher support $2.3M
JCU Data centre facilities, operations and power consumption plus researcher support $700K
Other members, CSIRO, NeCTAR, NICTA, TERN, DERM, DTMR, Ergon Energy, QCIF researcher support $2M
QCloud 19
Agreements Progress
27Mar2012
Smart Futures
• Signed with DEEDI, Jan2012; funds flow in July• Contingent on RDSI, NeCTAR
RDSI• Final adjustments in progress• Signature expected within April
NeCTARResearch
Cloud
• Feedback provided to NeCTAR• Signature expected within April
Operations (UQ,
JCU)
• Drafting agreement text• RDSI, NeCTAR agreements as back-to-back schedules
Data, service owners
• To be completed• Preliminary arrangements for early users
QCloud 20
RISK MANAGEMENT – QCIF
• Power upgrade at UQ DC2 (JCU also?)– Decision deferred to May
• Transfer to new UQ data centre– PACE building offer replacing DC2 – network
• Staff and skills availability• Software and skills maturity• Authentication, entitlements
27Mar2012
QCloud 21
RISK MANAGEMENT – NATIONAL
• Levels of interaction: Lead Agents, other nodes• National support and help desk structure• RDSI ReDS and DaSh Programs
27Mar2012
QCloud 22
RISK MANAGEMENT – LONG TERM
• RDSI value proposition to researchers• Growth past 2014• Business model for sustainability• Commercial service quality
27Mar2012