portable data management cloud for field science
TRANSCRIPT
PORTABLE DATA MANAGEMENT CLOUDFOR FIELD SCIENCE
UC San Diego, Calit2Yuma Matsui, Aaron Gidding, Thomas E. Levy, Falko Kuester, Thomas A. DeFanti
IEEE Cloud 2012, 6/24/2012
CONTENTS
Managing Big Data in Archaeology
Heterogeneous data
Need for data management system
Portability of Data Management Cloud
System in the Wild
DATA-DRIVENFIELD SCIENCE
DATA-DRIVENFIELD SCIENCE
I need data management infrastructure... but no fancy datacenter
and broadband network here.
PORTABLE DATA MANAGEMENT CLOUDNeed data management system that runs both on
Campus: powerful computers, high-speed network
Field sites: small computers, limited network
Need data management system that runs both on
Campus: powerful computers, high-speed network
Field sites: small computers, limited network
Cloud provides flexible computer infrastructure
virtualized environment, ease of deployment, scalability
Need data management system that runs both on
Campus: powerful computers, high-speed network
Field sites: small computers, limited network
Cloud provides flexible computer infrastructure
virtualized environment, ease of deployment, scalability
PORTABLE DATA MANAGEMENT CLOUDNeed data management system that runs both on
Campus: powerful computers, high-speed network
Field sites: small computers, limited network
Portable data management infrastructurebetween field sites and campus with cloud!
Cloud provides flexible computer infrastructure
virtualized environment, ease of deployment, scalability
Need data management system that runs both on
Campus: powerful computers, high-speed network
Field sites: small computers, limited network
Cloud provides flexible computer infrastructure
virtualized environment, ease of deployment, scalability
Need data management system that runs both on
Campus: powerful computers, high-speed network
Field sites: small computers, limited network
Cloud provides flexible computer infrastructure
virtualized environment, ease of deployment, scalability
Managing Big Data in Archaeology
Portability of Data Management Cloud
Virtualized environment
Data access
System in the Wild
PORTABILITY IN THE SYSTEM
Goal: streamline data processes over field sites and campus
Data collection
Data management
Data analysis
What is portability?
Portability of whole system environment
Portability of collected data
Data Collection Data Management
Data Analysis and
Visualization
Field Sites Campus DatacenterPortability
PORTABLE SYSTEM WITH CLOUD
IaaS
Fully controllable virtualized environment
Makes whole environment (data and programs) portable
Suitable for our field science needs
PaaS
SaaS
DATA ACCESS
Structured data: artifact/site metadata, artifact inventory data, and total station geo-data
Stored in a database
Accessible with JSON REST API
Raw measurement data: Photos, XRF (X-Ray Fluorescence), FTIR (Fourier Transform Infrared Spectroscopy), and LiDAR
Stored in an object storage
Accessible with S3-compatible REST API
Web-based data management application
All data are accessible with the web application or REST API. This makes data portable and consumable.
Managing Big Data in Archaeology
Between Cloud and Ground
System in the Wild
System components
System workflow
SYSTEM COMPONENTS
register
TotalStation
LIDAR
Artifact Data
copy
NetworkAttachedStorage
Cloud Storage
Photos
Field Sites Campus Datacenter
VisualizationFacility
(CAVE,OptIPortal)
Small Server
WebApp DB
Virtualization
IaaS Cloud
WebApp DB
Virtualization
SYSTEM WORKFLOW
Field sites
Various data are collected with insruments.
Structured data are put into the database through the web application.
Raw file data are temporarily stored in network-attached storage.
Campus
Data and programs from fields are moved to campus cloud infrastructure.
Data analyses and visualizations are executed with the collected data on high-performance computers.
Synchronize environments(VM copy and object storage registration)
register
TotalStation
LIDAR
Artifact Data
copy
NetworkAttachedStorage
Cloud Storage
Photos
Field Sites Campus Datacenter
VisualizationFacility
(CAVE,OptIPortal)
Small Server
WebApp DB
Virtualization
IaaS Cloud
WebApp DB
Virtualization
CONCLUSION AND FUTURE WORK
We developed a portable data management infrastructure for digital archaeology.
It is based on IaaS virtualized hosting environments and equipped with unified data access methods.
We used the system in an excavation in 2011.
Integration of the system with large-scale analysis and visualization is in progress.
