Multi-level Selective Deduplication for VMSnapshots in Cloud Storage
Wei Zhang*, Hong Tang†, Hao Jiang†, Tao Yang*, Xiaogang Li†, Yue Zeng†
* University of California at Santa Barbara† Aliyun.com Inc.
Motivations
• Virtual machines on the cloud use frequent backup to improve service reliability Used in Alibaba’s Aliyun - the largest public cloud
service in China• High storage demand
Daily backup workload: hundreds of TB @ Aliyun Number of VMs per cluster: 10000+
• Large content duplicates• Limited resource for deduplication
No special hardware or dedicated machines Small CPU& memory footprint
Focus and Related Work
• Previous work Version-based incremental snapshot backup
– Inter-block/VM duplicates are not detected. Chunk-based file duduplication
– High cost for chunk lookup• Focus on
Parallel backup of a large number of virtual disks. – Large files for VM disk images.
Contributions– Cost-constrained solution with very limited computing
resource– Multi-level selective duplicate detection and parallel backup.
Requirements
• Negligible impact on existing cloud service and VM performance
Must minimize CPU and IO bandwidth consumption for backup and deduplication workload
– (e.g. <1% of total resource).
• Fast backup speed Compute backup for 10,000+ users within a few hours
each day during light cloud workload.• Fault tolerance constraint addition of data deduplication should not decrease the
degree of fault tolerance.
Design Considerations
• Design alternatives An external and dedicated backup storage
system.
A decentralized and co-hosted backup system with full deduplication
BackupCloud service
backup
Cloud service
backup
Cloud service. . .
backup
Cloud service
Design Considerations
• Decentralized architecture running on a general purpose cluster• co-hosting both elastic computing and backup
service• Multi-level deduplication Localize backup traffic and exploit data parallelism Increase fault tolerance• Selective deduplication Use minimal resource while still removing most of
redundant content and accomplishing good efficiency
Key Observations
Inner-VM data characteristicsExploit unchanged data to localize
deduplicationCross-VM data characteristics
Small common data dominates duplicates
Zipf-like distribution of VM OS/user dataSeparate consideration of OS and user
data
VM Snapshot Representation
Data blocks are variable-sized
Segments are fix-sized
Processing Flow of Multi-level Deduplication
Data Processing Steps
• Segment level checkup. Use dirty bitmap to see which segments are modified.
• Block level checkup Divide a segment into variable-sized blocks,
and compare their signatures with the parent snapshot
• Checkup from common dataset (CDS) Identify duplicate chunks from CDS
• Write new snapshot blocks Write new content chunks to stoage.
• Save recipes Save segment meta-data information
Architecture of Multi-level VM snapshot backup
Cluster node
Status& Evaluation
• Prototype system running on Alibaba’s Aliyuan cloud. Based on Xen. 100 nodes and each has 16 cores, 48G memory,
25VMs. Use <150MB per machine for backup&deduplication
• Evaluation data from Aliyuan’s production cluster 41TB. 10 snapshots per VM Segment size: 2MB. Avg. Block size: 4KB
Data Characteristics of the Benchmark
• Each VM uses 40GB storage space on average
• OS and user data disks: each takes ~50% of space
• OS data 7 main stream OS releases: Debian, Ubuntu, Redhat, CentOS, Win2003
32bit, win2003 64 bit and win2008 64 bit.• User data
From 1323 VM users
Impacts of 3-Level Deduplication
Level 1: Segment-level detection within VMLevel 2: Block-level detection within VMLevel 3: Common data block detection across-VM
Impact for Different OS Releases
Separate consideration of OS and user data
Both have Zipf-like data distributionBut popularity growth differs as the cluster size/VM users increase
Commonality among OS releases
1G common OS meta datacovers 70+%
Cumulative coverage of popular user data
Coverage is the summation of covered data block size*frequency
Space saving compared to perfect deduplication as CDS size increases
100G CDS (1GB index) -> 75% of perfect dedup
Impact of dataset-size increase
Conclusions
• Contributions: A multi-level selective deduplication scheme among VM
snapshots– Inner-VM deduplication localizes backup and exposes more
parallelism– global deduplication with a small common data set appeared in
OS and data disks
Use less than 0.5% of memory per node to meet a stringent cloud resource requirement -> accomplish 75% of what perfect deduplication does.
• Experiments Achieve 500TB/hour on a 1000-node cloud cluster Reduce bandwidth by 92% -> 40TB/hour