[2c6]sqlite db 의 입출력 특성분석 : android 와 tizen 사례
DESCRIPTION
[2C6]SQLite DB 의 입출력 특성분석 : Android 와 Tizen 사례TRANSCRIPT
김명식 / Embedded Software Systems Lab.
Division of Computer Science & Engineering
Hanyang University
SQLite의 입출력 특성분석: Android 와 Tizen 사례
1. IntroductionSQLite in Mobile Platform
2. Analysis Analysis MethodExperiment Setup
3. ResultI/O Semantics (Size Distribution, Block Type, File Type)Tracing I/O on Database workload
4. DiscussionFilesystem journaling SQLite journal mode Platform DB usageOptimization Approaches
5. Conclusion
CONTENTS
1. Introduction
SQLite in Mobile Platform
1. Introduction
SQLite in Mobile Platform
What is SQLite스마트폰을 비롯한 다수 임베디드 장비에서 사용되고 있는 경량 DBMS 라이브러리
• Open Source , Support multiple OS’s
• Support Structured Query Language (SQL)
• ACID Compliant (Atomicity, Consistency, Isolation, Durability)
• Zero configuration
Mobile Platform 및 어플리케이션에서는 주로 SQLite를 사용해 DB 형태로 데이터를 저장
• The database is stored in the /data/data/<package>/databases folder
• Android OS version 및 제조사에 따라 탑재 SQLite 차이• < 2.1 : SQLite 3.5.9
• 2.2 - 2.3.3 : SQlite 3.6.22
• 3.0 4.0.3 : SQLite 3.7.4
• 4.1 4.2.x : SQLite 3.7.11
• 4.4.x : SQLite 3.7.13
• Latest SQLite Version: 3.8.5
SQLite in Mobile Platform
대부분 Mobile Platform에서 SQLite 사용Android, Tizen, iOS 다수의 Embedded Mobile Platform 에서 Sqlite를 사용
App 뿐만 아니라 Webkit과 같은 Platform 의 Framework 요소도 Sqlite 접근
SQLite in Mobile Platform
SQLiteSQLite
SQLiteSQLite
1. Introduction
SQLite in Mobile Platform
2. Analysis Analysis MethodExperiment Setup
IO Analysis Approach
User Domain
Kernel Domain
MobilePlatform
Linuxkernel
Mobile Applications
?
Core
Web Framework Native Framework
Native App Web App Hybrid App
Storage
SQLiteSQLite
MOST (Mobile Storage Analyzer) A tool for collecting IO attributes across layers
blktrace+α for tracing in real-time, debugfs+β for post reverse-mapping
IO Analysis Approach
2. LBA-to-file mapping
3. LBA-to-Process mapping
1. BLOCK IO Tracing
File systemFile system
Block I/OBlock I/O
ApplicationApplication
Specification of Evaluation Mobile
RD-PQ ( Tizen ) & Galaxy S3 (Android)
Workload & Filetype
File type Extension nameExecutable apk, tpkc, wgtd, exe, so
SQLite SQLite DB (.db)SQLite-temp SQLite-temp(.db-journal .db-wal .db-shm)Multimedia mp4, jpg, mp3, png, etcResources dat, xml, js, cache, etc
Others Others including directory entryc Tizen native app package ( tpk), d Tizen web app package ( wgt)
• Six file groups
Name (abbreviation) ScenarioContact1 (Cn1), Contact2 (Cn2) Adding name and dial number at contact list
Google (Br1), Daum (Br2), Naver (Br3) News reading, web searching from Website
Mail (Ml) Updating mailbox, Read and reply a mail
Camera (C), Camcorder (Cc) Take a picture, record movie
Media (Me), Music (Mus), Gallery (Gal) Play movie, music, image in localstorage
Youtube (You) Play web-streaming movie on youtube
Fishbowl (Fsb), Game (Gam) Browsing HTML5 Webapp benchmark site
• 14 scenarios on 7 applications
1. Introduction
SQLite in Mobile Platform
3. Result
IO Semantics (Size Distribution, Block Type, File Type)Locality, Buffered vs Synchronous Write
IO Semantics Sub-Type Tizen Android
IO Size IO Size 4KB 45% 65%
Block type Size D>J>M D>J>M
Popular large extent 128K 512K
Block type Meta and Journal 66.4(count) 57(size) 45.1(count) 37.3(size)
File type Sqlite and Sqlitetemp 75(count)54(size) 47.6(count) 40.2(size)
Sequentially Random count / Sequential Size
86 / 78 81.4 / 78
Buffered Write Sync WS 90 51.8
IO Semantics
SQLite and SQLite-temp write are dominant
Q&A
IO Size4K size 가 주요함, 안드로이드가 좀 더 높은 4K 비율
IO Blocktype실제 Data 못지 않게 저널과 메타가 비중이 높다
IO Filetype두 플랫폼 공통적으로 SQLite 관련 IO가 주요 비중
IO Sequentially횟수 측면에서는 Random IO, 크기는 Sequential IO가 주요
IO ProcessWrite Sync 가 주요, Android 는 Buffered Write 활용도 높음
I/O characteristics of Mobile Platform
To protect power failures
Generates small random, large sequential
Metadata and Journal are dominant
DB transactionis dominant
4KB is dominant
SQLite Journaling mode
SQLite Journaling Mode DELETE TRUNCATE PERSIST WAL
Number of fsync() calls 2 2 3 1
Number of IOs 9 8 12 3
EXT4 Journal size (metadata) 24 KB 16 KB 8 KB 16 KB
Total IO Volume 72 KB 64 KB 72 KB 36 KB
DELETE TRUNCATE
PERSIST WAL
Reference : Sooman Jeong et al. “I/O Stack Optimization for smartphones” USENIX ATC'13, SAN JOSE, CA, June 26~28, 2013Tizen use
Journal Mode
동작 적용 단말
ROLLBACK
DELETE
Transaction 마다 Journal file을 생성, 삭제함 GALAXY S Gingerbread, Linux Kernel 2.6.35.7
생성, 삭제하는 overhead가 큼.
TRUNCATEJournal file을 삭제하는 대신에, file size를 0으로 만듦. GALAXY S3 IceCream,
Linux Kernel 3.0 수정 사항이 Journal file에 append 되므로 delete 보다 개선
PERSIST
Journal file을 삭제하지 않고, size도 변경하지 않음. Journal header를 0으로 채움.
Tizen magnolia 2.2.1, Linux Kernel 3.0
수정 사항이 Append 되지 않고, existing block에 적용되어 TRUNCATE 보다 좀 더 성능 향상이 됨.
WAL WAL
Db의 복사본인 저널은 생성 하지 않고 메타 정보인 Wal 파일을 생성하여, 수정 사항을 Wal 파일에 sequential하게 append 함.
GALAXY S3 Jellybean 4.1.2Linux Kernel 3.0.31
수정 사항이 특정 threshold 까지 계속 sequential 하게 write 됨. 대부분의 scenario에서 성능 향상
SQLite Journaling mode
SQLite IO Pattern
2 fsync() and 8 writes in Android
3 fsync() and 12 writes in Tizen
Directory fsync()Directory fsync()
SQLite IO Pattern
Media.db
Meta of Foo.jpg
Media.db
• Before-and-after Binary Comparison when insert 1 record in media.db
• 사진 촬영 시 촬영된 사진 파일의 각종 메타 정보가 media.db에 추가됨
• MOST로 분석, 동작 전후의 db file의 binary Diff 하여 분석
• 다수회의 DB Transaction 이 조합되어 관찰됨
Foo.jpg
SQLite IO Pattern in Mobile Platform
Media.db
Media table in Media.db
Media table in Media.db
Media table in Media.db
Media table in Media.db
Iu.upload.dbAndroid 도 Tizen media.db와 유사한iu.upload.db 사용 (media record table, 29 field)
Q&A
■ B-Tree 구조를 가지고 Page 구조들 간 연결 되어 있음
■ Page (4KB) 내부는 Cell offset과 Cell로 구성
■ Cell Offset은 Page 선두에 위치, Cell 은 heap 처럼 Page 끝에서 부터 자라 올라옴
■ 각 Cell은 Cell Size/ID로 구성된 Cell Header와 Cell Size 만큼에 실제 Record header/Data 가 위치함.
SQLite File StructureFile signiture : 0x53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 (16 bytes).Page Size Definition: 4096 (0x10 00, big endian) Schema Table
DB Header
Schema Table
Pages
B-TreeTableN..
Pages
Page 1
B-TreeTable
index b-tree (index)Table b-tree (Contents)
....
Pages
Page header StructurePage Header
Offset 0
Cell 0
Offset 1
Cell 1
....
DB header Structure
Ref. 안랩 온라인 보안 매거진 월간 안(2014.04~ 2014.07)
Under 10 byte 27회
Under 100byte13회
More 400 bytes8회(shifting)
DB Binary Diff in a transaction
380 Bytes insert to the table in media.db
Media.db + media.db-journal + ext4 padding data
380 Bytes 132 KBx350 WAF
Update DB withBinary Shifting,fragmentation
380 Bytes 396 KBx1067 WAF
Media.db Only
Effect of DB transaction
eMMC Device
JEDEC mode (HS200, 400, UFS)
Block IO
IO Scheduler
Ext4 Filesystem
Data Journaling, Metadata
SQLite Library
Generation .db file DB journaling
Platform
DB for platform DB for manufacturer, carrier
App
DB for application DB from Web Access
DB insert
.db file .db-journal file
journal meta data journal meta data
DB transaction makesfacilitated diffusion of IO
DB Generates Excessive I/O
Flash MemoryDatabase DataOpen DB files(.db or .db-journal)
File System
Ext4 journalwrite() , fsync()
Insert a name,phone number
Adding one contacts generates 560 KB write traffic
2번의 journal fsync , 1번의 fsync, 1번의 journal fsync로구성
하나의 이름,전화번호 저장에 총 IO 580KB 발생
총 .contacts-svc.db에 3회의 db file fsync와 9회의 journal file fsync발생
1. Introduction
SQLite in Mobile Platform
4. Discussion
Filesystem journaling SQLite journal mode Platform DB usageOptimization Approaches
Q&A
수명
쓰기 내구성에 한계가 있는 낸드 플래시의 수명에 악 영향
성능
애플리케이션들이 발생시키는 DB 쓰기는 과도한 저널링 부하 생성
DB의 저널 파일을 파일 시스템에서 다시 저널링 하는 중복 저널 현상
사용자 앱/ 기기 제조사 /플랫폼의 유사 정보 중복 DB 접근
SQLite 연동시의 문제점
Q&A
Kernel 최적화Block I/O Scheduler
Page cache
Eliminating unnecessary metadata flushes using fdatasync()
Filesystem 최적화Ext4 mount option optimization (norelatime, Jounrnal_async_commit)
log-structured Filesystem : Flash friendly Filesystem (F2FS)
Discussion
fdatasync() for Metadata Flushes
fsync(fd0)
sizedata atimemtimedata
fsync(fd1) fdatasync(fd0)
sizedata atimemtimedata
fdatasync(fd1)
Page cache
Disk
Reference : Sooman Jeong et al. “I/O Stack Optimization for smartphones” USENIX ATC'13, SAN JOSE, CA, June 26~28, 2013
Fdatasync flush metadata only when file size changes
mtime = modify time, atime = access time
MetadataData
fsync()fsync() fdatasync()fdatasync()
fdatasync() for Metadata Flushes
size
data
data atimemtime dataPage cache
Disk
size
data data’ data’
mtime
fdatasync() fdatasync()
Why does fdatasync() brings more performance gain in update operation?
SQLite InsertSQLite Insert SQLite UpdateSQLite Update
Reference : Sooman Jeong et al. “I/O Stack Optimization for smartphones” USENIX ATC'13, SAN JOSE, CA, June 26~28, 2013
Q&A
SQLite 개선WAL 모드 적용
Multi-version B-tree(MVBT)의 적용, Tree 구조 개선을 통해 SQLite
transection 수행 시 update 되는 db-page 의 개수 최적화
Lazy Split, disabling sibling , Lazy Garbage Collection à Less fsync(),
Less dirty pages
Embedding Metadata, SQLite의 journal file 의 역할을 DB file로
embedding
WAL-mode 70%, Truncate mode 1,220% 향상.
Ref. KIM, W.-H., NAM, B., PARK, D., AND WON, Y. Resolving journaling of journal anomaly in android I/O: Multi-version
b-tree with lazy split. In Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST 14) (Santa
Clara, CA, 2014),USENIX, pp. 27328
Discussion
Q&A
New Architecture No Journaling ( iOS HFS+)
Quisi-NVRAM (DRAM to act as a battery backed up nonvolatile DRAM.)
Discussion
Ref. Hao Luo, Lei Tian and Hong Jiang , qNVRAM: quasi Non-Volatile RAM for Low Overhead Persistency Enforcement in
Smartphones, University of Nebraska, Lincoln, Hotstorage 14, USENIX
Junghoon Kim; Changwoo Min; Young Eom, "Reducing excessive journaling overhead with small-sized NVRAM for mobile
devices," Consumer Electronics, IEEE Transactions on , vol.60, no.2, pp.217,224, May 2014
The Performance of Persistent Page Cache and LazyFlush
speeds up the insert, update and delete transactions by up to 16.33x, 15.86x and 15.76x respectively.
Q&A
Platform필요한 쓰기 접근인지 트랜잭션 처리에 따른 IO 비용이 높은 DB를 사용
해야 하는지 고민 필요
DB 사용 시 schema, index,
저널링 모드 Effective journaling mode is WAL.
플랫폼 / 제조사 / App 의 중복된 접근에 대한 최적화 필요
Reorganize Web Database
Discussion
Q&A
Data Persistency in Web
Webkit 도 다양한 DB 쓰기 접근이 관찰됨
Web Storage
Web SQL DB: deprecated
HTML5 localstorage
only store a simple key-value pairs
Indexed DB
Object store 를 만들고 object 를 저장
단순 Key-Value Storage, SQL Query 대신 javascript index 를 사용
Reorganize Web Database
Databases.db
Storagetracker.db
Reorganize Database Usage
icon.db
Reorganize Database Usage
Q&A
Explore SQLite DB IO characteristics in Mobile Platform
- SQLite DB IO is dominant in Mobile.
- Excessive journaling I/O caused by SQLite and Ext4
- Need to rethink DB usage, DB file structure
Conclusion
THANK YOU
Q&A