testing database management systems · google’s os-fuzz project) sqlite’s test cases achieve...
TRANSCRIPT
![Page 1: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/1.jpg)
Testing Database Management Systems via Pivoted Query Synthesis
Manuel Rigger
Oct 18., 2019Workshop on Dependable and Secure Software
Systems 2019
@RiggerManuel @ast_eth
![Page 2: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/2.jpg)
2
Database Management Systems
PostgreSQL
![Page 3: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/3.jpg)
3
Database Management Systems
PostgreSQL
Who has heard about/used these Database Management Systems?
![Page 4: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/4.jpg)
4
Databases are Used Ubiquitously
“SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled inside countless other applications that people use every day.”
https://www.sqlite.org
![Page 5: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/5.jpg)
5
animal description picture
CatA cute toast
cat
Dog Cute dog pic
CatCat plants
(cute!)
animal_pictures
Relational Data Model
![Page 6: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/6.jpg)
6
animal description picture
CatA cute toast
cat
Dog Cute dog pic
CatCat plants
(cute!)
animal_pictures
A database schema describes the tables
(relations) in the database
Relational Data Model
![Page 7: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/7.jpg)
7
animal_pictures Structured Query Language (SQL) is a declarative DSL to query and manipulate data
SELECT picture, descriptionFROM animal_picturesWHERE animal = 'Cat'
AND description LIKE '%cute%'
Relational Data Model
![Page 8: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/8.jpg)
8
Database Management Systems
Database
Database Management System
(DBMS)
SELECT * FROM <table>WHERE <cond>
ClientApplication
![Page 9: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/9.jpg)
9
Database Management Systems
Database
Database Management System
(DBMS)
SELECT * FROM <table>WHERE <cond>
ClientApplication
row1 <cond>
row2 <cond>
row3 ¬<cond>
![Page 10: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/10.jpg)
10
Database Management Systems
Database
Database Management System
(DBMS)
SELECT * FROM <table>WHERE <cond>
ClientApplication
✓
row1 <cond>
row2 <cond>
row3 ¬<cond>
row1 <cond>
row2 <cond>
![Page 11: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/11.jpg)
11
Goal
Aim: Detect logic bugs in DBMS
![Page 12: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/12.jpg)
12
Database Management Systems
Database
Database Management System
(DBMS)
SELECT * FROM <table>WHERE <cond>
ClientApplication
✓
row1 <cond>
row2 <cond>
row3 ¬<cond>
row1 <cond>
row2 <cond>
![Page 13: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/13.jpg)
13
Database Management Systems
Database
Database Management System
(DBMS)
ClientApplication
row1 <cond>
row2 <cond>
row3 ¬<cond>SELECT * FROM <table>WHERE <cond>
row1 <cond>
row3 ¬<cond>
![Page 14: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/14.jpg)
14
Example Bug: SQLite
CREATE TABLE t1(c1, c2, c3, c4, PRIMARY KEY (c4, c3));INSERT INTO t1(c3) VALUES (0), (0), (0), (0), (0), (0),
(0), (0), (0), (0), (NULL), (1), (0);UPDATE t1 SET c2 = 0;INSERT INTO t1(c1) VALUES (0), (0), (NULL), (0), (0);ANALYZE t1;UPDATE t1 SET c3 = 1;SELECT DISTINCT * FROM t1 WHERE t1.c3 = 1;
![Page 15: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/15.jpg)
15
Example Bug: SQLite
CREATE TABLE t1(c1, c2, c3, c4, PRIMARY KEY (c4, c3));INSERT INTO t1(c3) VALUES (0), (0), (0), (0), (0), (0),
(0), (0), (0), (0), (NULL), (1), (0);UPDATE t1 SET c2 = 0;INSERT INTO t1(c1) VALUES (0), (0), (NULL), (0), (0);ANALYZE t1;UPDATE t1 SET c3 = 1;SELECT DISTINCT * FROM t1 WHERE t1.c3 = 1;
ANALYZE gathers statistics about tables, which are then used for query planning
![Page 16: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/16.jpg)
16
Example Bug: SQLite
CREATE TABLE t1(c1, c2, c3, c4, PRIMARY KEY (c4, c3));INSERT INTO t1(c3) VALUES (0), (0), (0), (0), (0), (0),
(0), (0), (0), (0), (NULL), (1), (0);UPDATE t1 SET c2 = 0;INSERT INTO t1(c1) VALUES (0), (0), (NULL), (0), (0);ANALYZE t1;UPDATE t1 SET c3 = 1;SELECT DISTINCT * FROM t1 WHERE t1.c3 = 1;
![Page 17: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/17.jpg)
17
Example Bug: SQLite
CREATE TABLE t1(c1, c2, c3, c4, PRIMARY KEY (c4, c3));INSERT INTO t1(c3) VALUES (0), (0), (0), (0), (0), (0),
(0), (0), (0), (0), (NULL), (1), (0);UPDATE t1 SET c2 = 0;INSERT INTO t1(c1) VALUES (0), (0), (NULL), (0), (0);ANALYZE t1;UPDATE t1 SET c3 = 1;SELECT DISTINCT * FROM t1 WHERE t1.c3 = 1;
c1 c2 c3 c4
NULL 0 1 NULL
0 NULL 1 NULL
NULL NULL 1 NULL
Expected result set
![Page 18: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/18.jpg)
18
Example Bug: SQLite
CREATE TABLE t1(c1, c2, c3, c4, PRIMARY KEY (c4, c3));INSERT INTO t1(c3) VALUES (0), (0), (0), (0), (0), (0),
(0), (0), (0), (0), (NULL), (1), (0);UPDATE t1 SET c2 = 0;INSERT INTO t1(c1) VALUES (0), (0), (NULL), (0), (0);ANALYZE t1;UPDATE t1 SET c3 = 1;SELECT DISTINCT * FROM t1 WHERE t1.c3 = 1;
c1 c2 c3 c4
NULL 0 1 NULL
0 NULL 1 NULL
NULL NULL 1 NULL
Expected result set
c1 c2 c3 c4
NULL 0 1 NULL
Actual result set
![Page 19: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/19.jpg)
19
Example Bug: SQLite
CREATE TABLE t1(c1, c2, c3, c4, PRIMARY KEY (c4, c3));INSERT INTO t1(c3) VALUES (0), (0), (0), (0), (0), (0),
(0), (0), (0), (0), (NULL), (1), (0);UPDATE t1 SET c2 = 0;INSERT INTO t1(c1) VALUES (0), (0), (NULL), (0), (0);ANALYZE t1;UPDATE t1 SET c3 = 1;SELECT DISTINCT * FROM t1 WHERE t1.c3 = 1;
A bug in the skip-scan optimization caused
this logic bug
c1 c2 c3 c4
NULL 0 1 NULL
0 NULL 1 NULL
NULL NULL 1 NULL
Expected result set
c1 c2 c3 c4
NULL 0 1 NULL
Actual result set
![Page 20: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/20.jpg)
20
Challenges
• DBMS are tested well
![Page 21: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/21.jpg)
21
Databases are Tested Well
SQLite (~150,000 LOC) has 662 times as much test code as source code
https://www.sqlite.org/testing.html
![Page 22: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/22.jpg)
22
Databases are Tested Well
SQLite (~150,000 LOC) has 662 times as much test code as source code
SQLite’s test cases achieve 100% branch test coverage
https://www.sqlite.org/testing.html
![Page 23: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/23.jpg)
23
Databases are Tested Well
SQLite (~150,000 LOC) has 662 times as much test code as source code
SQLite is extensively fuzzed (e.g., by Google’s OS-Fuzz Project)
SQLite’s test cases achieve 100% branch test coverage
https://www.sqlite.org/testing.html
![Page 24: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/24.jpg)
24
Databases are Tested Well
SQLite (~150,000 LOC) has 662 times as much test code as source code
SQLite is extensively fuzzed (e.g., by Google’s OS-Fuzz Project)
SQLite’s test cases achieve 100% branch test coverage
SQLite’s performs anomaly testing (out-of-memory, I/O error, power failures)
https://www.sqlite.org/testing.html
![Page 25: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/25.jpg)
25
Databases are Tested Well
SQLite (~150,000 LOC) has 662 times as much test code as source code
SQLite is extensively fuzzed (e.g., by Google’s OS-Fuzz Project)
SQLite’s test cases achieve 100% branch test coverage
SQLite’s performs anomaly testing (out-of-memory, I/O error, power failures)
https://www.sqlite.org/testing.html
Small. Fast. Reliable. Choose any three.
![Page 26: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/26.jpg)
26
Challenges
• DBMS are tested well
• Fuzzers are ineffective in finding logic bugs
![Page 27: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/27.jpg)
27
Existing Work: Fuzzers and Query Generators
DatabaseDatabase Management
System (DBMS)
RandomSQL Query
Fuzzer
AFL, SQLSmith, QGEN (Poess et al. 2014), …
![Page 28: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/28.jpg)
28
Existing Work: Fuzzers and Query Generators
DatabaseDatabase Management
System (DBMS)
RandomSQL Query
Fuzzer
SEGMENTATION FAULT
AFL, SQLSmith, QGEN (Poess et al. 2014), …
![Page 29: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/29.jpg)
29
Existing Work: Fuzzers and Query Generators
DatabaseDatabase Management
System (DBMS)
RandomSQL Query
Fuzzer
Fuzzers are effective in detecting bugs that result in crashes
SEGMENTATION FAULT
AFL, SQLSmith, QGEN (Poess et al. 2014), …
![Page 30: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/30.jpg)
30
Challenges
• DBMS are tested well
• Fuzzers are ineffective in finding logic bugs
• Knowing the precise result set for a query is difficult
![Page 31: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/31.jpg)
31
Differential Testing
Query Generator
PostgreSQL
RS1
RS2
RS3
RS1 = RS2 = RS3?
![Page 32: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/32.jpg)
32
Differential Testing
Query Generator
PostgreSQL
Differential testing applies only when systems implement the same language
RS1
RS2
RS3
RS1 = RS2 = RS3?
![Page 33: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/33.jpg)
33
Problem: Differential Testing
DBMS-specific SQL
Common SQL Core
Problem: The common SQL core is small
![Page 34: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/34.jpg)
34
Differential Testing: RAGS (Slutz 1998)
“[Differential testing] proved to be extremely useful,but only for the small set of common SQL”
![Page 35: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/35.jpg)
35
Constraint Solving (Khalek et al. 2010)
Idea: Use a solver to generate queries, generate data, and provide a test oracle
Query Generation Database and Test OracleGeneration
Could reproduce already reported bugs, injected bugs, but only one (potentially) new bug
![Page 36: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/36.jpg)
36
Challenges
• DBMS are tested well
• Fuzzers are ineffective in finding logic bugs
• Knowing the precise result set for a query is difficult
The problem of automatically testing DBMS has not yet been well addressed
![Page 37: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/37.jpg)
37
Approach: Pivoted Query Synthesis
Pivoted Query Synthesis (PQS)
Pivoted Query Synthesis is an automatic testing approach that can be used to effectively test DBMS
>100 bugs in widely used
DBMS
![Page 38: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/38.jpg)
38
Idea: PQSIdea: Construct an automatic testing approach considering only a single row
Column0 Column1 Column2
… … …
Valuei,0 Valuei,1 Valuei,2
… … …
Pivot Row
![Page 39: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/39.jpg)
39
Intuition
• Simpler conceptually and implementation-wise
Column0 Column1 Column2
… … …
Valuei,0 Valuei,1 Valuei,2
… … …
<cond>?
SELECT * FROM <table>WHERE <cond>
![Page 40: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/40.jpg)
40
Intuition
• Simpler conceptually and implementation-wise
• Same effectiveness as checking all rows
Column0 Column1 Column2
… … …
Valuei,0 Valuei,1 Valuei,2
… … …
✓
SELECT * FROM <table>WHERE <cond>
![Page 41: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/41.jpg)
41
Intuition
• Simpler conceptually and implementation-wise
• Same effectiveness as checking all rows
Column0 Column1 Column2
… … …
Valuei,0 Valuei,1 Valuei,2
… … …
SELECT * FROM <table>WHERE <cond>
![Page 42: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/42.jpg)
42
Intuition
• Simpler conceptually and implementation-wise
• Same effectiveness as checking all rows
![Page 43: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/43.jpg)
43
Intuition
• Simpler conceptually and implementation-wise
• Same effectiveness as checking all rows
• Precise oracle for a single row
![Page 44: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/44.jpg)
Approach
![Page 45: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/45.jpg)
45
Database GenerationRandomly Generate Database
![Page 46: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/46.jpg)
46
Database GenerationRandomly Generate Database
To explore “all possible database states” we randomly create databases
![Page 47: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/47.jpg)
47
Pivot Row SelectionRandomly Generate Database
SelectPivot Row
![Page 48: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/48.jpg)
48
Query GenerationRandomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
animal description picture
CatCat plants
(cute!)
SELECT picture, descriptionFROM animal_picturesWHERE animal = 'Cat'
AND description LIKE '%cute%'
![Page 49: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/49.jpg)
49
Verifying the ResultRandomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
DBMS
SELECT picture, descriptionFROM animal_picturesWHERE animal = 'Cat'
AND description LIKE '%cute%'
result set
pivot row
![Page 50: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/50.jpg)
50
Verifying the ResultRandomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
DBMS
✓
SELECT picture, descriptionFROM animal_picturesWHERE animal = 'Cat'
AND description LIKE '%cute%'
result set
pivot row
pivot row ∈ result set
![Page 51: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/51.jpg)
51
Verifying the ResultRandomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
DBMS
SELECT picture, descriptionFROM animal_picturesWHERE animal = 'Cat'
AND description LIKE '%cute%'
result set
pivot row
![Page 52: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/52.jpg)
52
Verifying the ResultRandomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
DBMS
SELECT picture, descriptionFROM animal_picturesWHERE animal = 'Cat'
AND description LIKE '%cute%'
result set
pivot row
pivot row ∉ result set
![Page 53: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/53.jpg)
53
Verifying the ResultRandomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
pivot row ∉ result set
The “containment oracle” is PQS’ primary oracle
![Page 54: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/54.jpg)
54
Approach
Randomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
![Page 55: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/55.jpg)
55
Approach
Randomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
![Page 56: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/56.jpg)
56
Approach
Randomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
![Page 57: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/57.jpg)
57
Approach
Randomly Generate Database
SelectPivot Row
Generate Query for the
Pivot Row
Verify that the Pivot Row is contained
How do we generate this query?
![Page 58: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/58.jpg)
58
How do we Generate Queries?
Generate an expression that yields TRUE for the pivot row
SELECT picture, description
FROM animal_pictures
WHERE
![Page 59: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/59.jpg)
59
How do we Generate Queries?
Randomly Generate
Expression
Evaluate Expression on
Pivot Row
Modify expression to
yield TRUE
Use in WHERE clause
![Page 60: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/60.jpg)
60
Random Expression Generation
animal description picture
https://www.sqlite.org/syntax/expr.html
animal_pictures
We first generate a random expression
![Page 61: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/61.jpg)
61
Random Expression Generation
animal = 'Cat'AND description LIKE '%cute%'
AND
= LIKE
animal 'Cat' description
'%cute%'
![Page 62: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/62.jpg)
62
Random Expression Generation
animal = 'Cat'AND description LIKE '%cute%'
Evaluate the tree based on the pivot row
AND
= LIKE
animal 'Cat' description
'%cute%'
![Page 63: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/63.jpg)
63
Random Expression Evaluation
AND
= LIKE
animal 'Cat' description
'Cat''%cute%'
Constant nodes return their assigned literal
values
'%cute%'
![Page 64: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/64.jpg)
64
Random Expression Evaluation
AND
= LIKE
animal 'Cat' description
'Catplants(cute!)'
'Cat' 'Cat''%cute%'
Column references return the values from the pivot row
'%cute%'
![Page 65: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/65.jpg)
65
Random Expression Evaluation
AND
= LIKE
animal 'Cat' description
'Catplants(cute!)'
'Cat'
TRUE TRUE
'Cat''%cute%'
Compound nodes compute their result
based on their children
'%cute%'
![Page 66: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/66.jpg)
66
Random Expression Evaluation
AND
= LIKE
animal 'Cat' description
'Catplants(cute!)'
'Cat'
TRUE TRUE
TRUE
'Cat''%cute%'
'%cute%'
![Page 67: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/67.jpg)
67
SELECT picture, description
FROM animal_pictures
WHERE animal = 'Cat' AND description LIKE '%cute%'
Query Synthesis
![Page 68: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/68.jpg)
68
Random Expression Evaluation
AND
= LIKE
animal 'Cat' description
'Catplants(cute!)'
'Cat'
TRUE TRUE
TRUE
'Cat''%cute%'
What about when the expression does not evaluate to TRUE?
'%cute%'
![Page 69: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/69.jpg)
69
Random Expression Evaluation
=
animal 'Dog'
'Cat'
FALSE
'Dog'
What about when the expression does not evaluate to TRUE?
animal = 'Dog'
![Page 70: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/70.jpg)
70
Random Expression Rectification
switch (result) {case TRUE:
result = randexpr;case FALSE:
result = NOT randexpr;case NULL:
result = randexpr ISNULL;}
![Page 71: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/71.jpg)
71
Random Expression Rectification
switch (result) {case TRUE:
result = randexpr;case FALSE:
result = NOT randexpr;case NULL:
result = randexpr ISNULL;}
animal = 'Dog'
FALSE
![Page 72: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/72.jpg)
72
Random Expression Rectification
switch (result) {case TRUE:
result = randexpr;case FALSE:
result = NOT randexpr;case NULL:
result = randexpr ISNULL;}
NOT(animal = 'Dog')
TRUE
![Page 73: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/73.jpg)
73
How do we Generate Queries?
SELECT picture, description
FROM animal_pictures
WHERE NOT(animal = 'Dog')
![Page 74: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/74.jpg)
Evaluation
![Page 75: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/75.jpg)
75
Tested DBMS
PostgreSQL
![Page 76: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/76.jpg)
76
Tested DBMS
PostgreSQL
We tested these (and other DBMS) in a period of 3-4 months
![Page 77: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/77.jpg)
77
DBMS
![Page 78: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/78.jpg)
78
DBMS
![Page 79: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/79.jpg)
79
DBMS
![Page 80: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/80.jpg)
80
Bugs Overview
DBMS Fixed Verified
SQLite 65 0
MySQL 15 10
PostgreSQL 5 4
Sum 85 14
Real Bugs
![Page 81: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/81.jpg)
81
Bugs Overview
DBMS Fixed Verified
SQLite 65 0
MySQL 15 10
PostgreSQL 5 4
Sum 85 14
99 real bugs: addressed by code or documentation fixes, or verified as bugs
Real Bugs
![Page 82: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/82.jpg)
82
Bugs Overview
The SQLite developers quickly responded to all our bug reports → we focused on this DBMS
Real Bugs
DBMS Fixed Verified
SQLite 65 0
MySQL 15 10
PostgreSQL 5 4
Sum 85 14
![Page 83: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/83.jpg)
83
Bugs Overview Real Bugs
DBMS Fixed Verified
SQLite 65 0
MySQL 15 10
PostgreSQL 5 4
Sum 85 14
All MySQL bug reports were verified quickly
![Page 84: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/84.jpg)
84
Bugs Overview Real Bugs
DBMS Fixed Verified
SQLite 65 0
MySQL 15 10
PostgreSQL 5 4
Sum 85 14
MySQL’s trunk is not available, and it has a long release cycle
![Page 85: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/85.jpg)
85
Bugs Overview Real Bugs
DBMS Fixed Verified
SQLite 65 0
MySQL 15 10
PostgreSQL 5 4
Sum 85 14
We found the fewest bugs in PostgreSQL and not all could be easily addressed
![Page 86: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/86.jpg)
86
Oracles
DBMS Containment Error SEGFAULT
SQLite 46 17 2
MySQL 14 10 1
PostgreSQL 1 7 1
Sum 61 34 4
Real Bugs
![Page 87: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/87.jpg)
87
Oracles
DBMS Containment Error SEGFAULT
SQLite 46 17 2
MySQL 14 10 1
PostgreSQL 1 7 1
Sum 61 34 4
Real Bugs
Containment Oracle
Our Containment oracle allowed us to detect most errors
![Page 88: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/88.jpg)
88
Result: Bug in SQLite3
CREATE TABLE t0(c1 TEXT PRIMARY KEY) WITHOUT ROWID;CREATE INDEX i0 ON t0(c1 COLLATE NOCASE);INSERT INTO t0(c1) VALUES ('A');INSERT INTO t0(c1) VALUES ('a');
Real Bugs
Containment Oracle
![Page 89: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/89.jpg)
89
Result: Bug in SQLite3
CREATE TABLE t0(c1 TEXT PRIMARY KEY) WITHOUT ROWID;CREATE INDEX i0 ON t0(c1 COLLATE NOCASE);INSERT INTO t0(c1) VALUES ('A');INSERT INTO t0(c1) VALUES ('a');
An index is an auxiliary data structure that should not affect the query’s result
Real Bugs
Containment Oracle
![Page 90: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/90.jpg)
90
Result: Bug in SQLite3
CREATE TABLE t0(c1 TEXT PRIMARY KEY) WITHOUT ROWID;CREATE INDEX i0 ON t0(c1 COLLATE NOCASE);INSERT INTO t0(c1) VALUES ('A');INSERT INTO t0(c1) VALUES ('a');
c1
'A'
'a'
Real Bugs
Containment Oracle
![Page 91: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/91.jpg)
91
Result: Bug in SQLite3
CREATE TABLE t0(c1 TEXT PRIMARY KEY) WITHOUT ROWID;CREATE INDEX i0 ON t0(c1 COLLATE NOCASE);INSERT INTO t0(c1) VALUES ('A');INSERT INTO t0(c1) VALUES ('a');
c1
'A'
'a'
SELECT * FROM t0; c1
'A'
Real Bugs
Containment Oracle
![Page 92: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/92.jpg)
92
Result: Bug in SQLite3
CREATE TABLE t0(c1 TEXT PRIMARY KEY) WITHOUT ROWID;CREATE INDEX i0 ON t0(c1 COLLATE NOCASE);INSERT INTO t0(c1) VALUES ('A');INSERT INTO t0(c1) VALUES ('a');
c1
'A'
'a'
SELECT * FROM t0; c1
'A'
SQLite failed to fetch 'a'!
Real Bugs
Containment Oracle
![Page 93: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/93.jpg)
93
Result: Bug in PostgreSQL
CREATE TABLE t0(c0 INT PRIMARY KEY, c1 INT);CREATE TABLE t1(c0 INT) INHERITS (t0);c0 c1
c0 c1
t0
t1
Real Bugs
Containment Oracle
![Page 94: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/94.jpg)
94
Result: Bug in PostgreSQL
CREATE TABLE t0(c0 INT PRIMARY KEY, c1 INT);CREATE TABLE t1(c0 INT) INHERITS (t0);INSERT INTO t0(c0, c1) VALUES(0, 0);
c0 c1
0 0
c0 c1
t0
t1
Real Bugs
Containment Oracle
![Page 95: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/95.jpg)
95
Result: Bug in PostgreSQL
CREATE TABLE t0(c0 INT PRIMARY KEY, c1 INT);CREATE TABLE t1(c0 INT) INHERITS (t0);INSERT INTO t0(c0, c1) VALUES(0, 0);INSERT INTO t1(c0, c1) VALUES(0, 1);
c0 c1
0 0
0 1
c0 c1
0 1
t0
t1
Real Bugs
Containment Oracle
![Page 96: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/96.jpg)
96
Result: Bug in PostgreSQL
CREATE TABLE t0(c0 INT PRIMARY KEY, c1 INT);CREATE TABLE t1(c0 INT) INHERITS (t0);INSERT INTO t0(c0, c1) VALUES(0, 0);INSERT INTO t1(c0, c1) VALUES(0, 1);
c0 c1
0 0
0 1
c0 c1
0 1
t0
t1The inheritance relationship
causes the row to be insertedboth in t0 and t1
Real Bugs
Containment Oracle
![Page 97: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/97.jpg)
97
Result: Bug in PostgreSQL
CREATE TABLE t0(c0 INT PRIMARY KEY, c1 INT);CREATE TABLE t1(c0 INT) INHERITS (t0);INSERT INTO t0(c0, c1) VALUES(0, 0);INSERT INTO t1(c0, c1) VALUES(0, 1);
c0 c1
0 0
0 1
c0 c1
0 1
t0
t1
SELECT c0, c1 FROM t0GROUP BY c0, c1;
c0 c1
0 0
Real Bugs
Containment Oracle
![Page 98: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/98.jpg)
98
Result: Bug in PostgreSQL
CREATE TABLE t0(c0 INT PRIMARY KEY, c1 INT);CREATE TABLE t1(c0 INT) INHERITS (t0);INSERT INTO t0(c0, c1) VALUES(0, 0);INSERT INTO t1(c0, c1) VALUES(0, 1);
c0 c1
0 0
0 1
c0 c1
0 1
t0
t1
SELECT c0, c1 FROM t0GROUP BY c0, c1;
c0 c1
0 0
PostgreSQL failed to fetch the row 0 | 1
Real Bugs
Containment Oracle
![Page 99: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/99.jpg)
99
Result: Bug in MySQL
CREATE TABLE t0(c0 TINYINT);INSERT INTO t0(c0) VALUES(NULL);c0
NULL
t0
Real Bugs
Containment Oracle
![Page 100: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/100.jpg)
100
Result: Bug in MySQL
CREATE TABLE t0(c0 TINYINT);INSERT INTO t0(c0) VALUES(NULL);c0
NULL
t0
SELECT * FROM t0WHERENOT(t0.c0 <=> 2035382037);
c0
FALSE
Real Bugs
Containment Oracle
![Page 101: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/101.jpg)
101
Result: Bug in MySQL
CREATE TABLE t0(c0 TINYINT);INSERT INTO t0(c0) VALUES(NULL);c0
NULL
t0
SELECT * FROM t0WHERENOT(t0.c0 <=> 2035382037);
c0
The MySQL-specific equality operator <=> malfunctioned for large numbers
FALSE
Real Bugs
Containment Oracle
![Page 102: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/102.jpg)
102
Oracles
DBMS Containment Error SEGFAULT
SQLite 46 17 2
MySQL 14 10 1
PostgreSQL 1 7 1
Sum 61 34 4
We also found many bugs using an Error oracle
Real Bugs
Error Oracle
![Page 103: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/103.jpg)
103
SQLite3 Bug
CREATE TABLE t1 (c0, c1 REAL PRIMARY KEY);INSERT INTO t1(c0, c1) VALUES(TRUE, 9223372036854775807), (TRUE, 0);UPDATE t1 SET c0 = NULL;UPDATE OR REPLACE t1 SET c1 = 1;SELECT DISTINCT * FROM t1 WHERE (t1.c0 IS NULL);
Real Bugs
Error Oracle
![Page 104: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/104.jpg)
104
SQLite3 Bug
CREATE TABLE t1 (c0, c1 REAL PRIMARY KEY);INSERT INTO t1(c0, c1) VALUES(TRUE, 9223372036854775807), (TRUE, 0);UPDATE t1 SET c0 = NULL;UPDATE OR REPLACE t1 SET c1 = 1;SELECT DISTINCT * FROM t1 WHERE (t1.c0 IS NULL);
Database disk image is malformed
Real Bugs
Error Oracle
![Page 105: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/105.jpg)
105
SQLite3 Bug
CREATE TABLE t1 (c0, c1 REAL PRIMARY KEY);INSERT INTO t1(c0, c1) VALUES(TRUE, 9223372036854775807), (TRUE, 0);UPDATE t1 SET c0 = NULL;UPDATE OR REPLACE t1 SET c1 = 1;SELECT DISTINCT * FROM t1 WHERE (t1.c0 IS NULL);
The INSERT and UPDATE statements corrupted the database
Database disk image is malformed
Real Bugs
Error Oracle
![Page 106: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/106.jpg)
106
Oracles
DBMS Containment Error SEGFAULT
SQLite 46 17 2
MySQL 14 10 1
PostgreSQL 1 7 1
Sum 61 34 4
We found only a low number of crash bugs, likely because DBMS
are fuzzed extensively
Real Bugs
SEGFAULTs
![Page 107: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/107.jpg)
107
Average Number of Statements
Half of all bugs can be reproducedwith only 4 SQL statements
Real Bugs
![Page 108: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/108.jpg)
108
SQLite3 Bug with a Single Statement Real Bugs
SELECT '' - 2851427734582196970;
-2851427734582196936
Subtracting a large integer from a string resulted in an incorrect result
![Page 109: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/109.jpg)
109
Discussion
• Are the bugs relevant?
![Page 110: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/110.jpg)
110
Discussion
• Are the bugs relevant?
The SQLite developers (inconsistently) assigned severity levels
Severity Level
#
Critical 14
Severe 8
Important 14
![Page 111: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/111.jpg)
111
Discussion
• Are the bugs relevant?
• Statement coverage
![Page 112: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/112.jpg)
112
Discussion
• Are the bugs relevant?
• Statement coverage
Low coverage 20%-50%, DBMS provide a lot more than pure database management
![Page 113: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/113.jpg)
113
Discussion
• Are the bugs relevant?
• Statement coverage
• Implementation effort
![Page 114: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/114.jpg)
114
Discussion
• Are the bugs relevant?
• Statement coverage
• Implementation effort
4,000-6,000 LOC per DBMS →significantly smaller than the DBMS
![Page 115: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/115.jpg)
115
Discussion
• Are the bugs relevant?
• Statement coverage
• Implementation effort
• Limitations
![Page 116: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/116.jpg)
116
Discussion
• Are the bugs relevant?
• Statement coverage
• Implementation effort
• Limitations
• Aggregate and window functions• Difficult-to-implement functionality
![Page 117: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/117.jpg)
117
Larger Picture
Pivoted Query Synthesis (PQS)
![Page 118: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/118.jpg)
118
Larger Picture
Pivoted Query Synthesis (PQS)
Metamorphic Testing
AggregateTesting
PQS is one of multiple DBMS testing approaches we have been working on
![Page 119: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/119.jpg)
119
Larger Picture
Pivoted Query Synthesis (PQS)
Metamorphic Testing
AggregateTesting
We have found about 15 bugs by a novel metamorphic testing approach that can
compute a precise result set
![Page 120: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/120.jpg)
120
Larger Picture
Pivoted Query Synthesis (PQS)
Metamorphic Testing
AggregateTesting
PQS is not applicable for testing aggregate and
window functions
![Page 121: Testing Database Management Systems · Google’s OS-Fuzz Project) SQLite’s test cases achieve 100% branch test coverage SQLite’s performs anomaly testing (out- ... SEGMENTATION](https://reader034.vdocuments.net/reader034/viewer/2022043018/5f3af34daaf42a0e347f3e44/html5/thumbnails/121.jpg)
121@RiggerManuel
Challenge: Precise Oracle is Difficult to ConstructAim: Find Logic Bugs in DBMS
Idea: Consider Only a Single Row PQS is Highly EffectiveCreate Expressions thatYield TRUE for the Pivot Row