test-driven synthesis · qual 1 qual 2 qual 3 andrew 01.02.2003 27.06.2008 06.04.2007 ben...
TRANSCRIPT
![Page 1: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/1.jpg)
Test-Driven Synthesis
Daniel Perelman1
Sumit Gulwani2 Dan Grossman1 Peter Provost3
1University of Washington
2Microsoft Research
3Microsoft Corporation
June 11, 2014
![Page 2: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/2.jpg)
TDD example1
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// do nothing
throw new
NotImplementedException();
}
1http://blog.8thlight.com/uncle-bob/2013/05/27/
TheTransformationPriorityPremise.html2 / 49
![Page 3: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/3.jpg)
TDD example1
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// return constant "word"
return "word";
}
1http://blog.8thlight.com/uncle-bob/2013/05/27/
TheTransformationPriorityPremise.html3 / 49
![Page 4: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/4.jpg)
TDD example1
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// return input string
return s;
}
1http://blog.8thlight.com/uncle-bob/2013/05/27/
TheTransformationPriorityPremise.html4 / 49
![Page 5: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/5.jpg)
TDD example1
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"
4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// return input if short
if(s.Len ≤ len)
return s;
else
return "Long\nWord";
}
1http://blog.8thlight.com/uncle-bob/2013/05/27/
TheTransformationPriorityPremise.html5 / 49
![Page 6: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/6.jpg)
TDD example1
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"
5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
1http://blog.8thlight.com/uncle-bob/2013/05/27/
TheTransformationPriorityPremise.html6 / 49
![Page 7: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/7.jpg)
TDD example1
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// wrap word to length len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + wrap(s[len:], len);
}
1http://blog.8thlight.com/uncle-bob/2013/05/27/
TheTransformationPriorityPremise.html7 / 49
![Page 8: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/8.jpg)
Google’s Real TDD2
2http://googletesting.blogspot.com/2014/04/
the-real-test-driven-development.html8 / 49
![Page 9: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/9.jpg)
Google’s Real TDD2 (April Fools’ Day joke)
2http://googletesting.blogspot.com/2014/04/
the-real-test-driven-development.html9 / 49
![Page 10: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/10.jpg)
Example domain: string transformations
“Spreadsheet Data Manipulation using Examples,CACM 2012,
Sumit Gulwani, William Harris, Rishabh Singh”⇓
“Spreadsheet Data Manipulation using Examples\nGulwani, S.; Harris, W.; Singh, R.\nCommunications of the ACM, 2012”
10 / 49
![Page 11: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/11.jpg)
Example domain: table transformations
Qual 1 Qual 2 Qual 3
Andrew 01.02.2003 27.06.2008 06.04.2007
Ben 31.08.2001 05.07.2004
Carl 18.04.2003 09.12.2009
⇓Andrew Qual 1 01.02.2003
Andrew Qual 2 27.06.2008
Andrew Qual 3 06.04.2007
Ben Qual 1 31.08.2001
Ben Qual 3 05.07.2004
Carl Qual 2 18.04.2003
Carl Qual 3 09.12.2009
11 / 49
![Page 12: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/12.jpg)
Example domain: XML transformations
<doc>
<p>1</p>
<p class='a'>2</p>
<p>3</p>
<p>4</p>
<p class='b'>5</p>
<p>6</p>
<p class='c'>7</p>
</doc>
⇒
<doc>
<p>1</p>
<p class='a'>2</p>
<p class='a'>3</p>
<p class='a'>4</p>
<p class='b'>5</p>
<p class='b'>6</p>
<p class='c'>7</p>
</doc>
12 / 49
![Page 13: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/13.jpg)
Workflow
domain expert user
DSL(library+grammar)
test cases
synthesize program
program P
done?
generatenew test
case
output programnoyes
13 / 49
![Page 14: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/14.jpg)
Workflow
domain expert user
DSL(library+grammar)
test cases
synthesize program
program P
done?
generatenew test
case
output programnoyes
14 / 49
![Page 15: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/15.jpg)
Workflow
domain expert user
DSL(library+grammar)
test casessequence
sythesize change
program P
done?
generatenew test
case
output programnoyes
15 / 49
![Page 16: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/16.jpg)
Contrast with genetic programming
domain expert user
DSL(library+grammar)
fitnessfunction f
sythesize change
program P
f (P) ≥ α
generatenew test
case
output programnoyes
16 / 49
![Page 17: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/17.jpg)
Workflow
domain expert user
DSL(library+grammar)
test casessequence
sythesize change
program P
done?
generatenew test
case
output programnoyes
17 / 49
![Page 18: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/18.jpg)
Workflow
domain expert user
DSL(library+grammar)
test casessequence
sythesize change
program P
done?
generatenew test
case
output programnoyes
18 / 49
![Page 19: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/19.jpg)
Modifying program
I Replace single subexpression onnew test case’s path
I Where to modify program
I What to replace with
19 / 49
![Page 20: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/20.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
20 / 49
![Page 21: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/21.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
21 / 49
![Page 22: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/22.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
22 / 49
![Page 23: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/23.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
23 / 49
![Page 24: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/24.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
24 / 49
![Page 25: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/25.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
25 / 49
![Page 26: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/26.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
26 / 49
![Page 27: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/27.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
27 / 49
![Page 28: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/28.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
28 / 49
![Page 29: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/29.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
29 / 49
![Page 30: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/30.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + s[len:];
}
30 / 49
![Page 31: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/31.jpg)
Where to modify
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
Program:wrap(string s, int len) {// wrap word to length len
if(s.Len ≤ len)
return s;
else
return s[0:len] +
"\n" + wrap(s[len:], len);
}
31 / 49
![Page 32: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/32.jpg)
What to put there
I DSL defines space of expressions
I Prefer expressions found inprevious program
I e.g., wrap(s[len:], len) containss[len:]
32 / 49
![Page 33: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/33.jpg)
Conditionals
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"
5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
test caseprogram path 1 2 3 4
""
"word" X"foobar" X"Long\nWord" X"Longer\nWord" Xs X Xs[0:len] +
"\n" + s[:len]
X X
guard 1 2 3 4
s == "word" Tlen == 6 T Ts.Len == len Ts.Len ≤ len T T
33 / 49
![Page 34: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/34.jpg)
Conditionals
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"
5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
test caseprogram path 1 2 3 4
""
"word" X"foobar" X"Long\nWord" X"Longer\nWord" Xs X Xs[0:len] +
"\n" + s[:len]
X X
guard 1 2 3 4
s == "word" Tlen == 6 T Ts.Len == len Ts.Len ≤ len T T
34 / 49
![Page 35: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/35.jpg)
Conditionals
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"
5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
test caseprogram path 1 2 3 4
""
"word" X"foobar" X"Long\nWord" X"Longer\nWord" Xs X Xs[0:len] +
"\n" + s[:len]
X X
guard 1 2 3 4
s == "word" Tlen == 6 T Ts.Len == len Ts.Len ≤ len T T
35 / 49
![Page 36: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/36.jpg)
Conditionals
Test cases:
1. wrap("word", 5)
== "word"
2. wrap("foobar", 6)
== "foobar"
3. wrap("LongWord", 4)
== "Long\nWord"4. wrap("LongerWord", 6)
== "Longer\nWord"
5. wrap("LongerWord", 2)
== "Lo\nng\ner\nWo\nrd"
test caseprogram path 1 2 3 4
""
"word" X"foobar" X"Long\nWord" X"Longer\nWord" Xs X Xs[0:len] +
"\n" + s[:len]
X X
guard 1 2 3 4
s == "word" Tlen == 6 T Ts.Len == len Ts.Len ≤ len T T
36 / 49
![Page 37: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/37.jpg)
Conditionals
Program:wrap(string s, int len) {// split string at len
if(s.Len ≤ len)
return s;
else
return s[0:len]
+ "\n"+ s[len:];
}
test caseprogram path 1 2 3 4
""
"word" X"foobar" X"Long\nWord" X"Longer\nWord" Xs X Xs[0:len] +
"\n" + s[:len]
X X
guard 1 2 3 4
s == "word" Tlen == 6 T Ts.Len == len Ts.Len ≤ len T T
37 / 49
![Page 38: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/38.jpg)
Loops
Program:wrap(string s, int len) {// wrap word to length len
if(s.Len ≤ len)
return s;
else
return s[0:len]
+ "\n"+ wrap(s[len:], len);
}
test caseprogram path 1 2 3 4
""
"word" X"foobar" X"Long\nWord" X"Longer\nWord" Xs X Xs[0:len] +
"\n" + s[:len]
X X
guard 1 2 3 4
s == "word" Tlen == 6 T Ts.Len == len Ts.Len ≤ len T T
38 / 49
![Page 39: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/39.jpg)
Loops
I Recursion (if in DSL)
I Higher-order functions (if in DSL)
I Direct support for certain loops:see paper for details
39 / 49
![Page 40: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/40.jpg)
Evaluation
I Three DSLs focused on end-user data transformation tasks.I String transformations (FlashFill+extensions)I Spreadsheet table transformationsI XML transformations
I Able to synthesize many examples from help forums in undera minute.3
I Good news: test case sequences are very short (easy for user)
I Bad news: test case sequences are very short (difficult toevaluate usefulness of sequences)
3https://homes.cs.washington.edu/˜perelman/publications/pldi14-tds.zip40 / 49
![Page 41: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/41.jpg)
Evaluation
I Three DSLs focused on end-user data transformation tasks.I String transformations (FlashFill+extensions)I Spreadsheet table transformationsI XML transformations
I Able to synthesize many examples from help forums in undera minute.3
I Good news: test case sequences are very short (easy for user)
I Bad news: test case sequences are very short (difficult toevaluate usefulness of sequences)
3https://homes.cs.washington.edu/˜perelman/publications/pldi14-tds.zip41 / 49
![Page 42: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/42.jpg)
Evaluation
I Three DSLs focused on end-user data transformation tasks.I String transformations (FlashFill+extensions)I Spreadsheet table transformationsI XML transformations
I Able to synthesize many examples from help forums in undera minute.3
I Good news: test case sequences are very short (easy for user)
I Bad news: test case sequences are very short (difficult toevaluate usefulness of sequences)
3https://homes.cs.washington.edu/˜perelman/publications/pldi14-tds.zip42 / 49
![Page 43: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/43.jpg)
Pex4Fun (now https://www.CodeHunt.com/)
domain expert
intro CS DSLtest cases
player modifies program
program
done?
generatenew test
case
win gamenoyes
43 / 49
![Page 44: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/44.jpg)
Pex4Fun (now https://www.CodeHunt.com/)
domain expert
intro CS DSLtest casessequence
synthesize change
program
done?
generatenew test
case
win gamenoyes
44 / 49
![Page 45: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/45.jpg)
Comparison to human players
0
20
40
60
80
100
0 20 40 60 80 100 120 140 160 180 200
Synth
esiz
er T
ime
(sec
onds)
User Time (seconds)
0
20
40
60
80
100
0 20 40 60 80 100 120 140 160 180 200
Synth
esiz
er T
ime
(sec
onds)
User Time (seconds)
Completion times comparableto human players
45 / 49
![Page 46: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/46.jpg)
Research question
Hypothesis:I Test sequence order matters. . .I . . . but not too much.
46 / 49
![Page 47: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/47.jpg)
Slowdown due to reordering test case sequence
1
10
100
0 0.2 0.4 0.6 0.8 1
Norm
ali
zed e
xecuti
on t
ime
Normalized inversions count
Robust to small reorderings,fails on larger reorderings
47 / 49
![Page 48: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/48.jpg)
Contributions
I Synthesis workflow inspired byTDD
I Search strategy enabled by thisworkflow, parameterized by a DSL
48 / 49
![Page 49: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/49.jpg)
Questions
49 / 49
![Page 50: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/50.jpg)
Backup slides
50 / 49
![Page 51: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/51.jpg)
Component-based synthesis example
Starting components:
I a+b
I 0
I x
I y
Generated components (1 step):
I 0+b
I x+b
I y+b
Generated components (2 steps):
I 0+0
I x+0
I y+0
I 0+x
I x+x
I y+x
I 0+y
I x+y
I y+y
51 / 49
![Page 52: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/52.jpg)
Component-based synthesis example
Starting components:
I a+b
I 0
I x
I y
Generated components (1 step):
I 0+b
I x+b
I y+b
Generated components (2 steps):
I 0+0=0
I x+0=x
I y+0=y
I 0+x=x
I x+x
I y+x
I 0+y=y
I x+y=y+x
I y+y
52 / 49
![Page 53: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/53.jpg)
Component-based synthesis example
Starting components:
I a+b
I 0
I x
I y
Generated components (1 step):
I 0+b
I x+b
I y+b
Generated components (2 steps):
I 0+0=0
I x+0=x
I y+0=y
I 0+x=x
I x+x
I y+x
I 0+y=y
I x+y=y+x
I y+y
53 / 49
![Page 54: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/54.jpg)
Component-based synthesis example
Starting components:
I a+b
I 0=[0,0]
I x=[10,-31]
I y=[-10,31]
Generated components (1 step):
I 0+b
I x+b
I y+b
Generated components (2 steps):
I 0+0=[0,0]
I x+0=[10,-31]
I y+0=[-10,31]
I 0+x=[10,-31]
I x+x=[20,-62]
I y+x=[0,0]
I 0+y=[-10,31]
I x+y=[0,0]
I y+y=[-20,62]
54 / 49
![Page 55: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/55.jpg)
Component-based synthesis example
Starting components:
I a+b
I 0=[0,0]
I x=[10,-31]
I y=[-10,31]
Generated components (1 step):
I 0+b
I x+b
I y+b
Generated components (2 steps):
I 0+0=[0,0]
I x+0=[10,-31]
I y+0=[-10,31]
I 0+x=[10,-31]
I x+x=[20,-62]
I y+x=[0,0]
I 0+y=[-10,31]
I x+y=[0,0]
I y+y=[-20,62]
55 / 49
![Page 56: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/56.jpg)
Loops
I Rule-based generation of test cases for loop body from testcases for loop
I For loop example:Factorial(1)=1
Factorial(2)=2
Factorial(3)=6
Factorial(4)=24
⇒Factorial body(2,1)=2
Factorial body(3,2)=6
Factorial body(4,6)=24
I Array by-element example:
CSum([6,2,8],[4,3,7])
=[10,15,30]⇒
CSum body(6,4,0,[])=10
CSum body(2,3,1,[10])=15
CSum body(8,7,2,[10,15])=30
56 / 49
![Page 57: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/57.jpg)
Loops
I Rule-based generation of test cases for loop body from testcases for loop
I For loop example:Factorial(1)=1
Factorial(2)=2
Factorial(3)=6
Factorial(4)=24
⇒Factorial body(2,1)=2
Factorial body(3,2)=6
Factorial body(4,6)=24
I Array by-element example:
CSum([6,2,8],[4,3,7])
=[10,15,30]⇒
CSum body(6,4,0,[])=10
CSum body(2,3,1,[10])=15
CSum body(8,7,2,[10,15])=30
57 / 49
![Page 58: Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2](https://reader035.vdocuments.net/reader035/viewer/2022063016/5fd5eeefd9f29b11f843898d/html5/thumbnails/58.jpg)
Loops
I Rule-based generation of test cases for loop body from testcases for loop
I For loop example:Factorial(1)=1
Factorial(2)=2
Factorial(3)=6
Factorial(4)=24
⇒Factorial body(2,1)=2
Factorial body(3,2)=6
Factorial body(4,6)=24
I Array by-element example:
CSum([6,2,8],[4,3,7])
=[10,15,30]⇒
CSum body(6,4,0,[])=10
CSum body(2,3,1,[10])=15
CSum body(8,7,2,[10,15])=30
58 / 49