constraint based synthesis for beginners
DESCRIPTION
Constraint Based Synthesis for Beginners. PSY 2012. Armando Solar- Lezama. CAP View of Synthesis. Code. Synthesis Methodology. A different approach. Synthesis Methodology. Code. Domain Specific Tool. Exhibit A: Synthesis of SQL. With Alvin Cheung and S am Madden. Motivation. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/1.jpg)
Constraint Based Synthesis for Beginners
PSY 2012
Armando Solar-Lezama
![Page 2: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/2.jpg)
CAP View of Synthesis
Synthesis Methodology Code
![Page 3: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/3.jpg)
A different approach
Synthesis Methodology
CodeDomain Specific Tool
![Page 4: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/4.jpg)
EXHIBIT A: SYNTHESIS OF SQLWith Alvin Cheung and Sam Madden
![Page 5: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/5.jpg)
Motivation•It turns out SQL is challenging to learn
Who would have thought?
•Frameworks simplify program/DB interface
You can access the DB without using SQL Which can lead to some interesting code...
![Page 6: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/6.jpg)
Examples: Explicit Selectpublic Set<Project> getUnfinishedProjects() { Set<Project> unfinishedP = new HashSet<Project>(); List<Project> projects = this.projectDao.getAllProjects();
for (Project project : projects) { if (!(project.getIsFinished())) { unfinishedP.add(project); } } return unfinishedP; }
Get list of projects from the DB
Select unfinished projects
SELECT * FROM Projects WHERE isFinished=FALSE
![Page 7: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/7.jpg)
Examples: Explicit Selectpublic List<WilosUser> getRoleUser(){ List<WilosUser> listUser = new ArrayList<WilosUser>(); List<WilosUser> user= this.userDao.getUsers(); List<Role> role = this.roleDao.getRoles(); for(int i = 0; i < user.size(); i ++){ for(int a = 0; a < role.size(); a++){ if(user.get(i).getRole_id(). equals(role.get(a).getRole_id())){ WilosUser userok = user.get(i); listUser.add(userok); } } return listUser;}
Find users in the Roles list and add them to the output
Start with users and
roles
SELECT u FROM users u, roles r WHERE u.roleId == r.id
![Page 8: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/8.jpg)
Why is this so bad?•These can be performance bottlenecks
Where performance matters, people write SQL
•More controversial arguments can be made
![Page 9: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/9.jpg)
Is this a synthesis problem?•It is
SynthesizerImperative code with loop nests
Equivalent Formula in Relational Algebra
Domain knowledge
about relational algebra
![Page 10: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/10.jpg)
Framing the problemList getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; int i = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } }return results;}
Precondition: truePostcondition:
![Page 11: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/11.jpg)
Framing the problem•We want to synthesize a post-condition!
•Challenges Defining the language Solving the synthesis problem Generating code
![Page 12: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/12.jpg)
Defining the language•Requirements
Should make reasoning about the program easy
Should have expressiveness comparable to SQL
![Page 13: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/13.jpg)
Language•Just like relational algebra...•...but with lists rather than sets
The program is written in terms of lists, not sets
• •
![Page 14: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/14.jpg)
InvariantsList getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; int i = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } }return results;}
![Page 15: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/15.jpg)
Verification•Easy if you know the invariants
Z3 can do it in seconds
•All you need are a few axioms:
:...
![Page 16: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/16.jpg)
Synthesis•Synthesis works best in a finite domain
•Invariant generation can be framed as a sketch
1) Model operations in terms of their source code
2) Construct the verification condition leaving invariant and post condition as unknowns
3) Create a sketch for unknowns4) Solve!
![Page 17: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/17.jpg)
Results so far•We have tried this with a dozen loop nests from real open source projects
•All solve in less than 7 minutes Slow, but not very optimized.
![Page 18: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/18.jpg)
EXHIBIT B: AUTOMATED GRADINGWith Rishabh Singh and Sumit Gulwani
![Page 19: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/19.jpg)
The real software problem•The Software Quality problem is a symptom
•The real problem:The demand for software in our society far
exceeds the supply of people skilled enough to produce it
![Page 20: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/20.jpg)
Three pronged attack
Make programmers more productive Make programming more accessible Reduce the cost of training the next generation
![Page 21: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/21.jpg)
Grading Programming Assignments•Test-cases based grading
No precise correctness correlation No student tailored feedback
•Manual grading by TAs Error-prone Time consuming Expensive
•Manual grading will not scale to 100K students
![Page 22: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/22.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
22
Buggy Program for Array Reverse
6:28::50 AM
![Page 23: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/23.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length-1; i < a.Length-1; i--){
b[count] = a[i];count++;
}return b;
} }
23
Buggy Program for Array Reverse
6:32::01 AM
![Page 24: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/24.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length-1; i < a.Length-1; i--){
b[count] = a[i];count++;
}return b;
} }
24
Buggy Program for Array Reverse
6:32::32 AM
No change! Sign of Frustation?
![Page 25: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/25.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i <= a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
25
Buggy Program for Array Reverse
6:33::19 AM
![Page 26: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/26.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){ Console.Writeline(i);
b[count] = a[i];count++;
}return b;
} }
26
Buggy Program for Array Reverse
6:33::55 AM
Same as initial attempt except Console.Writeline!
![Page 27: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/27.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){ Console.Writeline(i);
b[count] = a[i];count++;
}return b;
} }
27
Buggy Program for Array Reverse
6:34::06 AM
No change! Sign of Frustation?
![Page 28: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/28.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i <= a.Length; i--){ Console.Writeline(i);
b[count] = a[i];count++;
}return b;
} }
28
Buggy Program for Array Reverse
6:34::56 AM
The student has tried this before!
![Page 29: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/29.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
29
Buggy Program for Array Reverse
6:36::24 AM
Same as initial attempt!
![Page 30: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/30.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length-1; i < a.Length-1; i--){
b[count] = a[i];count++;
}return b;
} }
30
Buggy Program for Array Reverse
6:37::39 AM
The student has tried this before!
![Page 31: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/31.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i > 0; i--){
b[count] = a[i];count++;
}return b;
} }
31
Buggy Program for Array Reverse
6:38::11 AM
Almost correct! (a[i-1] instead of a[i] in loop body)
![Page 32: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/32.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i >= 0; i--){
b[count] = a[i];count++;
}return b;
} }
32
Buggy Program for Array Reverse
6:38::44 AM
Student going in wrong direction!
![Page 33: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/33.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
33
Buggy Program for Array Reverse
6:39::33 AM
Back to bigger error!
![Page 34: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/34.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
34
Buggy Program for Array Reverse
6:39::45 AM
No change! Frustation!
![Page 35: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/35.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
35
Buggy Program for Array Reverse
6:40::27 AM
No change! More Frustation!!
![Page 36: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/36.jpg)
using System;public class Program {public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){
b[count] = a[i];count++;
}return b;
} }
36
Buggy Program for Array Reverse
6:40::57 AM
No change! Too Frustated now!!! Gives up.
![Page 37: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/37.jpg)
AutoGrader•Automate grading
Find semantic errors Feedback to fix them
Students make similar mistakes
![Page 38: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/38.jpg)
Is this a synthesis problem?•It is
SynthesizerBuggy implementation
Corrections for student program
Error ModelReference Solution
![Page 39: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/39.jpg)
Array Reverse
i = 1
i <= a.Length
![Page 40: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/40.jpg)
Challenge 1: Different Algorithms
![Page 41: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/41.jpg)
Challenge 2: Scalability
1010 different possible candidate corrections
![Page 42: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/42.jpg)
Our Approach•Use data of previous student solutions
•Correction rules based on observed errors
•Create a set of candidate solutions using rules and find closest correct solution.
![Page 43: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/43.jpg)
Aren’t rewrite systems hard?•Can teachers really write rewrite rules?
•Angelic non-determinism helps Ambiguities and redundancies no longer matter
![Page 44: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/44.jpg)
Results: Problems Fixed
F1 F2 F3 F4 F50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Problems fixed
Array ReversePalindromeMaxFactorialisIncreasingSort
Error Models
Frac
tion
of
Prob
lem
s fix
ed
![Page 45: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/45.jpg)
Results: Performance
F1 F2 F3 F4 F50
5
10
15
20
25
30
35
Performance with Error Models
reversepalindromemaxisIncreasingfactorialsort
Error Models
Run
ning
Tim
e (i
n s)
![Page 46: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/46.jpg)
Results: Generalization
palindrome array max isIncreasing sort0
10
20
30
40
50
60
70
80
Generalization of Error models
Specific Error modelArray reverse error model
Num
ber
of fi
xes
![Page 47: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/47.jpg)
Feedback for Tutoring System
OK for Grading,
But not ideal for teaching
![Page 48: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/48.jpg)
Broad research agenda ahead•Transformative for students in under-funded schools
Reduce the resources required to support quality instruction Enable “true” distance education for programming courses
•Same technology could be used for automatic tutoring Identify errors stemming from deep misconceptions (e.g. not understanding difference in values vs. references)
Synthesize small examples that make misconceptions explicit
48
![Page 49: Constraint Based Synthesis for Beginners](https://reader036.vdocuments.net/reader036/viewer/2022062501/5681692e550346895de071d3/html5/thumbnails/49.jpg)
Sketch Tutorial on Saturday
http://bit.ly/sketch2012