1

INTERACTIVE PROGRAM TRANSFORMATIONS

Danny Dig

What do you see yourself doing after you earn your Bachelor’s?

A.  Get a job as a programmer/tester etc., on the tech track B.  Get a job with the government C.  Go to grad school (M.S. or Ph.D.) D.  Start my own company E.  Other

2

What can I do today so that I get my dream SE job next year?

Create a folder/CV with your CS361 work (professional documentation, process, TDD, Design Patterns) Write lots of code (read even more code) Contribute to OSS - your CV is your GitHub profile 3

Software Evolution as Done Today

Change is the heart of software development: - add features, fix bugs, support new hardware/UI/OS Today’s program development is very crude - change carried manually, through low-level edits - changes are almost never reused - versioning tools focus on changes to lines of code Change is too ad-hoc making software development error-prone, time-consuming, and $$$ - 2/3rd of software costs due to software evolution, some industrial surveys claiming 90%

4

My View of Tomorrow: Programming is Program Transformation

Change needs to move to a higher-level of abstraction Program transformations as first-class: -  most changes carried through automated program

transformations -  even manual edits become transformations -  programs as sequence of program transformations Q1: Analyze what software changes occur in practice? Q2: How can we automate them? Q3: Can we represent programs as transformations? Archive, retrieve, and visualize them? Q4: Can we infer higher-level transformations?

5

Interactive Source-to-Source Program Transformations

Programmer + Tool >> Tool | Programmer

search, remember, compute

domain knowledge

6

2

Interactive Analysis &

Transformations

Refactor end-user software

Retrofit Parallelism Introduce Asynchronous Code

Upgrade library-based programs

Overview of My Research

Generate tests

TOSEM15,STVR15, ECOOP14,ISSTA13, FSE13,ICST13,FSE12, ICSE11,SOFTW11, OOPSLA09, ICSE09, ASE09a

FSE14d,ICSM12

ICST13, TSE10, ASE09b, ISSTA08, FSE07

ECOOP13,ECOOP12, TSE08, ICSE08, ICSE07,ECOOP06, JSME06,ICSM05

ICSE14, FSE14

7

Refactoring Code for Asynchronous Execution on Mobile

By 2016, 300B apps downloaded annually Running I/O blocking operations synchronously freezes the UI and frustrates users My research enables retrofitting asynchronous execution via interactive refactoring (complex analysis and transformation) - inverting flow of control, callbacks - novel analyses determine non-interference with main thread Generated hundreds of patches to correctly introduce asynchrony, accepted by OSS developers

8

Our Refactorings for Parallelism

Refactorings for thread-safety - make class immutable [ICSE'11] - convert to Atomic* classes [ICSE'09] - use concurrent collections [ICSE'09] - infer region annotations [ASE'09] - atomic check-then-act operations [ICST’13]

Refactorings for throughput - parallel recursive divide-and-conquer [ICSE'09] - loop parallelism via ParallelArray [OOPSLA’10] - loop parallelism via lambda-enabled functional operators [FSE’13] Refactorings for scalability - Atomic*, concurrent collections [ICSE'09]

9

Interprocedural analyses: -  control, data-flow -  points-to -  constraint-based

10

x, y -> x + y λ expressions Hallmark of functional languages Some languages had λ from first version (Smalltalk, Python, JS)

λ retrofitted in OO languages (C++, C#, Java) - unobtrusive parallelism - collections

From Imperative to Functional Programming through Refactoring [Gyori, Franklin, Dig, Lahoda: FSE’13]

11

1. Anonymous Inner Class to lambda

2. for loop to functional operations

Cleaner code

Less clutter

How can I λ-ify my Java code?

LambdaFicator refactorings:

unobtrusive parallelism

explicit semantics

Advantages of iterating with functional operations

1.  private void redify() { 2.  int n = 4;// amount of

parallelism 3.  Thread[] threads = new

Thread[n]; 4.  final List<ElementRule> rules = 5. 

properties.getRules(); 6.  int size = rules.size(); 7.  for ( int i = 0; i < n; i++) { 8.  final int from = i * size / n; 9.  final int to = (i + 1) * size / n; 10.  threads[i] = new Thread(new

Runnable() { 11.  @Override 12.  public void run() { 13.  for ( Block b :

blocks) 14.   if(b.getColor() ==

BLUE) 15.   b.setColor(RED); 16.  } 17.  }); 18.  threads[i].start(); 19.  } 20.  for ( int i = 0; i < n; i++) { 21.  try { 22.  threads[i].join(); 23.  } catch

( InterruptedException ex) { 24.  //print error message 25.  } 26.  } 27.  }

private void redify() { for ( Block b : blocks) if(b.getColor() == BLUE) b.setColor(RED); }

private void redify() { blocks.parallelStream()

.filter(b -> b.getColor() == BLUE) .forEach(b -> { b.setColor(RED); });

}

External Iteration -> Internal Iteration

DEMO

3

13

Refactorings are widely applicable - 46% of for loops successfully refactored Refactorings improve code quality: - first refactoring removes boilerplate code (52% AST node reduction) - second refactoring makes loop semantics explicit LambdaFicator saves programmer effort - first refactoring changed avg. of 412 SLOC/project in 11 seconds - second refactoring changed avg. of 1368 SLOC/project in 11 seconds

Empirical evaluation We ran LambdaFicator on 10 projects, ~1Million SLOC

Refactoring meets Spreadsheets [Badame, Dig – ICSM’12]

Spreadsheets have been around for 4,000 years - Plimpton 322, Babylonian clay tablet showing the Pythagorean

triplets, 1800BC - contains a copy-paste error: number in row 9 is copied from 8 The number of spreadsheet end-users is at least 100M

=> 10x more errors than in professional programs 14

Spreadsheets that are changing are prone to errors

We studied a corpus of 3,700 real world spreadsheets

Spreadsheets are riddled with problems: - decreasing maintainability (e.g., duplicated expressions) - decreasing performance (e.g., redundant computation)

End-users have to maintain spreadsheets - life-span: 5 years, end-users/sheet: 13 [Hermans – ICSE’11]

Smells mask errors that cost [Burnett – CACM’04]

- TransAlta lost $24M in 2003 due to cut-and-paste error in Excel - The Fed made a copy-paste error in their consumer credit statement,

could have led to $4B loss 15

Our Refactorings

1.  Replace Awkward Formula 2.  Extract Literal 3.  Guard Call 4.  String to Dropdown 5.  Introduce Cell Name 6.  Make Cell Constant 7.  Extract Column

Implemented in our tool, REFBOOK, world’s first interactive refactoring tool for Microsoft Excel 16

17

Extract Column Refactoring

Extract Column Refactoring needs to change elements “similar” to the one selected

Need to ensure that all expressions in the column have the “same” formula, despite not being exact matches Precondition: - formulas in the column of selected expression are consistent Two formulas are consistent if their ASTs are isomorphic Refactoring steps for each row: - compute the corresponding expression (e.g., B2+C2+D2+E2) - insert corresponding expression in the new column - replace references to expression with the name of new cell 18

4

3-pronged Evaluation shows REFBOOK is useful

1.  Survey 2.  Controlled experiment 3.  Case study RQ1: Do refactorings improve spreadsheet quality? Yes, for 4 refactorings users preferred refactored sheets RQ2: Can REFBOOK make the process reliable? Yes, REFBOOK safe, manual refactorings had errors RQ3: Do automated refactorings improve human productivity? Yes, 2x-20x productivity improvement RQ4: Are the refactorings applicable? Yes, 32% of formulas are candidates for ExtractColumn

19

Practical Impact of My Research

Several of our tools ship with official release of Eclipse alone had 13M downloads in 2014 Influenced official concurrency libraries of Java and .NET Our tools used in industry Hundreds of accepted OSS patches http://learnparallelism.net (120,000 visitors)

20

Collaborators •  Alexandria Shearer (ugrad) •  Anda Bereckzy (ugrad) •  Lyle Franklin (ugrad) •  Alex Gyori (ugrad) •  Alex Sikora (ugrad) •  John Marrero (ugrad) •  Mihai Tarce (ugrad) •  Jack Ma (ugrad) •  Lorand Szakacs (ugrad) •  Sandro Badame (MS) •  Fredrik Kjolstadt (MS) •  Kely Garcia (MS) •  Kashif Manzoor (MS) •  Can Comertoglu (MS) •  Caius Brindescu (PhD) •  Mihai Codoban (PhD) •  Michael Hilton (PhD) •  Sergey Shmarkatiuk (PhD) •  Brett Daniel (PhD) •  Stas Negara (PhD) •  Cosmin Radoi (PhD) •  Semih Okur (PhD) •  Yu Lin (PhD) •  Rob Bochinno (PhD) •  Mohsen Vakillian (PhD) •  Stephen Heumann (PhD) •  Jeff Overbey (PhD) •  Ralph Johnson •  Darko Marinov •  Marc Snir •  Brian Bailey •  Vikram Adve •  Sarita Adve

•  Frank Tip (U of Waterloo) •  Julian Dolby (TJ Watson) •  Danny Soroker (TJ Watson) •  Ramón Cáceres (TJ Watson) •  Andreas Schade (TJ Watson) •  Susan Spraragen (TJ Watson) •  Alpana Tiwari (TJ Watson) •  Tien Nguyen (IA State) •  Tao Xie (NCSU) •  Kunal Taneja (NCSU) •  John Marrero (MIT) •  Michael Ernst (U Washington) •  Shay Artzi (MIT) •  Adam Kiezun (MIT) •  Marius Minea (UPT) •  Iulian Neamtiu (UC Riverside) •  Bill Griswold (UCSD) •  Vibhu Mohindra (ACL Wireless) •  Jan Lahoda (Oracle)

21

Future Work: Change-Oriented Programming Environment

Enable a software revolution similar with the industrial revolution Semi-automated transformations at center of software development -  View programs as a composition of transformations -  Enable programmers to write, script, modify, and replay

transformations

22