CEN 4072Software TestingChapter 1: How Failures Come To Be

Golden Rules• Software Testing begins with careful design.• Faulty software can cost stakeholders more than millions, or

even risk lives. • Testing is an important step in the development cycle.• Software that is poorly designed or put together is difficult to

test.• Software that is difficult to test or cannot be tested is useless.

Failures & Damage Caused• Knight Capital Group (2012) – Faulty stock trading algorithm

caused company to lose $440M in 30 minutes.• NASA Mars Climate Orbiter (1998) – Unit conversion failure,

caused $125M spacecraft to fail to stabilize and lost in space.• Patriot Missile Error (1991) – Software’s accuracy to detect

enemy attacks became worse the longer software ran, eventually failing to detect an attack which killed 28 American soldiers.

Debugging Facts• Software bugs are costing the U.S. economy an estimated

$59.5 Billion per year [RTI (2002)].• Developers estimate that improvements in debugging &

testing could reduce the cost of software by $22.5 Billion. [RTI (2002)]

• 50% of labor to develop a working program is spent on testing & debugging activities. [Beizer (1990)]

• Validation Activities (Debugging, Testing & Verification) can range from 50%-75% of development costs. [Hailpern & Santhanam (2002)].

Defects• Piece of code that can cause program to enter an infected

state.• Can be a miscalculation, a piece of code that did not consider

future or possible scenarios, etc.• Not necessarily a program failure.

From Defect(s) to Failure• Sequence of events cause defect to become failure.• Programmer creates defect – a piece of code is created that

did not predict future requirements/scenarios.• The defect causes an infection – the program runs, and

program enters a state different from what is intended.• The infection propagates – Functions dependent on the

results of the now infected state are likely to also become infected (unless overwritten, hidden or corrected).

• The infection causes a failure – the program produces a visible error, or acts incorrectly, and is thus, now a failure.

TRAFFIC Principal• Track the problem.• Reproduce the failure.• Automate & simplify the test case.• Find possible infection origins.• Focus on likely origins.• Isolate the infection chain.• Correct the defect.

Searching in Time & Space• We must find out which part of the state is infected (search in

space).• We also must find when the state becomes infected (search in

time).• This can be quite difficult, imagine trying to search in time &

space for a program with 1000’s of variables & references between them!

Sample Program - Sortingint main(int argc, char *argv[]){ int *a; int i;

a = (int *)malloc((argc - 1) * sizeof(int)); for (i = 0; i < argc - 1; i++) a[i] = atoi(argv[i + 1]);

shell_sort(a, argc);

printf("Output: "); for (i = 0; i < argc - 1; i++) printf("%d ", a[i]); printf("\n");

free(a); return 0;}

Sample Program (Cont’d)static void shell_sort(int a[], int size){ int i, j; int h = 1; do { h = h * 3 + 1; } while (h <= size); do { h /= 3; for (i = h; i < size; i++) { int v = a[i]; for (j = i; j >= h && a[j - h] > v; j -= h) a[j] = a[j - h]; if (i != j) a[j] = v; } } while (h != 1);}

There is a defect somewhere in this program, find it using the principles described below.

Sample Exercise• When ran, notice the input & output:• $ ./sample 11 14• Output: 0 11

• Expected Output: 11 14

• What’s going on here? Let’s Debug!

Sample Exercise• In this sample, the first three principles are straight forward.• Track – We have observed the problem, an unknown 0

appears on each run.• Reproduce – The sample program produces incorrect data

with any run.• Automate – The sample program is not very complex, and

does not need automation.

Find Origins• We observe that the program has two methods – main &

shell_sort.• main – Allocates variables, reads input, sorts input using

shell_sort then prints result.• shell_sort – Sorts inputted array.

• When the output is printed, we notice that the value of a[0] is incorrectly set to 0.

• Was the array incorrectly initialized or did something happen in shell_sort?

Observing Run/Search in Time• By printing to console using fprintf, we can note the values of

the array at each step of the program. • ./sample 11 14• Input: 11 14• Shell Sort Input: 11 14 0• Shell Sort Size: 3• Shell Sort Sorted: 0 11 14• Output: 0 11

• Shell Sort is called with a 3 element array?

Focus on Likely Origins• We hypothesize that shell_sort runs correctly.• The infection must be happening before shell_sort runs.• We notice that the size passed in shell sort is not equal to that

of the array. This is likely related to our defect.• Looking through shell sort, we notice that it accesses no local

variables.• We also notice the 3 element array outputted by shell_short is

correctly sorted.

Finding Causes• Array is correctly initialized with 2 elements at the start.• However, in main, before shell_sort is called, our array is of

size 2 while the passed value for size in shell_sort is 3. • shell_sort takes the 3 element array it gets [11, 14, 0] and in

fact correct sorts it as [0, 11, 14].• The input of size 3 causes shell sort to access uninitialized

space in a[], a[2] which happens to hold a value of 0.• During the sort, a[2] is swapped with a[0], setting a[0] = 0,

thus infecting the state.• We speculate this incorrectly passed size is to blame!

Correct Issue• We used argc-1 as the size of our array previously to initialize

it, however argc is passed into shell_sort, giving it the incorrect size.

• shell_sort(a, argc);

• We fix this by replacing this line with: • shell_sort(a, argc-1);

• We recompile the program & run it with the same input: • $ ./sample 11 14• Output: 11 14

• The failure no longer occurs, issue is resolved!

Efficiency of Testing• We were able to debug this sample manually.• All debugging problems could be solved manually.• However, manually debugging may take large amounts of

time.• We may miss defects. • Automated Debugging techniques make the task of testing

quicker & easier.

Automated Debugging: Program Slices• Separate part of the program that is relevant to the failure.• Deduce from code design what could or could not possibly in

different parts of the program.• Slice the program into parts relevant to the failure.

Automated Debugging: Observing the State• Stop the program to observe the entire state, line by line.• Allows programmer to view entire state at any given time.• This allows us to view values of variables at any given time.• No need to modify or recompile the program each time.

Automated Debugging: Watching State• Watch smaller part of a state to find changes during

execution.• More focused than observing the state.• Watch a particular variable to find the precise line of code

where its changed from sane to infected value.

Automated Debugging: Assertions• Specify expected values and check for them.• Especially before/after function call, we assert that the current

value of a particular variable is equal to the expected value.• If all assertions hold true, we can consider a chunk of code sane.• Thus allows us to focus on other potential infections.

// Return Sum of squares from 1 to nint sumOfSquares(int n){

ASSERT(n == 3);// Do Math// …ASSERT(sum == 14);return sum;

}

In Example: We assert input is 3 as expected, and that output is 14 as expected.

Automated Debugging:Anomalies• Generally, we assume a program works fine most of the time.• If a program fails, we observe key differences between the

passing runs & failing run. • These difference are called anomalies.

// Raises n to k’th power.int pow(int n, int k){

int result = n;for (int i = 0; i < k; i++)

result *= n;return result;

}

This method generally works fine. It works for 2^4, 5^2 and any positive integer for k.

However, what happens if we use a negative integer or zero for k?

Automated Debugging:Cause-Effect Chains• Identify which variable or assignment in each state transition

contributed to failure.• Although causes are not necessarily defects/failures, they can

help narrow down which variables may be related to the failure.

Automated Debugging:Simplified Input• Simplify the input so that each part contributes to the failure.• Using a technique called delta debugging, which automatically

narrows down what’s different between passing & failing run.• For simplified input, delta debugging is used to find only

inputs that contributed to the failure.

For Example: Suppose we have some program that transitions to infected each time when a sequence of even numbers in inputted.

Manually, we may be have tested this using a more complex input like:0 1 2 4 5 7 -1 2 4 6 8 1 1 10

Simplified Input may take this & a passing set of input & create:0 2 4 6 8

Recap• Importance of Software Design & Testing• Cost of Software Failures• Defects, Infections & Failures• TRAFFIC Principle• Manual Debugging• Automated Debugging

Bibliography• Zeller, A. (2009). Why Programs Fail: A Guide to Systematic

Debugging(2nd ed.). Burlington, MA: Morgan Kaufmann.• Harley, N. (2014, May 29). 10 of the most costly software

errors in history. Retrieved September 1, 2015.