Pareon Verify

Test suite for dynamic software verifi cation

Find the hard-to-fi nd errorsother tools don’t fi nd

Copyright © 2015 Vector Fabrics B.V

“We are going to defi nitely incorporate the Vector Fabrics toolset as an important part of our quality off ering“

Jan AsselmanQuality team leader / Altran Intelligent Systems

2 Copyright © 2015 Vector Fabrics B.V

Avoid software errors and high repair costs with Pareon Verify

Test suite for dynamic software verification finds difficult-to-detect software errors and makes lengthy debugging sessions a thing of the past.

The amount of software in embedded systems, cell phones, or transportation vehicles such as cars and airplanes is exploding in size and importance. This development has led to a exponential growth in the number of potential software errors or “bugs”.

It is more important than ever to effectively and thoroughly test software before it is shipped. With the common use of C++ nowadays, it is also common to have more dynamic behavior and multithreading. This leads to a new category of errors that is hard to detect using the traditional testing tools and methods.

Errors are expensiveHaving to repair software errors post-release in pro-duction or in the field is a costly process. Expensive, even without taking into account any damage to the brand and the public image of the manufacturer.

Thus it is crucial that as many errors in the code as possible are detected and corrected before the prod-uct enters the market.

Cost of an errorA study by NASA [1] shows the cost of fixing a software error found in the field is 100x more expensive than the same error found already during development.

Development Testing In the field

The mounting costs of a software defect.

3Copyright © 2015 Vector Fabrics B.V

100 million lines of complex codeIt is not uncommon for a car or an airplane to have over 100 million lines of code running. As the amount and the of complexity of code increases, so does the number of errors in the code.

Another component to drive up the complexity and the number of errors in code is multicore usage. More and more processors are already multicore, which brings new challenges to software correctness and verifi cation.

Increasing dynamic behaviorModern software testers are challenged to cope with the ever increasing complexity and the ever increasing dynamic behavior of the software. Dynamic behavior - for example dynamic memory use and multithread-ing - is hard to test for. Bugs in dynamic behavior tend to be hard to reproduce. Examples of these errors include buff er overfl ows, memory leaks, and invalid memory accesses.

HeisenbugsA whole new category of software errors is nowadays labelled as a “Heisenbug”: named after the uncer-tainty principle elaborated by the German theoretical physicist Werner Heisenberg. Heisenbugs are hard to diagnose because they tend to disappear when under scrutiny: debuggers or so-called “printf-debugging” tend to adjust the run-time circumstances enough to make the bug disappear.

For instance, a Heisenbug could relate to a timing problem: in a debugger, timings change and the bug does not appear. This makes the subsequent detection and resolution of these errors an ex-tremely complex and thus costly process.

For the reasons mentioned above, validating software without dynamic analysis is no longer an eff ective strategy. Software projects are growing in size, the software itself is becoming more dynamic and the hardware used to operate the software is becoming increasingly sophisticated. Without employing dynam-ic analysis, the complex software errors will remain undetected in the product to cause problems later.

Automated dynamic analysis by Pareon VerifyThis trend led Vector Fabrics to develop Pareon Verify - a tool for dynamic software verifi cation. Pareon Verify allows for automated dynamic analysis of coding errors in large volumes of source code. This process is able to detect software errors that cannot be identi-fi ed using other tools or methods.

Increasing size and complexity of applicationscause more errors to occur in software.

10024

148,5

19,4

21,4

30,1

20,6100

2414

8,5

19,4

21,4

30,1

20,6

Growing size and complexity of code increases the number of errors in code.

Car SofwareAverage high-end car

F-35 Fighter Jet2013

Boeing 787Total fl ight software

*million lines of code

% Single processor

% Multi processor

% Multi core

% Multi core & Multi processor

% Don’t know

Copyright © 2015 Vector Fabrics B.V4

The advantages of dynamic code analysisDynamic analysis - or runtime analysis, as it’s some-times called - can be seen as an MRI scanner for software. By looking inside the application while it executes, it is possible to detect errors in the internal behavior.

Dynamic analysis observes at the application’s internal structures and behavior and observes where errors originate. In doing so, it pinpoints the primary cause of the error. This way, dynamic analysis identifies errors that remain undetected during other tests.

Limits of existing methodsErrors in dynamic behavior are notoriously difficult to detect with traditional testing tools and methods. Errors in the dynamic behavior don’t trigger when the application is run in its normal testing environment, but cause problems when environment, timing or circumstances change.

Static analysis is limited in its capability to predict the behavior of dynamic structures and cannot foresee all possible combinations of inputs, code paths, dynamic memory and thread timing in the full application scope.

Unit testing is limited in scope to a smaller part of the application, and cannot detect errors that are caused by the global behavior or the environment of the application.

Dynamic analysis to helpDynamic analysis is not hindered by these limitations: as it observes the application’s actual operation while being executed, it can see all the combinations of dynamic and global behavior that are triggered during the execution, regardless of complexity.

Dynamic analysis looks at the entire application and is not limited to a local unit scope in its error detection. Additionally, as it looks inside the application to see errors in individual operations in a white-box fashion, it reports errors in an application even if that error does not cause the application to fail its black-box test on a particular test case.

Therefore, dynamic code analysis is a part to a comprehensive software testing process. It com-plements static code analysis and unit testing. Dy-namic analysis offers the key advantage: it reveals the most expensive errors that would otherwise remain undetected.

The role of the dynamic analysis in the software verification process.

Dynamic analysis

Unit & system tests

Static analysis

Coding style checks

Compilation

Heisenbugs

Functional errors

Coding errors

Coding style violations

Syntax errors

Cost

of a

n er

ror

Copyright © 2015 Vector Fabrics B.V 5

How it works?Dynamic analysis works by instrumenting the target application to see the behavior inside the application. The instrumentation in the application binary creates a stream of events, or a trace, that is captured by the analyzer and studied for errors in the behavior. The analyzer then reports the found errors in a clear and concise way with detailed references to source code for easy correction of the error.

The workflow of Pareon Verify is as follows:1. Compile

Use Pareon’s instrumenting C and C++ compiler as a drop-in replacement for your (cross)compiler. Set-up your build in minutes.

2. Execute Execute the instrumented program on a host PC or target embedded board. Run with representa-tive input data or test suite.

3. Analyze Conveniently analyze the resulting execution trace on your host PC. Minimal memory and perfor-mance impact from the analysis on the target!

4. Resolve bugs Get detailed messages for each error. Instantly see how to fix the problem. Variable names, full call stacks. No false positives!

Pareon Verify detects a range of software errors, in-cluding Heisenbugs.. Pareon Verify can be completely integrated into the development process so that new software changes can be continuously screened for new errors.

Invalid memory access Invalid pointer usage and buffer overflows e.g. through bad pointer arithmetic

Array out-of-bounds Access outside a buffer or using wrong indexes into a multidimensional array

Uninitialized memory readsUsing variables that have not been written yet, generally leading to data corruption or crashes

Use-after-free Use of dangling pointers, pointing to memory that no longer exists

Race conditionsMultiple threads accessing the same data without proper synchronization, generally leads to data corruption

Memory leaks Will cause system to slow down or run out of memory

Hard-to-detect errors Pareon Verify finds.

6 Copyright © 2015 Vector Fabrics B.V

Example Multicore errors Computer hardware increasingly uses multicore processors. Not just PCs, but also cell phones have up to 8 cores inside nowadays. This causes problems on the software side as multicore software development is harder. So-called “data races” already occurred on single core CPUs when interacting with interrupts, but are now much more widespread due to the multi-threaded programs actually running threads in parallel.

A data race arises when two threads access the same variable without proper synchronization. As a result, threads could read stale or just wrong data - data corruption will happen.

In general, multi-threaded programs will have more dynamic behavior and thus will be more susceptible to dynamic software errors.

“We are glad we are now able to catch some really difficult bugs much earlier“

Daniele LacameraLead developer / PicoTCP

Read and write from different threads to shared memory.

Data race, read of uninitialized data because the read happens before the write is done.

Thread A Thread BShared

memory

?

Read

Thread A

Tim

e

Tim

e

Thread BShared

memory

?

Read

Write10

10

Write10

10

Thread A Thread BShared

memory

?

Read

Thread A

Tim

e

Tim

e

Thread BShared

memory

?

Read

Write10

10

Write10

10

7Copyright © 2015 Vector Fabrics B.V

Solving errors early through continuous integrationMany professional software development compa-nies deploy a development process called “contin-uous integration”. Upon on check-in of a new piece of code by a developer or at least every night, a central server will take all source code, build it and run a set of tests on it. If any of the tests break, the just submitted code is rejected and the develop-er has to fix it. This allowed to find a lot of errors earlier in the process: whilst the developer is still working on it, or just finished. The code is still fresh on his mind, fixes will be faster.

Using Pareon Verify, it is now also possible to deploy dynamic error finding during continuous in-tegration. Instead of just receiving a “pass” or “fail” from the automated tests, Pareon Verify will also report if the code flow leading to the verdict was correct. Pareon Verify reports if the test result is only correct by accident. Pareon Verify also reports if there were unwanted side effects. The benefit of this early detection is that developer can correct the error immediately - even before it triggers a test case to fail and a crash to happen, saving a lot of time and costs in the future.

To reduce maintenance costs, it is essential that errors be removed from a software design as early as possible. A well-known Japanese car manufacturer notably had to issue recalls for problems that were linked to software errors in a number of its models. It is estimated that the recalls cost the company two billion dollars, not to mention the damage to its corporate image.

Benefits of Pareon VerifyLess errors in the fieldBy capturing the errors before the hit the field, companies can reduce maintenance costs of software. The customer satisfaction and the quality image of the brand will improve when errors are no longer triggered in the field. An error found during testing is 5x more expensive than one found during development, and it can be up to 100x more expensive when found in the field - not to mention impacts to customer satisfaction and brand image.

Improved time to marketOften, during the integration and system testing phase’ the number of bugs found by the test team only grows. Developers cannot tell when they are done fixing all the issues as one issue might be hiding the next. As a result, project and release schedules tend to slip, delaying the release or the end of the project. With Pareon Verify, companies will be able to cut significantly back on the time spent on debugging and correcting errors, which in turn reduces the risk of project delays. In prac-tice, this can mean weeks’ worth of time-saving and a more predictable test and integration phase.

SummaryBy using Pareon Verify, software can be brought to market more quickly, cheaper and with fewer error reports from customers. Software developed with Pareon Verify is of a higher quality and is complet-ed sooner. This allows companies to improve the quality image of their software.

http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf

For more information about Pareon Verify, please visit: http://pareonverify.com

About Vector Fabrics

Software is an essential part of virtually every modern product or service. Vector Fabrics specializes in devel-opment and testing tools for complex software that is used in embedded systems, including automotive, IOT devices, network equipment, consumer electronics, aerospace, medical and industrial systems, and more.

The Pareon family of dynamic analysis tools provides customers with deep insight into how their software will actually behave when running, ensuring a higher level of reliability, safety and performance. Using these tools, businesses are able to prevent costly software bugs and improve the time-to-market for new products.

Phone +31 40 8200 960 Email info@pareonverify.comHogeweg 39A5301 LJ ZaltbommelThe Netherlands

www.pareonverify.com

Copyright © 2015 Vector Fabrics B.V All Rights Reserved.