4 December 2001

S

E

E

S

C

O

A STWW - Programma

Debugging of Real-Time Embedded Systems: Experiences from

SEESCOAMichiel Ronsse

RUG-ELIS

S

E

E

S

C

O

A

Problem description

embedded systems have become increasingly sophisticated

debugging becomes a major problem

types of bugs computational errors synchronisation errors performance errors

testing and debugging is responsible for a huge part of the development time

definitive need for appropriate debugging tools

S

E

E

S

C

O

A

Debugging embedded systems

two test/debug phases during the development during the actual use

an embedded system typically runs continuously interactive debugging? can have hard and soft real-time constraints Heisenbugs parallel and/or distributed systems: absence of a global clock uses lots of input repeatability? cyclic debugging? hardware related problems: code in ROM, limited amount of

RAM

Solution: hardware add-ons for on-chip debugging techniques

S

E

E

S

C

O

A

On-chip debugging techniques (I)

Logic Analyser

ROM monitor

ROM emulator

In-Circuit Emulator

Background Debug Mode

JTAG

S

E

E

S

C

O

A

almost all contemporary embedded processors have add-ons for debugging

as embedded processors get faster, these add-ons get closer to (or in) the processor

these add-ons take up valuable chip area (up to 10%)

these add-ons are also available in the consumer produsts, making on-site testing possible

hardware manufacturers believe in design for debugability

On-chip debugging techniques (II)

S

E

E

S

C

O

A

Embedded software

Nowadays, software engineering methods designed for `business’ applications are also used for embedded systems: Use of `higher’ languages (C, C++, Java) Use of reusable components Multithreaded applications ...

Debugging/maintaining these complex applications becomes a difficult task

Design for DebugabilityDesign for Debugability

S

E

E

S

C

O

A

Debugging software

Most important notion: Repeatability cyclic debugging Observability what is going on?

These are two problems for embedded systems: Non-determinism present in most embedded systems Low observability (real-time constraints, ...)

S

E

E

S

C

O

A

Causes of non-determinism

Sequential programs: input (keyboard, disk, network, certain system calls (e.g. gettimeofday(), …)

Parallel programs: race conditions: two threads accessing the same shared variable (memory location) in an unsynchronised way and at least one thread modifies the variable

S

E

E

S

C

O

A

Execution Replay

Goal: make repeated equivalent re-exections possible

Method: two phases Record phase: record all non-deterministic events during an

execution in a trace file Replay phase: use trace file to produce the same execution

Question: what & where to trace?

S

E

E

S

C

O

A

Example code

#include <pthread.h>

unsigned global=5;

thread2(){ global=global+6; }thread3(){ global=global+7; }

main(){pthread_t t2,t3;pthread_create(&t2, NULL, thread1, NULL);pthread_create(&t3, NULL, thread2, NULL);pthread_join(t2, NULL);pthread_join(t3, NULL);printf(“global=%d\n”, global);

}

S

E

E

S

C

O

A

Possible executions

L(5)

global=12 global=18global=11

L(5)

L(5)

L(5)

L(5)

L(11)S(11)

S(12) S(11)S(12)

S(11)

S(18)

A

A

A

A

A

A

S

E

E

S

C

O

A

Example code II

#include <pthread.h>

unsigned global=5;

thread2(){lock(); global=global+6; unlock();}thread3(){lock(); global=global+7; unlock();}

main(){pthread_t t2,t3;pthread_create(&t2, NULL, thread1, NULL);pthread_create(&t3, NULL, thread2, NULL);pthread_join(t2, NULL);pthread_join(t3, NULL);printf(“global=%d\n”, global);

}

S

E

E

S

C

O

A

Possible executions II

global=18

L(5)

L(11)

S(11)

S(18)

A

A

global=18

L(5)

L(11)

S(11)

S(18)

A

A

S

E

E

S

C

O

A

Race conditions

Three types of conflicts: load/store store/load store/store

Two types: synchronisation races:

doesn’t allow the use of cyclic debugging techniquesis not a bug, is desired non-determinism

data races:doesn’t allow the use of cyclic debugging techniquesis a bug, is undesired non-determinism

S

E

E

S

C

O

A

Tracing all memory operations

•Introduces an intolerable overhead, both in time and space

•Due to out-of-order stores, the order in which threads see events can differ

•Will replay both synchronization and data races.

S

E

E

S

C

O

A

Tracing synchronization operations

•The only events that are assured to be seen in the same order are synchronisation operations

•Synchronization operations are a subset of all memory operations => lower overhead

•Will replay only synchronization races.

•Will fail if data races occur.

•Solution

•Record: trace synchronization operations

•Replay: check for data races (once)

S

E

E

S

C

O

A

Record phase

T2

Main

T34

55

6

8

6

7

9

8

12

13

14

10

11

1

2

33

4

Main: 1,2,3,4,5,13,14,...T2: 3,6,7,8,9,...T3: 4,5,6,8,10,11,12,...

S

E

E

S

C

O

A

Replay phase T2Main T3

4

5

8

6

7

9

12

13

14

10

11

1

2

3

Main: 1,2,3,4,5,13,14,...T2: 3,6,7,8,9,...T3: 4,5,6,8,10,11,12,...

S

E

E

S

C

O

A

Implementation

RecPlay for Solaris (SPARC) and Linux (x86)

Uses JiTI for dynamic instrumentation

Record overhead: 1.6%

S

E

E

S

C

O

A

Instrumenting Java classes

JVM

Generic Debugger Backend

class 1class 1

class 1class

JVM instrumenting classfiles

JVMPI

Socket

Cache

S

E

E

S

C

O

A

The RecPlay system for Java

Machine 2

Machine 1

JVM

JVM instrumenting

classfiles for

record or replay

Component System

...

Trace

I/O

S

E

E

S

C

O

A

Current Situation

Record/replay for synchronization operations works

To be added: Race detection

Tracing of input