debugging of real-time embedded systems: experiences from seescoa
DESCRIPTION
Debugging of Real-Time Embedded Systems: Experiences from SEESCOA. Michiel Ronsse RUG-ELIS. Problem description. embedded systems have become increasingly sophisticated debugging becomes a major problem types of bugs computational errors synchronisation errors performance errors - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/1.jpg)
4 December 2001
S
E
E
S
C
O
A STWW - Programma
Debugging of Real-Time Embedded Systems: Experiences from
SEESCOAMichiel Ronsse
RUG-ELIS
![Page 2: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/2.jpg)
S
E
E
S
C
O
A
Problem description
embedded systems have become increasingly sophisticated
debugging becomes a major problem
types of bugs computational errors synchronisation errors performance errors
testing and debugging is responsible for a huge part of the development time
definitive need for appropriate debugging tools
![Page 3: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/3.jpg)
S
E
E
S
C
O
A
Debugging embedded systems
two test/debug phases during the development during the actual use
an embedded system typically runs continuously interactive debugging? can have hard and soft real-time constraints Heisenbugs parallel and/or distributed systems: absence of a global clock uses lots of input repeatability? cyclic debugging? hardware related problems: code in ROM, limited amount of
RAM
Solution: hardware add-ons for on-chip debugging techniques
![Page 4: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/4.jpg)
S
E
E
S
C
O
A
On-chip debugging techniques (I)
Logic Analyser
ROM monitor
ROM emulator
In-Circuit Emulator
Background Debug Mode
JTAG
![Page 5: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/5.jpg)
S
E
E
S
C
O
A
almost all contemporary embedded processors have add-ons for debugging
as embedded processors get faster, these add-ons get closer to (or in) the processor
these add-ons take up valuable chip area (up to 10%)
these add-ons are also available in the consumer produsts, making on-site testing possible
hardware manufacturers believe in design for debugability
On-chip debugging techniques (II)
![Page 6: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/6.jpg)
S
E
E
S
C
O
A
Embedded software
Nowadays, software engineering methods designed for `business’ applications are also used for embedded systems: Use of `higher’ languages (C, C++, Java) Use of reusable components Multithreaded applications ...
Debugging/maintaining these complex applications becomes a difficult task
Design for DebugabilityDesign for Debugability
![Page 7: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/7.jpg)
S
E
E
S
C
O
A
Debugging software
Most important notion: Repeatability cyclic debugging Observability what is going on?
These are two problems for embedded systems: Non-determinism present in most embedded systems Low observability (real-time constraints, ...)
![Page 8: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/8.jpg)
S
E
E
S
C
O
A
Causes of non-determinism
Sequential programs: input (keyboard, disk, network, certain system calls (e.g. gettimeofday(), …)
Parallel programs: race conditions: two threads accessing the same shared variable (memory location) in an unsynchronised way and at least one thread modifies the variable
![Page 9: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/9.jpg)
S
E
E
S
C
O
A
Execution Replay
Goal: make repeated equivalent re-exections possible
Method: two phases Record phase: record all non-deterministic events during an
execution in a trace file Replay phase: use trace file to produce the same execution
Question: what & where to trace?
![Page 10: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/10.jpg)
S
E
E
S
C
O
A
Example code
#include <pthread.h>
unsigned global=5;
thread2(){ global=global+6; }thread3(){ global=global+7; }
main(){pthread_t t2,t3;pthread_create(&t2, NULL, thread1, NULL);pthread_create(&t3, NULL, thread2, NULL);pthread_join(t2, NULL);pthread_join(t3, NULL);printf(“global=%d\n”, global);
}
![Page 11: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/11.jpg)
S
E
E
S
C
O
A
Possible executions
L(5)
global=12 global=18global=11
L(5)
L(5)
L(5)
L(5)
L(11)S(11)
S(12) S(11)S(12)
S(11)
S(18)
A
A
A
A
A
A
![Page 12: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/12.jpg)
S
E
E
S
C
O
A
Example code II
#include <pthread.h>
unsigned global=5;
thread2(){lock(); global=global+6; unlock();}thread3(){lock(); global=global+7; unlock();}
main(){pthread_t t2,t3;pthread_create(&t2, NULL, thread1, NULL);pthread_create(&t3, NULL, thread2, NULL);pthread_join(t2, NULL);pthread_join(t3, NULL);printf(“global=%d\n”, global);
}
![Page 13: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/13.jpg)
S
E
E
S
C
O
A
Possible executions II
global=18
L(5)
L(11)
S(11)
S(18)
A
A
global=18
L(5)
L(11)
S(11)
S(18)
A
A
![Page 14: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/14.jpg)
S
E
E
S
C
O
A
Race conditions
Three types of conflicts: load/store store/load store/store
Two types: synchronisation races:
doesn’t allow the use of cyclic debugging techniquesis not a bug, is desired non-determinism
data races:doesn’t allow the use of cyclic debugging techniquesis a bug, is undesired non-determinism
![Page 15: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/15.jpg)
S
E
E
S
C
O
A
Tracing all memory operations
•Introduces an intolerable overhead, both in time and space
•Due to out-of-order stores, the order in which threads see events can differ
•Will replay both synchronization and data races.
![Page 16: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/16.jpg)
S
E
E
S
C
O
A
Tracing synchronization operations
•The only events that are assured to be seen in the same order are synchronisation operations
•Synchronization operations are a subset of all memory operations => lower overhead
•Will replay only synchronization races.
•Will fail if data races occur.
•Solution
•Record: trace synchronization operations
•Replay: check for data races (once)
![Page 17: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/17.jpg)
S
E
E
S
C
O
A
Record phase
T2
Main
T34
55
6
8
6
7
9
8
12
13
14
10
11
1
2
33
4
Main: 1,2,3,4,5,13,14,...T2: 3,6,7,8,9,...T3: 4,5,6,8,10,11,12,...
![Page 18: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/18.jpg)
S
E
E
S
C
O
A
Replay phase T2Main T3
4
5
8
6
7
9
12
13
14
10
11
1
2
3
Main: 1,2,3,4,5,13,14,...T2: 3,6,7,8,9,...T3: 4,5,6,8,10,11,12,...
![Page 19: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/19.jpg)
S
E
E
S
C
O
A
Implementation
RecPlay for Solaris (SPARC) and Linux (x86)
Uses JiTI for dynamic instrumentation
Record overhead: 1.6%
![Page 20: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/20.jpg)
S
E
E
S
C
O
A
Instrumenting Java classes
JVM
Generic Debugger Backend
class 1class 1
class 1class
JVM instrumenting classfiles
JVMPI
Socket
Cache
![Page 21: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/21.jpg)
S
E
E
S
C
O
A
The RecPlay system for Java
Machine 2
Machine 1
JVM
JVM instrumenting
classfiles for
record or replay
Component System
...
Trace
I/O
![Page 22: Debugging of Real-Time Embedded Systems: Experiences from SEESCOA](https://reader036.vdocuments.net/reader036/viewer/2022062321/56813aef550346895da365e4/html5/thumbnails/22.jpg)
S
E
E
S
C
O
A
Current Situation
Record/replay for synchronization operations works
To be added: Race detection
Tracing of input