1 sigrace: signature-based data race detection abdullah muzahid, dario suarez*, shanxiang qi &...

1

SigRace: Signature-Based Data Race Detection

Abdullah Muzahid, Dario Suarez*, Shanxiang Qi & Josep Torrellas

Computer Science DepartmentUniversity of Illinois at Urbana-Champaign

http://iacoma.cs.uiuc.edu

*Universidad de Zaragoza, Spain

2

Debugging Multithreaded Programs

“Debugging a multithreaded program has a lot in

common with medieval torture methods”

-- Random quote found via Google search

3

Data Race

• Two threads access the same variable without intervening synchronization and at least one is a write

T1 T2

lock L

unlock Lx++

x++

• Hard to detect and reproduce

• Common bug

4

Dynamic Data Race Detection

• Mainly two approaches

5


• Mainly two approaches– Lockset: Finds violation of locking discipline

6


• Mainly two approaches– Lockset: Finds violation of locking discipline

– Happened-Before: Finds concurrent conflicting accesses

7

Happened-Before Approach

8


[0, 0] [0, 0]

Thread 0 Thread 1

9


Lock L

[0, 0]

[1, 0]

[0, 0]

Thread 0 Thread 1

10


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

11


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

12


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

13


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[0, 1]

14


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

+

[0, 1]

15


• Epoch : sync to sync

Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

16



Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

a

b

c

d

e

17


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

a

b

c

d

e


18


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

a

b

c

d

e


19


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

a

b

c

d

e


20


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

a

b

c

d

e


• a, b happened before e

21


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

a

b

c

d

e



22


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

a

b

c

d

e



• c, d unordered

23


Lock L

Unlock L

[0, 0]

[1, 0]

[2, 0]

[0, 0]

Thread 0 Thread 1

Lock L

[2, 1]

a

b

c

d

e



• c, d unordered

= x

x =

Data Race

24

Software Implementation

• Need to instrument every memory access– 10x – 50x slowdown

– Not suitable for production runs

25

Hardware Implementation

26


P1 P2

… …C1 C2

27


P1 P2

… …TS TS

C1 C2

28


P1 P2

… …xts1

TS TS

C1 C2

29


P1 P2

… …x xts1 ts2

ts2 …TS TS

C1 C2

WR

30


P1 P2

… …x xts1 ts2

ts2 …

ts2

check

TS TS

C1 C2

WR

31

Limitations of HW Approaches

32


P1 P2

… ……

C1 C2

• Modify cache and coherence protocol

33


P1 P2

… ……

C1 C2


• Perform checking at least on every coherence transaction

ts1ts2

ts2

check

34


P1 P2

… …C1 C2


• Perform checking at least on every coherence transaction

• Lose detection ability when cache line is displaced or invalidated

35

Our Contributions

• SigRace: Novel HW mechanism for race detection based on signatures– Simple HW

• Cache and coherence protocol are unchanged– Higher coverage than existing HW schemes

• Detect races even if the line is displaced/invalidated

• SigRace finds 150% more injected races than a state-of-the-art HW proposal

– Usable on-the-fly in production runs

36

Outline

• Motivation• Main Idea• Implementation• Results• Conclusions

37

Main Idea

Address Signature + Happened-before

38

Hardware Address Signatures

[Ceze et al, ISCA06]

…P

erm

ute

…

ROB

addressld/st

39


40


• Logical AND for intersection

• Has false positives but not false negatives

41

Using Signatures for Race Detection

sync

sync

Epoch

42


sync

sync

Epoch SigTS

43


Block

• Block is a fixed number of dynamic instructions (not a cache block or basic block or atomic block)

sync

sync

Epoch Sig1TS1

44


Block

sync

sync

Epoch

Race Detection Module

Sig1TS1

45


Block

sync

sync

Epoch ØTS1


46


Block

sync

sync

Epoch

Sig2TS1


47


Block

sync

sync

Epoch

Sig2TS1


48


Block

sync

sync

Epoch

ØTS1


49


Block

sync

sync

Epoch

Sig3TS2


50


Block

sync

sync

Epoch

Sig3TS2


51


Block

sync

sync

Epoch

Sig3TS2


52


Block

sync

sync

Epoch

Sig3TS2 Sig Ո Sig


53

On Chip Race Detection Module (RDM)

54


P1 P2

Q1 Q2

Chip

RDM

55


P1 P2

T1 R1 W1

Q1 Q2

Chip

RDM

56


P1 P2

T1 R1 W1

Q1 Q2

Chip

RDM

57


P1 P2

T1 R1 W1

Q1 Q2

Chip

RDM

58


P1 P2

T2 R2 W2

Q1 Q2

T1 R1 W1

Chip

RDM

59


P1 P2

T2 R2 W2

Q1 Q2

T1 R1 W1

Chip

RDM

60


P1 P2

Q1 Q2

T2 R2 W2T1 R1 W1

Chip

RDM

61


P1 P2

Q1 Q2

T2 R2 W2T1 R1 W1

TJ RJ WJ

If T2 & TJ unordered

R2 WJՈ

W2 WJՈ

W2 RJՈ

Else stop

Chip

RDM

62


P1 P2

Q1 Q2

T2 R2 W2T1 R1 W1 TJ` RJ` WJ`

If T2 & TJ` unordered

R2 WJ`Ո

W2 WJ`Ո

W2 RJ`Ո

Else stop

Chip

RDM

63


P1 P2

Q1 Q2

T2 R2 W2T1 R1 W1

TJ” RJ” WJ”

If T2 & TJ” unordered

R2 WJ”Ո

W2 WJ”Ո

W2 RJ”Ո

Else stop

Chip

RDM

64


P1 P2

Q1 Q2

T2 R2 W2T1 R1 W1

TJ”` RJ”` WJ”`

If T2 & TJ”` unordered

R2 WJ”`Ո

W2 WJ”`Ո

W2 RJ”`Ո

Else stop

Chip

RDM

65


P1 P2

Q1 Q2

T2 R2 W2T1 R1 W1



R2 WJ”`Ո

W2 WJ”`Ո

W2 RJ”`Ո

Else stop

Chip

RDM

Done in Background

66


P1 P2

Q1 Q2

T2 R2 W2T1 R1 W1



R2 WJ”`Ո

W2 WJ”`Ո

W2 RJ”`Ո

Else stop

Chip

RDM

False Positives

67

Re-execution

• Needed for

– Discard if a false positive

– Identify the accesses involved

68

Support for Re-execution

• Save synchronization history in TS Log

– Timestamp at sync points

• Log inputs (interrupts, sys calls, etc)

• Take periodic checkpoints: ReVive [Prvulovic et al, ISCA02]

69

Modes of Operation

• Normal Execution

• Re-Execution: Bring the program to just before the race

• Race Analysis: Pinpoint the racy accesses or discarding the false positive

70

SigRace Re-execution Mode

• Can be done in another machine

71



• Periodic checkpoint of memory state

sync

sync

sync

sync

sync

checkpointT0 T1 T2

72




sync

sync

sync

sync

sync

checkpoint

s1

s2Data Race

T0 T1 T2

73




sync

sync

sync

sync

sync

checkpoint

s1

s2Data Race

Ո

Conflict Sig Conflict Sig

T0 T1 T2

74




sync

sync

sync

sync

sync

checkpoint

s1

s2Data Race

Ո


checkpointT0 T1 T2 T0 T1 T2

75




sync

sync

sync

sync

sync

checkpoint

s1

s2Data Race

Ո


sync

sync

sync

sync

sync

checkpointT0 T1 T2 T0 T1 T2

Use the TS Log

76

SigRace Analysis Mode

sync

sync

sync

sync

sync

checkpointT0 T1 T2

77


sync

sync

sync

sync

sync

checkpointT0 T1 T2

78


sync

sync

sync

sync

sync

checkpointT0 T1 T2

Conflict SigConflict Sig

ldՈ

79


sync

sync

sync

sync

sync

checkpointT0 T1 T2


ldlog

Ո

80


sync

sync

sync

sync

sync

checkpointT0 T1 T2


ldlog

sync sync

Ո

81


sync

sync

sync

sync

sync

checkpointT0 T1 T2

log

sync sync

log

82


sync

sync

sync

sync

sync

checkpointT0 T1 T2

log

sync sync

log

• Pinpoints racy addresses or,

• Identifies and discards false positives

83

Outline


84

New Instructions

• collect_on– Enable R and W address collection in current thread

85

New Instructions

• collect_on– Enable R and W address collection in current thread

• collect_off– Disable R and W address collection in current thread

86

New Instructions

• sync_reached

87

New Instructions

• sync_reached– Dump TS, R and W

P

RDM

Network

TS R W

88

New Instructions


– Clear signatures

P

RDM

Network

TS Ø Ø

89

New Instructions


– Clear signatures

– Update TS

P

RDM

Network

TS` Ø Ø

90

Modifications in Sync Libraries

91


• Synchronization objectvariable timestamp

92


• Synchronization object

• Unlock macro

variable timestamp

UNLOCK (‘{

}’)

unlock($1.lock)

93



• Unlock macro

variable timestamp

UNLOCK (‘{

}’)

unlock($1.lock)

sync_reached

RDM

Network

TS R WP

94



• Unlock macro

variable timestamp

UNLOCK (‘{

}’)

unlock($1.lock)

sync_reached

RDM

Network

TS Ø ØP

95



• Unlock macro

variable timestamp

UNLOCK (‘{

}’)

unlock($1.lock)

sync_reached

RDM

Network

TS Ø ØP

96



• Unlock macro

variable timestamp

UNLOCK (‘{

}’)

unlock($1.lock)

sync_reached$1.timestamp = TS lock TS

97



• Unlock macro

variable timestamp

UNLOCK (‘{

}’)

unlock($1.lock)

sync_reached$1.timestamp = TS

98



• Unlock macro

variable timestamp

UNLOCK (‘{

}’)

unlock($1.lock)

sync_reached$1.timestamp = TS

AppendtoTSLog(TS) TS LogTS

99



• Lock macro

variable timestamp

LOCK (‘{

}’)

lock($1.lock)

…

…

100



• Lock macro

variable timestamp

LOCK (‘{

}’)

lock($1.lock)

…

TS = GenerateTS (TS,$1.timestamp)

lock timestamp

…

101



• Lock macro

variable timestamp

LOCK (‘{

}’)

lock($1.lock)

…

TS = GenerateTS (TS,$1.timestamp)

lock timestamp

…

Transparent to Application Code

102

Other Topics in Paper

• Easy to virtualize

• Queue Overflow

• Detailed HW structures

103

Outline


104

Experimental Setup

• PIN – Binary Instrumention Tool• Default parameters

• Benchmarks: SPLASH2, PARSEC

– # of proc: 8

– Signature size: 2 Kbits

– Block size: 2,000 ins

– Queue size: 16 entries

– Checkpoint interval: 1 Million ins

105

Race Detection Ability

• Three configurations

– SigRace Default

– SigRace Ideal : Stores every signature between 2 checkpoints

– ReEnact [Prvulovic et al, ISCA03]: Cache based approach with timestamp per word

106


App Ideal

SigRace

Default SigRce

ReEnact

Cholesky 16 16 16Barnes 11 11 6Volrend 27 27 18Ocean 1 1 1Radiosity 15 15 12Raytrace 4 4 3Water-sp 8 4 2Streamcluster

13 12 13

Total 95 90 70

107


• More coverage than ReEnact

App Ideal

SigRace

Default SigRce

ReEnact


13 12 13

Total 95 90 70

108


• More coverage than ReEnact

• Coverage comparable to ideal configuration

App Ideal

SigRace

Default SigRce

ReEnact


13 12 13

Total 95 90 70

109

Injected Races

• Removed one dynamic sync per run• Each application runs 25 times with diff sync

elimination

110

Injected Races

111

Injected Races

• More overall coverage than ReEnact

112

Injected Races

• More overall coverage than ReEnact– 150% more coverage

113

Injected Races

• More overall coverage than ReEnact– 150% more coverage

114

Conclusions

• Proposed SigRace: – Simple HW

• Cache and coherence protocol are unchanged– Higher coverage than existing HW schemes

• Detect races even if the line is displaced/invalidated

– Usable on-the-fly in production runs

• SigRace finds 150% more injected races than word-based ReEnact

115

SigRace: Signature-Based Data Race Detection

Abdullah Muzahid, Dario Suarez*, Shanxiang Qi & Josep Torrellas

Computer Science DepartmentUniversity of Illinois at Urbana-Champaign

http://iacoma.cs.uiuc.edu

*Universidad de Zaragoza, Spain

116

Back Up Slides

117

Execution Overhead

• No overhead in generating signatures (HW)• Additional instructions are negligible• Main overheads

– Checkpointing (ReVive – 6.3%)

– Network traffic (63 bytes per 1000 ins - compressed)

– Re-execution (depends on false positives & race position)• Can be done offline

118

Network Traffic Overhead

63

[ 1 cache line

119

Re-execution Overhead

• Instructions re-executed until the first true data race is analyzed are shown as overhead

• In this process, it may also encounter many false positive races

• Instructions re-executed to analyze only the true race are shown as true overhead

• Instructions re-executed to filter out the false positives are shown as false overhead

120

Re-execution Overhead

22%

Modest overhead

121

False Positives

• Parallel bloom filters with H3 hash function

1.57%

Low False Positive

122

Virtualization

123

Virtualization

• RDM uses as many queues as the number of threads

• Timestamp is accessed by thread id

• Thread id remains same even after migration

• Timestamps, flags, conflict signature are saved and restored at context switch

• RDM intersects incoming signatures against all other threads’(even inactive ones) signatures

• Threads can be re-executed without any scheduling constraints

124

Scalability

• For small # of proc., scalability is not a problem

• The operation of RDM can be pipelined– Simple repetitive operation

• Network traffic (compressed message) around 63Bytes/thousand ins

• Checkpoint is an issue.

1 sigrace: signature-based data race detection abdullah muzahid, dario suarez*, shanxiang qi &...

Documents

approach slide

approach lock

spain slide

b c d e slide

data race slide

d unordered slide

hardware implementation

approach epoch