1

4/3/2001 1

Design Examples of DistributedApplications Based on the Use of

Synchronized Clocks

Randy Coverstone and John C. EidsonAgilent Laboratories

Ptolemy Conference, Berkeley,March 22-23, 2001

4/3/2001 4/3/2001 2

Agenda

• Measurement and control based on explicittime

• Techniques for synchronizing clocks• Example systems based on synchronized

clocks• Simulation of distributed, synchronized

clock based systems

2

4/3/2001 4/3/2001 3

Centralized SystemsPC

Network

• Master/slave protocol

•Usually message based design

• Slaves are sequentially polledin a loop.

Scalability of such a system is poor:as the number of channels isincreased,

• Processing load on the controllerincreases,• Complexity of the control programincreases,• Maximum channel sampling ratedecreases,• Ability to sample channels“simultaneously” degrades, and• The worst-case network loadincreases.

[Reading]

Externaltriggerinput

Externaltriggerinput

Externaltriggerinput

BUT: We are familiar withthis style!

4/3/2001 4/3/2001 4

Distributed Systems With ClocksPC

Network

• Master/slave protocol permitted, but not required.

• Increased flexibility in sampling schedule.

• Network messages may now contain accurate time-stamps generated at the point of sensing/actuation

Improved scalability:

• Complexity of the control programremains the same,• Maximum rate at which each channelcan be sampled depends on each node,not the controller,• The minimum time to sample all thechannels is reduced to the amount oftime to sample one channel.• To add more channels you simplyadd more nodes,• Simultaneous acquisition is possible.

[Reading, timestamp,variable, physical units,uncertainty,ID, location, calibrationinfo]

3

4/3/2001 4/3/2001 5

Use of Explicit Time inMeasurement and Control

• Basic uses of time (relative or absolute)– Time stamping system events for debugging– Time stamping data to allow correlation with

other data or events• Coordination of measurement: (sampling and

triggering)• Coordination of action (time based behaviors)

– Timed execution scripts– Temporal mutual exclusion: (time slots)

4/3/2001 4/3/2001 6

Sampling/triggering OptionsCentral clock With distributed clocks

Sequential

SimultaneousRequires simultaneous sample and hold or ext trigger wires.

Differentrates for eachnode

Requires oversampling and decimation.

4

4/3/2001 4/3/2001 7

Timed Execution Scriptnetwork message: start_experiment(time stamp = a)

Script_1:When(t= =a){Initialize_node( );Sample(10khz,a);}When(t= =a+0.0005){Setpoint_1 = 27;}

Script_2:Sample(100khz,now);Start_circular_buffer( );When(t= =a+0.001){Freeze_circular_buffer( );Send_data(a-

0.001,a+0.001);}

4/3/2001 4/3/2001 8

A start into the literature• Atul Adya, Robert Gruber, Barbara Liskov, Umesh

Maheshwari: Efficient Optimistic Concurrency ControlUsing Loosely Synchronized Clocks. SIGMODConference 1995: 23-34

• Barbara Liskov: Practical Uses of Synchronized Clocksin Distributed Systems. Distributed Computing 6(4):211-219 (1993)

• Barbara Liskov, Liuba Shrira, John Wroclawski: EfficientAt-Most-Once Messages Based on Synchronized Clocks.TOCS 9(2):125-142 (1991)

• Real-Time Systems: Design Principles for DistributedEmbedded Applications, Hermann Kopetz, KluwerAcademic Publishers, 1997.

5

4/3/2001 4/3/2001 9

Use of explicit timerepresentation requires:

• A real time clock and associated applicationdevices in each distributed device

• The local time of all clocks to agree towithin some system defined uncertainty

• Clock time is monotonic and ‘continuous• The process synchronizing the clocks to be

a provided system service

4/3/2001 4/3/2001 10

Node Block DiagramNetwork

1451.2driver

STIM dataacquisition

TEDS

STIM

NACKNTRIG

• CHANNEL_ZERO correction timing• Calibration factors• Calibration date

[Reading, timestamp, variable, physical units, uncertainty, ID, location, calibration info]

• NTRIG requests a measurement• NACK indicates request is done

Reading is corrected and stored in buffer. 1451.2

driver

Correctionengine

Data buffer

Real-timeclock

Timestampadjustment

Timestamp

Sample timetriggerregister

Clock syncengine

Node firmwareNCAP

ApplicationCode

6

4/3/2001 4/3/2001 11

Degrees of Time Synchronization• Day• Hour• Minute• Second• 10 msec• millisecond• microsecond• < microsecond

Requires hardware assiste.g. Agilent PTP, TTP/C,GPS,IRIG

No hardware assist requirede.g. Network Time Protocol (NTP)

4/3/2001 4/3/2001 12

Removing jitter

Application layer

Network protocolstack

msecs of delay andjitter

Time packetrecognizer

Application layer

Network protocolstack

Time packetrecognizer

Physical layer Physical layer

< 100 nsecs of delayand jitter

7

4/3/2001 4/3/2001 13

PTP Time Synchronization

Step 1:• Time sync packet from

master to slaves• All TPR’s note the time

this packet appears• Step 2:• Follow-up packet from

master to slaves• All slaves compute

offset and correct theslave clock

Micro-A 10:00AM

Micro-B 10:05AM

TPR-A 10:00AM

TPR-B 10:05AM

Time sync packet

Master Clock

Slave Clock

Micro-A10:00AM

Micro-B10:05AM10:00AM

TPR-A TPR-B

Follow-up packetSend time = 10:00AM

Clockcorrection

circuits

Clock

0:05

4/3/2001 4/3/2001 14

Selecting best master clockSelection criteria• Tracability to UTC• Precision/jitter• Path length• Preferred status

GPS

Example system #1

Example system #2

Repeater

Repeater

Router

Spider

Repeater Repeater

8

4/3/2001 4/3/2001 15

Seconds tick deviations between the clocks of two system instruments

0

20

40

60

80

100

120

140

160

-215 -185 -155 -125 -95 -65 -35 -5 25 55 85 115 145 175 205 235 265 295 325 355

Deviation- nanoseconds

Rel

ativ

e fr

equ

ency

Master

Slave

HP J2610A

HP J2610B

HP J2610B

HP J2610B

σ =68nsmean = 37ns

Seconds tick deviations between theclocks of two system instruments

4/3/2001 4/3/2001 16

Example: Fermentation Control

9

4/3/2001 4/3/2001 17

Fermentation Control

4/3/2001 4/3/2001 18

Gigabit Switch

BuildEvent PC

BuildEvent PC

BuildEvent PC

BuildEvent PC

16 Front End unit

optionalPC

switch10 / 100 MbpsE

ther

net L

ink

Optional PC for local

Processing

16 Front End unit

optionalPC

switch10 / 100 MbpsE

ther

net L

ink

Optional PC for local

Processing

16 Front End unit

OptionalPC

switch10 / 100 MbpsE

ther

net L

ink

Optional PC for local

Processing

24 planes tracker

Load perFront-EndUnit

Switch10/100(Mbits/s)

GigabitSwitchload

102 KBps

(10 Mbps)

1.6 Mbps

(100 Mbps)

38 Mbps

LoadperPC (4)

10 Mbps

1st

leve

l

2

nd l

evel

3

trd

leve

l

Ethernet Network-based DAQ and smart sensorsfor the neutrino long base line OPERA experiment.S. Gardien, C. Girerd, S. Katsanevas, J. Marteau.-CERN

72000 channels120Mbits/sec aggregate

10

4/3/2001 4/3/2001 19

Fig. 2 General block diagram of an Ethernet Capable Frontend Module.

Analog Front End ADC

ControlLogic Time

Stamping

FIFO

Ethernet CapableDevice

Ethernet

PPS

0

63

Fro

m P

MT

or

HP

DOr trigger

Readout logic signals

OptionalHard reset

CalibrationMonitoring ...

CERN-front end

4/3/2001 4/3/2001 20

Example: Data Acquisition

11

4/3/2001 4/3/2001 21

Sample Triggers From Several Instruments

100 triggers/sec10 msec/division

12 Hz sine wave

Clock master

Clock slave

Clock slave

4/3/2001 4/3/2001 22

Sample Triggers From Several Instruments

0.48 triggers/sec200 nsec/division

Clock master

Clock slave

Clock slave

1 usec

12

4/3/2001 4/3/2001 23

Example: Motor Controlcourtesy Indramat

4/3/2001 4/3/2001 24CCD-Box with CLC-Motion control card

HMIConsol

PLC

LWL-Ring with SERCOSInterface

Data-Bus

I / O

InfeedPrintunit 2 Outfeed

Chillrolls

ABRFB

3FU

3FL

Decentrale drive concept

Printunit 3

Printunit 4

Printunit 1

Folder

Commercial printing press(slide courtesy of Indramat)

Decentralized DriveConcept

13

4/3/2001 4/3/2001 25

Flexo Printing Press(Unit Construction)(slide courtesy of Indramat)

60mph =

1 inch/millisecond =

1 mil/microsecond

4/3/2001 4/3/2001 26

Simulation of Time Based SystemsDevice Under Test ( real world time)

Measurement andControl Device

CLOCK-1

Measurement andControl Device

CLOCK-2

Measurement andControl Device

CLOCK-3

Sampling basedon clocks 1, 2, 3

Messaging based on network timing