9/9/2015 13:17 1 can real-time systems be built with off the shelf components? krithi ramamritham...

04/19/23 13:16 1

Can Real-Time Systems be built with

Off the Shelf Components?

Krithi Ramamritham

Real-Time Systems LaboratoryUniversity of Massachusetts, Amherst& Indian Institute of Technology, Bombay

04/19/23 13:16 2

Talk Outline Using off-the-shelf components for

Real-Time applications? Future application characteristics Problems and challenges

Characteristics and limitations of OTS OSs

OTS component based solutions for distributed Real-Time applications User-level scheduling of communicating RT tasks

Conclusions, recommendations, future

04/19/23 13:16 3

EWS

Network

Field Devices

Field Network

PLC

Field Devices

Field Network

PLC

Operator Commands

Video/audio

Device monitoring

Background

Server

M.Services

LynxOS

M. Services

VxWorks

M. Services M. Services M. ServicesM. Services

OperatorStations

M. Services NT

Industrial Control Environment

04/19/23 13:16 4

Future Application Characteristics

Interactions and communication

Human/device to human/device

Co-existence of multiple types of media Small control and signal data Periodic updates Bursty file/page/image access and transfer Continuous media

Computer and communication technology advances

04/19/23 13:16 5

What Are the New Challenges?

Communication centered Operating System is no-longer standalone

Support for real-time and non-real-time co-existence

Integrated, end-to-end solutions required Application-level control of system resources

Need to support new applications and technologies

04/19/23 13:16 6

System Research to Meet the Challenges

Network Architecture

Network

Interface

Design

Resource

ManagerScheduling

Algorithms

Scheduler

Middleware

Services

Traffic Management

Rate Control

04/19/23 13:16 7

Real-Time Spectrum

Hard

Soft

04/19/23 13:16 8

Real-Time OS Spectrum

Hard

Soft

Real-Time Operating System

General-PurposeOperatingSystem

VxWorks, Lynx, QNX, ...Intime, HyperKernel, RTLinux

(Windows NT, Linux)

(Windows NT, Linux)

04/19/23 13:16 9

Using General Purpose Operating

Systems

GPOS offer some capabilities useful for real-time system builders

RT applications can obtain leverage from existing development tools and applications

Some GPOSs accepted as de-facto standards for industrial applications

04/19/23 13:16 10

Windows NT -- for RT applications?

Scheduling and priorities Preemptive, priority-based scheduling

non-degradable priorities priority adjustment

No priority inheritance No priority tracking Limited number of priorities No explicit support for guaranteeing timing constraints

04/19/23 13:16 11

Windows NT -- for RT applications? (contd.)

Quick recognition of external events Priority inversion due to Deferred Procedure Calls (DPC)

I/O management Timers granularity and accuracy

High resolution counter with resolution of 0.8 sec. Periodic and one shot timers with resolution of 1 msec.

Rich set of synchronization objects and communication mechanisms. Object queues are FIFO

04/19/23 13:16 12






04/19/23 13:16 13

Goals - I

Evaluate the real-time capabilities of NT.

Identify areas where NT is (not) suitable for real-time applications.

Determine to what extent the unpredictable parts of NT can be “masked”.

Offer recommendations to designers using NT.

04/19/23 13:16 14

Priority Model

Real-timeclass

2625242322

16 Idle

Above NormalNormalBelow NormalLowest

Highest31 Time-critical

Dynamicclasses

15 Time-critical

14131211

15

High class

1 Idle

987

11

Normal class10

5432

6

Idle class

ThreadLevel

04/19/23 13:16 15

Thread Priority = Process class + level

Real-timeclass

2625242322

16 Idle

Above NormalNormalBelow NormalLowest

Highest31 Time-critical

Dynamicclasses

15 Time-critical

14131211

15

High class

1 Idle

987

11

Normal class10

5432

6

Idle class

ThreadLevel

04/19/23 13:16 16

Scheduling

Threads scheduled by executive.

Priority based preemptive scheduling.

Interrupts

Deferred Procedure Calls (DPC)

System anduser-level threads

04/19/23 13:16 17

Servicing an interrupt

Power Failure...

APCNormal Exec.

Dispatch/DPC

Device X...

InterruptDispatch Table

1

Device interrupts

InterruptServiceRoutine

DPCRoutineDevice Driver

DPC

DPC DPC

DPC FIFO Queue

DPC DPC

DPC FIFO Queue

user-level threads

Interrupts

DPC

04/19/23 13:16 18


Power Failure...

APCNormal Exec.

Dispatch/DPC

Device X...


1

Device interrupts

2

Transfercontrol to ISR



DPC

DPC DPC

DPC FIFO Queue

DPC DPC

DPC FIFO Queue

user-level threads

Interrupts

DPC

04/19/23 13:16 19


Power Failure...

APCNormal Exec.

Dispatch/DPC

Device X...


1

Device interrupts

2




DPC

DPC DPC

DPC FIFO Queue

DPC DPC

DPC FIFO Queue

user-level threads

Interrupts

DPC

04/19/23 13:16 20


Power Failure...

APCNormal Exec.

Dispatch/DPC

Device X...


1

Device interrupts

2




DPC

DPC DPC

DPC FIFO Queue

DPC DPC

DPC FIFO Queue

user-level threads

Interrupts

DPC

3

Stop int.. andqueue DPC

04/19/23 13:16 21


Power Failure...

APCNormal Exec.

Dispatch/DPC

Device X...


1

Device interrupts

2




DPC

DPC DPC

DPC FIFO Queue

DPC DPC

DPC FIFO Queue

user-level threads

Interrupts

DPC

3


04/19/23 13:16 22


Power Failure...

APCNormal Exec.

Dispatch/DPC

Device X...


1

Device interrupts

2




3


DPC

DPC DPC

DPC FIFO Queue

DPC DPC

DPC FIFO Queue

4 Task level drops and DPC can execute

user-level threads

Interrupts

DPC

04/19/23 13:16 23


Power Failure...

APCNormal Exec.

Dispatch/DPC

Device X...


1

Device interrupts

2




3


DPC

DPC DPC

DPC FIFO Queue

DPC DPC

DPC FIFO Queue


5

Transfercontrol to driver’s

DPC

user-level threads

Interrupts

DPC

04/19/23 13:16 24


Power Failure...

APCNormal Exec.

Dispatch/DPC

Device X...


1

Device interrupts

2




3


DPC

DPC DPC

DPC FIFO Queue

DPC DPC

DPC FIFO Queue


5

Transfercontrol to driver’s

DPC

6

Execution of DPCroutine

user-level threads

Interrupts

DPC

04/19/23 13:16 25

I/O Handling

I/O request is sent to device driver. Device completes operation and interrupts. Complete I/O request.

Buffered I/O Direct I/O

APC

Device

System

User’s space

Device

System

User’s space

(Keyboard, mouse) (disk, network)

04/19/23 13:16 26

Prototype Software Architecture

Operator StationAcquisition &Control Equipment

Heartbeat Timer Heartbeat Timer

Command

Ack.

RealVideo

Receiver

Producer

ConsumerBuffer

Highest Priority

Normal Priority

Real-Time Class

04/19/23 13:16 27

Performance Metrics

Round Trip Time (RTT) as seen by the operator input.

Rate of execution of sensor data processing entities.

Quality of the video output.

04/19/23 13:16 28

Workload

Number of sensor data streams (2-20).

Period of new sensor values (10-1000 ms).

Period of control messages (10-100 ms).

Amount of work done in processing data.

One Video and audio.

04/19/23 13:16 29

Operator Command Performance

Real-Time Priority Class

0

10

20

30

40

50

60

0 200 400 600 800 1000

Sample no

RTT

(ms)

High Priority Class

0

10

20

30

40

50

60

0 200 400 600 800 1000

Sample no.

RTT

(ms)

Normal Priority Class

0

10

20

30

40

50

60

0 200 400 600 800 1000

Sample no.

RTT

(ms)

• Two 1KB Sensor data streams • 1 second update rate• 30 ms period for control messages• No work

04/19/23 13:16 30

Operator command Performance

• NT Scheduler - same RT priority • 16 Sensor data streams (1KB) • Update rate (100, 200, 500, 1000 ms)• 90 ms period for control messages• work

Real-Time Priority Class

01020304050607080

0 200 400 600 800 1000

Sample #

RT

T (

mse

c)

04/19/23 13:16 31

Operator command Performance(user-level scheduling)

Real-time Priority Class

0

20

40

60

80

0 500 1000 1500 2000

Sample #

RTT

(ms)

• Rate Monotonic with limited levels• 16 Sensor data streams (1Kb)• Update rate (100, 200, 500, 1000 ms)• 50 ms period for control messages• work

Real-time Priority Class

0

20

40

60

80

0 200 400 600 800 1000

Sample #

RT

T (

ms)

• Time-cognizant dispatcher + plan• 8 Sensor data streams (1Kb)• Update rate (100, 200, 500, 1000 ms)• 100 ms period for control messages• work

04/19/23 13:16 32

Design principles and

recommendations:

Do not depend on NT scheduler to accomplish timing behavior in interactive applications.

Utilize user-level scheduling to achieve higher predictability.

If possible, characterize duration of I/O activity and its frequency.

Lock pages in memory for real-time threads. Manage and control utilization of systems

resources.

04/19/23 13:16 33

Use a General Purpose OS for RT?

It is possible to improve the predictability of real-time tasks. It is not possible to “mask” all sources of unpredictability

within NT “as is”. Designer needs to be aware of the effects DPC queue on any

user thread. I/O handling.

Prototype application demonstrates the uses of the recommendations.

K. Ramamritham, C. Shen, O. González, S. Sen, S.B. Shirgurkar. Using Windows NT for Real-Time Applications. In Proc. of the 4th IEEE Real-Time Technology and Applications Symposium, Denver, CO, June 1998.

04/19/23 13:16 34






04/19/23 13:16 35

Challenges in SupportingCommunicating Real-Time Tasks

Network ArchitectureNetwork

Interface

Design

Resource

Manager

Scheduling

AlgorithmsScheduler

Middleware

Services

Traffic Management

Rate Control

04/19/23 13:16 36

Communications Middleware

Transporting multiple media types, control, data, visual,

image, alarms, video and audio

Integrating control with visual monitoring

Enabling plug-and-play

QoS management

04/19/23 13:16 37

Writer DPA_1

DPA_nReader_i

Network

Node 1 Node 2

ReMA

ReMA

ReMA = Reflective Memory Area

Real-Time Channel-based Reflective Memory (RT-CRM)

DPA = Data Push Agent

DRA = Data Receive Agent

DRA_n

04/19/23 13:16 38

RT-CRM Operation Modes

Synchronous/Asynchronous

Writer DPA_1

DPA_nReader_i

Network

Node 1 Node 2

ReMA

ReMA

Blocking / Non-blocking

04/19/23 13:16 39

Network

Node 1 Node 2

Using MidART services

Write Cmd

DPA

Process Cmd

DPA

Result

• Synchronous mode

04/19/23 13:16 40

End-to-End QoS Provisions

Writer DPA_1

DPA_nReader_i

Network

Node 1 Node 2

ReMA

ReMA

Memory-to-Memory

Application-to-Application

04/19/23 13:16 41

GPOS Problems

No priority inheritance Among user processes/threads

No Priority tracking From user to system/network threads

Limited number of priorities

No explicit support for guaranteeing timing constraints

04/19/23 13:16 42

Problem alleviated:

No priority inheritance

and limited priorities

No priority tracking

Hidden protocol stack

Priority inversion

Server-based User Level Scheduling

Solution:

Modified Dual Priority

Scheduling

04/19/23 13:16 43

Dual Priority Scheduling (York)

Middle Band

Low Band

High BandThree

Priority

Bands Non-real-time tasks

Real-time tasks before promotion time

Real-time tasks after promotion time

04/19/23 13:16 44

Dual Priority in MidART

Middle Band

Low Band

High Band

Time Critical Band

Four

Priority

BandsNon-real-time tasks

Real-time tasks before promotion time

Critical real-time tasks

Real-time tasks after promotion time

04/19/23 13:16 45

Communication Server

ReMA

DPA

ReMA

DPA

ReMA

DPA

High

Promoted

Middle

Low

QueuesRateControllers

DataPusher

To network

@ promotion time

Request = actual messages Queues

limited priorities

Calculation of promotion time

04/19/23 13:16 46

Admission control

To calculate the schedulability T_i:

Promotion Timei = Deadlinei - worst case response timei

worst case response timei = worst case delay (wi) + release jitteri

wi m+1 = Ci + So + [ w

i m + Jj C

j ] j hp(i) Tj

Computation time Blocking time Interference of high priority tasks

04/19/23 13:16 47

Rate Control with Uniform Message Size

Bounding the priority inversion time due to non-preemptive message transmission.

The “optimal” message size is a platform dependent parameter.

Receiving node capability is the key - the known live-lock problem.

Match value between the system timer granularity and the data push latency. (E.g., in our case, 8KB)

04/19/23 13:16 48

Sending Time for Data Pusher

0

2

4

6

8

10

12

14

16

0.5 1 2 4 8 9 10 12 16 24 32 48

Size of ReMA (KB)

Tim

e (

ms

ec

)

1K

8K

Single

04/19/23 13:16 49


ReMA

DPA

ReMA

DPA

ReMA

DPA

High

Promoted

Middle

Low


DataPusher

To network

@ promotion time

STOP

Uniform message size to allowtransmission with fix preemption points

04/19/23 13:16 50


ReMA

DPA

ReMA

DPA

ReMA

DPA

High

Promoted

Middle

Low


DataPusher

To network

Uniform message size to allowtransmission with fix preemption points

04/19/23 13:16 51

Problem alleviated:

No priority inheritance

and limited priorities

No priority tracking

Hidden protocol stack

Priority inversion

Server-based User Level Scheduling

Solution:

Modified Dual Priority

Scheduling

Real-time data pusher

Rate control

Uniform message size

Integration of real-time and non-real-time applications on the same computing and communication platform

04/19/23 13:16 52

Industrial Control Environment

EWS

Network

Field Devices

Field Network

PLC

Field Devices

Field Network

PLC

OperatorStations

Periodic Monitoring (64KB @ 10 msec)

Command and control (1KB @ 20 msec)

Server

04/19/23 13:16 53

Data sending period without User level scheduling

0

5

10

15

20

25

30

0 100 200 300 400 500 600 700 800 900

# Sample

Tim

e (

mse

c)

Command and ControlPeriodic Monitoring

04/19/23 13:16 54

Data sending period with User level scheduling

0

5

10

15

20

25

30

0 100 200 300 400 500 600 700 800 900 1000

# Samples

Tim

e (

ms

ec

)

Command and control

Periodic monitoring

04/19/23 13:16 55

Summary: User Level Communication

Communication Scheduler

End-to-End Task Model Local scheduling of precedence constrained distributed

tasks

Integration of real-time and non-real-time tasks

C. Shen, O. González, K. Ramamritham, and I. Mizunuma. User Level Scheduling of Communicating Real-Time Tasks. In Proc. Of fifth IEEE Real-Time Technology and Applications Symposium, Vancouver, Canada, June 1999.

04/19/23 13:16 56

Conclusion

Characterization and understanding of limitations imposed by GPOS.

Development and evaluation of an efficient user-level scheduling scheme for communicating real-time tasks Support coexistence of real-time and non-real-time

using general purpose operating systems and networks

Ongoing work for QoS management support at the middleware level

04/19/23 13:16 57

Incorporation of Quality of Service

QoS management is needed to support multiple real-time

applications on single platform Operating system

Application

Dynamic QoS management. Extend QoS guarantees provided by network to end point

applications

Predictable performance for real-time and multimedia applications

according to user QoS parameters.

04/19/23 13:16 58

System Research to Meet the Challenge

Network ArchitectureNetwork

Interface

Design

Resource

Manager

Scheduling

AlgorithmsScheduler

Middleware

Services

Traffic Management

Rate Control

Qos

Management

04/19/23 13:16 59

Research Questions

What type of support is needed to optimize resource use?

How to implement efficient QoS adaptation at the middleware

level?

The real problem

Integrated end-to-end Quality of Service management

From the mouse to the wire (User to Network)

From the wire to the glass (Network to Display)

04/19/23 13:16 60

Problems

Specification of user QoS requirements

Efficient adaptation mechanisms

Mode change

Integrated scheduling of multiple resources

Resource reservation

Account for interrupt in a GPOS

Support for management of trade-offs between different types of

requirements

dependability vs. performance

9/9/2015 13:17 1 can real-time systems be built with off the shelf components? krithi ramamritham...

Documents