ee514 – real-time computing 1 real-time linux department of electrical engineering the hong kong...

EE514 – Real-Time Computing 1

EE514 – Real-Time Computing

Real-Time Linux

Department of Electrical Engineering

The Hong Kong Polytechnic University


Real Time Linux Operating Systems

Swaminathan SivasubramanianDependable Computing & Networking Laboratory

Iowa State University

[email protected]@iastate.edu


Linux as a Real-Time OS

Traditional RT Systems used custom built systems – which were not extensible I.e. tough to develop new applications

However, as technology improved, generic real-time OS became acceptable

In OS suited for extensible development Linux looks more appealing

3


•Why Linux?• Linux (and its Real Time versions) are

free!!• Linux (and its Real Time versions) are

Open Source!!• Easy for developing RT applications

4


Why not Linux?

Linux didn’t have any corporate support until now

Linux, is a very good general purpose operating system, but not so for real-time OS (Why??)

Because, the design motive of a conventional OS and RTOS is different

5


• Linux as your real-time solution?

• Could increase priority for “real-time” tasks and assume they get scheduled

• Problem – Linux optimizes average case whereas an RTOS should work under worst case assumptions

6


Linux – A Simplified View

7


Linux – conflicts with RT constraints Coarse grained synchronization – long intervals when a

task has exclusive use of data ( as fine grained – leads to lot of overhead reducing the average case performance)

Linux batch all operations for efficient use of H/W E.g freeing a list of pages when memory is full

reducing the worst case performance Linux doesn't preempt low-priority task during system call Linux will make high priority tasks wait for low priority to

release resources

8


Real Time Linux approaches

Modify the current Linux kernel to guarantee RT constraints Used by KURT

Make the standard Linux kernel run as a task of the real-time kernel Used by RT-Linux, RTAI

9


Modifying Linux kernel

Advantages Most problems, such as interrupt handling,

already solved Less initial labor

Disadvantages No guaranteed performance RT tasks don’t always have precedence over

non-RT tasks.

10


Running Linux as a process of a second RT kernel Advantages

Can make hard real time guarantees Easy to implement a new scheduler

Disadvantages Initial port difficult, must know a tremendous

amount about underlying hardware Running a small real-time executive is not a

substitute for a full fledged RTOS

11


RTAI Overview

Based on Real-Time Hardware Abstraction Layer (HAL) (also used in Windows NT)

HAL exports Linux data and functions related to hardware

HAL defines an interface between RTAI and Linux

The Windows NT hardware abstraction layer (HAL) refers to a layer of software that deals directly with your computer hardware.

12


RTAI – A Simplified View

13


RTAI Overview (continued)

Software architecture Interface to Linux hardware management

(HAL) 3 basic components: dispatcher, scheduler,

fifos 1 interface used in user tasks to initialize and

start the components RTAI is basically an interrupt dispatcher

(reroutes interrupts to Linux if necessary) e.g: Disk interrupt

14


RT-Linux Overview

Open source Linux project Supports x86, PowerPC, Alpha Patch of the regular Linux kernel (simply

install the patch and recompile the kernel) Provides an RT API for developers Runs Linux kernel as lowest priority

process

15


•RT-Linux Task Structure

16


•RT-Linux Interrupt Dispatcher

17


RT-Linux Overview (continued)

RT tasks are coded as modules Modules are inserted and removed at

users discretion Extremely good at handling periodic tasks Communicates with non-RT kernel and

other RT tasks via fifo queues Tools are provided for graphical analysis of

RT execution

18


Problems with RT-Linux

Currently no support for aperiodic tasks Not very useful for complex RT systems Currently limited to simple problems

19


Montavista – A Faster Response Linux Kernel Montavista Inc. provides a linux solution for

embedded systems The solution’s aim is to make the Linux

kernel fully preemptable It identifies the points where priority

inversion occurs in Linux and makes those points fully preemptable

A good embedded solution not a complete RT solution.

20


Real-Time Linux A class of Linux systems which could meet the real-time

requirements.

There are a number of such real-time Linux systems available:

RTLinux – hard real-time

RTAI – hard real-time

MontaVista – soft real-time

and many more …

Many of them are developed as research projects and were released to public freely.

Some of them are commercialized though.


RTLinux or RTCore The first real-time Linux system developed as a research

project over a decade ago.

Subsequently commercialized to become a real-time core (RTCore) of a commercial Linux called Wind River Linux.

RTLinux or RTCore is a hard real-time microkernel that runs the entire Linux operating system as a fully preemptable process.

The technologies developed to enable the real-time operation are patented and can’t be used freely.


RTAI RTAI stands for Real-Time Application Interface and is a

Linux extension to hard real-time application.

It is initially developed as a research project in Italy for enhancing the RTLinux functionalities at that time.

Subsequently, new technologies were invented to replace all the RTLinux code adopted from the initial RTLinux prototype to overcome any patent problems.

The new technologies is collectively referred as ADEOS.

Current version is 3.8.1 released in May 2010.

Still actively under development and maintained by the original developers in Italy and the Internal community.


Beginning of Linux – as a UNIX clone


Linux System Linux kernel

In Sept 1991, Linus Torvalds, a second year student of Computer Science at the University of Helsinki, developed the preliminary kernel of Linux, known as Linux version 0.0.1, and released to the Internet.

Version 1.0, 2.0, 2.2, 2.4, and 2.6 were released in 1994, 1996, 1999, 2001, and 2003, respectively.

Current version is 2.6.37.2 released on 24 Feb 2011.

Linux system Software from the GNU project + Linux kernel = Linux system

Redhat, Fedora, CentOS, Ubuntu, etc. are popular variants of Linux distributions.


Evolution of Linux Early Linux is Not Designed for Real-Time Processing

Early Linux (1.x Kernel) installations on retired Windows PCs Old/Obsolete hardware useful under Linux due to efficiency of O/S

Linux outperformed Windows in reliability and uptime (still does)

Linux Design: Fairness, Throughput and Resource-Sharing Basic Unix development design principles applied in Kernel

Heavily (over)-loaded systems continue to make progress

Does not drop network connections or starve users / applications

Fairness- and Resource-Sharing Design is Linux's Strength contributed to make Linux competitive and popular in the enterprise-

server and development-application environments

Gave rise to RedHat and others.

Essential to the evolution of Linux, endemic of UNIX legacy


Why Linux in Real-Time Systems? Not because of the Kernel’s Real-Time Performance!

UNIX-legacy Operating Systems were designed with operating principles focused on throughput and progress

User tasks should not stall under heavy load System resources must be shared fairly between users

Fairness, progress and resource-sharing conflict with the requirements of time-critical applications

VIP vs. General Admission

UNIX systems (and Linux) are historically not Real-Time OS

Linux has lagged many commercial Unix's in Real-Time performance-enhancement and Real-Time capabilities

Solaris, LynxOS, QNX SCO


28

Real time computing requirements

Scheduling Hardware Interface Communication and

Synchronization Memory Management Task Management

Timing Configuration External

Communications Error Reporting Embedded Features File system Reliability


Why Real-Time in Linux Systems (Embedded)

Most Important Factors Inhibiting Linux Adoption

Data from VDC, “Linux’s Future in the Embedded Systems Market”, June 2004


Real-Time in Handheld & Embedded Systems

Cost / Performance / Power / Weight Compromise Competitive, High-Volume, Low-margin Markets

Maximum Feature-set, Add-ons, Responsive UI feel

Device specs: minimal CPU & Memory & Battery Powered

Minimal CPU = High CPU utilization

High CPU load + Time-Critical functionality RT specs

Real-time Requirements will never be alleviated by Improvements in Hardware Performance / Efficiency

Software utilizing latest hardware technologies easily keep up with, and usually out-paces, advances in hardware technology

If you don't believe that, go shopping (for a mobile phone)


Real-Time Linux 2.6 Enablers Pro-Audio Performance Requirements

Audio Community Involved in Kernel-Preemption since 2.2 Audio Community strongly Endorsing RT technology

Embedded Application Domain Single-Chip, Mobile Applications (Wireless / Cellular Handsets) Predictable OS performance eliminates HW design uncertainty

Reliable Prototyping and Improved Product Scheduling

Multimedia Carrier (QOS) Application Domain Telephony, Audio / Video / Multimedia / Home Entertainment

Fine-Granular Preemption improves SMP scalability Mainstreaming of SMP Technology

Dual / Quad / Octa - Core Intel, AMD, PPC, Arm


Real-Time and Linux Kernel Evolution Gradual Kernel Optimizations over Time

SMP Critical sections (Linux 2.x) Low-Latency Patches (Linux 2.2) Preemption Points / Kernel Tuning (Linux 2.2 / 2.4) Preemptible Kernel Patches (Linux 2.4) Fixed-time “O(1)” Scheduler (Linux 2.6) Voluntary Preemption (Linux 2.6)

In 2003-04 Linux 2.6 RT Technology Regressed Early Linux 2.6 Real-Time Performance was worse than 2.4

Kernel Performance Audio Community and others balked at moving to 2.6 Kernel

Base


Real-Time Inhibitor: Critical Section Locking

Linux 2.6 Kernel Critical Sections are Non-Preemptible Critical sections protect shared resources, e.g. hardware

registers, I/O ports, and data in RAM Critical sections are shared by Processes, Interrupts and CPUs. Effective protection is provided by the Spin-Lock Subsystem Critical sections must be locked and unlocked Locked critical sections are not preemptible Linux 2.6 Kernel has 11,000 critical sections Exhaustive Kernel testing to identify worst-case code paths Labour-intensive cleanup of critical sections No control over 3rd party drivers Worst-case after cleanup still not acceptable Maintenance, community education, policing / regression testing


Real-Time Inhibitor: Interrupt Handlers Linux 2.6 Kernel: Unbounded IRQ subsystem latencies

Task-Preemption latency increases with hardware-interrupt load

Interrupts cannot be preempted

No Priorities for Interrupts IRQ Subsystem always preempts tasks unconditionally

Unbounded SoftIRQ subsystem (“Bottom Half Processing”) Activated by HW IRQs (Timers, SCSI, Network)

SoftIRQs re-activate, iterate

Driver-level adaptations Network Driver NAPI adaption reduces denial of service (D.o.S.)

effects of high packet loads


Real-Time Inhibitor: Legacy Locking Existing Locking Subsystems are not Priority-Aware

System semaphore Counting semaphore used to wake multiple waiting tasks No support for priority inheritance No priority ordering of waiters

Big Kernel Lock (BKL) Originally non-preemptible, now preemptible using system

semaphore Can be released by blocking tasks, re-acquired upon wake-up No priority-awareness, or priority inheritance for contending tasks

RCU (Read-Copy-Update) Locks in Network subsystem Read-optimized cached locking requiring race-free invalidation

Read – Write Locks Classical blocking / starvation issues with no priority awareness


The Fully Preemptible Linux Kernel Dramatic Reduction in 2.6 Preemption Latencies

Multiple Concurrent Tasks in Independent Critical Sections Generally Fully Preemptible “No Delays”

Non-preemptible: Interrupt off paths and lowest-level interrupt management Non-preemptible: Scheduling and context switching code

Design Flexibility Provides Full Access to Kernel Resources to RT Tasks Supports existing driver and application code User-space Real-Time

Optimization Flexibility RT Tasks designed to use Kernel-resources in managed ways

can reduce or eliminate Priority-Inheritance delays

Adequate Instrumentation Latency timing, latency triggers & stack tracing, histograms


Kernel Evolution: Preemptible Code

Kernels 2.2-2.4

PreemptibleKernel 2.4Kernel 2.6

Preemptible Non-Preemptible

Kernel 2.0

Real-Time Kernel 2.6


Real-Time Linux 2.6 Performance Real-Time Linux 2.6 Kernel Performance

Far exceeds most stringent Audio performance requirements

Enables sub-millisecond control-loop response

Enables Hard Real Time for qualified RT-aware Applications

SMP Kernel Performance

SMP-safe code is by definition preemptible

Any code that allows concurrent execution by multiple CPUs, also allows context switching and therefore preemption

Increased preemptible code surface in the Kernel also increases SMP throughput / efficiency


Real-Time Linux 2.6 Performance (Cont)

No Preemption Preemptible

Real-Time Preemption

MontaVista Linux 4.0 Target machine:

Intel® Celeron® 800 MHz CPU utilization during test:

100% most of the time Test Duration:

20 hours


RTLinux / RTAI Architecture

Hardware abstraction layer real-time interrupt dispatcher, real-time scheduler Inter processes communication services

Real-time plugin

System libraries

Linux kernelDrivers

I/O

I/O

Hardware interrupts

Software interrupts

Real Time task

User’s task

Real-time Scheduler

HARDWARE


Interrupt Latency The time stamp is frozen when the interrupt is generated The current time is read when the interrupt is

acknowledged The difference gives the interrupt latency

Dual-Xeon 3.2GHz

Dual-core Opteron 2GHz


Freescale – dual-kernel, real-time solutions (BSP) for Linux (Case studies) Solutions provide sub-20 microsecond precision with

hard, real-time scheduling guarantees but at the expense of increased programming effort and memory protection

These techniques offer less than 250 µsec latency performance for greater than 99.5 percent of all scheduled task under demanding system test loads

Under real-world conditions, performance is near the 99.5 percent, 100 µsec mark

BSP : board support service

42


Linux kernel improvements for real time

Features and performance enhancement for upgrading standard Linux kernel to challenge real time products and applications: Enhanced schedulers Virtual memory Shared memory Portable operating system interface X (POSIX) timers Real-time signals POSIX asynchronous I/O POSIX threads Quality of service capabilities Low latency/preemptable kernel modifications

43


Quality of sercice

As for quality of service, Linux v2.4 offers a sophisticated component for bandwidth management called Traffic Control. The component supports methods for classifying, prioritizing and limiting both incoming and outgoing traffic. Linux can perform: Limit bandwidth for certain computers Help fairly share bandwidth Protect the Internet from abuses Restrict access and do routing based on user ID, MAC

address, source IP address and so on

44


Linux network subsystem (Traffic control)

45


Queueing discipline- traffic control

Queueing discipline provides a method to enqueue a packet

46


Low-Latency patches for Linux

The kernel latency is a quantity used to measure the difference between the theoretical schedule and the actual one

In a standard Linux kernel, the maximum latency is equal to the maximum length of a system call plus the processing time of all the interrupts that fire before returning to user mode. This value can be as large as 100 ms

47


Low-Latency Linux (Red-Linux, some Real-time Linux) This approach corrects the monolithic structure by

inserting explicit rescheduling points inside the kernel. In this approach, when a thread is executing inside the kernel it can explicitly decide to yield the CPU to some other thread

In this way, the size of non-preemptable sections is reduced, thus decreasing the latency

The consistency of kernel data is enforced by suing cooperative scheduling

Since the low-latency patch has been carefully hand-tuned for quite a long time, it performs surprisingly well

48


Preemptable Linux (used in most real-time systems) Removes the constraint of a single execution flow

inside the kernel. It is not necessary to disable preemption when an execution flow enters the kernel

To support full kernel preemptability, kernel data must be explicitly protected using mutexes or spinlocks

The Linux preemptable kernel pathc, maintained by Robert Love and sponsored by MontaVista

49


Evaluation

50


Probability density function of the latency in the low-latency kernel

51


Summary of the Freescale case studies Linux is rapidly evolving to include more and more

features that have explicit value to the embedded developer

Key enhancements to the Linux kernel, which perform identically for all practical purposes

The developer’s choice regarding which should be implemented can be based on independent criteria including the ability to update/modify existing device drivers for compatibility with the real-time implementation, and the availability of each real-time technique for the selected architecture and kernel version

52


References

RT-Linux : http://www.rtlinux.org RTAI :

http://www.aero.polimi.it/projects/rtai/contrib.htm

Montavista: http://www.mvista.com Linux as a real-time operating system –

Freescale semiconductor, David Beal, Nov/2005

53

EE514 – Real-Time Computing Real Time Linux 54

EE514 – Real-Time Computing Real Time Linux 55

ee514 – real-time computing 1 real-time linux department of electrical engineering the hong kong...

Documents