1. Enea Linux and Light-weight ThreadingReal-time Linux for Next Generation Networking InfrastructureLight-weight Run-time Threading for improved real-time behavior Michael ChristoffersonDirector Marketing, Networking FTF China, 2012

2. Enea LinuxA flexible approach to embedded Linux based on Eneas 25+ year experiencewith Tier 1s and many others in the global telecom market Enea Linux experience Linux based solutions for last 6 years Enea LINX, Enea Element, EneadSPEED, Enea BMPT, Linux Competence Center and Android (services) And with major silicon vendor Linuxs : Freescale, NetLogic, Cavium,ARM, And 3PP integrated Linux solutions MontaVista, TimeSys, Q1 2012 But until now, all based on other Linux distros Enea Linux is powered by Yocto open source configuration andbuild technology Custom tailored solutions that address the specific challenges inthe telecom and networking space Linux packages that apply Hardened Linux solution for customers platform and rqmtsQ3 2012 3. Why Enea Linux? 4. Enea Linux uses YoctoYocto is not an embedded Linux distribution, it creates a custom ones. The YoctoProject is an open source collaboration project that provides templates, toolsand methods to help create custom Linux-based systems for embedded productsregardless of the hardware architecture. Yocto automates how source is fetched from a Yocto packagesvariety of upstream sources or from localproject repositories. Updating to a new versionof a Linux package is easy.Linux tools Enea packages Yocto features a powerful customizationYoctoarchitecture that allows the choice of a widevariety of footprint sizes as well as control overLinux Hardened linuxthe choice or absence of components such asruntime packagesgraphics subsystems, visualization middleware,networking, and other services. HW vendorpackages 5. Enea Linux Packages - Customization Enea Linux now has over 150 packagesthat are specially selected for the real Enea Linux Customer Xtime embedded telecom / networking Availablespace Packages Middleware Expected to grow to around 200packages in the futureCustomer Y Customer driven Supported packages available onrequest Customer ZCustomer Only Gets Desired Packages Integrated and supported on their HW platform 6. Linux Development Environment Host support Ubuntu, Fedora, Debian and SUSE Graphical configuration and build using Openconf planned Qemu processor emulation Debug, Profiling, Analysis Tools Freeze mode (JTAG from 3PP), but Eclipse LTTng oProfile gcov, & gprof Valgrind More integrated Eclipse based tools planned for future 7. HW Support in Enea Linux Freescale Now: P2020/2041, P3041, P4080 Future: P5020 (e5500) T4240 (e6500) More e5500, e6500 support as well as e500v2 LSI Now: ACP34xx (PPC476) Intel Now: x86 AMCC Now: PPC440 ARM Future: A15 8. Enea Linux Add-ons 9. Linux User Space Multi-threading The Enea LWRT InnovationLight-weight Runtime Threads 10. Polling Question 1 Do you think that the Linux implementations that you usenow or have studied deliver the kind of real-timeperformance and behavior that you need? Yes? No? Enea Confidential 11. 3 Ways to Tackle Realtime in Linux:We can add a realtime kernel We can simply avoid using kernelWe can rework the internals ofunderneath Linux:functionality in situations thatLinux: causes realtime problems: User Mode Runtime Multi-threading PTREADSLinux Kernel Linux KernelLinux Kernel Realtime Kernel Examples includes hypervisor, We can partition a single Linux Might require significant changesXenomai, RTLinux etc.instance and separate realtimecompared to standard Linux. The end result is two separate from non-realtime. Most common example is thesystems/environments. We can configure processes andPREEMPT_RT patch. We cannot completely utilize the interrupts to run with core affinity. Taking 3.0.27 as an example,Linux eco-system (e.g. tools) in the We make minor modifications toPREEMPT_RT patches 500+realtime domain. the kernel to avoid running kernellocations in the kernel, with threads/timers on realtime cores. 11,500+ new lines of code in total. We can avoid using/calling the kernel, and rely on user-mode functionality instead. 12. Linux Multi-threading with PthreadsPthreads Model Requires less overhead than "forking" or spawning App Appa new process because the system does notinitialize a new system virtual memory space andenvironment for the process.PThread PThreadPthreads Under the Hood Threads exist in two separate execution spaces inLinux user space and kernel space. User-space Processthreads are created with the pthread library APIand mapped to kernel threads. All thread scheduling therefore must involve acontext switch from user space to kernel spaceLinux Cores So the real question is: can Linux multi-threading be done better? 13. Why is Multi-threading Performance an Issue? Isnt Linux is real time? What about Bare Metal Linux? Bare Metal Linux for multicore devices focuses on direct HW access from user spaceBare MetalSingle threadAppApp Achieves bare metal via a single thread App Good for simple apps like IP Forwarding, transcoding, etc Multi-threading is the time honored embedded solution to serious real-time problems wherein an application has multiple PThread PThread PThread purposes Add run-time debug and profiling? Add higher level networking protocols ProcessProcess Add device and run-time configuration and management? Multi-threading has always been a part of Linux Real-time Extensions for Linux have focused on faster and Linux Cores Linux Coresmore deterministic process AND thread scheduling amongother thingsComplex protocols and applications require multi-threading for max performance and isused widely in Linux 14. Polling Question 2 Is multithreading in Linux a concern for your applications interms of performance and/or robustness? Yes? No? 15. The LWRT Concept Better to partition the system intoseparate realtime critical an non- We improve performancecritical domains.and realtime characteristics It is often the Linux kernel itself that under Linux by partitioningintroduces realtime problems. the system into logical A combination of partitioning,combined with a user-modedomains, and by avoid usingenvironment that allow us to avoid the Linux kernel itself moreusing the kernel can improve than necessary.performance and realtimecharacteristics compared to astandard Linux. 16. What is LWRT? LWRT is a Light-Weight Runtime (Environment) built upon Linux. LWRT runs in most part in user-space,and is a essentially a library linked to the application.Application The LWRT environment provides better performanceand realtime characteristics compared to the standard LWRT EnvironmentPOSIX/Linux environment. Linux Kernel CoreCore0 N 17. How does LWRT work?LWRT partitions the system into one realtime domain and one non-realtime domain. Non-realtime ProcessesRealtime Processes LWRT adds a user-mode runtime environment, includingan optimized user-mode scheduler.LWRT Environment Pthread Pthread LWRT migrates some specific kernel functionality (e.g.timers) away from the realtime domain.Linux KernelLWRT adds a kernel module to catch and forward LWRT Kernel Moduleinterrupts to the user-mode environment.CoreCore 0 N 18. What are the benefits of LWRT?LWRT provides low latency and high throughput. LWRTdoes not depend on the PREEMPT_RT patch, and doesnot affect throughput negatively.LWRT provides a solution that is unencumbered byNon-realtime ProcessesRealtime ProcessesGPL, even for interrupt driven code which can beplaced in user-space without any major penalty.LWRT EnvironmentLWRT provides optimized APIs for realtimeapplications, and allows the same application to usePthread Pthreadthe POSIX/Linux APIs when realtime doesnt matter.LWRT provides very good (i.e. low-latency) interruptresponse time, all the way up to user-mode.Linux KernelLWRT KernelModuleLWRT is an all-Linux solution, based on a single LinuxKernel. Thus, almost all tools from the existing Linux CoreCoreecosystem will be available.0 N 19. LWRT ServicesLWRT Architecture Multiple threads per core Low inter-thread signaling overhead Deterministic scheduling Cheap thread creation Low scheduling overhead Atomic signal/sleep Cheap context switches 20. So LWRT is a Different Multi-threading Model A Light-Weight Thread Environment (LWT)implemented in user-space over a single pthread A user-mode scheduler in the scope of a Linux process LWTLWTLWT - No kernel thread context switchPThread PThread PThread - Preemptive, priority based scheduler - 100% Linux, utilizes same Linux tools as restof system Linux Process - Complete access to normal POSIX/libC Support for message-passing between threads (andprocesses). With simple buffer management (forCoreCore Coremessages), and interrupt handlingSo what is the Enea LWRT performance advantage over Pthreads? 21. LWRT vs Pthreads Demo CoreStart GUI0ForkShared MemorySpawn Ping Core Enea LWTLWT1 Demo Pong LWTSpawnPing Core PThreads PThread2DemoPong PThread Measurements done on x86 using TSC (time stamp counter) Demo HW: AMD Phenom II N620 Dual-Core processor @ 2.8GHz. Linux: OpenSuSE 12.1, 32 bit 22. LWRT vs Pthreads - Context Switch Overhead100000 Pthreads LWRT 80000 LWRT has much better 60000 performance i.e. lower scheduling latency 40000 20000 LWRT has much better real-time characteristics, i.e. less variance. Clock cycles 0 0 5001000 1500 2000 2500 3000 3500 40004500 23. LWRT vs PthreadsInter-thread Communications Ping-Pong benchmarkmeasuring schedulingtime/latency Histogram show one-triplatency LWT examplebased on signaling Pthread example based onsemaphores In the demo, the user caninteractively play with coreaffinity and schedulingpolicy 24. LWRT vs PthreadsScheduling Behavior under System LoadScheduling Jitter Due to Load 25. Comparing LWRT and PREEMPT_RT PREEMPT_RT applies to the entire system, whereasLWRT partitions the system. PREEMPT_RT improves latency at the expense ofthroughput. LWRT on the other hand is built upon theidea that one part of the system might be optimizedfor latency, and one part might be more focused onthroughput. PREEMPT_RT makes significant modifications to theLinux kernel (e.g. 500+ locations, 11,500+ lines). LWRTmakes minimal modifications, and tries to use as muchof standard kernel as possible. PREEMPT_RT assumes that all device drivers havebeen adapted to the patch. LWRT puts no specialrequirements on device driver. LWRT achieves same interrupt latency as PREEMP_RT,with better end-to-end performance (time) 26. Summary - Why LWRT?LWRT combines the best of two worlds Linux and RTOS Characteristics Linux Linux + LWRT RTOS A native all-Linux solution A vast ecosystemDeterminism The advantages of the Extensive HW supportLow overhead RTOS in Linux user-space Rich toolset Same tools as the rest of theFine-grained thread model Established programmingLinux-based system Priority based scheduling model Access to the normal POSIX/libC. Better performance and real-time characteristics for any multi-threaded application eithercontrol plane or data plane 27. More Good News Remember the three ways to boost Linux Real-time Performance?Enea Can Support All of Them:A realtime kernel based on the Enea Prototypes of LWRT show very good We can offer PREEMPT_RT as partHypervisor offers excellent realtimerealtime characteristics with anof Enea Linux:characteristics:almost unmodified kernel.LWRT Linux KernelLinux Kernel Linux Kernel Realtime Kernel / Hypervisor Examples includes LWRT doesnt exclude PREEMPT_RT. These technologies are nothypervisor, Xenomai, necessarily competing, but may actually be used in combination.RTLinux etc. However, depending on the situation, an LWRT solution might actually The end result is twoachieve better results than PREEMPT_RT.separatesystems/environments. We cannot completelyutilize the Linux eco-system (e.g. tools) in therealtime domain. 28. See us in the Exhibit Hallfor an Enea Linux and LWRT Demoalong with other Enea technology:- Packet Acceleration - Linux Base Station Platform - Hypervisor - Element HA Booths 1430-1435