Multiple AGL profiles consolidationwith L4RE Hypervisor
By Iot.bzh & KernKonzept
AMM @ Tokyo, March 19
Johann [email protected]
Jakub Jermář[email protected]
March 2019 L4RE: consolidating multiple AGL profiles 2
IoT.bzh team
● Fully dedicated to AGL● https://iot.bzh/en/● http://github.com/iotbzh
Workshop in Lorient
LORIENT
vannes
3L4RE: consolidating multiple AGL profiles March 2019
Typical AGL architecture
● One board, one kernel :
LINUX
APPLICATIONS
SERVICES low-can
CAN
NOTE : for this PoC runs on Renesas Gen3 hardware (H3, M3, Salvator X, ...)
4L4RE: consolidating multiple AGL profiles March 2019
● CAN subsystem isolation using L4Re hypervision:
● RT_CAN instance runs on a RAM disk, IVI on mmc (SD card).
Targeted Architecture
L4Re hypervision
LINUX
APPLICATIONS
SERVICES low-can
LINUX-RT
low-can
CAN
host-IVI
host-RT_CAN
virtio-sock
5L4RE: consolidating multiple AGL profiles March 2019
PoC Goals● Purpose:
– CAN isolation leveraging hypervisor
– Isolate priority service from IVI
– Possible certification of a critical subsystem
● Perspective:,– Better availability of CAN statistics,
– Possible certification of RT_CAN instance.
– Binder for smaller trusted OS (certification)● NewGen RTOS: Zephyr, Mbed● Legacy RTOS: FreeRTOS, VxWorks, QNX● L4Re native server module, native trust zone service, ..
– Binder for other Linux flavours● Android● Adaptive Autosar
6L4RE: consolidating multiple AGL profiles March 2019
Communication between AGL instances
● L4Re provides 2 solutions outside TCP/IP.– Virtio-console:
● No kernel patch needed,● No AGL services change required at source code (config update only),● Require a “socket proxy” (proxso) to emulate sockets over virtio-console.
– L4Re’s VSOCKs:● Impose to patch the kernel to add the virtio_sock module,● Change required to application,● Applications/Services can then rely directly on L4 virtio sockets.
7L4RE: consolidating multiple AGL profiles March 2019
L4Re’s virtio vsocks
4.14.35 kernel + L4 patch
L4 VSOCKS
4.14.35 kernel + L4 patch
L4 VSOCKS
L4Re
UVMM UVMM
CANCAN
App
App
App
App agl-service-can-low-level
Renesas R-Car Gen 3
8L4RE: consolidating multiple AGL profiles March 2019
L4Re virtio console
4.14.35 kernel
virtio-console
4.14.35 kernel
virtio-console
L4Re
UVMM UVMM
CAN
proxso proxso
CAN
App
App App
Appagl-service-can-low-level
Renesas R-Car Gen 3
9L4RE: consolidating multiple AGL profiles March 2019
Not as simple as it should
● Multiple problems, related to :– Actual hardware / firmware (far simpler with QEMU)
– Yocto / AGL
– L4Re● No hard showstopper but:
– Multiple room for improvement (virtio performance/integration)
– Integration cost is far from being negligible
– Portabilty/Maintainability does not come for free
10L4RE: consolidating multiple AGL profiles March 2019
Roadmap of the PoC Around 10 weeks FTE
● Learn L4Re (configuration, build process, …)● Hack binder to support virtio● Generate AGL images (from yocto):
– IVI : ● close to an agl-demo-platform standard image, but without CAN capabilities (driver options disabled
in kernel .config)
– RT_CAN : based on core-image-minimal target in yocto● Features : agl-netboot, agl-appfw-smack ● Apply PREMPT_RT kernel patch● Disable as much options as possible in kernel .config (except CAN)
– Apply L4_VSOCKS kernel patch on both images● Assemble everything in a u-boot image● Run it on a RCar Gen3 board
11L4RE: consolidating multiple AGL profiles March 2019
UBOOT Hard-Firmware issues
● U-Boot : binary (kernel) loaded in memory must be <8MiB (default value), or else u-boot complains.– Size can be configured (CONFIG_SYS_BOOTM_LEN), but
ultimately, the problem is u-boot needs to copy the binary to the “entry point” address declared in the header, them jump to this address (this binary is NOT rellocatable).
– Better solution : load the binary “in place” (i.e. in such a way that no copy is needed). Target address is then:
● entry_point – header_length (u-boot header is 64 bytes)● 0x51 00 00 00 – 0x40 == 0x50 FF FF C0● ext4load 0:1 0x50ffffc0 ${uImage}
12L4RE: consolidating multiple AGL profiles March 2019
RAM Hard-Firmware Issues
● Memory Mapping : some RAM zones are used by “obscure” features like “CMA for lossy comp” (48MiB from 0x54 00 00 00 to 0x57 00 00 00),
● Another zone has been detected further, but not its exact boudaries, nor the feature that uses it,
● Bytes in those zones are “corrupted”, ● Corruption is detected by u-boot (CRC error),● As for now, we believe it’s safe to load uImage after 0x5A 00 00 00● Need to change the entry point address, which is configuration in L4Re
13L4RE: consolidating multiple AGL profiles March 2019
L4Re / Hardware issues (Firmware)
● Problem with mapping the A53 in L4Re startup script.– Original setup :
● IVI: 2 A57 + 2 A53● RT_CAN + L4Re: 2 A53
– Quick hotfix : ● Deploy VMs only on two A57, one VM on each. ● Obviously not a valid option for the IVI instance.
● Nota: Seems to be fixed with newer firmware versions
14L4RE: consolidating multiple AGL profiles March 2019
L4Re / Hardware issues (RAMDISK)
● Problem to deploy ramdisks big files from uImage– Needed because we cannot share access to the SD card (L4Re)
– Related to memory mapping (RAM banks that are not contiguous).
– KernKonzept managed to deploy the 240MiB ramdisk for RT_CAN.
– IoT.bzh approach implements 2nd partition on the SD card. One served by the IVI instance over NBD (agl-netboot):
● NOT an ideal architecture:● RT_CAN can’t boot/work if IVI is down● Somewhat “breaks” isolation
15L4RE: consolidating multiple AGL profiles March 2019
VIRTIO VSOCK patch issue
● Patch L4 VSOCK : only applies on kernels 4.18 and 4.19 – Not applicable out-of-the-box on our 4.14.35 kernel (few API
changes, ...),
– Managed to backport, but still generated a Kernel Oops when using VSOCKs.
– Not a fully standard implementation
16L4RE: consolidating multiple AGL profiles March 2019
Yocto / AGL (1)
● core-image-minimal is still ~520MiB (!!)– After manually stripping down, still ~240MiB…
● 90 packages removed using packaging system● 3000+ files removed manually (including dnf and dependencies)
– A really “minimal” target (BSP+AGL appfw) should be < 128MiB (<64MiB).● Need to define a restricted list of packages ● Each package (at least in this list) should support a minimal installation mode (no
debug symbols, …)● Customization of packages (for instance, no need to have a full HW in udev)● Kernel optimisation, install only needed modules, ...
17L4RE: consolidating multiple AGL profiles March 2019
Preemp_RT for Yocto / AGL
● PREEMPT_RT patch.– Today : nothing to enable / apply it in yocto,
– Very tricky, no version for kernel 4.14.35,
– Renesas kernel is not “vanilla”, which adds difficulties.
Ref: Thierry Dresden presentation on AGL and Real Time https://iot.bzh/en/publications/32-2018/78-agl-real-time-architecture-options-for-critical-profiles
March 2019
M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
18www.kernkonzept.com
L4Re HypervisorJakub Jermář
Kernkonzept
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
19
L4Re Highlights Mature technology 20+ years of operating systems research and application in industry TU Dresden, Germany Kernkonzept Small & secure Orders of magnitude smaller TCB than Linux
Secure capability-based component architecture Multiplatform 32-bit/64-bit x ARM/MIPS/x86
L4Re Microkernel
VM2
sigma0
moe
uvmm
ned
io
VM1
uvmm
~31 kloc
~1 kloc
~5 kloc
~2.5 kloc
~20 kloc
~11 kloc~3.6 kloc
2+ mloc
l4re libs ~35 klocuclibc
libstdc++
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
20
L4Re Highlights Virtualization Full-virtualization x paravirtualization Hardware pass-through Per-guest untrusted user-level VMM VIRTIO P2P link components Optee scenarios Real-time Multi-server Microkernel L4Re native applications POSIX subset
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
21
L4Re Highlights Power management features Suspend to RAM Enable/disable CPUs Dual-licensing model Open source, CLA Commercial licensing
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
22
L4Re Usecases In automotive EB corbos Hypervisor AGL on L4Re Not only in automotive Security products Intrusion detection systems Safety products Consumer electronics Smart factories IoT
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
23
Open source GitHub https://github.com/kernkonzept Snapshots http://l4re.org/download/snapshots/ Tutorials https://github.com/kernkonzept/manifest/wiki#tutorials QEMU Advent Calendar 2018 http://www.qemu-advent-calendar.org/2018/#day-21 FOSDEM Microkernel devroom
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
24
AGL on L4Re Combining AGL and L4Re expertise Over a distance of 1300km Different user experience from x86 desktop virtualization Lots of configuration due to HW pass-through Platforms’ variability Detailed knowledge of the target platform needed Familiarity with the guest kernel is a plus
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
25
Hardware Challenges Issues with the M3/H3 boards Failures to netboot a large image over TFTP All Cortex-A53 CPUs disabled in older M3 firmware Virtualization not enabled on one H3 board Perfect HW partitioning vs. HW dependencies Shared system, reset and clock controllers
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
26
Software Challenges How does the CAN-only VM get its rootfs? AGL image size vs. available memory agl-minimal-image ~520MB Guest drivers ignoring DT configuration
MM_LOSSY_SHARED_MEM_ADDR
www.kernkonzept.com
March 2019M I C R O K E R N E L M A D E I N G E R M A N Y
L4RE: consolidating multiple AGL profiles
27
Come to see our Under the Hood of AGL Virtualization on L4Re BoF!
28L4RE: consolidating multiple AGL profiles March 2019
Conclusions
● Good collaboration with KernKonzept● About L4Re :
– works well, despite some limitations in actual implementation
– Ultra-lightweight code base (and good quality, but some core “legacy” components, like Fiasco kernel, sigma0 pager, etc. would deserve a little lifting to take advantage of modern C++ features)
– Significant learning curve when no/limited knowledge on “microkernels”
– A fully compliant VIRTIO VSOCK implementation would help● About Virtualisation
– It open new doors but it’s not a magic wand people dream of
– Integration cost and time should not be ignore
– Add an extra layer to debug that might impose unwanted cross team collaboration