new evaluating the real-time capabilities of zephyr on …1472100/... · 2020. 9. 30. ·...

Malardalen UniversitySchool of Innovation Design and Engineering

Vasteras, Sweden

Thesis for the Degree of Master of Science in Computer Science withspecialisation in Intelligent Embedded Systems 30.0 credits

EVALUATING THE REAL-TIMECAPABILITIES OF ZEPHYR ON UPSQUARED - A PILOT STUDY WITH

ZEPHYR AND ACRN

Charlie [email protected]

Jimmy [email protected]

Examiner: Thomas NolteMalardalen University, Vasteras, Sweden

Supervisors: Anna FriebeMalardalen University, Vasteras, Sweden

Shaik MohammedMalardalen University, Vasteras, Sweden

September 25, 2020

Charlie Kotro, Jimmy Makkeli Evaluating the RT capabilities of ACRN and Zephyr on UP2

Abstract

In this study we have investigated if using Zephyr (User VM) and ACRN (Hypervisor) on the UPSquared (N3350) board are suitable for use in real-time scenarios. This was achieved by conductinga series of tests on Zephyr’s scheduling API functions such as busy-wait and sleep under differentsystem configurations. The tests involved running a task in isolation and taking timestamps ofthe current system cycle count before and after each function. To do this, we used the CommonTrace Format functionality in Zephyr to collect execution traces, which were then analyzed usingBabeltrace to convert the binary trace into human-readable form. The difference between the beforeand after timestamp were compared to the configured clock frequency to see if the system behaved asexpected. Our findings in this report suggest that Zephyr and ACRN on the UP Squared board arenot suitable for use in real-time scenarios due to large differences in system cycle count dependingon the configuration.

i


Table of Contents

1. Introduction 21.1. Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1..1 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2. Background 42.1. Hypervisors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2. ACRN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3. Zephyr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4. Open-source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.4..1 Quality assessment of open-source software . . . . . . . . . . . . . . . . . . 62.5. Software tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.5..1 Common Trace Format - CTF . . . . . . . . . . . . . . . . . . . . . . . . . 72.5..2 SEGGER SystemView . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.5..3 DTrace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.6. Babeltrace 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3. Related work 83.1. Hypervisors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2. Quality assessment for open-source projects . . . . . . . . . . . . . . . . . . . . . . 83.3. Quality models for open-source projects . . . . . . . . . . . . . . . . . . . . . . . . 93.4. Risk management for open-source projects . . . . . . . . . . . . . . . . . . . . . . . 93.5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4. Method 11

5. Ethical considerations 13

6. Implementation and problems 146.1. Zephyr problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146.2. ACRN problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146.3. Test program implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

7. Results 17

8. Discussion 21

9. Conclusions 22

10.References 23

Appendix A Appendix A: Raw cycle count data 251.1. ACRN Debian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261.2. ACRN Debian strip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261.3. ACRN Clearlinux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271.4. ACRN Clearlinux strip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281.5. ACRN Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281.6. ACRN Ubuntu strip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291.7. Zephyr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301.8. Zephyr strip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301.9. Zephyr tickless . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311.10. Zephyr tickless strip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Appendix B Appendix B: Implementation source code 33

ii


List of Figures

1 Overwiew of different hypervisor types . . . . . . . . . . . . . . . . . . . . . . . . . 42 High level overview of ACRN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Device Passthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Trace results of Sleep compared to the baseline in a logarithmic scale (x-axis). . . . 185 Trace results of Sleep compared to the baseline. . . . . . . . . . . . . . . . . . . . . 186 Trace results of Busy-wait compared to the baseline. . . . . . . . . . . . . . . . . . 197 Trace results of Busy-wait compared to the baseline. . . . . . . . . . . . . . . . . . 198 Trace results of Loop compared to the baseline. . . . . . . . . . . . . . . . . . . . . 209 Trace results of Loop compared to the baseline. . . . . . . . . . . . . . . . . . . . . 20

iii


Acknowledgements

We want to thank our supervisors Anna Friebe and Shaik Mohammed, for all their help duringthis thesis. They listened to us whenever we had problems during every phase of the work, theyalso provided us with a lot of feedback as well as help on how to proceed forward. We would alsolike to thank our families for always providing support and giving us that nudge forward.

1


1. Introduction

Open-source software (OSS) is software that is freely available to anyone, and you are allowed tomake changes to the software. This is different compared to closed-source software, which is usuallybought, and you are not allowed to freely change the software. Both types of software come withpros and cons. This is also true when it comes to the development of both types of software. Open-source software development is different from traditional development, and an example where thesetwo approaches to development differ is when it comes to the documentation. The documentationfor closed-source software tends to be more extensive than OSS documentation. Many studies haveshown that OSS development does not use many of the best practices, but does so without loss inquality [1]. But OSS development does keep some of the best practices such as central ownership,system testing, and planning and strategy meetings, among others [1]. It is not easy to select anOSS for your system since there are no set standards for determining the quality of the software[2]. Open-source software, have several benefits such as lowered development and maintenancecosts, and the ability to customize the software [3].

Virtual Machine Monitors (VMM), also called Hypervisors, sit between the hardware and theoperating system and manage resource distribution. This allows the system to run multiple otheroperating systems on the same hardware without them affecting each other [4]. ACRN is an open-source lightweight type-1 hypervisor, which means that it runs directly on the hardware as opposedto running on an operating system. It was built for development for the Internet of Things (IoT)[5]. ACRN was developed as collaborative work between the Linux foundation and Intel becauseof Intel’s expertise in virtualization [6]. Zephyr OS [7] is an open-source operating system forresource-constrained systems. It has a small memory footprint and supports a lot of features,such as multiple scheduling algorithms, among others. Another project from the Linux foundationis Akraino-edge stack, which has an integration project called Time-Critical Edge Compute [8],which uses both Zephyr and ACRN. The three main reasons for opting to use Zephyr and ACRNare: both systems are considered lightweight, made for IoT development, and there is no previousresearch of using them together.

Edge-computing is a paradigm in which the resources are brought to the edges of the computationalnetworks. By shifting where the computation and storage take place, it becomes possible tolower latency and decrease the bandwidth usage [9]. Some examples of edge-computing stacks areAkraino Edge Stack, Apache Edgent, and Azure IoT Edge [9]. Because of the increasing demandsfor more responsive applications and less usage of bandwidth, the real-time capabilities of edgecomputing are suitable to facilitate these types of applications or services [10]. Some examples ofapplications that would require real-time performance are multiplayer games, cloud games, andstreaming services [10].

As mentioned earlier in this section, both ACRN and Zephyr are made for IoT development.But according to some researchers, most existing hypervisors have a hard time fulfilling real-timerequirements [5]. Therefore it becomes interesting to investigate if these two systems can be usedfor real-time systems.

1.1. Problem formulation

This study will focus on finding out if Zephyr and ACRN are suitable for use in real-time systemswhen ran on UP Squared. It is not known how suitable these systems are for use in real-timesystems, and to our knowledge, there is no previous research that has used these two systemstogether in real-time systems. Therefore, this study will focus on the aspects related to commonfunctionalities that are used when scheduling tasks in the system, such as sleep functions and busy-waits. By using a hypervisor, extra overhead is added to the system because of this extra layerthat each request has to go through. But this extra amount of overhead is not known, thereforethis needs to be studied to be able to determine if the systems can be used in real-time scenarios.

2


1.1..1 Research questions

This is our research question that we are going to look into:

• RQ: What is the performance of common scheduling API methods when run on Zephyrstandalone and when Zephyr is run via ACRN?

Our research question was chosen because we are trying to determine if ACRN and Zephyr can beused for real-time systems, and therefore it is important to investigate how these systems performwhen scheduling tasks, which is an important part of a real-time system. As real-time systemsneed to be deterministic, it is important to know by how much these functions can differ from thespecifications.

3


2. Background

In this section, we will talk about topics that are related to this work, such as hypervisors andquality assessment for open-source software. Hypervisors are relevant as we are using a hypervisorourselves, and we are evaluating whether it is ready to be used in real-time scenarios. But to beable to use the hypervisor, we need to have an operating system, which the hypervisor can run,and therefore we opted to use Zephyr, as this is also a system which has not been used in this typeof work before. By assessing the quality of these two systems, we can determine if they can beused for real-time systems. A common method for gathering data related to scheduling is to usevarious tracing tools such as Babeltrace and Common Trace Format. Both the hypervisor and theRTOS we have chosen are open-source systems.

2.1. Hypervisors

Hypervisors, or Virtual machine monitors, can be divided into two categories: type-1 (bare-metal)or type-2 (hosted hypervisor). The type-1 hypervisor interfaces directly with the hardware anddoes not need an operating system, unlike the type-2 versions [11]. Figure 1 shows the differencesbetween type-1 and 2 hypervisors. Some examples of type-1 hypervisors are XenServer, KVM,and VMware ESXi [12]. And some examples of type-2 hypervisors are VMware -workstation, -fusion, and VirtualBox [13]. Hypervisors allow multiple different virtual operating systems to runat the same time while still having access to the same hardware. The hypervisor manages andmonitors the system resources while also handling requests for these resources. An advantage ofusing hypervisors is the isolation they give, in the sense that if one of the guest operating systemsis having problems, these problems do not affect the other guest systems [11].

Figure 1: Overwiew of different hypervisor types [14].

4


2.2. ACRN

This section contains information about ACRN from the paper ACRN: A Big Little Hypervisorfor IoT Development [5] and the ACRN documentation [15].

When it comes to IoT usages, it is not easy for existing solutions to handle real-time constraints.Figure 2 shows a high-level overview of ACRN and the different operating systems it can run.ACRN is developed from scratch to be a lightweight type-1 hypervisor. This means that ACRNhas a small code size and impact on performance, which is critical for industrial usage. Being atype-1 hypervisor, ACRN does not have to rely on RTOS services. The developers argue that thismakes porting it to other hardware architectures easier since there is no need to port an RTOSfirst, which can be a problem for other hypervisors since it would require extensive source codemodification. ACRN also has the benefit of not binding to a specific RTOS, which means thatit works with various operating systems on demand. Another benefit is it being open source andfree. [5]

Figure 2: High level overview of ACRN [15].

ACRN uses x86 as a starting point, compared to other solutions which do not have mature supportfor x86 embedded systems [5]. Some of the features that ACRN supports are rich I/O virtualization,device pass-through and, mediated pass-through. According to ACRN documentation, it is possibleto allocate a physical device exclusively to a guest operating system, which allows almost near-native performance [15]. In order to avoid complex CPU-emulation most CPU features such asCPUID, MSR, etc. are passed through. It also uses Intel VT-x to help with CPU virtualization.[5]

Figure 3: Device Passthrough [15].

The ACRN hypervisor supports three types of VMs: Service VM, User VM, and Real-Time VM,each able to run an OS. The User VM and RTVM are isolated in order to guarantee that theperformance of the application is running in real-time. By using dedicated pass-through, whichcan be seen in Figure 3, the RTVM minimizes the use of emulated devices at boot time. The

5


hypervisor passes through access to hardware for the RTVM to optimize real-time performanceand to avoid additional latency from the Service OS. [5]

To achieve isolation, the CPU core and memory partitioning are spatially isolated. This meansthat the hardware can not be accessed by a VM without it being assigned to it. Therefore, otherVMs have no access to process this data. [5]

ACRN can be run using four different scenarios: SDC (Software Defined Cockpit) with 2 VMs, IU(Industrial Usage) without Safety VM, IU with Safety VM, and Logical Partition. However, forour UP Squared board, the only verified scenario is SDC with 2 VMs.

2.3. Zephyr

Zephyr is a small-footprint kernel intended for use in embedded systems that are resource con-strained. These systems include sensors that are wearable, smart-watches, and IoT applications,among others. It supports a lot of different architectures and boards. Zephyr supports several dif-ferent scheduling algorithms and kernel services. According to the documentation, the system alsofeatures memory protection in the form of stack-overflow protection, kernel and object permissiontracking, and thread-level memory protection. [7]

2.4. Open-source

Open-source software is software that is available to anyone, and you are allowed to change, in-spect and redistribute the modified software [16]. This type of software is often supported by acommunity, which helps out with tasks such as debugging, testing, and even developing new fea-tures. A few of the advantages for companies using OSS is that it can lower the development costs,maintenance costs, and the company can change the software to suit their needs. But this does notmean that adopting OSS is risk-free, as studies have shown that most open-source projects end upin failure [3]. Keeping most open-source projects alive can be quite hard since the group of coredevelopers is usually small and without financial backing [17]. But they also need the help of thecommunity to debug, test, and maintain the software. Maintaining this community can be hard ifthe contributors do not feel that they can learn something new or take on challenges. Recognitionwas also a motivating factor for contributors as studies have shown that most programmers arelooking for social status [1].

2.4..1 Quality assessment of open-source software

Models and metrics are suitable methods to assess the quality of software [2]. To assess the qualityof software means to look at the different parts that make up the software, e.g. the documentationand the source code. But as mentioned before in the introduction, the lack of any standards makesthis quite challenging [18]. Models and metrics are not equally useful for everybody, e.g. modelsare more useful to system-designers than the end-user [2]. An advantage that metrics have isthat they can be used by anybody as they are just measurements. Some studies have shown thatover time as the software is undergoing development, the quality tends to decline due to increasedcomplexity. This means that it becomes harder to maintain the software. A lot of different factorscontribute to this loss of quality, such as lines of code, high coupling, or lower cohesion in methods[19].

2.5. Software tracing

Software tracing is a method of logging and recording execution data for a program. Some examplesof tracing software are Common Trace Format, SEGGER SystemView, and DTrace. Softwaretraces are useful for debugging as they contain valuable information about the execution of thesystem. One of the challenges with software tracing is getting valuable system information withoutaffecting the system in question. When a purely software-based tracing approach is used, it affectsthe execution of the software as the original source code is extended with tracing calls. This is

6


an important factor to take into consideration for real-time systems, as the system behavior ischanged. This is not the case with hardware-implemented tracing methods. [20]

2.5..1 Common Trace Format - CTF

Common Trace Format is a binary tracing format resembling C. It uses a declarative languagecalled Trace Stream Description Language (TSDL) and it allows for different stream layouts. Apacket contains:

• A header

• A context (which is optional)

• Zero or more events which call all contain a header, stream specific context (optional), anevent specific context (optional), and a payload.

The different data types that are supported in CTF are integers, null-terminated strings of bytes,and floats, among others. All of the mentioned information is taken from the homepage [21].

2.5..2 SEGGER SystemView

SEGGER SystemView is a program that can record and visualize what happens in the systemduring run-time. SystemView can record various events that happen in the system, such as Inter-rupt Service Routines, timers, and user-defined events, among others. According to the developer’shomepage, the system limits the amount of overhead introduced in the system by only recording asmall amount of data for each event. The developers also argue that when the system runs 10000events per second the overhead from SystemView is less than 1%. [22]

2.5..3 DTrace

DTrace is another tracing software that allows the user to trace all software according to theirwebsite. It uses the language D for writing scripts. DTrace stands for Dynamic Tracing and isreferring to that it can run and change tracing scripts during program execution. Another featurethat DTrace supports is static tracing, where the user defines trace-points before deployment. Byproviding filters that can filter the data in the kernel before the data is sent to the user, DTracecan reduce the overhead of presenting and gathering data. [23]

2.6. Babeltrace 2

Babeltrace 2 is the reference implementation of CTF [21, 24]. It is an open-source program thatconverts or processes traces. Babeltrace 2 is a completely rewritten version of the old library thatcomes with a lot of changes, some of these changes are full plugin support, Babeltrace 2 can runon any major operating system, and expanded certain features of the old library.

7


3. Related work

3.1. Hypervisors

J. Hwang et al. [25] compared four different hypervisors to each other. The researchers comparedthe different hypervisors with regards to CPU, memory, disk I/O, and network I/O. The resultsfrom the study showed that there was no clear indication of which hypervisor that was the best,rather that it depended on the workload. This was also the case in another study made by S.Shirinbab et al. [12] They compared three different type-1 hypervisors with each other when itcomes to CPU-utilization and Disc-utilization during different scenarios. The results showed thatthe three different hypervisors performed differently in each case and that there was no conclusiveevidence that one hypervisor was better than the others. This was not the results in the studyby S. Pawar and S. Singh [26], in which they compared VMware ESXi 6.0 and Citrix XenServer6.5. In their study, they compared the two hypervisors based on CPU, memory, disk, and systemup-time, on the guest operating system Microsoft Windows Server 2008 R2 (64-bit). The resultsshowed that the ESXi hypervisor was significantly better than XenServer.

J. Ruh and W. Steiner [27] compared the system latencies of the hypervisors: KVM, ACRNand, Xen RTDS. They evaluated each of the hypervisors based on four requirements, and theydetermined that none of the hypervisors available fulfilled all of the requirements. Based on this,the authors state that none of the hypervisors qualified as deterministic virtualization. Experimentsusing the RTEval tool was conducted to measure the system latency of each hypervisor. Eachhypervisor was tested twice, the first test involved running one VM, and the second test involvedrunning two VMs at the same time. Their results showed that KVM and ACRN had low averagesystem latencies, but could have peaks up to 1.968ms. On the other hand, Xen had higher averagesystem latency, but a maximum latency of 267µs. From these results, they drew the conclusionthat KVM and ACRN are not suitable for acting as control VMs because of the peaks in latency,almost reaching two milliseconds.

3.2. Quality assessment for open-source projects

Jospeh Tassone et al. [2] reviewed the body of research to find out which characteristics that affectthe quality of open-source projects the most. There are four main characteristics in which open-source projects (OSPs) differ from closed-source projects (CSPs): different levels of programmers,frequent beta releases, platform independence, and support for many run- and compile-time con-figurations. OSPs also differ from CSPs in regard to the development process as they do not usemore traditional processes such as the Waterfall model and that the development process changesover time as the project grows. An advantage OSPs have over CSPs is that they are available toanyone, which means that the larger the community around the project becomes, the more peoplecan help out. Two different approaches are used to measure quality: models and metrics. Butthese approaches are useful for different people in the project. Metrics is a useful tool to determineif there are problems within the project and can also be used to analyze errors. Models are usedto bring together different attributes related to quality to get a final result of the quality of thesoftware. An advantage of models is the utilization of metrics to establish a larger picture overthe process. But a disadvantage is that they can not be used by anyone, as they need to haveall the information needed to establish a working model. The authors conclude that both metricsand models are suitable for determining the quality of the software. Therefore, it is preferable toconcentrate effort on the idea of maintainability. Care should be taken when gathering metrics, asthere are a lot of tools that are not comparable with each other, and there are no standards setyet.

Other researchers have also looked at the differences between OSPs and CSPs but from anotherpoint of view. Mark Aberdour [1] wrote an article about what lessons can be learned and howthe development process is different between OSPs and CSPs. Some of the highlighted differencesbetween OSPs and CSPs were: effort put into planning, level of documentation, the methodologyregarding testing, peer review, and people management. Peer review was something that con-tributed a great deal to the quality of the code if the comments and the code got quick responses,

8


which helped keep the contributors interested. When it comes to managing the project, OSPshave a different mindset. In OSPs, the development process is more people-focused, which meanspeople volunteer to work on certain parts of the system. This results in a culture that fosterscreativity and innovation compared to more traditional development processes. The community ishighlighted as a key element for success in OSPs, as it is the community that builds new features,test and debug the system. But to sustain the community, the project must attract volunteeringprogrammers and motivate people to work by giving opportunities to take on challenges, learn newtechnology or tools. The author concludes that the study raised many questions but also foundsome guidelines for open-source software, some of which were: a large sustainable community, codemodularity, good documentation, rapid release cycles, and that the culture and the environmentare equally important as the system design.

3.3. Quality models for open-source projects

Normi Sham Awang Abu Bakar and Nadiah Arsat [16] studied which factors affected the quality ofopen-source systems the most. In the study, McCall’s Quality Factor Model was used to determinethe quality of two different open-source systems used by the Malaysian government: MyMeetingand MyTaskManager. The researchers also used six object-oriented measurements by Chidamberand Kemerer to measure quality. The PHP Depend tool was used to visualize the different statisticsof the two different systems. One result from the study was that there is no correlation between thenumber of lines of code and the depth of the inheritance tree. Another result was that the weightedmethods per class increased as the lines of code increased, which suggested to the researchers thatthe lines of code are a good indicator of complexity and coupling. This meant that the factors thataffected the quality the most are: correctness, maintainability, efficiency, and usability.

Won Jun Sung et al. [18] proposed in their work a quality model for selecting open-source software(OSS) components. This model determines the quality of an OSS component by looking at thedifferent characteristics of the component. The model has four main categories: functionality,reusability, usability and portability, and ten sub-categories. The researchers conclude that themodel provides basic measurements for the quality and that it is expected to be used to gatherpractical data.

3.4. Risk management for open-source projects

Xavier Franch et al. [3] propose a framework for risk management in OSS adoption processes.This proposed framework will provide the companies with tools and methods that are centeredaround a community-based development approach. Their framework builds upon the conceptsof business and software ecosystems. These different ecosystems handle different aspects of thedevelopment, the business ecosystem handles all things related to business such as production,distribution, and costs, while the software ecosystem handles all matters about software such aslicensing problems and release schedules. The platform provided by the framework is dividedinto two tiers: the decisional and the technological. The technological tier provides the necessaryobservations needed for the decisional tier to be able to make any decisions. One important dutyof the decisional tier is risk management, by utilizing the data provided from the technological tierand the business, it can detect potential risks related to the ecosystems. One of the most commonmistakes companies make when adopting OSS is having insufficient risk management, which canlead to costly failures. A scenario-based approach is planned to validate the framework.

Nguyen Duc Linh et al. [28] also studied the risks that come with using open-source software. Theresearchers studied different methods for assessing the risk involved with using OSS components ina system. Two of the different methods were: Qualitative and Quantitative analysis. In qualitativerisk assessment, each risk is evaluated based on the rate that they would occur and the severity.Some examples of tools that can help with qualitative risk management are the risk matrix andrisk breakdown structure. In quantitative risk assessment, a risk matrix is also used, which theproject manager uses. Risk assessment needs to be re-evaluated every time a major risk occurs,after phase end-reviews, and when the project needs to be re-planned. When a component has

9


been selected, different methods can be used to integrate it into the project, some examples thatthe researchers list are: having an expert as the supervisor, using a version control system, andbackup systems periodically. A case study was conducted in order to learn what difficulties thereare when integrating OSS components. The results from the case study showed that the communityfor Broadleaf was not strong, and Broadleaf’s products were not diverse. They also found that thecause for selecting unsuitable components came down to three different causes. The first cause wasthat the team had not fully defined their requirements. The second cause was that the team hadonly checked the homepage for Broadleaf and not checked the reviews of the system. The thirdand last cause was that the team had not compared Broadleaf to any similar products.

3.5. Summary

All of these different topics are connected to our work in at least one way. As mentioned earlierin the introduction text in the background section, both systems we have chosen to work withare open-source. There are various risks involved when selecting OSS and also the quality can bevarying as well. These risks can also be higher depending on the quality of the proposed softwaree.g. if the documentation is not understandable, then risks for failure might increase and thereforethe company might opt to select another software instead. One of the ways to assess the qualityof a software is to use metrics and models. These can be used to evaluate different parts of thesystem as well as different aspects such as portability, quality of documentation, and how easy thesystem is to use among others.

10


4. Method

We evaluated if Zephyr and ACRN can be used for real-time systems by conducting a series oftests on different configurations, e.g. different service VMs, clock frequencies, and tick-rates, whichcan be seen in Table 1. We ran the tests using two different clock frequency and tick-rate baselineconfigurations, these were 19.2 MHz with a tick-rate of 10000 ticks/s, and 25 MHz with a tick-rateof 100 ticks/s. These are the default settings when configuring Zephyr stand-alone and Zephyr forACRN during the build process. We then used these clock frequencies to calculate the differencebetween expected clock cycles taken and actual clock cycles taken. We used clock cycles as thesegave us the most accurate results, but also because many system functionalities like deadlines, andbusy-waits, etc. depend on clock cycles. As we have no expected value for the loop, we chose touse the Zephyr binary that was closest to having the correct cycle count (for sleep and busy-wait)as a baseline value for the loop. Strip means that the compiled executable does not contain debuginformation.

UART was used for serial communication and Common Trace Format (CTF) for creating traces.We opted to use CTF because it was readily available in Zephyr, and we did not have to buy anyadditional resources. We had planned to use SEGGER SystemView as it could present the resultsvery nicely. However, we could not find any information regarding if it was possible to use it onour board or not.

Configuration Clock frequency (MHz) Tick-rate (Ticks/s)ACRN Debian 19.2 10000

ACRN Debian Strip 19.2 10000ACRN Clearlinux 19.2 10000

ACRN Clearlinux Strip 19.2 10000ACRN Ubuntu 19.2 10000

ACRN Ubuntu Strip 19.2 10000Zephyr non-tickless 19.2 10000

Zephyr non-tickless Strip 19.2 10000Zephyr tickless 19.2 10000

Zephyr tickless Strip (Baseline for loop) 19.2 10000ACRN Debian 25 100

ACRN Debian Strip 25 100ACRN Clearlinux 25 100

ACRN Clearlinux Strip 25 100ACRN Ubuntu 25 100

ACRN Ubuntu Strip 25 100Zephyr non-tickless 25 100

Zephyr non-tickless Strip 25 100Zephyr tickless 25 100

Zephyr tickless Strip (Baseline for loop) 25 100

Table 1: Table showing the different configurations that we ran tests on.

The board we used is an UP Squared with an Apollo Lake Intel N3350 processor, 2 GB DDR4RAM, 32 GB eMMC. The versions of Zephyr and ACRN we have used are Zephyr v2.3.0 andACRN v2.0 with SDC as the scenario configuration. We created a test program which ran a singlethread containing three test functions. The functions that we tested were sleep, busy-wait, anda loop which ran for 100000 iterations. The sleep and busy-wait were configured so that theywould run for one second. To reduce the risk for errors, we ran the different functions four timeseach and averaged the results. This can be seen in Table 2, which shows an example using thebaseline configuration running the sleep function. During each test, we ran the test program forabout a minute before stopping the execution. While the test program was executing, we collectedthe tracing data in a text file. When we ran Zephyr directly on the UP Squared, we were ableto receive the CTF trace via the UART connection. However, when we ran ACRN, we could not

11


use the UART interface to capture the data, so we had to in the service VM pipe the CTF tracefrom a terminal to a text file. Because Zephyr printed out some extra text when it booted up thekernel, we had to remove it before we could read it with Babeltrace. To do this we used the sedcommand.

We created a set of timestamp functions in CTF to be able to see when a specific function hadbeen executed. These functions had a name that corresponded to the executed function e.g.ctf before sleep. We had to configure Zephyr to enable CTF, and we did so by going into aconfigurations menu when building the RTOS. In this menu, we had to go down to the optionTracing and enable CTF. We changed the size of the CTF buffer to 16 KB because we saw that insome cases there were some gaps in the trace. We also changed the tracing thread waiting periodto zero as this lead to a more responsive tracing experience. CTF produces a binary output, andwe used Babeltrace to read it. The default format for each timestamp is hh:mm:ss:ns, and becauseof this, we used a flag that converted them into clock cycles instead. Then to calculate the timetaken for each function, we calculated the difference between the before and after timestamps, andthis result was divided by the expected cycle count for each tested function, which can be seen inTable 3. This results in a number that describes how many times longer it took to execute eachfunction on every configuration compared to the baseline.

Before sleep After sleep Difference269221246 288424130 19202884288424317 307627332 19203015307627512 326829871 19202359326830052 346032877 19202825

Table 2: Example showing how the data was collected. This is the sleep function running onZephyr Strip tickless.

Configuration Sleep duration (cycles) More cycles taken relative to expected valueZephyr tickless Strip 19202770,75 1,00014431

ACRN Debian 7553942456 393,4345029ACRN Clearlinux 5961251437 310,4818457

Table 3: Calculated differences from the expected values.

12


5. Ethical considerations

In this work, there are no ethical considerations needed to be taken into account because the dataor the results will not be related to any company or person.

13


6. Implementation and problems

In this section, we will talk about the installation of Zephyr and ACRN, the various problems wehad during the installation phase. We will also talk about the solutions that we found to certainproblems and also how some problems are still unresolved.

6.1. Zephyr problems

This section will describe the problems we had with Zephyr.

• The sleep function does not sleep for the specified amount of time.

• The busy-wait function does not work with every configuration.

• Installing Zephyr when following the guide from the documentation.

• UART support for earlier versions for tracing data.

When compiling the system, it is possible to specify whether the system should use a tickless kerneland idle. Tickless idle disables periodic system clock interrupts when the kernel is idle. Ticklesskernel disables periodic clock system generation and turns the kernel into an event-driven kernel.We built the binaries with both tickless enabled and disabled and measured the time it took toperform functions such as sleep and busy-wait.

One of the problems we had was timing related when using the sleep and busy-wait functions. InZephyr v2.2.99 (Tickless), the sleep was shorter than specified. This problem seems to be fixedin v2.3 (Tickless). However, when tickless is not used, v2.2.99 would sleep for a shorter time andv2.3 for a longer time than specified. The busy-wait function seems to work with tickless enabledand disabled for v2.3, but only with tickless enabled for v2.2.99. When disabled, the thread wouldstop responding, and after a while, the system enters the idle state without finishing the remaininginstructions in the thread.

We also had problems installing Zephyr SDK v2.3 on another computer, following the guide in thedocumentation. The problem was related to an environment variable that we had to set, but thiswas missing from the guide. The solution to this problem was to go back to the guide for version2.2, and in that guide, that step was explained.

Due to UART support being introduced in v2.2, we have not been able to test these functions onearlier versions of Zephyr. This is because we rely on UART to get our tracing data. This wouldnot have been a problem if we had an alternative way of getting these data.

6.2. ACRN problems

This section will describe the problem we had using ACRN.

• Efibootmgr not saving the boot order correctly.

• Huge time differences in functions compared to running only Zephyr.

• Documentation and configuration files.

One step in the installation process of the hypervisor is to create a custom entry efibootmgr inLinux. The entry for the hypervisor is required to be the first entry, or the hypervisor will not beinitialized correctly during boot. The problem we had is that changes such as removing entries orchanging the order of them would not persist when rebooting. Because of this, we had to spendtime changing the boot order and rebooting until the board managed to save it. This was notdocumented when we started our work but was later found by another person that posted a pullrequest and updated the documentation to include this. Later we would find a solution to thisproblem that worked on our board (N3350). In the BIOS for the board, there is a setting called

14


UEFI harddisk drive BBS priorities that allowed us to force a change to the boot order. Thisallowed the hypervisor to initialize correctly on the first try when installing the hypervisor.

When running our test program on Zephyr using ACRN as the hypervisor, we noticed huge timedifferences in the tested functions compared to running Zephyr standalone. With one setup, thesleep function took ≈310.44 times as many cycles, the busy-wait ≈1.11 times and our simple loop≈27.35 times. Then with another setup, the sleep took ≈2.97 times as many cycles, while thebusy-wait and loop function took ≈1.56 times and ≈27.77 times. We measured the current clockcycle before and after each function and calculated the difference. This difference was then dividedby the cycles expected for the sleep and busy-wait functions, while using Zephyr v2.3 tickless asour baseline for the loop. We chose this as our baseline since the cycles reported for each functionclosely matched the cycles we expected it to take.

When we noticed this cycle difference, we posted an issue on the ACRN GitHub. We receivedan answer from one of the developers asking us about which ACRN and Zephyr version we wereusing. We were running acrn-2019w35.1-140000p with Zephyr 2.2.99. This was the only versionof ACRN that we had managed to install on our board. The developer then told us that theyhad only verified Zephyr v2.1 on ACRN v2.0 and asked us if we could test it again using theseversions. However, because of an update to ACRN, our board (N3350) was no longer the defaultboard when building, because it had been changed to the newer (N4200). This led to a problembecause the configuration files needed to build the hypervisor on our board for our scenario didnot exist. After discussing this, we received a configuration file which made it possible for us tobuild v2.0 for the N3350 board. But we could still not get measurements for Zephyr v2.1 since itlacked the UART support we relied on to get our trace measurements. Because of this, we had touse newer versions of Zephyr. Since the hypervisor was up and running, we wanted to get backto solving our original problem, posted on July 2nd, 2020, about the cycle differences. But as ofwriting this, we have yet to receive a response.

When installing ACRN, we had problems following some of the documentation. The guide usedfor our board (N3350) referenced the installation guide for another board (nuc7i7dnb) and notedchanges that had to be made in the installation process. This could make it difficult to keep trackof what changes had to be made and when to make these changes. There was also no mentionof the flag that had to be used in order to build the hypervisor for the UP Squared board (BothN3350 and N4200). We had to check the makefiles in order to find this flag. As mentioned before,we were told that Zephyr v2.1 had only been verified for ACRN v2.0. According to the developeranswering us in the GitHub issue, it was also only verified to run with the logical partition scenario.However, the documentation states that the only scenario that has been verified for Up Squared(both N3350 and N4200) is SDC with 2 VMs. The N3350 board did have scenario configurationfiles for logical partition mode but not SDC, while N4200 had configuration files for both.

15


6.3. Test program implementation

The pseudo-code for the program used to take all our measurements can be seen in Listing 1. Theactual code can be found in Appendix B, Listing 2. This program consists of two loops that testthe API functions sleep and busy-wait. There is also a loop doing a simple calculation. At thestart and end of each iteration, we measure the current system clock cycle by taking a timestamp.These timestamps are then used in order to calculate the number of cycles each function took sowe can compare that to the cycles they are expected to take given the system configuration.

1 void func ( )2 {3 i n t i = 4 ;45 // S leep6 whi le ( i−−) {7 t imes tamp cur r ent cyc l e ( ) ;8 s l e e p (1SECOND) ;9 t imes tamp cur r ent cyc l e ( ) ;

10 }1112 i = 4 ;13 // Busy−wait14 whi l e ( i−−) {15 t imes tamp cur r ent cyc l e ( ) ;16 busy wait (1SECOND) ;17 t imes tamp cur r ent cyc l e ( ) ;18 }1920 i = 4 ;21 // Loop22 whi l e ( i−−) {23 t imes tamp cur r ent cyc l e ( ) ;24 f o r ( i n t j = 0 ; j < 100000; j++) {25 v o l a t i l e i n t a = 2 ;26 a = a ∗ a ;27 }28 t imes tamp cur r ent cyc l e ( ) ;29 }30 }

Listing 1: Pseudo-code of the implementation .

16


7. Results

From our results, we can see that depending on the configuration, especially with regards to clockfrequency and tick-rate, the results changed when running ACRN. In Table 4, we can see thatthe sleep function took between 310 to 465 times more cycles than the expected number of cycles,while the busy-wait took barely any extra cycles at all. Finally, the loop took 27 times more cyclesthan the baseline. These results were taken when the system was running with a clock frequencyof 19.2 MHz and a tick rate of 10000 ticks/s.

Configuration Sleep Busy-Wait LoopACRN Debian 393,4345029 1,070640299 27,33158789

ACRN Debian Strip 402,8886401 1,111302643 27,32981885ACRN Clearlinux 310,4818457 1,107642656 27,34719077

ACRN Clearlinux Strip 307,2372667 1,000026185 27,08388763ACRN Ubuntu 447,1061502 1,076953177 27,21614067

ACRN Ubuntu Strip 465,5921071 1,101838633 27,33703651

Table 4: Difference from the baseline when running the test program on ACRN with a clockfrequency of 19.2 MHz and a tick rate of 10000 ticks/s.

Interestingly enough, when the clock frequency was changed to 25 MHz and the tick rate to 100ticks/s, the results were dramatically different from previous results, which can be seen in Table 5.When ran under this setup, we saw that the sleep function took around 2.25 to 3.98 times morecycles, the busy-wait did not change much (1.18 as maximum compared to 1.11 as the maximumfor 19.2 MHz), and the loop took around 41 times more cycles than the baseline.

Configuration Sleep Busy-Wait LoopACRN Debian 3,92751806 1,08976253 40,62336083

ACRN Debian Strip 3,65007764 1,0000202 41,04018613ACRN Clearlinux 2,28071853 1,19650823 41,05499154

ACRN Clearlinux Strip 2,24608165 1,08700284 40,80430415ACRN Ubuntu 2,66058113 1,18279499 40,67364636

ACRN Ubuntu Strip 3,98172415 1,0462964 40,62801396

Table 5: Difference from the baseline when running the test program on ACRN with a clockfrequency of 25 MHz and a tick rate of 100 ticks/s.

When running Zephyr directly on the board using a clock frequency of 19.2 MHz and a tick-rateof 10000 ticks/s, the cycle count for each test function did not differ that much from the expectedvalues, with the exception of the sleep function. When running Zephyr in non-tickless mode, thesleep function slept for about 40% more cycles than expected, but this did not happen when usingthe tickless mode. This can be seen in Table 6. But when the clock frequency and tick-ratewere changed to 25 MHz and 100 ticks/s, the results were a little different. The sleep functionactually woke up around 30% cycles earlier than expected, which can be seen in Table 7. Thecycles taken for the busy-wait and loop did not differ much between different configurations andnot even between the different clock frequencies and tick-rates.

Configuration Sleep Busy-Wait LoopZephyr non-tickless 1,405909128 1,000016745 1,061562412

Zephyr non-tickless Strip 1,405874466 1,000018867 1,060005661Zephyr tickless 1,000152891 1,000017135 0,999823097

Zephyr tickless Strip 1,00014431 1,000017526 1

Table 6: Difference from the baseline when running the test program on Zephyr directly on theboard with a clock frequency of 19.2 MHz and a tick rate of 10000 ticks/s.

17


Configuration Sleep Busy-Wait LoopZephyr non-tickless 0,69324044 1,00000043 1,000105753

Zephyr non-tickless Strip 0,6932399 1,00000043 1,000052876Zephyr tickless 0,69326735 1,00000042 1

Zephyr tickless Strip 0,69326683 1,00000044 1

Table 7: Difference from the baseline when running the test program on Zephyr directly on theboard with a clock frequency of 25 MHz and a tick rate of 100 ticks/s.

It is easier to see the differences in cycles taken when looking at a bar chart. We have separatedeach function and system configuration into its own charts, which can be seen below. Since thecycle difference was so large in the Sleep Difference 19.2 MHz chart, we had to use a logarithmicscale for the X-axis. This is not the case for the other charts.

1 10 100 1000

ACRN Debian

ACRN Debian Strip

ACRN Clearlinux

ACRN Clearlinux Strip

ACRN Ubuntu

ACRN Ubuntu Strip

Zephyr

Zephyr Strip

Zephyr Tickless

Zephyr Tickless Strip

Sleep Difference (19200000)

Figure 4: Trace results of Sleep compared to the baseline in a logarithmic scale (x-axis).

0 0,5 1 1,5 2 2,5 3 3,5 4 4,5

ACRN Debian

ACRN Debian Strip

ACRN Clearlinux


ACRN Ubuntu

ACRN Ubuntu Strip

Zephyr

Zephyr Strip

Zephyr Tickless


Sleep Difference (25000000)

Figure 5: Trace results of Sleep compared to the baseline.

18


0,94 0,96 0,98 1 1,02 1,04 1,06 1,08 1,1 1,12

ACRN Debian

ACRN Debian Strip

ACRN Clearlinux


ACRN Ubuntu

ACRN Ubuntu Strip

Zephyr

Zephyr Strip

Zephyr Tickless


Busy Wait Difference (19200000)

Figure 6: Trace results of Busy-wait compared to the baseline.

0,9 0,95 1 1,05 1,1 1,15 1,2 1,25

ACRN Debian

ACRN Debian Strip

ACRN Clearlinux


ACRN Ubuntu

ACRN Ubuntu Strip

Zephyr

Zephyr Strip

Zephyr Tickless


Busy Wait Difference (25000000)

Figure 7: Trace results of Busy-wait compared to the baseline.

19


0 5 10 15 20 25 30

ACRN Debian

ACRN Debian Strip

ACRN Clearlinux


ACRN Ubuntu

ACRN Ubuntu Strip

Zephyr

Zephyr Strip

Zephyr Tickless


Loop Difference (19200000)

Figure 8: Trace results of Loop compared to the baseline.

0 5 10 15 20 25 30 35 40 45

ACRN Debian

ACRN Debian Strip

ACRN Clearlinux


ACRN Ubuntu

ACRN Ubuntu Strip

Zephyr

Zephyr Strip

Zephyr Tickless


Loop Difference (25000000)

Figure 9: Trace results of Loop compared to the baseline.

20


8. Discussion

Many of the problems encountered during the installation of these systems stem from the docu-mentation being somewhat unclear at times. This lead to it being hard to figure out what couldhave gone wrong when the system did not function correctly. Also, the biggest hurdle was findingout how to consistently be able to install or re-install the hypervisor while also managing to get itto boot correctly every time. Most of the problems related to the documentation are easily fixedby just adding the missing information to the existing pages. This would not only make it easierto use the systems but also increase the quality of the documentation.

The differences in cycles taken between running Zephyr on the board directly and when run viaACRN are unexpectedly large. This suggests to us that there is a problem either with runningZephyr via ACRN or them running on our board. We did not have access to the other supportedboards for ACRN to make similar tests. Another problem for us was that Zephyr standalone wasnot supported on the NUC-line of boards that ACRN supported. Because of this, we could notperform any comparisons between the boards even if we had them.

When measuring the sleep duration on one ACRN configuration, the cycle count was very largecompared to another configuration (the highest being ≈465 times more cycles than expected). Forthe busy-wait function, the difference to the baseline was not that large, with ≈1.19 times morecycles than expected being the highest. The basic loop behaved differently depending on whichclock frequency and tick-rate we used. when running under ACRN with a clock frequency of 19.2MHz and a tick-rate of 10000 ticks/s, the loop took 27 times more clock cycles than expected onall configurations. But when the clock frequency was changed to 25 MHz and the tick-rate to 100ticks/s, the loop took around 41 times more clock cycles than expected on all configurations. Thiscan lead to problems since the system does not seem to behave as one would expect. The differentfunctions do not seem to exhibit the same behavior when it comes to the increases and decreasesin cycles taken to perform them. That is, the cycles taken for the sleep, busy-wait, and loop donot increase for all of them on one configuration. Rather sometimes one performs better on oneconfiguration and worse on another independently from each other.

We know that software-based tracing methods affect the execution on the hardware by addingtracing functions in the source code. However, we do not know how much CTF affects our testprogram when it is run on the systems. That is why we think it would be good to investigate howother tracing methods affect the results.

21


9. Conclusions

We have answered our research question about the performance of common scheduling API methodswhen run on Zephyr and ACRN by performing a series of tests. According to our results, the currentversions of Zephyr and ACRN are not ready to be used for real-time systems, at least not on theUP Squared board. This is because depending on the configuration, the time taken to performeach function varied greatly. When ACRN ran Zephyr using a 19.2 MHz clock frequency and atick-rate of 10000 ticks/s, the highest results for the sleep function was 465 times more cycles thanthe baseline, while the lowest result was 307 times more cycles, which is a substantial disparity.While the busy-wait ran for almost the specified time, producing results that only varied a littlebit, with the lowest being 1.000026 times more cycles and the highest being 1.11 times more cycles.The loop took around 27 times more cycles than expected, which is a very unexpected outcomehere. However, when the clock frequency was changed to 25 MHz and a tick rate of 100 ticks/s,the results became closer to the expected amount of time, but still off by a large margin. The sleepfunction on this configuration took around two to four times more cycles than the baseline, and thebusy-wait produced similar when running these settings (1.000020 - 1.19 times more cycles than thebaseline). The time it took for the loop did not change much between the different configurations,with 40.623 times more cycles being the lowest and 41.05 times more cycles being the highest.From these results, we can clearly see that the system does not behave in a deterministic manner,which is one of the more important attributes for a real-time system.

When we ran Zephyr directly on the board, the results for the busy-wait and the loop did not changemuch between the different configurations, and clock frequency, and tick-rate. The sleep function,however, changed behavior completely between the different setups. When running Zephyr non-tickless with a clock of 19.2 MHz and 10000 ticks/s, the sleep function slept for too long, around40%, but this did not happen in tickless mode. When the clock frequency was changed to 25 MHzand tick-rate to 100 ticks/s, the sleep function slept around 30% less than expected. This is alsosomething that is considered non-deterministic behavior.

We would argue that based on the problems we encountered during the installation of each systemthat the documentation for both systems needs more work in order to make them clearer andunderstandable. For both systems, the documentation is still usable but can be hard to follow oruse. However, some sections are missing some crucial information that needs to be added.

22


References

[1] M. Aberdour, “Achieving quality in open-source software,” IEEE Software, vol. 24, no. 1, pp.58–64, 2007.

[2] J. Tassone, S. Xu, C. Wang, J. Chen, and W. Du, “Quality assessment of open source software:A review,” in 2018 IEEE/ACIS 17th International Conference on Computer and InformationScience (ICIS), 2018, pp. 411–416.

[3] J. Franch Gutierrez, A. Susi, M. C. Annosi, C. P. Ayala Martınez, R. Glott, D. Gross,R. Kenett, F. Mancinelli, P. Ramsany, C. Thomas et al., “Managing risk in open sourcesoftware adoption,” in ICSOFT 2013: Proceedings of the 8th International Joint Conferenceon Software Technologies, 2013, pp. 258–264.

[4] O. Osanaiye, S. Chen, Z. Yan, R. Lu, K. R. Choo, and M. Dlodlo, “From cloud to fogcomputing: A review and a conceptual live vm migration framework,” IEEE Access, vol. 5,pp. 8284–8300, 2017.

[5] H. Li, X. Xu, J. Ren, and Y. Dong, “Acrn: A big little hypervisor foriot development,” in Proceedings of the 15th ACM SIGPLAN/SIGOPS InternationalConference on Virtual Execution Environments, ser. VEE 2019. New York, NY,USA: Association for Computing Machinery, 2019, p. 31–44. [Online]. Available:https://doi-org.ep.bib.mdh.se/10.1145/3313808.3313816

[6] Intel R© testimonial: Introducing acrnTM: A big little hypervisor for iot development.[Online]. Available: https://projectacrn.org/intel-testimonial/

[7] Zephyr project: Introduction. [Online]. Available: https://docs.zephyrproject.org/latest/introduction/index.html

[8] Time-critical edge compute. [Online]. Available: https://wiki.akraino.org/display/AK/Time-Critical+Edge+Compute

[9] J. Liang, F. Liu, S. Li, and Z. Cai, “A comparative research on open source edge computingsystems,” in Artificial Intelligence and Security, X. Sun, Z. Pan, and E. Bertino, Eds. Cham:Springer International Publishing, 2019, pp. 157–170.

[10] K. Bilal and A. Erbad, “Edge computing for interactive media and video streaming,” in 2017Second International Conference on Fog and Mobile Edge Computing (FMEC), 2017, pp. 68–73.

[11] N. Alam, “Survey on hypervisors,” Indiana University, Bloomington, School of Informaticsand Computing, 2009.

[12] S. Shirinbab, L. Lundberg, and D. Ilie, “Performance comparison of kvm, vmware andxenserver using a large telecommunication application,” in Cloud Computing. IARIA XPSPress, 2014.

[13] V. Timcenko, B. Djordjevic, S. B. Rakas, and N. Davidovic, “Performance examinationof type-2 hypervisors: Case of particular database application in a virtual environment,”in Proceedings of the International Conference on Information Systems and Design ofCommunication, ser. ISDOC ’14. New York, NY, USA: Association for ComputingMachinery, 2014, p. 122–126. [Online]. Available: https://doi-org.ep.bib.mdh.se/10.1145/2618168.2618187

[14] Wikipedia: Hypervisor. [Online]. Available: https://en.wikipedia.org/wiki/Hypervisor

[15] Introduction to project acrn. [Online]. Available: https://projectacrn.github.io/latest/introduction/index.html

23

https://doi-org.ep.bib.mdh.se/10.1145/3313808.3313816

https://projectacrn.org/intel-testimonial/

https://docs.zephyrproject.org/latest/introduction/index.html

https://docs.zephyrproject.org/latest/introduction/index.html

https://wiki.akraino.org/display/AK/Time-Critical+Edge+Compute

https://wiki.akraino.org/display/AK/Time-Critical+Edge+Compute



https://en.wikipedia.org/wiki/Hypervisor

https://projectacrn.github.io/latest/introduction/index.html

https://projectacrn.github.io/latest/introduction/index.html


[16] N. S. Awang Abu Bakar and N. Arsat, “Investigating the factors that influence the quality ofopen source systems,” in The 5th International Conference on Information and Communica-tion Technology for The Muslim World (ICT4M), 2014, pp. 1–6.

[17] G. Avelino, E. Constantinou, M. T. Valente, and A. Serebrenik, “On the abandonment andsurvival of open source projects: An empirical investigation,” in 2019 ACM/IEEE Interna-tional Symposium on Empirical Software Engineering and Measurement (ESEM), 2019, pp.1–12.

[18] W. J. Sung, J. H. Kim, and S. Y. Rhew, “A quality model for open source software selection,”in Sixth International Conference on Advanced Language Processing and Web InformationTechnology (ALPIT 2007), 2007, pp. 515–519.

[19] G. Singh, “Metrics for measuring the quality of object-oriented software,” SIGSOFTSoftw. Eng. Notes, vol. 38, no. 5, p. 1–5, Aug. 2013. [Online]. Available: https://doi-org.ep.bib.mdh.se/10.1145/2507288.2507311

[20] C. Koehler, A. Mayer, and M. Wurm, “Combined hardware and software tracing of real andvirtual embedded system parts,” in Proceedings of the 19th International Conference MixedDesign of Integrated Circuits and Systems - MIXDES 2012, 2012, pp. 340–345.

[21] The - common trace format. [Online]. Available: https://diamon.org/ctf/#specification

[22] Systemview. [Online]. Available: https://www.segger.com/products/development-tools/systemview/

[23] About dtrace. [Online]. Available: http://dtrace.org/blogs/about/

[24] Babeltrace - a trace manipulation toolkit. [Online]. Available: https://babeltrace.org/#bt2-get

[25] J. Hwang, S. Zeng, F. y. Wu, and T. Wood, “A component-based performance comparisonof four hypervisors,” in 2013 IFIP/IEEE International Symposium on Integrated NetworkManagement (IM 2013), May 2013, pp. 269–276.

[26] S. Pawar and S. Singh, “Performance comparison of vmware and xen hypervisor on guest os,”IJICSE, vol. 2, pp. 56–60, 2015.

[27] J. Ruh and W. Steiner, “The need for deterministic virtualization in the industrial internetof things,” in Proceedings of the Workshop on Fog Computing and the IoT, ser. IoT-Fog’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 26–30. [Online].Available: https://doi.org/10.1145/3313150.3313222

[28] N. D. Linh, P. D. Hung, V. T. Diep, and T. D. Tung, “Risk management inprojects based on open-source software,” in Proceedings of the 2019 8th InternationalConference on Software and Computer Applications, ser. ICSCA ’19. New York, NY,USA: Association for Computing Machinery, 2019, p. 178–183. [Online]. Available:https://doi-org.ep.bib.mdh.se/10.1145/3316615.3316648

24



https://diamon.org/ctf/#specification

https://www.segger.com/products/development-tools/systemview/

https://www.segger.com/products/development-tools/systemview/

http://dtrace.org/blogs/about/

https://babeltrace.org/#bt2-get

https://babeltrace.org/#bt2-get

https://doi.org/10.1145/3313150.3313222



A Appendix A: Raw cycle count data

Here is the raw data containing all the cycle counts measure before and after the function aswell as the difference between them. Each section contains all the measurements taken for eachconfiguration. Because of the size of these tables we could not fit them on this page. You can findthem on the next pages.

25


1.1.

ACRN

Debian

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce23

3183

7746

9956

2033

2776

2436

5581

32547609462

32566809949

19200487

32629845052

32630042612

197560

9956

2037

6517

6339

9029

876

7778

6533

32566812503

32586066888

19254385

32630042858

32630233364

190506

1763

3990

769

2517

3830

035

7539

8392

6632586069522

32607954920

21885398

32630233531

32630424102

190571

2517

3830

529

3254

7608

971

7373

7784

4232607957508

32629842413

21884905

32630424257

32630618120

193863

Tab

le8:

Con

figu

rati

on

:1000

tick

s,19200000H

z

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce27

2325

7699

2843

5037

0012

0246

001

3116011401

3141011896

25000495

3224998210

3225188869

190659

2843

5041

3229

4092

6060

9742

1928

3141014485

3174989260

33974775

3225189064

3225382166

193102

2940

9265

4930

2405

8765

8313

2216

3174991937

3199992416

25000479

3225382392

3225576416

194024

3024

0592

4731

1601

0908

9195

1661

3199994999

3224995503

25000504

3225576581

3225767065

190484

Tab

le9:

Con

figu

rati

on

:100

tick

s,25000000H

z

1.2.

ACRN

Debian

strip

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce27

8738

1016

1039

3517

079

7606

1360

6333729230327

33748922642

19692315

33814588521

33814786059

197538

1039

3517

549

1814

6230

790

7752

7132

4133748925191

33770810604

21885413

33814786311

33814976818

190507

1814

6231

088

2615

7270

444

8011

0393

5633770813142

33792697796

21884654

33814976984

33815167546

190562

2615

7270

947

3372

9229

849

7571

9589

0233792700334

33814585995

21885661

33815167702

33815361545

193843

Tab

le10:

Con

figu

rati

on

:1000

tick

s,19200000H

z

26


Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce13

1043

782

2192

4277

488

1989

92485192513

510192945

25000432

593339311

593533272

193961

2192

4325

731

2815

591

9357

2334

510195545

543330402

33134857

593533466

593726574

193108

3128

1607

339

3939

597

8112

3524

543333020

568333546

25000526

593726799

593917285

190486

3939

4006

348

5192

004

9125

1941

568336156

593336661

25000505

593917448

594111486

194038

Tab

le11:

Con

figu

rati

on

:100

tick

s,25000000H

z

1.3.

ACRN

Clearlinux

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce40

5264

3619

1006

7686

159

6015

0425

4027897651281

27917242022

19590741

27982728720

27982922850

194130

1006

7686

648

1601

7405

535

5949

7188

8727917244650

27939125798

21881148

27982923118

27983120959

197841

1601

7405

991

2197

5320

019

5957

9140

2827939128384

27960815847

21687463

27983121132

27983311618

190486

2197

5320

497

2789

7650

789

5922

3302

9227960818427

27982726031

21907604

27983311852

27983502336

190484

Tab

le12:

Con

figu

rati

on

:1000

tick

s,19200000H

z

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce23

5861

7710

2420

8904

7762

2727

672586691212

2611691699

25000487

2706352232

2706

550260

198028

2420

8909

5924

7617

5571

5528

4612

2611694205

2647492426

35798221

2706550511

2706

744563

194052

2476

1760

5825

3144

8463

5527

2405

2647495038

2672495590

25000552

2706744822

2706

935309

190487

2531

4488

8225

8669

0951

5524

2069

2672498010

2706349573

33851563

2706935472

2707

129337

193865

Tab

le13:

Con

figu

rati

on

:100

tick

s,25000000H

z

27


1.4.

ACRN

Clearlinuxstrip

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce22

1180

2961

8217

8763

5460

0607

3393

25807626921

25826827408

19200487

25884438802

25884629455

190653

8217

8768

0813

9021

8575

856

8430

8950

25826829843

25846030356

19200513

25884629658

25884823534

193876

1390

2186

220

1982

9988

948

5927

8027

2825846032841

25865233308

19200467

25884823690

25885014176

190486

1982

9989

393

2580

7626

408

5977

6370

1525865235807

25884436351

19200544

25885014410

25885204894

190484

Tab

le14:

Con

figu

rati

on

:1000

tick

s,19200000H

z

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce31

7551

1020

3230

5159

4555

0049

253400121141

3425121656

25000515

3508831530

3509

025560

194030

3230

5164

2932

8578

1655

5526

5226

3425124117

3458822845

33698728

3509025772

3509

218913

193141

3285

7821

0933

4430

7144

5852

5035

3458825536

3483826056

25000520

3509219173

3509

409661

190488

3344

3076

4034

0012

0619

5581

2979

3483828548

3508829069

25000521

3509409824

3509

603856

194032

Tab

le15:

Con

figu

rati

on

:100

tick

s,25000000H

z

1.5.

ACRN

Ubuntu

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce25

7593

4843

1126

6979

285

8691

0444

4236913689212

36932889636

19200424

36996409742

36996603931

194189

1126

6979

806

1983

3180

637

8566

2008

3136932892179

36952631521

19739342

36996604191

36996794773

190582

1983

3181

140

2839

5986

880

8562

8057

4036952634161

36974518818

21884657

36996794960

36996988943

193983

2839

5987

344

3691

3688

669

8517

7013

2536974521396

36996406977

21885581

36996989135

36997179618

190483

Tab

le16:

Con

figu

rati

on

:1000

tick

s,19200000H

z

28


Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce39

6526

5973

4038

7109

6973

4449

964231326108

4264207924

32881816

4349616521

4349

810850

194329

4038

7114

9441

0556

4549

6685

3055

4264210717

4289211214

25000497

4349811121

4350

001645

190524

4105

5650

5241

7067

1225

6510

6173

4289213906

4314214448

25000542

4350001875

4350

195759

193884

4170

6716

9242

3132

5581

6065

3889

4314217102

4349613746

35396644

4350195924

4350

386407

190483

Tab

le17:

Con

figu

rati

on

:100

tick

s,25000000H

z

1.6.

ACRN

Ubuntu

strip

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce69

9063

043

9519

8696

9588

2080

6652

36456538850

36475739278

19200428

36541170036

36541367689

197653

9519

8701

9818

7956

5901

792

7578

8819

36475741713

36497392692

21650979

36541367936

36541558525

190589

1879

5659

522

2752

7326

353

8731

6668

3136497395253

36519280279

21885026

36541558682

36541749184

190502

2752

7326

842

3645

6538

368

8929

2115

2636519282795

36541167569

21884774

36541749335

36541943245

193910

Tab

le18:

Con

figu

rati

on

:1000

tick

s,19200000H

z

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce23

4048

9948

2447

4731

6010

6983

212

2738664406

2763665038

25000632

2843304546

2843497813

193267

2447

4736

2425

5527

2179

1077

9855

52763667636

2788668139

25000503

2843498055

2843692183

194128

2555

2727

2426

4556

0543

9028

7819

2788670706

2818298701

29627995

2843692384

2843882866

190482

2645

5610

8327

3866

3912

9310

2829

2818301374

2843301884

25000510

2843883025

2844073505

190480

Tab

le19:

Con

figu

rati

on

:100

tick

s,25000000H

z

29


1.7.

Zephyr B

efor

esl

eep

Aft

ersl

eep

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce28

6466

711

3134

6883

027

0021

19394441257

413641611

19200354

471243375

471250884

7509

3134

6901

134

0450

251

2698

1240

413641901

432842209

19200308

471251073

471258577

7504

3404

5043

436

7437

604

2698

7170

432842389

452042717

19200328

471258757

471266261

7504

3674

3778

539

4441

077

2700

3292

452042899

471243195

19200296

471266442

471273929

7487

Tab

le20:

Con

figu

rati

on

:1000

tick

s,19200000H

z

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce25

0620

983

2679

1227

617

2912

93319945073

344945088

25000015

419945151

419949881

4730

2679

1228

828

5256

507

1734

4219

344945097

369945106

25000009

419949890

419954618

4728

2852

5651

730

2600

765

1734

4248

369945114

394945123

25000009

419954625

419959353

4728

3026

0077

531

9945

059

1734

4284

394945132

419945142

25000010

419959360

419964088

4728

Tab

le21:

Con

figu

rati

on

:100

tick

s,25000000H

z

1.8.

Zephyrstrip

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce28

8952

972

3159

6887

627

0159

04396924860

416125209

19200349

473727037

473734537

7500

3159

6905

734

2942

681

2697

3624

416125389

435325845

19200456

473734717

473742182

7465

3429

4286

436

9930

499

2698

7635

435326030

454526394

19200364

473742363

473749854

7491

3699

3068

039

6924

676

2699

3996

454526575

473726855

19200280

473750034

473757538

7504

Tab

le22:

Con

figu

rati

on

:1000

tick

s,19200000H

z

30


Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce33

9710

338

3570

0155

017

2912

12409034375

434034389

25000014

509034450

509039180

4730

3570

0156

237

4345

842

1734

4280

434034397

459034407

25000010

509039189

509043917

4728

3743

4585

339

1690

151

1734

4298

459034414

484034424

25000010

509043924

509048652

4728

3916

9016

140

9034

361

1734

4200

484034432

509034441

25000009

509048659

509053386

4727

Tab

le23:

Con

figu

rati

on

:100

tick

s,25000000H

z

1.9.

Zephyrtick

less

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce33

5219

875

3544

2342

719

2035

52412032340

431232666

19200326

488834383

488841450

7067

3544

2360

937

3627

685

1920

4076

431232848

450433206

19200358

488841631

488848696

7065

3736

2786

539

2830

326

1920

2461

450433387

469633740

19200353

488848877

488855939

7062

3928

3050

741

2032

160

1920

1653

469633923

488834202

19200279

488856120

488863185

7065

Tab

le24:

Con

figu

rati

on

:1000

tick

s,19200000H

z

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce22

8650

413

2459

4239

717

2919

84297977195

322977209

25000014

397977268

397981998

4730

2459

4240

926

3287

313

1734

4904

322977218

347977227

25000009

397982007

397986734

4727

2632

8732

328

0632

269

1734

4946

347977235

372977245

25000010

397986742

397991470

4728

2806

3227

929

7977

180

1734

4901

372977252

397977261

25000009

397991477

397996204

4727

Tab

le25:

Con

figu

rati

on

:100

tick

s,25000000H

z

31


1.10.

Zephyrtick

less

strip

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce26

9221

246

2884

2413

019

2028

84346033059

365233333

19200274

422835133

422842201

7068

2884

2431

730

7627

332

1920

3015

365233514

384433874

19200360

422842382

422849448

7066

3076

2751

232

6829

871

1920

2359

384434057

403634409

19200352

422849629

422856694

7065

3268

3005

234

6032

877

1920

2825

403634589

422834949

19200360

422856874

422863939

7065

Tab

le26:

Con

figu

rati

on

:1000

tick

s,19200000H

z

Bef

ore

slee

pA

fter

slee

pD

iffer

ence

Bef

ore

Loop

Aft

erL

oop

Diff

eren

ceB

efore

Loop

Aft

erL

oop

Diff

eren

ce22

5924

510

2432

1643

017

2919

20295251242

320251256

25000014

395251316

395256046

4730

2432

1644

326

0561

350

1734

4907

320251265

345251275

25000010

395256055

395260782

4727

2605

6136

027

7906

313

1734

4953

345251282

370251293

25000011

395260790

395265518

4728

2779

0632

329

5251

226

1734

4903

370251300

395251309

25000009

395265525

395270252

4727

Tab

le27:

Con

figu

rati

on

:100

tick

s,25000000H

z

32


B Appendix B: Implementation source code

This is the full version of the implemented function that we use to test sleep, busy-wait and oursimple loop. The program consists of a single thread running three loops. Each loop will performa a test and a timestamp is collected before and after the function. The ctf-functions are customCTF messages that contains the current system clock cycle. This makes it easier to see whichtimestamps that belong to which functions, as well as if the timestamp is before or after it. TheseCTF messages are captured over UART and stored in a binary trace file that can be translatedinto a human-readable format by Babeltrace.

1 void func ( void ∗p1 , void ∗p2 , void ∗p3 )2 {3 i n t i = 4 ;45 // S leep6 whi le ( i−−) {7 c t f b e f o r e s l e e p ( k c y c l e g e t 3 2 ( ) ) ;8 k s l e e p (K MSEC(1000 ) ) ;9 c t f a f t e r s l e e p ( k c y c l e g e t 3 2 ( ) ) ;

10 }1112 i = 4 ;13 // Busy−wait (1 second )14 whi l e ( i−−) {15 c t f b e f o r e bu s ywa i t ( k c y c l e g e t 3 2 ( ) ) ;16 k busy wait (1000000) ;17 c t f a f t e r b u s ywa i t ( k c y c l e g e t 3 2 ( ) ) ;18 }1920 i = 4 ;21 // Loop22 whi l e ( i−−) {23 c t f b e f o r e l o o p ( k c y c l e g e t 3 2 ( ) ) ;2425 f o r ( i n t j = 0 ; j < 100000; j++)26 {27 v o l a t i l e i n t a = 2 ;28 a = a ∗ a ;29 }30 c t f a f t e r l o o p ( k c y c l e g e t 3 2 ( ) ) ;31 }32 }

Listing 2: The implementation of the test program.

The k sleep function takes K MSEC as a parameter, K MSEC converts the number of millisecondsto the correct format in the k timeout t data structure, which is required for the k sleep function.

In this case, the k sleep function will sleep for one second. The k busy wait function takes mi-croseconds as input and will also run for one second. However, there are certain instances wherethe k busy wait function will run for more time than specified by the input. We did not measurethe time taken for one iteration of the loop in order to set the number of iterations to a numberthat would also take one second.

33

new evaluating the real-time capabilities of zephyr on …1472100/... · 2020. 9. 30. ·...

Documents