The paper describes ESI’s current view on the field of System-Performance Engineering (SysPE). SysPE encompasses modeling formalisms, methods, techniques, and industrial practices to design systems for performance, where performance is taken integrally into account during the whole system life cycle. Industrial SysPE state of practice is generally model-based. Due to the rapidly increasing complexity of systems, there is a need to develop and establish model-driven methods and techniques. To structure the field of SysPE, the paper identifies: (1) industrial challenges motivating the importance of SysPE, (2) scientific challenges that need to be addressed to establish model-driven SysPE, (3) important focus areas for SysPE and (4) best practices. A survey was conducted to collect feedback on our views. The responses were used to update and validate the identified challenges, focus areas, and best practices. The final result is presented in this paper. Interesting observations are that industry sees a need for better design-space exploration support, more than for additional performance modeling and analysis techniques. Also tools and integral methods for SysPE need attention. From the identified focus areas, scheduling and supervisory control is seen as lacking established best practices.
The paper will be presented as a part of Industry Session 2 at ESWEEK on October 12. The second talk of that session presents why and how ITEC, Nexperia, a world-leading manufacturer of semiconductor equipment, is moving towards model-driven system-level development. The session ends with a moderated Q&A. Since ESWEEK is an online event this year, you can register for 20 USD if you want to attend the conference and the session.
Update: The video of the Industry session is now available:
The accepted paper is entitled “Synthetic Portnet Generation with Controllable Complexity for Testing and Benchmarking” and presents a heuristic-driven method for synthetic generation of random portnets, a kind of Petri Nets suitable for modelling software interfaces in component-based systems. The method considers three user-specified complexity parameters: the expected number input and output places, and the prevalence of non-determinism in the skeleton of the generated net. An implementation of this method is available as an open-source Python tool. Experiments demonstrate the relations between the three complexity parameters and investigate the boundaries of the proposed method. This work was helpful for the DYNAMICS project, as it allowed us to synthetically generate a large number of interfaces of varying complexity that we could use to evaluate the scalability of existing academic tools for adapter generation.
It is my great pleasure to announce that our paper “An Empirical Survey-based Study into Industry Practice in Real-time Systems” has appeared at the 41st IEEE Real-Time Systems Symposium (RTSS). The paper presents results and observations from a survey of 120 industry practitioners in the field of real-time embedded systems. The survey provides insights into the characteristics of the systems being developed today and identifies important trends for the future. The survey aims to inform both academics and practitioners, helping to avoid divergence between industry practice and fundamental academic research.
This work is a dear pet project of mine that has been a long time in the making. Once I joined ESI (TNO), I started reading papers and attending conferences in the modelling community. I came across empirical survey-based research that systematically investigated industry trends, needs and practices, and that studied adoption and perceived benefits and drawbacks of different technologies and methodologies. I immediately found this line of work incredibly useful as it elevated my understanding of what happened in industry from a collection of anecdotes based on conversations with a few people in a few companies to something that could capture the experience of hundreds of people across industrial domains. I also had the feeling that this line of work provided all the citations I needed for the introduction of my papers, as it helped me position my own work on modelling in a broader industrial reality.
Empirical research is an established research direction in social science, but also in technical fields, such as software engineering and to a lesser extent system engineering. However, there was no work like this in the area of real-time systems. I decided to change this and pitched the idea to Rob Davis, Mitra Nasri, and Geoffrey Nelissen and Sebastian Altmeyer during a meeting in Amsterdam in May 2019. They substantially improved on my ideas and did a lot of very good work and almost a year and a half later, the paper is available for you to read. We could not fit everything we had to say into the RTSS paper, so there is also a supporting technical note entitled “A Comprehensive Survey of Industry Practice in Real-time Systems“.
RTSS Session
A separate session was dedicated to this work on the last day of RTSS 2020. The session began with a 25 minute paper presentation, which is available here:
The paper presentation was followed by a panel discussion involving three industry practitioners from the three main industrial domains covered by the survey: Marcelo Lopez Ruiz (Microsoft), representing the consumer electronics industry, Simon Schliecker (Volkswagen), representing automotive, and Stephen Law (Rolls-Royce), providing an avionics perspective. The panel discussed four key questions relating to the survey results:
Q1. What important characteristics of real-time systems highlighted in the survey results are the most relevant with respect to your industry? And what other important characteristics are there that were not picked up by the survey?
Q2. What are the most relevant trends in real-time systems development in your industry now, and looking ahead over the next 10 years?
Q3. Did anything surprise you in the survey and its results? And why?
Q4. Given the results of the survey, and your own experience, what recommendations would you to give to the academic community? Which areas should we work more or less on? What assumptions should we make or not make?
The opening statements from the panelists related to the four questions was pre-recorded and followed by a live discussion. The pre-recorded part of the panel is available here:
The session finished after one hour, before there was time to take questions from the audience. A separate Zoom room was created for this purpose and to allow the interaction to continue, which it did for another hour! We were very pleased with the interest in this paper and in the session.
Emerging Research Direction
I hope that this work is the first of many empirical research papers in real-time systems. There are many ways to continue with this line of work. First of all, others need to replicate our results to validate that they hold for different populations. For this purpose, we will be happy to transfer the survey we made on SurveyMonkey, such that it can be reused. Secondly, our survey was very broad and covers real-time systems across many application domains. More specific questions could be obtained if the focus was on a single domain, although the main challenge will be finding enough representative participants with a narrow focus. Thirdly, surveys are only one way of conducting empirical research. Another method sometimes used in software engineering is to use interviews, allowing more in-depth questions to be asked. However, the drawback of this method is that it is more time consuming to interview are large number of participants and to encode and analyze the results.
This direction in real-time system research is just emerging and we hope it will grow and become a well-established part of the research conducted in the community. This would help us better understand the industry we are trying to serve and help us close the gap between academic research and industry practice. A first important step is that this direction is recognized by all main conferences and journals in the area of real-time systems and explicitly included in the call for papers. You can play an important part here by helping us communicate the value of empirical research to others in our community and beyond.
An article entitled “Response Time Analysis of Multiframe Mixed-Criticality Systems with Arbitrary Deadlines” has been accepted for publication in Real-time Systems journal. This work is first authored by Ishfaq Hussain and is another collaboration with my former colleagues at CISTER. The article extends our RTNS 2019 paper “Response Time Analysis of Multiframe Mixed-Criticality Systems” that received both an Outstanding Paper Award and a Best Student Paper Award. The RTNS paper presented a schedulability analysis for the multi-frame mixed-criticality model, extending the static and dynamic analysis techniques for mixed-criticality scheduling and the schedulability analysis for multi-frame task systems.
The accepted journal article extends the RTNS paper by generalizing the proposed schedulability analyses from a constrained-deadline task model to the more general, but also more complex, model with arbitrary deadlines. The corresponding optimal priority assignment for our schedulability analysis is also identified. In experiments with synthetic workloads, the proposed analyses are compared in terms of scheduling success ratio, against the frame-agnostic analyses for the corresponding variants of the Vestal model.
The paper addresses the problem of satisfying real-time requirements in industrial systems using unpredictable hardware and software, which limit or entirely prevent the application of established real-time analysis techniques. To this end, we propose PReGO, a generative methodology for satisfying real-time requirements in industrial commercial-off-the-shelf (COTS) systems. We report on our experience in applying PReGO to a use-case: a Search & Rescue application running on a fixed-wing drone with COTS components, including an NVIDIA Jetson board and a stock Ubuntu/Linux. We empirically evaluate the impact of each integration step and demonstrate the effectiveness of our methodology in meeting real-time application requirements in terms of deadline misses and energy consumption.
Anna Minaeva, who recently received her PhD degree, just had a journal article entitled “Control Performance Optimization for Application Integration on Automotive Architectures” accepted for publication in IEEE Transactions on Computers. This article is the result of a HiPEAC collaboration grant that Anna was awarded back in 2016 to visit the group of Samarjit Chakraborty at TU Munich. I am very happy to see that this grant resulted in a joint publication in a prestigious journal and hope to collaborate with Samarjit again in the future.
The article addresses the problem of generating a time-triggered schedule for a number of independently developed automotive applications on a number of shared resources, such that their control performance only suffers minimal degradation. The three main contributions are: 1) a constraint programming model that solves the problem optimally, exploiting properties of the problem to reduce the computation time; 2) a fast heuristic called Flexi that only has a minor impact on the optimality of the solution; and 3) an experimental evaluation of the scalability and efficiency of the proposed approaches on a case study, in addition to several synthetic datasets. The results show that the heuristic provides a solution on average 5 times faster, finding a feasible solution in 31% more problem instances than the optimal approach within a time limit, while only sacrificing 0.5% of the control performance quality for the largest dataset.
The paper describes applied research from an industrial ESI project with goal of enabling continuous evolution of software in service-oriented architectures through automatic detection and correction of service incompatibilities. Towards this, the paper has three main contributions: 1) the state-of-the-art in the areas of specification of service interfaces, and detection and correction of incompatible service interactions is surveyed, 2) directions for a methodology to detect and correct incompatible interactions that is currently under development are discussed, and 3) the methodology is discussed in the context of a simplified industrial case study from the defense domain.
This article addresses the problem of resource sharing in mixed-criticality systems through temporal isolation by extending the state-of-the-art Single-Core Equivalence (SCE) framework in three ways: 1) we extend the theoretical toolkit for the SCE framework by considering EDF and server-based scheduling, instead of partitioned fixed-priority scheduling, 2) we support uneven memory access budgets on a per-server basis, rather than just on a per-core basis, and 3) we formulate an Integer-Linear Programming Model (ILP) guaranteed to find a feasible mapping of a given set of servers to processors, including their execution time and memory access budgets, if such a mapping exists. Our experiments with synthetic task sets confirm that considerable improvement in schedulability can result from the use of per-server memory access budgets under the SCE framework.
Overall, I greatly appreciate that key conferences in the real-time community are starting to allow journal articles to be presented. This increases the exposure of these works that are often longer and better edited. It is also helpful for researchers at the institutes where conference publications are not considered a relevant KPI. You can argue the validity of this reasoning in areas of computer science where conferences are highly competitive with 20-30% acceptance rates, but it is reality for some researchers. An interesting thing with the MODELS conference is that they collaborate with the SOSYM journal such that some accepted articles in the journal gets a full slot at the conference. This is a nice way to highlight good articles and to appreciate the work done by both authors and reviewers.
The paper “Response Time Analysis of Multiframe Mixed-Criticality Systems” has been accepted at RTNS 2019. This work is the next in our mixed-criticality research line, in collaboration with my former colleagues at CISTER. It continues our work on the multi-frame task model, also considered in our RTCSA paper this year. The multi-frame model describes tasks that have different worst-case execution times for each job, following a known pattern, which can be exploited to reduce the cost of the system. Existing schedulability analyses fail to leverage this characteristic, potentially resulting in pessimism and increased system cost.
In this paper, we present a schedulability analysis for the multi-frame mixed-criticality model. Our work extends both the analysis techniques for Static Mixed-Cricality scheduling (SMC) and Adaptive Mixed-Criticality scheduling (AMC), on one hand, and the schedulability analysis for multi-frame task systems on the other. Our proposed worst-case response time (WCRT) analysis for multi-frame mixed-criticality systems is considerably less pessimistic than applying the SMC, AMC-rtb and AMC-max tests obliviously to the WCET variation patterns. Experimental evaluation with synthetic task sets demonstrates up to 63.8% higher scheduling success ratio compared to the best of the frame-oblivious tests.
A paper “Memory Bandwidth Regulation for Multiframe Task Sets” has been accepted at RTCSA 2018. This paper aims to reduce cost of real-time systems where the worst-case execution times of tasks vary from job to job, according to known patterns. This kind of execution behavior can be captured by the multi-frame task model. However, this model is optimistic and unsafe for multi-cores with shared memory controllers, since it ignores memory contention, and existing approaches to stall analysis based on memory regulation are very pessimistic if straight-forwardly applied.
This paper remedies this by adapting existing stall analyses for memory-regulated systems to the multi-frame model. Experimental evaluations with synthetic task sets show up to 85% higher scheduling success ratio for our analysis, compared to the frame-agnostic analysis, enabling higher platform utilization without compromising safety. We also explore implementation aspects, such as how to speed up the analysis and how to trade off accuracy with tractability.
