An Empirical Survey-based Study into Industry Practice in Real-time Systems @ RTSS 2020

It is my great pleasure to announce that our paper “An Empirical Survey-based Study into Industry Practice in Real-time Systems” has appeared at the 41st IEEE Real-Time Systems Symposium (RTSS). The paper presents results and observations from a survey of 120 industry practitioners in the field of real-time embedded systems. The survey provides insights into the characteristics of the systems being developed today and identifies important trends for the future. The survey aims to inform both academics and practitioners, helping to avoid divergence between industry practice and fundamental academic research.

This work is a dear pet project of mine that has been a long time in the making. Once I joined ESI (TNO), I started reading papers and attending conferences in the modelling community. I came across empirical survey-based research that systematically investigated industry trends, needs and practices, and that studied adoption and perceived benefits and drawbacks of different technologies and methodologies. I immediately found this line of work incredibly useful as it elevated my understanding of what happened in industry from a collection of anecdotes based on conversations with a few people in a few companies to something that could capture the experience of hundreds of people across industrial domains. I also had the feeling that this line of work provided all the citations I needed for the introduction of my papers, as it helped me position my own work on modelling in a broader industrial reality.

Empirical research is an established research direction in social science, but also in technical fields, such as software engineering and to a lesser extent system engineering. However, there was no work like this in the area of real-time systems. I decided to change this and pitched the idea to Rob Davis, Mitra Nasri, and Geoffrey Nelissen and Sebastian Altmeyer during a meeting in Amsterdam in May 2019. They substantially improved on my ideas and did a lot of very good work and almost a year and a half later, the paper is available for you to read. We could not fit everything we had to say into the RTSS paper, so there is also a supporting technical note entitled “A Comprehensive Survey of Industry Practice in Real-time Systems“.

RTSS Session

A separate session was dedicated to this work on the last day of RTSS 2020. The session began with a 25 minute paper presentation, which is available here:

The paper presentation was followed by a panel discussion involving three industry practitioners from the three main industrial domains covered by the survey: Marcelo Lopez Ruiz (Microsoft), representing the consumer electronics industry, Simon Schliecker (Volkswagen), representing automotive, and Stephen Law (Rolls-Royce), providing an avionics perspective. The panel discussed four key questions relating to the survey results:

Q1. What important characteristics of real-time systems highlighted in the survey results are the most relevant with respect to your industry? And what other important characteristics are there that were not picked up by the survey?

Q2. What are the most relevant trends in real-time systems development in your industry now, and looking ahead over the next 10 years?

Q3.  Did anything surprise you in the survey and its results? And why?

Q4. Given the results of the survey, and your own experience, what recommendations would you to give to the academic community? Which areas should we work more or less on? What assumptions should we make or not make?

The opening statements from the panelists related to the four questions was pre-recorded and followed by a live discussion. The pre-recorded part of the panel is available here:

The session finished after one hour, before there was time to take questions from the audience. A separate Zoom room was created for this purpose and to allow the interaction to continue, which it did for another hour! We were very pleased with the interest in this paper and in the session.

Emerging Research Direction

I hope that this work is the first of many empirical research papers in real-time systems. There are many ways to continue with this line of work. First of all, others need to replicate our results to validate that they hold for different populations. For this purpose, we will be happy to transfer the survey we made on SurveyMonkey, such that it can be reused. Secondly, our survey was very broad and covers real-time systems across many application domains. More specific questions could be obtained if the focus was on a single domain, although the main challenge will be finding enough representative participants with a narrow focus. Thirdly, surveys are only one way of conducting empirical research. Another method sometimes used in software engineering is to use interviews, allowing more in-depth questions to be asked. However, the drawback of this method is that it is more time consuming to interview are large number of participants and to encode and analyze the results.

This direction in real-time system research is just emerging and we hope it will grow and become a well-established part of the research conducted in the community. This would help us better understand the industry we are trying to serve and help us close the gap between academic research and industry practice. A first important step is that this direction is recognized by all main conferences and journals in the area of real-time systems and explicitly included in the call for papers. You can play an important part here by helping us communicate the value of empirical research to others in our community and beyond.

License to Teach

It is my great pleasure to announce that I have received my University Teaching Qualification (UTQ), essentially my license to teach. The UTQ (Basiskwalificatie Onderwijs, BKO in Dutch) is a proof of teaching skills for university teaching staff, which allows you to demonstrate a proven ability to develop and teach courses at university level. It is recognized by all Dutch universities.

One of the most important things I have learned during the UTQ trajectory is the constructive alignment method for course design. I already knew that aligning goals, execution, and assessment was an important part of course design, yet challenging to do in practice. I appreciate that the constructive alignment workshops not only communicated the importance of this alignment, but also provided the tools (e.g. specification tables and tips for assessment) to identify and address misalignments. Constructive alignment may not be the only way to approach course design, but I see it as a very useful method and a good starting point from which further exploration can be done.

One of my stated goals was to learn how to make my lectures more interactive. This is something I knew was important to further improve my teaching, but I did not quite know how to achieve. This was addressed by the workshop on activating teaching that came paired with the excellent list of 50 Classroom Assessment Teachniques (CATs)  by Angelo and Cross. Based on this material, I have now significantly increased the level of interaction and formative assessment in my course Embedded Software and Systems.

Lastly, the module about organizing teaching was very important to lift my view from my course and start looking more at the program that provides the context, in my case the Master of Software Engineering program. Being aware of this context not only makes my course better, but also the program, resulting in a better educational experience for the students. Interviewing people in the program for this module also provided a trigger to go talk to colleagues and build a network that will help both my research and education at UvA going forward.

Design Methodologies for Cyber-physical Systems

In this short two minute presentation, I introduce myself and my fundamental and academic research into design methodologies for cyber-physical systems. I sketch a high-level view of the problem and outline a direction based on model-based engineering in which my previous work into domain-specific languages and analysis non-functional behavior fits. For a more elaborate description of my research, please have a look at my research page.

Paper Accepted at GPCE 2020

A paper entitled “PReGO: a Generative Methodology for Satisfying Real-Time Requirements on COTS-based Systems – Definition and Experience Report” was accepted for publication at the 19th International Conference on Generative Programming: Concepts & Experiences (GPCE). This is my first paper in collaboration with my new colleagues at the University of Amsterdam, and it describes work done in the TeamPlay project, a three-year research project funded by the EU Horizon 2020 research and innovation programme.

The paper addresses the problem of satisfying real-time requirements in industrial systems using unpredictable hardware and software, which limit or entirely prevent the application of established real-time analysis techniques. To this end, we propose PReGO, a generative methodology for satisfying real-time requirements in industrial commercial-off-the-shelf (COTS) systems. We report on our experience in applying PReGO to a use-case: a Search & Rescue application running on a fixed-wing drone with COTS components, including an NVIDIA Jetson board and a stock Ubuntu/Linux. We empirically evaluate the impact of each integration step and demonstrate the effectiveness of our methodology in meeting real-time application requirements in terms of deadline misses and energy consumption.

Bachelor Thesis on Synthetic Interface Generation Defended

Mohammed (Mo) Diallo just defended his bachelor thesis entitled “Towards the Scalability of Detecting and Correcting Incompatible Service Interfaces“. This work is carried out in the context of a project between ESI (TNO) and Thales that developed a five-step methodology for automatic detection and correction of behavioral incompatibilities resulting from evolving software interfaces (see paper for more details). Mo’s thesis provides a starting point for evaluating the scalability of the proposed methodology. An essential ingredient towards this is the ability to synthetically generate interfaces of various complexity. The thesis has two main contributions: 1) a notion of interface complexity in terms of inputs, outputs and non-determinism is defined and the relation between these parameters is studied, and 2) the methodology for a ComMA interface generator using user-supplied complexity parameters, and its implementation in a supporting tool, is introduced.

I would like to thank Mo for the excellent work he delivered in this thesis, and I am happy that he will continue working over summer to extend it.

Announcement of Appointment as Professor at UvA

The press release announcing my appointment as Professor at the University of Amsterdam is finally ready. Time to make them and ESI (TNO) proud!

The Chair of Design Methodologies for Cyber-Physical Systems focuses on two research areas. The first area considers design methodologies for cyber-physical systems in which abstraction, provided by models used for specification, analysis, simulation, or synthesis, play an essential role. While this area applies to cyber-physical systems in general, the second area focuses on design aspects of real-time systems. Together, these two areas capture much of my existing work in both academic (TU/e, CTU Prague, CISTER) and applied research (ESI) in different application domains and industries in which I have worked, e.g. avionics (Airbus), consumer electronics (Philips & NXP), and defense (Thales). They are also broad enough to sustain a long-term effort towards managing complexity of cyber-physical systems. For more information about the research, click the ‘Research‘ button in the menu at the top of the page.

My first mission will involve developing and teaching a course on Embedded Software and Systems, a course that is extremely relevant to our work at ESI. The course is primarily aimed at students following the Master in Software Engineering and teaches the fundamentals of embedded system development. This includes modelling systems using StateCharts, Petri Nets, Data-flow graphs, and Domain-Specific Languages, embedded hardware, functional and timing verification, and design-space exploration. I will also explain the industrial reality behind some of these aspects by drawing on my experience from projects at ESI.

During the course, the students will get practical experience with model-based engineering as they work in groups to program a LEGO Mindstorm Rover using Stateflow to autonomously follow a path, while avoiding obstacles. From this batch of students, I am hoping to find some promising ones that can help us make the next innovative steps in model-based engineering for complex cyber-physical systems for their thesis project.