Virtual Continuous Testing – New Horizons for Modern Software-based Systems Development

Over the course of time, the indispensability of information and communications technology as an integral part of modern software-based systems has also given rise to the demand for more sophisticated development approaches. Among those, the established V-Model and its novel interpretation [1] constitute a trendsetting concept of a reference process for sustainable systems development across diverse segments of the manufacturing and automotive sector as well as other industries. In this blog post, we want to take a look at the associated emerging field of Virtual Continuous Testing (VCT) and its impact on how leading-edge products and services, whose system-level integration is increasingly growing in complexity, may be developed in the years to come.

State of the Practice in the Development of Software-based Systems

Adopted from the systems engineering domain, the common comprehension of the V-Model includes the strict division of a single development lifecycle into two consecutive parts, the Specification and Design phase followed by the Integration and Testing phase (see Figure 1). A well-understood consequence of this traditional interpretation is that results provided by evaluation activities of the second development phase are usually not available before the system is fully designed and implemented. Hence, possible integration issues or changes required due to, e.g., stakeholder demands or legal restrictions, especially at the end of the process, usually imply expensive rework on the aspired development output. In the worst case, this can make the associated product fail completely even before it has entered the intended market or has been deployed at the customers requesting it in the first place.
At least in the case of the automotive sector, Hardware-in-the-Loop (HiL) simulation testbeds are a common and mandatory means to support quality assurance, comply with regulations, and counteract undesired implications resulting from requirement misconceptions, inappropriate design decisions, and erroneous implementations. However, exclusively relying on physical test environments remains costly because hardware prototypes for the developed product need to be available in advance and the required testing equipment is generally expensive, limited in terms of adaptability as well as shareability, and hard to maintain.

*Figure 1.* Product development lifecycle according to the traditional V-Model enhanced by the shift left/front-loading approach in the context of virtual continuous engineering.

Cutting-Edge Development of Software-based Systems

A recent and more resource-efficient enhancement of the aforementioned matter is the idea behind the so-called Shift Left or Front-Loading approach, which suggests that the evaluation of non-functional/functional aspects of every release candidate shall be performed much earlier and more often. In this regard, we refer to VCT in terms of simulation-based X-in-the-loop (XiL) tests, where each relevant requirement (RiL), model (MiL), software (SiL), and virtual hardware (vHiL) of the system under test can be considered in the loop with its respective adjacent environmental context. This is meant to support any non-technical/technical design decision already in the early stages of the development process as part of the first phase in question. To this end, dedicated software-driven simulation technologies exist, each usually supporting a specific development stage, which in turn yields generally testable artifacts. These include, among others, requirements specifications, architecture models, executable code, and virtual hardware mockups every version of which can be prepared as input for the actual exploration, prognosis, validation, and verification tasks to be performed according to VCT.

*Figure 2.* Product development lifecycle according to the recent II-Model inherently embodying the continuous engineering aspect of VCT.

At the Virtual Engineering department at Fraunhofer IESE, we summarize VCT and further engineering-related perpetual activities, which can be enhanced by means of virtual abstraction, under the notion of Virtual Continuous Engineering, and illustrate it in the so-called II-Model (see Figure 2). Essentially, the two pillars of the development process are executed in parallel rather than in series, and this is done continuously on an iterative basis. Now, to utilize the methodology behind this model and VCT, in particular, for actual product and service development across arbitrary industries, we apply and offer a corresponding solution: our Virtual Continuous Integration Platform (VCIP), which is based on an event-driven simulation and model coupling framework called FERAL [2].

Virtual Continuous Testing: Abstraction and confidence levels for different evaluation aspects of virtual testing technologies — *Figure 3.* Abstraction and confidence levels for different evaluation aspects of virtual continuous testing technologies.

Virtual Continuous Testing Solutions

The majority of simulators and tools for the purpose of virtual validation and verification have always tended to focus on specific abstraction and confidence levels in terms of the system under test. MatLab Simulink, for instance, and most macro-level simulation tools, like the driving simulator CARLA, rather target the mere functional behavior while abstracting away from any low-level details. For example, when validating an automated driving function of a vehicle, such as Adaptive Cruise Control (ACC), a purely functional model does not necessarily need to replicate each and every data register shift at any involved microcontroller in order to reveal any discrepancies regarding the expected operational behavior. Other tools like ns-3 and OMNeT++ specifically focus on the interaction level, i.e., the data exchange and infrastructure between system components in view of specific network topologies and communication protocols. Micro-simulation tooling, in turn, concentrates on the hardware and physical level where even lowest-level operations of digital integrated circuits or signal propagation phenomena can be virtually reproduced with high fidelity, but usually at the expense of high consumption in terms of memory and computational resources.
With regard to the abstraction degree, unlike other tools, our VCIP solution does not focus on any of those levels in particular but rather on the integration of any focused simulation models and corresponding runtime environments that can execute such models in an interleaved manner in order to get exactly that level of confidence that is required to substantiate significant design and implementation decisions. In this context, VCIP is neither a feature-complete evaluation environment nor a testing tool chain by itself. It rather provides a reference architecture for building application use case-specific virtual continuous testing solutions by means of its instantiation (see Figure 4).

Virtual Continuous Testing: VCIP reference architecture — *Figure 4.* VCIP reference architecture for building use case-specific VCT solutions.

At its core resides the FERAL simulation framework, which uses so-called nestable domains based on different semantics for the structuring and execution of simulation models representing the actual system (components) under test. Interchange connectors allow for combining FERAL’s execution environment with co-simulators for bidirectional data transfers as well as with simulation control and observation utilities, e.g., to synchronize simulation time with the real wall-clock time. Import connectors can be used to include further component models like black-box models from mockup providers, such as Functional Mockup Units (FMUs), as well as to import customizable parameter settings as required for the generation of simulation scenarios. Finally, we can connect downstream systems like loggers and visualizers via export connectors for the post-processing and analysis of simulation results [3].

Virtual Continuous Testing: Examplar vCIP instantiation for VCT-based system evaluation — *Figure 5.* Examplary VCIP instantiation for VCT-based system evaluation in an automotive context.

Now, all these facilities can be used to instantiate the VCIP reference architecture with any combination of co-simulators, runnable code, virtual hardware models, analysis tools, and whatever is needed for the virtual continuous testing of any kind of software-based system that is to be validated or verified against a conceptual idea or detailed requirements specification, respectively.

Figure 5 illustrates a typical example of a VCT-based evaluation setup meant for distributed system component testing in the automotive sector. This setup is enabled by unified connectors for the interconnection of selected co-simulation tools hosting virtual Electronic Control Units (vECUs) and the import of a black-box vECU model as FMU along with a JSON-based property file defining the simulation scenario configuration.

Do you want to learn more about FERAL and our solutions in the context of virtual engineering and simulation-based testing?

Visit our website and do not hesitate to contact us! Our expert „Virtual System Integration“, Adam Bachorek, and our interdisciplinary scientific teams will be pleased to advise and support you – from conception and design to implementation and deployment.