Reliable and efficient quality assurance with FERAL

Powerful verification and validation of safety-critical systems using simulation and AI-supported test case generation in virtual test benches based on VCIP/FERAL and the Req2Test connector extension, which enables automated derivation of test cases from product requirements and their transfer into executable test case specifications based on industry standards such as ASAM OpenTestSpecification and OMG ReqIF.

The central challenge in the development of complex safety-critical systems is the reliable and efficient safeguarding of their functional and qualitative properties. Proof of safety in the development of modern motor vehicles in particular requires a robust verification and validation concept with traceability embedded in the development process.

An important part of the safety argumentation is reliable and traceable evidence from test activities, which is based on the implementation and execution of a large number of generally requirement-based test cases at different levels of abstraction. For each system requirement, the derivation including design and implementation of typically one or more test cases including stimuli, sequences and scenarios as well as their linkage with architecture and test result artifacts is required.

These artifacts are an elementary prerequisite for the effective application of virtual continuous testing in the context of modern product development (see blog post: https://www.iese.fraunhofer.de/blog/virtual-continuous-testing/).

However, repeated adaptation of the test case implementation and the linking of development artifacts due to changed requirements and identification of errors are commonplace in almost every iteration of the product development cycle. In addition, these activities for the provision of required development artifacts are still carried out manually today, despite extensive efforts to automate processes, due to the required expertise of test engineers, which causes very high effort and costs (see Figure 1).

The aim is therefore to increase the efficiency of these necessary activities by means of AI support with the promising use of large language models (LLMs).

Our solution concept defines 4 steps to increase efficiency in virtual system evaluation based on VCIP/FERAL and the use of AI technology (see Figure 2):
 

1.    LLM-supported derivation of test cases from requirements

A pre-trained large language model that has been fine-tuned for this task (fine-tuned LLM) is used to automatically generate test case descriptions on the basis of textual requirements that are available in the standardized OMG ReqIF format. These descriptions are stored in a structured form using DDIs (Digital Dependability Identity) for the purpose of formal traceability.
 

2.   LLM-based conversion of derived test cases into executable test specifications

Derived test case descriptions are automatically converted into standard ASAM OpenTestSpecification (XOSC/OTX) formats using another fine-tuned LLM, whose integration into the virtual test bench is supported by a VCIP/FERAL connector extension. This drastically reduces the effort required for the design and implementation of executable test cases and qualifies the virtual test bench for the use of ASAM-compliant tool chains. 3.    Experten-basierte Überprüfung der LLM-konvertierten Testspezifikation

3.   Expert-based review of the LLM-converted test specification

Test engineers review the converted test specification, adapt it as required and release it for execution on the virtual test bench. This step ensures that the suitability and correctness of the test specification is not impaired by the use of AI, because ultimately subject matter experts check the content again for plausibility and executability, but without the initial effort that recurs in every development iteration.
 

4.   VCIP/FERAL-based test case execution

The approved test specification, including the test case instances, is executed in the virtual test bench for the current product increment using the VCIP/FERAL connector extension and the corresponding test results are generated. This evidence is in turn automatically linked to the associated development artifacts such as requirements and architecture descriptions using DDIs and is then used to make decisions for the next development iteration.

Embedding this extension for the verification and validation concept in the product development process enables a drastic reduction in manual effort, because the majority of test cases are already available for execution within a very short time, allowing test engineers to focus their expertise on plausibility checks and special cases rather than on recurring, time-consuming preparation of test artifacts.

To implement the AI-supported solution concept, virtual test benches based on VCIP/FERAL are supplemented by two connector components. Firstly, the “Req2Test” connector is used to convert ReqIF-formatted requirements into ASAM standard-compliant test specification formats (XOSC/OTX). On the other hand, the script-based “Configuration” connector is used to import such a test specification for the purpose of configuration, including parameterization and parameterization of the test environment in preparation for the subsequent test execution.