LLM-based Qualification of Software

Software qualification is the process of validating and verifying that a software system meets the requirements and specifications set out for it, ensuring it performs as expected under certain conditions. It is a part of software testing that focuses on evaluating the capabilities and characteristics of a program to ensure it meets its intended use.

The influence of laws and legal aspects on software qualification is significant, especially in industries that are heavily regulated. Legal requirements can dictate the standards and procedures that must be followed during the software development and qualification processes. Certain sectors, such as healthcare, finance, and aviation, have strict regulatory requirements for software. Non-compliance can lead to legal penalties, including fines and sanctions. Legal frameworks define the liability of software manufacturers and developers in cases where software failure leads to financial loss, injury, or death. Ensuring that software is fully qualified reduces this legal risk.

The qualification process for software is very complex and labor-intensive. Requirements can evolve during the development process, requiring ongoing adjustments to the testing and qualification plans. Additionally, software needs to be tested in various environments to ensure it works under different conditions, which can be resource-intensive.

The challenges of software qualification for autonomous driving functions are enormous. The software and the system itself are very complex. In addition, self-driving cars pose a potential risk to people in other cars or walking on the street. The regulatory requirements for such software are therefore very high. Up to now, the qualification of a piece of software can only be started at the end of the product development process. A real vehicle is used to drive around a defined area, and all possible situations that could occur when driving in a real environment have to be simulated. This requires a huge number of test kilometers. The process is very time- and resource-intensive. For example, the vehicle has to “read” road signs for speed limits and adjust its speed accordingly. If the law were to change – for example, if the speed limit in a city were to be reduced from 50 km/h to 30 km/h – the whole qualification process would have to be repeated. To reduce manual testing, Fraunhofer IESE’s Virtual Engineering department proposes the use of simulations to test and qualify such software.

Instead of driving a real vehicle on the road, we propose simulating the situations necessary to meet the legal and technical requirements. This is where our FERAL simulation platform comes in. The platform is capable of creating all kinds of simulation scenarios and connecting simulation models, tools, and software. In addition, FERAL is able to inject errors into the simulation to test all kinds of software failures and system behavior.

To speed up the development of these scenarios, we also propose the use of large language models (LLM) in this context. The AI would analyze the scenarios and test results and then give feedback to the tester regarding, for example, how to interpret different test results or what caused problems during the simulation, or even suggest changes to the software to meet requirements.