LLM-based Configuration of Virtual Testbeds

Increasingly automated software-based systems require stringent quality assurance, which, despite facing more and higher demands, must be designed with sustainability in mind when it comes to efficiency and scalability in the first place. In this context, we are developing AI-supported methods that can be used for the dynamic configuration of virtual testbeds based on the established VCIP reference architecture and the FERAL simulation framework. Our goal is to improve the prediction quality of the underlying models while reducing resource consumption during the evaluation of associated system functionality through optimized parameterization and coupling of relevant simulation components. The excessive energy demand due to suboptimal testbed configuration in the continuous operation of such complex input-output chains is to be significantly reduced by means of selected Machine Learning (ML) techniques and the deployment of Large Language Models (LLMs). This shall be achieved by fine-tuning pre-trained baseline LLMs whose parameter spaces are focused on domain-specific expert knowledge and thus cause only a fraction of their original resource requirements while, at the same time, allowing for the establishment of more resource-efficient development environments.

As the complexity of highly automated and autonomous driving-related functions increases, so do the requirements for the quality assurance and qualification of the underlying automotive software. Virtual testbeds are therefore being used more and more frequently to test these functions, as they are able to systematically carry out a large number of tests and systematically investigate a wide variety of scenarios using simulation. However, numerous simulation variants are often required in order to achieve meaningful test results. This is particularly relevant when highly automated or autonomous driving functions have to be qualified for use in a vehicle, or when the integration and quality assurance of that software is automated using CI/CD pipelines. To this end, virtual testbeds are increasingly being used. They can generate realistic test environments by coupling simulation models embodying the application surroundings and system components under test. The appropriate use of a virtual testbed, especially regarding its configuration, constitutes a decisive step in this process. By integrating virtual testbeds into the CI/CD process, the testing of software changes can be made more efficient, which speeds up product development and ensures higher product quality.

To operationalize the methodology behind the LLM-based configuration of virtual testbeds, Fraunhofer IESE is developing a generative assistance software with open interfaces that automates the configuration process through LLM-supported selection, parameterization, and integration of established simulation models as the foundation for performing virtual experiments based on holistic and authentic scenarios. The adoption of this technology in the development environments of our partner companies will strengthen their international competitiveness in the long term, as they will benefit significantly from the time and cost savings achieved in the virtual validation of their products thanks to increased test process automation. Furthermore, the solution approach of this application-driven undertaking can be generalized to other fields of application, which promotes cross-workgroup collaboration on joint ML models and interdisciplinary networking of the specialist disciplines involved and creates added value for AI-related research and practical implementations.

In recent years, Fraunhofer IESE has entered into diverse collaboration partnerships with renowned industry companies and research institutions in the context of this seminal topic, including Bosch, Ansys/CADFEM, aSR, SETLabs, Vector, DataArt, and ZF, to name just a few. The joint participation in corresponding activities around strategic developments, scientific research, and commercial commissions has already resulted in many positive synergies regarding exploitable results, like scientific publications, the organization of specialist events, and the development of marketable technology demonstrators. These endeavors have now resulted in the submission of a proposal for a research project, named LIASON, which defines the time and technical framework of the upcoming collaboration work items of all the active and associated parties involved.