The evolution of vehicle infotainment systems has dramatically enhanced the driving experience. This is achieved by incorporating advanced functionalities such as navigation, multimedia, and connectivity options. Therefore, meticulous testing is crucial to ensure the reliability and functionality of these sophisticated systems. Virtual testing offers an effective and safe avenue to assess these systems in a controlled setting. This promotes early detection of issues and enables cost-effective enhancements. In this post, we detail how we implemented an infotainment module to enhance the testing and validation capabilities of the Fraunhofer IESE Driving Simulator.
The motivation behind our infotainment system development
A few months ago, one of our industry customers approached us with an intriguing request. They wanted to integrate FERAL [1] into their infotainment development process as a virtual continuous integration tool. However, they had a prerequisite – they needed to see the infotainment system in action.
This request posed a significant challenge for us. We needed to determine the communication interface layers for FERAL. Additionally, we had to find a way to obtain real-time vehicle data without using actual hardware. After much deliberation, we concluded that the optimal solution would be to implement the infotainment graphical user interface (GUI) alongside our driving simulator’s main screen, which displays critical vehicle data.
To make the simulation as realistic as possible, the data detected by CARLA sensors had to be seamlessly transferred to the simulator. It also needed to be updated in real time. Additionally, we needed to address real communication issues, making communication protocols and signal emulation viable options. Given these requirements, we extended our driver simulator to meet these needs. This provided comprehensive testing capabilities at various layers of the Android Automotive Framework.
This request offered us an exciting opportunity to innovate and deliver a robust solution that meets our client’s needs. It also allowed us to improve our testing facilities.
Infotainment system implementation
In this section, we explore how we enhanced our driving simulator by integrating an infotainment module. This module extracts crucial vehicle information from the CARLA simulator and displays it in real time through a graphical user interface (GUI).
We’ll walk you through the entire process, from data collection and extraction to the technical aspects of the GUI. You’ll learn how we ensure real-time accuracy and usability, making our infotainment system both advanced and user-friendly.
Data collection and preparation
Before extracting information, it was necessary to complete the vehicle model definition as thoroughly as possible. This was crucial in order to achieve realistic vehicle behavior. We used the “coupe_2020” from the CARLA vehicle_blueprints library. It contains the physical information of the Mercedes Benz Coupe model. Additionally, we extended the model’s definition to include the torque-rpm curve and fuel consumption information.
The data flow from CARLA to the infotainment system utilizes the CARLA Python API for CARLA release 0.9.14, enabling real-time vehicle and environmental data transmission. We focused on five data blocks: instrument cluster, Advanced Driver Assistance Systems (ADAS), lights, weather, and navigation information. Here is an overview of the infotainment variables associated with each data block:
| Instrument cluster | ADAS | Lights | Time & Weather | Navigation |
|---|---|---|---|---|
| Speed Odometer Power Failure State Reverse State Current Gear RPM Fuel level km remaining Speed limit Engine Tempera-ture Driving State | ACC State LCOA State LKA State Distance to front | Brake Reverse Position Low Beam High Beam Left Blinker Right Blinker Fog Lights Interior Lights | Current time Temperature Time of day Cloudiness Precipitation Wind Intensity Fog Density Wetness Rain Intensity Cloudiness | Location Map |
Instrument cluster block
Regarding the instrument cluster block, several variables were directly extracted from CARLA using the Python APIs. However, some key values could not be directly obtained and required computation. These values were computed as follows:
- RPM: Based on the speed, final ratio, gear ratio, and wheel radius. All parameters were obtained directly from CARLA.
- Power: Calculated from the current torque, determined by interpolating the RPM within the torque-RPM vehicle curves.
- Odometer: Measured in kilometers, assuming simple car dynamics based on speed and time.
- Fuel Level: Computed from the kilometers traveled and a constant consumption ratio to ascertain the total fuel consumed in liters.
ADAS state information
The ADAS state information was directly received from the respective external Ansys SCADE and JAVA models already integrated into our driving simulator (see Ref. [2]).
Environmental conditions
Extracted using the carla.World library:
- Current Time: Derived from the timestamp variable of
carla.WorldSnapshotand formatted as HH:MM:SS. - Time of Day: Categorized into Noon, Sunset, and Night based on the sun’s altitude angle.
- Rain Intensity and Wetness: Determined from precipitation and wetness values, respectively, with categorized states such as HardRain, MidRain, SoftRain, and NoRain.
- Cloudiness: Displayed as “Cloudy” or “Clear” based on the cloudiness value.
Vehicle light information
Managed using the Vehicle and VehicleLightState libraries for setup and retrieval of light state directly from the driving simulator controls.
Navigation data
Extracted using the get_map and get_location methods from CARLA:
- Road Maps: Detailed information about roads, lanes, waypoints, and junctions.
- Vehicle Location: Current position of the vehicle.
Data storage
All data is stored in a Python dictionary and serialized for access and use in FERAL. This integration enhances simulation and supports vehicle infotainment testing.
Graphical User Interface (GUI)
The GUI mimics a head-up display (HUD) and the main infotainment panel of a modern car, both displayed on the driving simulator windshield.
Head-up display (HUD)
The HUD, shown in Figure 1, provides essential information such as distance to the car ahead. It also displays the current velocity and upcoming road details. The HUD is located in the driver’s sight for quick access to the information.
Main infotainment panel
The main panel layer, shown in Figure 2, is structured into six sections corresponding to the above data blocks. There is also an additional section for basic entertainment options. The panel spans the entire screen width at the bottom of the simulator screen, occupying about one-fifth of the vertical screen space. Key features include icons for light and ADAS activation states, which change color from white to cyan when activated.
-
Instrument cluster
The instrument cluster features five analog gauge widgets positioned directly in front of the driver. These widgets update in real time to display the speedometer, engine temperature, fuel level, power, and gear. Additional information such as RPM, current time, and odometer readings are shown via customized labels. Dynamic icons provide visual feedback by animating and changing color based on related variable changes. They also update their value accordingly. For instance, a speed limit signal appears when approaching a limit in the CARLA scenario, while alert messages are displayed in the center of the panel in the event of a collision or system error.
-
Navigation
For navigation, we utilized the matplotlib library to convert waypoints from Carla into a detailed navigation map. As shown in Figure 3, this section displays the complete scenario road, with a vehicle icon marking the current position. Weather information is also presented through labels accompanied by icons that change according to current conditions.
-
Connectivity and utility
The applications panel provides basic icons for accessing the driver’s cell phone, enabling functionalities such as reading messages, making calls, and listening to music. This enhances the connectivity and utility of the infotainment system.
This GUI design not only delivers essential driving information but also enriches the driving experience with useful features. Currently, the entertainment section is under development. Future integrations are expected to include more communication capabilities and testing options. These will extend to Android Automotive connectivity components such as Bluetooth, Wi-Fi, mobile networks, and GPS.
Role of FERAL
FERAL plays a crucial role in bridging the data collection module with the infotainment GUI through a virtual vehicle network, simulating a realistic communication environment. It supports various vehicle communication protocols including CAN, CAN-FD, FlexRay, and Automotive Ethernet, ensuring accurate and timely data transmission to the infotainment system. This integration streamlines the process, enhancing the overall efficiency of the simulation setup.
Preliminary testing
The infotainment system was successfully deployed and integrated into the driving simulator to evaluate its performance in handling challenging real-time data simulation. Preliminary testing indicated effective synchronization with simulated data. This enhances the capabilities for testing and validating infotainment systems under different driving conditions and scenarios.
During the preliminary testing phase, various scenarios from the CARLA simulator were employed. This was done to assess the synchronization and operational speed of the infotainment system. The results indicated that the infotainment system functions as anticipated, effectively syncing with the simulated data. This new feature enhances the capabilities for testing and validating infotainment systems under different driving conditions and scenarios.
Furthermore, FERAL provides a vital testing environment to examine the impact of realistic vehicle network communications, including CAN, CAN-FD, FlexRay, and Automotive Ethernet, on the development and reliability of infotainment technologies. This aspect of FERAL allows developers to ensure that the infotainment system not only meets functional requirements but also adheres to performance standards in realistic automotive communication environments.
Video Link: Infotainment module working in real time in the driving simulator
Conclusions and outlook
Our methodology for simulating and testing vehicle infotainment systems represents a significant advancement in automotive technology development. By tackling the complexities of real-time data simulation and network communication, we ensure robustness. This makes our infotainment systems well-suited for modern vehicles. Looking ahead, we plan to integrate our implementation with Android Automotive applications and introduce fault injection techniques into the communication network to further solidify the reliability and security of our infotainment systems in real-world scenarios.
References
- Kuhr T, Forster T, Braun T et al. (2013) FERAL — Framework for simulator coupling on requirements and architecture level. In: 2013 Eleventh ACM/IEEE International Conference on Formal Methods and Models for Codesign (MEMOCODE 2013), pp 11–22
- Bachorek A, Meiler M (eds) (2023) A Virtual Testbed for Detecting Feature Interactions in ADAS/AD Function Design