SAFE AI – Use of Safe Artificial Intelligence

• What is SAFE AI and the engineering of reliable AI systems?
• What are the opportunities for companies through the use of SAFE AI?
• What challenges need to be considered when using Safe AI?
• How can reliable AI systems be developed?
• How do we provide concrete support at Safe AI?
• Why should your company collaborate with us on Safe AI?

Dependability of a system describes its ability to avoid unacceptable failures in the provision of a service or functionality (Jean-Claude Laprie). Dependability includes, for example, the availability or reliability of a service, but also, in particular, functional safety in order to avoid catastrophic consequences for the users and the environment (i.e., serious or even fatal accidents as a result of failures). But the integrity and security of a system or its maintainability are also quality properties mentioned in the context of dependability. 

Accordingly, the engineering of safe AI systems involves using systems and software engineering principles to systematically guarantee safety during the construction, verification/validation, and operation of the AI system and, in particular, to also take into account legal and normative requirements right from the start.

To this end, it is important to understand that AI systems are developed in a fundamentally different way than traditional software-based systems. Often, the requirements on the AI system cannot be described completely, and the system must function dependably in an almost infinite application space. This is where established methods and techniques of classical software engineering reach their limits and new, innovative approaches are required. In an AI component, the functionality is not programmed in the traditional way, but created by applying algorithms to data. This results in a model that is merely executed by the software at runtime. One approach in this regard is for example, Machine Learning (ML) or, more specifically, Deep Learning (DL). The resulting model (e.g., an artificial neural network) is generally not comprehensible to humans due to its structure and inherent complexity, and thus the decisions made by an AI system are often not understandable.

Typical application areas are systems where the dependability of the AI system – i.e., managing the risk of failures or functional insufficiencies – is a concern.

Smart Mobility

Highly automated and autonomous vehicles will completely change the areas of mobility and logistics (transport of goods) and give rise to new digital services. But the risks of such a vehicle must be acceptable for users and for people interacting with the vehicle (such as passers-by). What exactly constitutes an acceptable risk is currently still the subject of debate, e.g., also by ethics committees. Basic elements that are already widely accepted are a positive risk balance and the principle of ALARP (As Low as Reasonably Practicable). Positive risk balance ensures that the risks for life and limb are less than if a human were to operate the system manually. With ALARP, it must be demonstrated that everything that can reasonably be done to minimize risks – depending on feasibility and effort – is actually being done.

Industry 4.0

AI is one of the pillars of the fourth industrial revolution to realize smart factories that allow individualized products to be created in a highly flexible manner. In addition to scenarios such as Production-as-a-Service or Predictive Maintenance, new smart machines are also being used. Driverless transport systems plan their routes independently and adapt them to demand. Cobots support factory workers and can be trained efficiently for a wide variety of tasks. Nowadays, mechanisms such as light barriers and safety cages are used that completely deactivate the system should a human enter the restricted zone. However, this does not allow the desired level of close interaction with humans to be achieved. In the future, smarter protective functions will also be needed when using such systems and interacting with them so that humans are not exposed to any increased risk.

Digital Health

Today already, physicians are supported by AI systems in the prevention, diagnosis, and therapy of diseases. Here, AI is often responsible for advanced image and data analysis (e.g., in the interpretation of a CT image). It enables analyses in terms of quality and quantity that cannot be achieved by conventional means. Even in surgeries, the first robotic systems are already in use today, but they are controlled by humans. However, the more medical professionals rely on such systems and the higher the degree of autonomy of an AI system in the future, the more important it becomes to control the risk of failures and functional insufficiencies of the AI.

Trustworthiness is a prerequisite for people and societies to develop, deploy, and use AI systems (European Commission’s Ethics Guidelines for Trustworthy AI). Similar to aviation or nuclear energy, it is not only the components of the AI system that need to be trustworthy, but the socio-technical system in its overall context. A holistic and systematic approach is required to ensure trustworthiness. According to the guidelines of the European Commission, trustworthy AI consists of three aspects that should be fulfilled throughout the entire lifecycle:

1. Compliance with all applicable laws and regulation
2. Adherence to ethical principles and values
3. Robustness both from a technical and social perspective

Safety assurance of the AI system refers, in particular, to the aspect of robustness of the AI system from the technical perspective in compliance with applicable laws and standards.

Back to Overview

Figure 1 shows our view of the fundamental process areas in a W-model. Here, we distinguish different activities in the lifecycle of an AI system (x-axis) and different stakeholders involved in the various phases: from the management level to the operative/technical level. The presentation of the activities and levels is based on VDE-AR-E 2842-61, which defines a reference model for trustworthy AI (developed by DKE, the organization responsible for the development of standards in VDE and DIN). Basically, the process is oriented towards a holistic systems engineering approach. Depending on the application area, it is common that individual system components delivered as a black box by suppliers must be integrated into the overall system.

Competence Development

The development process is typically preceded by a ramp-up phase in which the competencies required in the company are identified and developed. For the management level, it is important in this context to understand both the concrete possibilities offered by AI systems for their own business and the stumbling blocks encountered in implementation. For the operative/technical level, on the other hand, knowledge about concrete techniques/methods/tools for the implementation plays an important role.

Construction and V&V

The overall process starts with the development of the business strategy and the elaboration of the concrete potentials/added values expected from an AI system. From this, the technical requirements on the system level and a concrete solution concept are derived. Depending on the criticality of the system, further analyses are carried out on the system level (e.g., regarding functional safety) in order to take the regulatory and legal requirements into account during development. Following conventional systems engineering approaches (such as ISO/IEC 15288 or 12207), the system is decomposed into subsystems, which can consist of software and hardware. The software subsystems, in turn, consist of classical software components and components that contain AI and represent a normal or safe function of the system.

Today, software components are developed in an agile development process (such as Scrum), which takes place in parallel to hardware development. AI components are usually also developed in a highly iterative process, but this is more oriented to common methods from the field of data science (such as CRISP-DM). This means that the model is improved incrementally/iteratively until it meets the specified quality criteria. In this context, we speak of the evaluation or hardening of the AI model. The individual components of the system are then integrated into the overall system and verified or validated as a whole before the system can be released to the outside world after satisfying defined acceptance and release criteria (e.g., according to required testing and auditing processes). In practice, this overall process for construction and verification/validation is also iterated several times, depending on the size and complexity of the system and the development project.

Operation

The lifecycle of an AI system does not end with the deployment of the system. In the operation phase, the aim is to monitor the performance of the system and learn from the real behavior how to improve the AI system and/or the AI model. In the case of AI systems that continue to learn at runtime, i.e., that change their behavior on the basis of experience gained, further dynamic risk management procedures are also necessary.

Back to Overview

As a general rule, we advise and support companies in engineering dependable AI systems and accompany them throughout the entire lifecycle: from AI strategy via AI development to the assurance of AI including AI validation and AI auditing and compliance with legal and normative requirements. We are multidisciplinary in our approach, i.e., we have competencies in the area of software and systems engineering as well as in data science and innovation strategy.

In our research in the context of Safe AI, we have developed concrete solution components for key challenges, which we are transferring into practice. Figure 2 shows a classification of solution components in the depicted process areas of the W-model.

AI Potential Analysis

The added value of AI for a company is worked out within the framework of potential analyses. Using established templates (such as Value Proposition or BMC), the impact of AI on the business model is elaborated. We compile relevant norms, standards, and quality guidelines for the application context and show which AI systems are already possible today and which requirements will have to be met in the future depending on the criticality. During this process, we also identify the gap between the company’s current setup and its future direction and develop a strategy. In the context of prototypes, promising application scenarios are evaluated in terms of profitability, demand, and technical feasibility (e.g., with regard to the availability of data and the quality of the prediction).

Multi-Aspect Safety Engineering

Especially for the development of highly automated and autonomous systems, not only functional safety (ISO 26262) plays an important role, but also the safety of the intended functionality (SOTIF, ISO/PAS 21448) and the systematic development of a safe nominal behavior. In a model-based approach, we use state-of-the-art safety engineering methods and techniques to model the system holistically, to analyze it, and to manage the identified risks. Assurance or safety case analyses are used to argue that the system is dependable. Evidences are used to argue that regarding the assurance or safety case, the system behaves in such a way that the risk acceptance threshold is not exceeded.

Safety Supervisor Architecture

Another central element for the dependability of a system is the system architecture. Here, architectures exist, for example, that place a kind of safety container around the AI. These Safety Supervisor or Simplex architectures override the AI’s decision if it would lead to dangerous behavior of the AI system. For this purpose, it is important to know, for example, the uncertainty of the AI recommendation under certain conditions of use and to determine the risk of an incorrect decision based on this. Such approaches are interesting for Machine Learning components in general and particularly important if we are talking about AI systems that continue to learn over their lifetime and can therefore adapt their behavior. Since it is potentially impossible to know at the design time of an AI system which data the AI will use as a basis for further learning, appropriate guardrails for the behavior must be defined and incorporated into the system.

Digital Dependability Identities (DDI)

A Digital Dependability Identity (DDI) is an instrument for centrally documenting all specifications that are relevant for system dependability in a uniform and exchangeable manner and for delivering this specification together with a system or system part. This is especially important if system parts from different manufacturers/suppliers along the supply chain have to be integrated, and/or if dynamic integration of different system parts takes place at runtime (as in collaborative driving). The backbone of DDI is an assurance case that structures and links all specifications to argue under which conditions the system will operate dependably. The nature of the evidences in the argumentation chain varies widely. They may concern the safety architecture or the management of uncertainties of the AI component, for example the assurance that data is used that sufficiently covers the intended context of use.

Dynamic Risk Management (DRM)

DRM is an approach for the continuous monitoring of the risk of a system and its planned behaviors as well as of the safety-related capabilities. DRM monitors the risk, optimizes the performance, and ensures safety by means of continuous management informed by runtime models. One ingredient of DRM of AI-based systems can also be the output of an Uncertainty Wrapper.

Uncertainty Management

Uncertainty is an inherent problem in data-based solutions and cannot be ignored. The Uncertainty Wrapper is a holistic, model-independent approach for the identification of situationally reliable predictions of uncertainties in AI components. These uncertainties may stem not only from the AI model itself, but also from the input data and the context of use. The Uncertainty Wrapper can be used, on the one hand, during the development of the AI model to manage the identified uncertainties by taking appropriate measures and making the model safer. On the other hand, at runtime of the AI system, the wrapper enables a holistic view of the current uncertainty of the AI component. This assessment can in turn be used by a safety container to override the AI and bring the system to a lower-risk state.

Training

Last but not least, we are also active in providing qualification for the corresponding competencies within the company: from Enlightening Talks for management via continuing education and training programs on becoming a Data Scientist to special training for dependable AI systems (e.g., Safe AI seminar).

Back to Overview

Artificial Intelligence (AI) and Machine Learning (ML) have great potential to improve existing products and services – or even to enable completely new products and services. However, assuring and verifying essential quality properties of AI systems, and in particular properties relating to dependability, is a major challenge. Those who find and implement adequate solutions in this regard have a clear competitive edge! We are convinced: With our methods and solution components, we can actively and effectively support you on this path!

End-to-end: We accompany companies throughout the entire lifecycle: from AI strategy via AI development to AI assurance including AI validation and AI auditing and the fulfillment of legal and normative requirements.

Multidisciplinary: We are multidisciplinary in our approach, i.e., we have competencies in software and systems engineering as well as in data science and innovation strategy. Accordingly, we offer a one-stop service where all relevant competencies regarding Safe AI are available.

Innovative: As a Fraunhofer Institute, we are represented in innovative research projects for Safe AI at the German and EU level, so we know the current state of the art and state of the practice regarding processes, technologies, and tools for Safe AI.

Informed: We are involved in roadmap and standardization activities on Safe AI through associations and standardization bodies, so we have access to current discussions and are aware of future changes of the legal and normative requirements.

One-stop provider: As a member of the Fraunhofer Big Data and Artificial Intelligence (AI) Alliance, we have access to extensive resources (data scientists, software packages, and computing infrastructure). This allows us to act as a one-stop provider and offer even more complex projects in the context of Safe AI.