Measuring the Performance Efficiency of Eclipse BaSyx

Industrie 4.0 (I4.0) is about end-to-end digitalization between IT (information technology) and OT (operation technology), i.e., the shopfloor. For example, the production of small lot sizes requires the ability to efficiently change production. Modern approaches, such as service-oriented manufacturing and Digital Twins, support this. With this technology, process changes can be implemented much faster than with traditional automation. In order to support realistic deployments, an Industrie 4.0 middleware needs to support a large number of Digital Twins. In this post, our experts from Fraunhofer IESE provide an overview of the performance efficiency of Eclipse BaSyx, an open-source Industrie 4.0 middleware that enables the realization of Digital Twins based on Asset Administration Shells.

Performance Efficiency is one of the key quality attributes to be considered when designing an Industrie 4.0 solution. The ISO 25010 [1] standard defines Performance Efficiency as “the performance relative to the amount of resources used under stated conditions”. It is composes of the following sub-characteristics:

• Time behavior: Degree to which the response, processing times and throughput rates of a product or system, when performing its functions, meet requirements.
• Resource utilization: Degree to which the amounts and types of resources used by a product or system, when performing its functions, meet requirements.
• Capacity: Degree to which the maximum limits of a product or system parameter meet requirements.

In this post, we will describe the results of the time behavior of Digital Twins implemented with Eclipse BaSyx. Monitoring of the resource utilization also took place, but does not need a major focus for now.

The Asset Administration Shell in a nutshell

The Asset Administration Shell (AAS) is a digital representation of an asset. It has one or multiple IDs that describe the asset. Every AAS should have at least one worldwide IDs that uniquely identify the asset. In addition, AASs may represent additional IDs that are unique in their context, e.g., a serial number. An AAS consists of multiple submodels that describe the represented real-world asset from the viewpoint of different use cases.

The submodels therefore focus on specific kinds of information: For example. one AAS submodel defines a digital nameplate for the asset, another submodel provides documentation, a third submodel describes the current status of a resource and connects, for example, to a device to obtain this information. Depending on the kind of asset that is represented by the AAS, and depending on the environment, an AAS may provide different sets of AAS submodels. AAS submodels may be standardized or not. Important submodels that are imperative for realizing digital supply chains are being standardized under the umbrella of the IDTA. Other, possibly more specific submodels are probably not covered by a standard, but stakeholders have agreed on them.

About Eclipse BaSyx

Eclipse BaSyx is an Industrie 4.0 middleware that implements the infrastructure for AASs and provides the necessary tools for creating AASs. Furthermore, it offers tools for the import of AAS types and instances, and for the dynamic instantiation of AASs. In addition, Eclipse BaSyx enables active AASs, which yield a behavior to autonomously collect and aggregate data, or to implement harmonized interfaces to devices by acting as device drivers. In this way, for example, characteristic curves do not need to be implemented in the process logically anymore (and therefore do not need to be changed when the device type changes).

Eclipse BaSyx implements three main AAS components: the AAS Server, a repository for hosting AAS; the Registry, which locates AASs on the AAS server(s) based on the AAS ID, and the Data Bridge, which connects data sources to AAS properties. Furthermore, Eclipse BaSyx provides SDKs for Java, C#, and C++ that allow implementers to create AAS-enabled applications.

To ensure scalability, Eclipse BaSyx stores all information in an external information provider. For the following benchmarks, we selected MongoDB, an NoSQL database. Besides, BaSyx supports backends such as SQL, standard HDD, memory (for testing), and S3 storage. The communication between users, other software systems, and AAS Servers and Registries can be implemented using different technologies, like http-rest interfaces, MQTT, OPCUA, and many more.

What is being tested by Performance Tests

For measuring the performance efficiency of Eclipse BaSyx, we tested the following:

1. How long does it take between sending an http-request to the AAS-Server and receiving an answer back, i.e., what is the response time of the system? This response time depends on the constraints of the software and the hardware on which BaSyx is running.
2. Implicit effects of the network, i.e., delay of requests. As an infrastructure for BaSyx, our experts set up a Kubernetes cluster to ensure that the parameters of a network with industrial complexity can be tested.

Test scenarios for testing the Performance Efficiency of Eclipse BaSyx

Since it is not enough to perform only one benchmark to make a statement about the system, we defined a variety of scenarios. In every one of them, we adapted several parameters to mimic data flows in a real production system and determined the following parameters for our testing:

1. Number of clients or virtual users (VU): Real production environments consist of a large number of independent virtual users. Our tests ran with 10 – 30,000 VU’s simultaneously.
2. Frequency of VU’s: Requests are sent at different frequencies without a predictable order. To cover this, we varied the frequency requests sent by the VU’s from 1 – 100 Hz. We additionally benchmarked VUs without frequency restrictions.
3. Hardware: We executed our benchmarks on a Kubernetes system, which will be described in more detail in the next section.
   1. We varied the virtualization (configuration) of the virtual machines (VM’s) as well as the
   2. Physical hardware to determine the dependency of the software on the available resources
4. Number of AASs present in the system: The larger and more complex a system becomes, the more AASs the middleware has to process. In this regard, we tested the Eclipse BaSyx software with 200 – 8000 active AASs plus submodels (SM).
5. Kind of request: We tested
   1. Read requests and
   2. Write requests to check for any performance differences. For the same reason, we tested the influence of
6. Randomness More precisely, the difference between:
   1. All requests target the same AAS and
   2. Every request targets a random AAS

Defining the test scenarios

Based on these testing requirements, we defined a test order for all possible combinations of these parameters, as shown in Figure 1.

First of all, we defined the hardware setups. For this purpose, we used the same physical hardware for all benchmarks, but configured different virtualizations, i.e., with a varying numbers of nodes (VMs) present in the cluster and different amounts of computational resources. We started with one node and four CPU cores + 16 GB RAM and a second cluster with four nodes and four CPU cores + 16 GB RAM per node, as depicted in Figure 2. On each of these hardware configurations, we executed multiple test series with different numbers of AASs. Regarding the application behavior, we tested read and write operations, and we targeted either the same AAS or a random one using Eclipse BaSyx version 1.2.

Performance test setup

For testing the performance of Eclipse BaSyx, we chose Kubernetes. It allows hosting code within container software on VMs (virtual machines), called nodes. The infrastructure for the Kubernetes clusters is hosted by our partner NetApp, which also uses Digital Twins and BaSyx.

Kubernetes offers:

1. Automated health and readiness checks
2. Automated networking between software components
3. Constant and defined distribution of computational resources across containers
4. Monitoring of the resource consumption of the containers

To eliminate network delays to the greatest possible extent, we decided to host the testing software in another Kubernetes cluster in the same data center (cf. Figure 3). The connection speed between the two clusters was > 10 GB/s. Another reason for using a second Kubernetes cluster for the testing software was its own resource consumption. Kubernetes ensures that the executed software always has sufficient computational resources and provides an intelligent resource distribution, as long as technically possible. This minimizes the impact of the testing infrastructure on the test results.

After defining the general setup, the testing system itself had to be designed. As previously mentioned, our goal was to simulate the behavior of a real factory environment. Therefore, we defined the number of VUs that communicate with BaSyx components over the http-rest interface, as shown in Figure 4. In doing so, we also measured and saved the duration of each request. We used K6, an open-source project, for loading testing applications and for orchestrating the testing environment. A K6 configuration defines the Kubernetes resources, for example the number of VUs and the simulated applications for a benchmarking scenario.

In our case, the duration of each request by each VU was measured and saved to an InfluxDB. We visualized the results after each benchmark with Eclipse Grafana. The resource consumption of each container, the so-called metrics, were monitored through Rafay, a management software/platform for Kubernetes clusters.

Performance Efficieny of Eclipse BaSyx – results

In Figures 5 – 7 below, the Grafana dashboard depicts a test series with 10-1000 VUs with a frequency of 10 requests per second per VU. The Eclipse BaSyx middleware was hosted on a cluster with four nodes, with four CPU cores and 16 GB RAM for each node. Each test lasted for five minutes, using write requests to query a random AAS.

Then we conducted a review for every test series. This led to the determination and verification of the following observations:

1. It makes no difference for the duration of the request if only one AAS is targeted with all VUs or if each request targets a random AAS.
2. It makes almost no difference for the duration of the request duration whether a write or read request was performed.
3. Regarding the duration of the request, the number of AASs made no difference.
4. Quadrupling the computational resources yielded a quadrupling of the number of processed requests, which indicates linear growth.
5. The minimum duration of a request was 2 ms due to network limitations.
6. With the hardware configuration used, Eclipse BaSyx responded to 1000 VUs (with 10 requests per second per VU, so 10,000 request/s) within 60 ms on average
7. The RAM consumption of the AAS Server was influenced by the number of AASs targeted by requests. If the number of AASs was increased by a factor of 40, the RAM consumption increased by a factor of 1.6.
8. The number of AASs did not affect the CPU consumption of the AAS Server

Conclusion

In a nutshell, we conclude that Eclipse BaSyx is able to handle over 10,000 requests per second, with little hardware consumption. In addition, the growth rate of the hardware consumption related to the growth rate of the number of AASs present in the system also looks positive. More specifically, every AAS in the system was found to be actively used and therefore present in the RAM of the AAS Server. In a real industrial setup, a subset of AASs would be actively used and the rest would be stored in the data backend until they are needed.

Despite these positive results, the resource consumption related to AASs in use is a potential bottleneck. This information will be considered for future development processes with regard to Eclipse BaSyx.

Tests with bigger clusters, more computational resources, and better hardware will be able to show the full potential of Eclipse BaSyx.

 

 

[1] ISO 25000 Standard. „ISO/IEC 25010 Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models.“ ISO/IEC, (2017).