Robotics Humanoids Put to the Test

Related Vendors

FAULHABER Drive Systems

Taiwan Machine Tool & Accessory Builders' Association (TMBA)

SIMTOS (Seoul International Machine Tool Show)

To use humanoid robots in industrial applications, they must meet application-relevant criteria. The Fraunhofer IPA has developed a benchmark that allows manufacturers and end-users to analyze humanoids in terms of energy efficiency, cleanroom suitability, and data security.

Humanoid robots fascinate with their human-like appearance. However, there is a significant gap between spectacular staging and actual capabilities. "For end users as well as manufacturers, a look behind the facade sometimes built by marketing agencies is essential," explains Simon Schmidt, Head of Automated Systems at Fraunhofer IPA. "The market is too volatile and non-transparent to reliably assess and evaluate humanoids for individual applications."

The Fraunhofer Institute for Manufacturing Engineering and Automation IPA has therefore developed a comprehensive benchmark for the standardized analysis of humanoid robots. The modular benchmark covers six application-relevant criteria and is based on internationally recognized industry standards.

Aligned with Industry Standards

The modular structure of the benchmark allows manufacturers, end-users, and software providers to specifically test the areas relevant to their application. Where possible, the benchmarking aligns with established and internationally recognized industry standards that have been in place for decades – for example, ISO 14644 for cleanroom suitability or ISO 10218 and ISO TS 15066 for functional safety.

The benchmark is divided into six key areas:

1. Technologies and Basic Capabilities: Examination of installed sensors, AI models, gripper types, as well as tests on walking speed, gripping forces, and manageable loads. Objective measurements are recorded using a 3D tracking system and force sensors.

2. Complex Capabilities: Evaluation of practical generic tasks such as climbing stairs, overcoming obstacles, movement and force precision, as well as reaction speed. The tests are intentionally designed to be challenging to allow comparability with future model generations.

3. Cleanroom Suitability: Evaluation of particle emission according to ISO 14644-14, outgassing behavior, and cleanability—crucial for use in the semiconductor, pharmaceutical, or food industries.

4. Functional Safety: Central to human-robot collaboration. Stability on various surfaces, force limitation during collisions, obstacle detection, and system behavior in case of failures are tested. Collision tests are conducted using the same force sensors as for collaborative industrial robots.

5. Cybersecurity: Four modules examine vulnerability management, secure lifecycle, network security, and penetration resistance—a critical factor given the increasing legal requirements.

6. Energy Efficiency: Measurement of battery life and power consumption in various scenarios (standing, walking, walking with incline and load). The results enable realistic deployment planning and optimization of charging cycles.

Benchmark Already successfully Applied

Using the Unitree G1 as an example, Fraunhofer IPA applied the benchmark comprehensively for the first time. The technical basis was a Unitree G1 EDU-4 with Dex3-1 3-finger hands and firmware version 1.04, delivered in May 2025. While the robot demonstrated good self-stabilization and potential suitability for ISO Class 5 cleanrooms, significant limitations were also revealed. During collisions, forces exceeding 500 Newtons can occur—far above the pain thresholds permitted by standards. Additionally, researchers identified a critical Bluetooth security vulnerability in the software version available at the time of testing, which allowed complete remote control by attackers. This vulnerability has since been resolved. In terms of energy efficiency, maximum operating times were shown to be 2 hours and 49 minutes on a single charge while standing, and 1 hour and 49 minutes in a typical scenario involving both standing and walking.

"Users can directly interpret the results and thus find the right humanoid for the right application," emphasizes Werner Kraus, head of research at the Fraunhofer IPA. The benchmark makes humanoids comparable not only with each other but also with proven automation components. This is particularly important because:

- demographic change necessitates the use of automation in previously manual areas;

- significant investment decisions require solid, objective evaluation criteria;

- safety standards for humanoids are not expected until 2028 (ISO 25785-1);

- regulatory requirements for cybersecurity are increasing;

- sensitive production environments require reliable data for contamination prevention.

The Fraunhofer IPA plans to test more humanoids and build a comparison database. Manufacturers and users can now commission individual benchmark modules up to comprehensive examinations and benefit from the existing infrastructure and expertise.

Deploying Humanoids in Industrial Operations With NVIDIA

Siemens and Humanoid Bring Physical AI to the Factory Floor