Machine learning Robots learn grasping from humans

Related Vendors

FAULHABER Drive Systems

Wuxi InfiMotion Technology Co., Ltd.

The IFL at the Karlsruhe Institute of Technology and the IAS at the University of Stuttgart are jointly developing a robot that learns human skills through imitation. For this purpose, they have established the ICM Future Lab Hapt-X-Deep, a research infrastructure unique in Germany.

Edgar Welte snaps his fingers in the air and brings the robot one step closer to being human. At least that's how it should be in the near future. For now, the researcher only remotely controls the robot and the humanoid Shadow Dexterous Hand with the Hapt-X virtual reality glove, which is grasping a tool right next to it.

Doctoral student and junior professor Rania Rayyes have established the ICM Future Laboratory Hapt-X-Deep at the Institute for Conveying Technology and Logistics Systems (IFL) of the KIT to develop a robotic gripping system that learns human tasks through imitation. Germany's first laboratory with the complete system from the company Shadow Robot is located in Karlsruhe (Germany), but the Institute for Automation Technology and Software Systems (IAS) at the University of Stuttgart is also involved in the research work.

The institutes are jointly developing a learning robot with a gripping system that can quickly, flexibly, reliably, and safely respond to changing requirements, product designs, or materials. Their research infrastructure, costing 212,000 US dollars, was funded by the ICM.

Imitation learning: Robots learn like apprentices

Programming robots for industrial tasks is currently a lengthy process. Writing the code is followed by testing and then rewriting until the machine performs a process reliably. These learning phases are too long for future production technologies, which rely on rapid adaptability. "Getting a robot ready for use should not take longer than training a new employee in the future," says Edgar Welte.

To achieve this goal, the team aims to teach robots to learn from humans. Just as apprentices learn from their mentors, their robotic gripping system should eventually learn from an operator how to use new tools, handle different materials, perform entire tasks, or respond to changes in production processes. "We employ autonomous imitation learning and deep reinforcement learning for our system. The robot intuitively and immediately learns skills through interaction with humans," explains Rania Rayyes. This approach shortens not only the initial programming process but also retooling times.

Hapt-X-Deep advances human-machine communication

Human learning is an exchange of information. The participants communicate with each other, interpret visual information, and ask follow-up questions. In the Hapt-X-Deep Future Lab, the researchers communicate with the robot via data. Edgar Welte generates data through his movements with the Hapt-X-Glove and instructs the six-axis cobot on how to rotate and grasp tools or workpieces. With its 20 actuators, the Shadow Dexterous Hand can almost exactly replicate human grasping movements, better than any other gripper on the market. The pressure sensors in the mechanical fingers provide direct feedback to the operator. Edgar Welte feels small air cushions in the Hapt-X-Glove when the weight increases or the tool is incorrectly positioned in the hand. He immediately adjusts his grip, and the gripper mimics his actions.

 The researcher corrects the robot's error in real time. This will also be an important function when the machine can later learn independently, relying on its artificial intelligence. "Through immediate corrections, we can quickly expand the capabilities of our system and save months of work on reprogramming," explains Rania Rayyes.

The Hapt-X-Deep Future Lab has a dual function. It serves as the hardware for researching and developing software-supported technologies, while also acting as a demonstrator for their functionality. "For me, it's a unique opportunity to work with such a high-quality lab for my research," says Edgar Welte. With his dissertation on "Interactive Imitation Learning for Dexterous Manipulation," he will make a decisive contribution to the development of the overall system. Additionally, Hapt-X-Deep is expected to host projects on sensor technology, alternative teleoperation methods like gesture recognition via motion tracking, or the control of individual fingers.

This also includes collaborations with companies like Bosch AI or other academic institutes. "Hapt-X-Deep is intended to open as many doors as possible in research," explains Rania Rayyes. One of these leads to the IAS at the University of Stuttgart to the group of junior professor Andrey Morozov. As part of collaborative projects, the researchers in Stuttgart will develop methods to ensure the safety and reliability of the adaptable robot.

Robots optimize themselves through error detection

Fault injection via digital twins will be used as a testing procedure for the system, with deep learning systems helping the robot to recognize anomalies independently and thus predict errors. Eventually, it should also be able to autonomously correct these errors based on what it has learned. This part of the software is being developed in Stuttgart and tested in the real-world lab of the IFL at KIT. Hapt-X-Deep is a state-of-the-art research infrastructure and simultaneously an interface through which the ICM brings together the best researchers in the country to jointly develop groundbreaking solutions for human-robot learning.