GARMI: Assistant robot for care

Assistant robot with AI GARMI interacts with doctors and patients via ChatGPT

2024-05-22 From | Translated by AI 4 min Reading Time

Related Vendors

GARMI is an assistant robot for the care and supervision of people that understands commands via ChatGPT. It can also already grasp objects. However, it will be some time before it can be used safely in nursing homes.

Assistant robot communicates via ChatGPT: Doctors and patients can interact with GARMI.(Image: Andreas Heddergott / TU Munich) — Assistant robot communicates via ChatGPT: Doctors and patients can interact with GARMI.
(Image: Andreas Heddergott / TU Munich)

The assistant robot GARMI (GERiatric Assistant Munich) is an assistant robot being developed by the Technical University of Munich at the Geriatrics Research Center in Garmisch-Partenkirchen. As the researchers from the Munich Institute of Robotics and Machine Intelligence (MIRMI) of TUM demonstrate at the ICRA 2024 Robotics Fair in Yokohama, Japan, the robot not only understands various commands via ChatGPT, but also autonomously carries out different tasks and skills such as gripping objects, safe maneuvering, and communicating with patients.

In addition, it communicates live with doctors who are conducting telemedical examinations. "GARMI is now able to execute various individual skills that we have taught him over the past few years safely and on demand via ChatGPT," explains Dr. Abdeldjallil Naceri, the head of the geriatronics project.

Human-robot interaction

A digital twin is used to try out the movements of the robot in a simulation before the real interaction with humans in order to avoid collisions. Artificial Intelligence (AI) helps GARMI to grab and hand over cups and glasses without spilling liquid. Finally, ChatGPT acts as a link in the communication between robot, patients, physiotherapists, and doctors.

Researcher Naceri compares the technology installed in GARMI to developments in autonomous driving: "It takes many steps to develop a new feature like the autonomous parking assistant in a vehicle," the researcher said. "Just like in care robotics, the technology must be absolutely reliable and safe since it's used around people."

Grasping and precise remote movement

To enable GARMI to grasp objects much like humans, the researchers have combined a camera, a robot arm with seven joints and an artificial hand, as well as artificial intelligence. First, the camera takes a picture of the object that GARMI is supposed to grasp and identifies it as a cup, bottle, or ball using neural networks.

Since the camera only sees the object from one side, the system supplements non-visible areas of a cup, for example, by comparing what it sees with other images and reconstructs the complete 3D object. The researchers represent the probabilities with which the object is depicted as it really looks with a color-graded graphic, a heatmap. This allows the ideal position for the hand to be determined, which is supposed to grab a cup. In nine out of ten cases, the complex system is now capable of doing so. "If it works with a cup, our system can transfer this to all other cup shapes," says Naceri.

Interact remotely

To be able to research whether doctors can work together with patients via teleoperation, the researchers designed a special experiment. For this, they drew simple shapes on a digital graphics tablet. GARMI was also equipped with a pen in one hand and a camera in the other.

In a room away, GARMI was supposed to transfer onto a canvas what the researchers had previously drawn - so project a simple drawing into a complex robotic system. It turned out that the best circles, squares, and triangles were formed when GARMI autonomously used the camera.

In a new research paper, the researchers show how tools can be maneuvered past an object. The challenge is to keep an eye on the distances on the one hand, and to correctly assess the mobility of the robot arm with all its joints on the other hand. If this is successful, the robot can even dodge balls that are flying towards it.

Robot reacts within one millisecond

GARMI processes information at a rate of one millisecond. This applies to perception, interaction, and navigation alike. The force sensors of the robot arms register the slightest touches and react immediately. If a person accidentally bumps into the arm of the robot, it stops for safety reasons within one millisecond.

To prevent accidents, humans and robots first meet as digital twins in a virtual environment. This is essential because the assistant robot can theoretically reach speeds of up to 20 km/h in a nursing home. In the computer simulation, GARMI uses the Safety Motion Unit to register when a person gets too close to it through its sensors, and moves slower. When the person moves away, it moves faster again.

Subscribe to the newsletter now

Don't Miss out on Our Best Content

Business E-mail

Please enter a valid mailadress.

By clicking on „Subscribe to Newsletter“ I agree to the processing and use of my data according to the consent form (please expand for details) and accept the Terms of Use. For more information, please see our Privacy Policy. The consent declaration relates, among other things, to the sending of editorial newsletters by email and to data matching for marketing purposes with selected advertising partners (e.g., LinkedIn, Google, Meta)

Date: 08.12.2025

Naturally, we always handle your personal data responsibly. Any personal data we receive from you is processed in accordance with applicable data protection legislation. For detailed information please see our privacy policy.

Consent to the use of data for promotional purposes

I hereby consent to Vogel Communications Group GmbH & Co. KG, Max-Planck-Str. 7-9, 97082 Würzburg including any affiliated companies according to §§ 15 et seq. AktG (hereafter: Vogel Communications Group) using my e-mail address to send editorial newsletters. A list of all affiliated companies can be found here

Newsletter content may include all products and services of any companies mentioned above, including for example specialist journals and books, events and fairs as well as event-related products and services, print and digital media offers and services such as additional (editorial) newsletters, raffles, lead campaigns, market research both online and offline, specialist webportals and e-learning offers. In case my personal telephone number has also been collected, it may be used for offers of aforementioned products, for services of the companies mentioned above, and market research purposes.

Additionally, my consent also includes the processing of my email address and telephone number for data matching for marketing purposes with select advertising partners such as LinkedIn, Google, and Meta. For this, Vogel Communications Group may transmit said data in hashed form to the advertising partners who then use said data to determine whether I am also a member of the mentioned advertising partner portals. Vogel Communications Group uses this feature for the purposes of re-targeting (up-selling, cross-selling, and customer loyalty), generating so-called look-alike audiences for acquisition of new customers, and as basis for exclusion for on-going advertising campaigns. Further information can be found in section “data matching for marketing purposes”.

In case I access protected data on Internet portals of Vogel Communications Group including any affiliated companies according to §§ 15 et seq. AktG, I need to provide further data in order to register for the access to such content. In return for this free access to editorial content, my data may be used in accordance with this consent for the purposes stated here. This does not apply to data matching for marketing purposes.

Right of revocation

I understand that I can revoke my consent at will. My revocation does not change the lawfulness of data processing that was conducted based on my consent leading up to my revocation. One option to declare my revocation is to use the contact form found at https://contact.vogel.de. In case I no longer wish to receive certain newsletters, I have subscribed to, I can also click on the unsubscribe link included at the end of a newsletter. Further information regarding my right of revocation and the implementation of it as well as the consequences of my revocation can be found in the data protection declaration, section editorial newsletter.

ChatGPT uses a list of commands

The AI tool ChatGPT works as a translator between technology and human. It has learned various commands such as "Start the rehab", "Show me the weather for tomorrow", or "Call the doctor", and GARMI uses this tool to communicate with patients. "Currently, the researchers have a list of 15 to 20 commands that trigger certain actions.

"Potentially, we can expand it indefinitely," says robotics researcher Naceri, "we are thus one of the first institutes where robots and humans interact with each other using ChatGPT."

Robot hands are to take over finer tasks

The new universal GARMI is now active in a sample apartment in Garmisch-Partenkirchen. The main area of research will be the further development of hands that can take on increasingly finer tasks. It will be several years before GARMI is used in nursing homes. (heh)