Agricultural Robotics How Greenhouse Robots Should Respond to Voice Commands in the Future

Related Vendors

University of Bonn

Taiwan Machine Tool & Accessory Builders' Association (TMBA)

SIMTOS (Seoul International Machine Tool Show)

As part of the “AgriLoop” project, researchers from the University of Bonn (Germany) and Taiwan are developing a system designed to control greenhouse robots using natural language. Experts would be able to check on the condition of plants, issue instructions, and correct the robots’ actions.

Researchers at the University of Bonn, in collaboration with partners from Taiwan, are developing a system for collaboration between experts and greenhouse robots as part of the project “Learning Expert-Guided Crop Intervention through Human-in-the-Loop LLM Robotics,” or “AgriLoop” for short. With the help of artificial intelligence, 3D sensors, and natural language processing, robots will in the future be able to understand instructions such as “Examine this plant more closely” or “Remove this leaf” and translate them into safe actions. On the German side, the project is receiving 463,000 euros ($537,080) in funding from the Federal Ministry of Research, Technology, and Space (BMFTR).

Connecting Language with Spatial Perception

Modern greenhouse production faces challenges such as a shortage of skilled workers, increasing operational complexity, and the need for more sustainable cultivation methods. Robotic systems are already capable of monitoring plant populations and performing specific tasks. However, they often lack the ability to incorporate the experiential knowledge of plant experts and translate it into concrete actions.

This is where “AgriLoop” comes in. The project combines language models, image-language models, and vision-language-action models with physically embedded robotics and three-dimensional perception of crop stands. Experts will be able to use a natural-language user interface to ask questions about the condition of the plants, give instructions, and specifically influence the robots’ behavior.

To do this, the robot maps its surroundings using cameras and other sensors. The data is used to create a 3D model of the greenhouse, in which individual plants and plant parts are located. The AI links this model to the verbal instruction and uses it to generate an executable robot movement.

Humans Remain Part of the System

One example is the command “Remove this leaf.” For a human, the context usually makes it clear which plant and which leaf are being referred to, and how the action should be performed without damaging the plant. A robot, on the other hand, must first interpret the language, identify the correct object, determine its position, and plan an appropriate movement.

Until now, these abilities have often been developed separately. “People naturally associate language with their environment and with experiential knowledge,” explains Prof. Dr. Maren Bennewitz of the University of Bonn. “For a robot, this is much more difficult. It must interpret speech, identify the correct object in its environment, determine its spatial position, and derive an appropriate movement from that information.”

“AgriLoop” therefore takes a “human-in-the-loop” approach. Experts provide guidance, correct the robots, and evaluate their decisions. The system is designed to use this feedback to better adapt its actions to the situation at hand. The goal is not to replace human expertise, but to make it usable for robotic systems.

German-Taiwanese Cooperation

The German side is contributing robot platforms, methods for three-dimensional perception and reconstruction, real greenhouse environments, and expertise in crop production to the project. Among other things, the Taiwanese partners are developing components for multimodal speech processing, dialogue systems, the mapping of instructions to specific objects and actions, and learning from expert feedback.

In Bonn, Bennewitz and her doctoral student, Rohit Menon, are working on the project. Another doctoral position is also being funded by the project. “AgriLoop” is a partner project of the PhenoRob Cluster of Excellence and is based in the Transdisciplinary Research Area “Sustainable Futures” at the University of Bonn.

One goal of the project is to develop highly efficient AI models that can be deployed directly on robot platforms. This would eliminate the need for the systems to rely on large data centers for every request. The data collected in the project from communication between humans and robots is also intended to contribute to the further development of agricultural robotics systems.

The Demonstrator Is Intended to Illustrate the Approach

By the end of the project, a prototype will demonstrate how a greenhouse robot processes verbal instructions and translates them into specific movements. The initial focus is on integrating the individual technical components into a cohesive system and testing it under real-world conditions.

In the long term, such systems could assist specialists with time-consuming or repetitive tasks, such as inspecting plants or carrying out targeted maintenance measures. More precise interventions could also help ensure that water, pesticides, and other resources are used more efficiently. However, further research is needed before these systems can be put into practical use.

Four Questions for Maren Bennewitz

Why haven't robots been able to understand such instructions so far?

People naturally associate language with their surroundings and with their experiential knowledge. When a professional says, “Remove this leaf,” she recognizes which plant and which leaf are being referred to, and also knows how to go about it without damaging the plant. For a robot, this is significantly more difficult. It must interpret the language, identify the correct object in its environment, determine its spatial position, and derive an appropriate movement from that information. Until now, multimodal AI models, 3D spatial perception, and physical robot control have mostly operated in isolation from one another. “AgriLoop” aims to combine these capabilities into a seamless system.

How is "AgriLoop" supposed to solve the problem?

The robot uses cameras and sensors to map its surroundings and creates a 3D model of the plants in the greenhouse. At the same time, the system processes verbal instructions from experts via a user interface.

Language models and vision-language models integrate this information. They cross-reference what was said with the specific detail in the 3D model to which the instruction refers. This integration is intended to result in a specific and safe robotic action. For example, the sentence “Examine this plant more closely” could result in the robot making a targeted movement toward a specific plant.

Why Do We Still Need People?

Especially when working with plants, not every decision can be planned in advance. Experts draw on experience-based knowledge that depends on the specific situation. For example, they assess how healthy a plant appears, which leaf should be removed, or where special care is needed.

Humans therefore provide guidance, correct the robot, and evaluate its decisions. The system is designed to use this feedback to better adapt its actions to the requirements of the greenhouse. The goal is to make human expertise available to robotics, not to replace the experts.

What specific changes could this bring about?

In the long term, such systems could assist skilled workers with time-consuming and repetitive tasks, such as monitoring plant populations or carrying out targeted maintenance measures. This would be particularly relevant in situations where there is a shortage of workers or where large populations need to be checked regularly.

More precise interventions could help plan the use of resources in greenhouses in a more targeted manner. “AgriLoop” is initially developing the technical foundations and aims to use a demonstrator to show that collaboration between humans and robots is fundamentally possible on this basis.