Artificial Intelligence Dangerous Stubbornness: How a Single AI Agent Can Manipulate Entire Systems

Related Vendors

FAULHABER Drive Systems

Taiwan Machine Tool & Accessory Builders' Association (TMBA)

SIMTOS (Seoul International Machine Tool Show)

AI systems are increasingly working in teams to solve complex tasks. But this collaboration poses an underestimated danger: Researchers at the CISPA Helmholtz Center warn of manipulative AI agents that can hijack the decisions of entire networks simply by being “stubborn.”

Imagine a neighbor who stubbornly insists that you’re making unbearable noise at night. At first, the other residents disagree. But the neighbor doesn’t let up, repeating his accusations over and over again and skillfully building connections within the building. Eventually, the mood shifts—and everyone believes him.What sounds like a classic neighborhood dispute is, according to Samira Abedini, a researcher at the CISPA Helmholtz Center for Information Security, the perfect metaphor for a massive, new security problem in the world of artificial intelligence. Together with her team, she has uncovered how the malicious influence of a single AI agent can spread like a virus and manipulate the decisions of an entire AI group.

When AIs Work as a Team—and Influence One Another

The days when a single, massive AI system had to do everything on its own are over. Developers are increasingly turning to so-called multi-agent systems: specialized AIs share the work. In software development, for example, Agent A plans, Agent B writes the code, and Agent C handles error checking.For this digital teamwork—whether in office assistance, planning, or programming—to work, the agents must communicate with one another. What’s more, they must be able to adapt their own assessments to the insights of the others. But it is precisely this openness to outside arguments that is now becoming a gateway for sabotage.

Hacking Without Code: The Power of Persistence

“If we have an open network, AI agents from different providers can collaborate within it, for example,” explains Samira Abedini. “One of these providers could try to gain an advantage by configuring its agent to steer the network’s collective decision in a certain direction.”
The alarming thing about this is that this attack looks completely different from what cybersecurity experts are used to. There’s no malware, no stolen passwords, no overwritten code. “The agent neither takes control of the other agents nor alters their instructions,” says the researcher. “It simply exploits the regular communication between the agents, persistently advocates a specific position, and thereby attempts to change the others’ assessment.” In other words, the attacker uses the system against itself.
It was already known in research that such manipulation is possible in principle. But Abedini and her team weren’t satisfied with a mere before-and-after comparison. “For us, the important question was what happens in between,” she emphasizes.

Sociology Meets Algorithms: The Mathematics of Opinion Formation

To reveal the hidden dynamics between the machines, the researchers used an unusual tool: the Friedkin-Johnsen model. This mathematical model actually originates from the social sciences and describes how people change their opinions in social networks. It measures how stubbornly someone clings to their convictions, how susceptible their counterpart is to influence, and who holds how much power within the network. The surprise: The human model could be applied almost perfectly to the interacting language models (LLM agents). The researchers can now mathematically predict with precision under what conditions an AI attack will be successful.

Researchers can now mathematically predict with precision the conditions under which an AI attack will be successful.

The Boss Takes Center Stage: Why Architecture Determines Safety

The experiment clearly showed that manipulation is only successful if the agent is extremely “stubborn” and, at the same time, has sufficient influence over others who are willing to adjust their opinions. How “stubborn” an agent acts depends on the language model, the prompts, and the task.But the biggest vulnerability is the network itself. In a star-shaped network, where all agents communicate via a central hub, the attacker has an easy time of it if they occupy precisely this position of power. If they’re on the periphery, their influence fizzles out. In completely open networks, where everyone communicates with everyone else, it comes down to the weight of trust. Furthermore, the larger the group, the harder it is for a troublemaker to cause disruption. Abedini’s conclusion is clear: “This means that the very architecture of a multi-agent system itself becomes a security issue.”

The Solution: A Digital Lie Detector

To protect the system from the stubborn neighbors in the AI world, the CISPA team tested a “dynamic trust mechanism.” A central control authority repeatedly assigns the agents small test tasks for which it already knows the solutions. If an agent repeatedly gives the wrong answer—or lies—its internal “influence weight” decreases. From then on, its arguments carry less weight, and the other agents no longer listen to it. The mechanism showed initial success in experiments, but Abedini believes more research is needed for more complex scenarios. In any case, the goal of their future work is clear: the AI agents must continue to be able to listen to good advice from their colleagues—without blindly trusting every stubborn loudmouth.