Data protection No backup, no mercy—the fundamentals for more resilience in production

Related Vendors

Auvesy GmbH

FAULHABER Drive Systems

Wuxi InfiMotion Technology Co., Ltd.

Cyberattacks threaten production sites. Effective data protection is essential. How can modern backup strategies and version control help companies avoid production outages and become more resilient?

Stefan Jesse is Group CEO and spokesperson of the Executive Board at AMDT.

More and more production facilities are facing attacks from cybercriminals. According to the current Threat Intelligence Report, the proportion of hacker attacks on manufacturing plants and industrial companies rose to 41 percent in the first half of 2024. In the previous year, the proportion was still 20 percent. Among the affected companies are prominent names like the battery manufacturer Varta or the automotive company Volkswagen, which faced alleged hacker attacks from China. The European Union has recognized the potential threat and aims to improve network and information security in various sectors such as energy, transport, or banking, as well as in production and processing with the NIS-2 directive.

In addition to external threats, internal manufacturing errors can also lead to costly rejects or recalls and thus to a loss of reputation. Companies, therefore, need security strategies that comprehensively protect both IT components that control data and OT components that operate hardware. But what is the status quo of industrial companies in Germany? What can they learn from global IT disruptions like the Crowdstrike incident? And why is version control so important as part of a disaster recovery plan?

More production resilience through disaster recovery and version control

There are many documented cases where seemingly small errors have led to massive economic consequences. The following example clearly illustrates this: A cleaning agent manufacturer produced five million bottles of household cleaner over several weeks, which were then shipped to North America. However, because a particular additive was not included in the production, the product was not only unusable but also harmful to health. An employee made a typo, which resulted in an error creeping into the machine control configuration.

Since even short downtimes can lead to enormous costs, quickly restarting production is of essential importance. Ideally, a disaster recovery strategy employs backup and versioning software. This allows snapshots of the project and program files of an industrial production control system to be taken at specific times. These can be conducted, for example, at the start of a shift or at any other meaningful time. In the event of a disruptive incident, the current version can be compared with any other version, and a desired state can be restored.

IT/OT interface as a gateway

In modern industrial enterprises, the combination of information technology (IT) and operational technology (OT) is essential. While IT components control the data, OT components operate the hardware. Unfortunately, traditional IT security mechanisms often do not work in OT facilities, making them attractive entry points for attackers.

Production environments are often very complex and fragmented because they include multiple device families from manufacturers such as Siemens, Rockwell, Fanuc, or Mitsubishi. Additionally, some control systems run on Windows XP or even older operating systems. Many companies attempt to tackle this heterogeneity with network segregation. This means building an IT fence around the industrial control systems. However, even if the pasture's fence is secure, the sheep are far from safe.

Double protection: electric fence and sheep guard

A practical example: One of the world's largest dairy companies operates a dairy plant in Europe where a tanker truck full of milk is processed every two minutes—24 hours a day, seven days a week. The milk producer's very advanced technology works with a shop floor template based on a Cisco Catalyst switcher. Since this OT component could serve as the main entry point, it must be protected with backups and version controls. By combining the switcher with the backup and versioning software Octoplant, the setup can be monitored in any Network Operation Center. Thus, to stay with the imagery, both the pasture fence and the sheep are protected.

Lessons from the Crowdstrike mishap—no backup, no mercy

The Crowdstrike incident led to significant disruption. Domain controllers had to be restored, which naturally play a central role in Active Directory domains. But what lessons can we learn from the global IT disruption? In the brave new world of AI, we still try to optimize that last ounce. However, we should first do our homework. What is common practice on personal smartphones, creating a backup of one's own data, should also be standard in the industry. Unfortunately, the saying holds true: No backup, no mercy. Additionally, an overview of all deployed assets and version control that versions all software inventories is of great importance.

A practical example from not too distant past: An automobile manufacturer with a production facility the size of a small town used to secure backups on USB sticks before introducing Octoplant. Engineers would ride a bicycle to the machine, make the backup on a stick, collect them in a basket, and then register them into the file system. It is obvious that such a system reaches its limits.

Fast restart of production

Industrial robots have long since made their way into the automotive industry and typically feature a small box reminiscent of a Raspberry Pi. If the program code stored here is incorrect, the robot will do what seems right to it. To prevent this, the code must be secured and versioned as a whole. This way, it can be compared with the last working version. The comparison shows where the coordinates deviate. Has the temperature changed, the speed, or the throughput? It is important to back up the entire production code as well as the files uploaded to the machine in the language of production control. This is precisely what modern backup and versioning tools achieve.

Disaster recovery also promotes ecological sustainability

The initial example of the misproduced household cleaners vividly shows the economic impact that a lack of security in production processes can have. Added to this is the ecological component. In this case, an individual error and a missing warning mechanism resulted in millions of bottles of household cleaner being not only unusable. The fact that 2,000 trucks drove through North America also led to unnecessary greenhouse gas emissions.

The case highlights the overall economic threat posed by a lack of supply chain resilience. Not too long ago, toilet paper was a highly sought-after commodity for which some resorted to unscrupulous means. What if baby formula is in short supply next time? Therefore, highly automated logistics or production systems should also be protected with software. A logistics company from the States, for example, processes more than 30,000 packages per hour at each of its logistics centers. During the Covid period, this logistics company ensured that vaccines were delivered in the correct quality. Modern backup and versioning solutions thus also help ensure high supply chain resilience.

How we can avoid the most common mistakes

The assumption that IT will handle everything is a fallacy. Those responsible for information processing are often unfamiliar with protocols like Profinet or Profibus. In addition, access to the devices is often very individual and not known to IT. Industrial control systems are much more heterogeneous and complex compared to IT devices. A lot of domain-specific and in-depth technical OT knowledge is necessary, as well as the understanding that all control systems truly need to be secured. Those responsible should not rely on solutions deemed "good enough," as backups sometimes occur on USB sticks or with attached versioning notes on the file system, which can prove inadequate in an emergency.

As part of NIS2, a significant improvement can be expected in this regard, as companies must demonstrate that they are up to date with the latest technology. The required audit protocol, for example, provides this necessary evidence by showing that a backup of production data is available at any time and at any status.