Why Software Is the Actual Safety Factor Creating Trust in AI-Driven Vehicles

Related Vendors

dataTec AG

Taiwan Machine Tool & Accessory Builders' Association (TMBA)

SIMTOS (Seoul International Machine Tool Show)

From pedestrian detection to lightning-fast driving decisions, artificial intelligence is now at the forefront of safety-critical functions. But this capability also comes with a significant challenge: how can safety be ensured? This is where ISO/PAS 8800 comes into play.

Artificial intelligence (AI) is no longer just a function in modern vehicles—it is the driving force behind innovations. Even safety-critical functions are influenced by AI. And this is where the danger lies. The ISO/PAS 8800 is intended to provide a solution. This new specification is not a reinvention of the wheel. Instead, it represents a crucial advancement that bridges the gap between traditional safety standards for the automotive industry—such as ISO 26262 and SOTIF—and the complex nature of AI. The message is clear: AI safety is a system-level issue that must be continuously developed, validated, and managed.

Fundamental Shift from "Error" to "Inadequacy"

Conventional systems fail due to identifiable errors. AI, on the other hand, can function exactly as intended and still be unsafe. Its problems are often attributed to "inadequacies": gaps in training data, inability to generalize, or limited understanding of the real world.

While a conventional software error is like a broken gear that can be seen, measured, and replaced, an inadequacy in AI is more like a map missing a road. It is incomplete but not defective. A vehicle may turn onto the missing road, and the consequences are dangerous. This distinction is crucial. Safety is no longer just about eliminating software errors but also about understanding and managing uncertainties. This shift fundamentally changes how we demonstrate safety.

Proof of Safety Instead of Mere Test Report

One of the key insights from ISO/PAS 8800 is that AI safety cannot be reduced to a single metric or a final test report. An AI system must be safety-assured through various pieces of evidence, including requirements, architecture, datasets, test results, runtime safeguards, and operational monitoring. It is not about perfection but about identifying risks. Consider a pedestrian recognition model with an accuracy of 99.9 percent. That sounds impressive, but in practice, it means hundreds of errors. Thanks to ISO/PAS 8800, superficial metrics are questioned. What lies behind the 0.1 percent? The answer is always a story, not a number.

Data is a safety-critical asset. In an AI system, the dataset is part of the specification. If critical scenarios are missing, the system will fail. Therefore, the standard prescribes a structured lifecycle for datasets. In this process, data is treated with the same care as hardware or software. This requires continuous validation, maintenance, and refinement.

For this reason, a structured lifecycle for datasets must be fulfilled. Data must be treated with the same care as hardware or software. This requires continuous validation, maintenance, and refinement. If data is not under version control, not traceable, and not actively managed, it is impossible to establish a safety assurance.

New Approach for Verification and Validation

AI systems cannot be tested with conventional methods. Their high-dimensional inputs, non-deterministic behavior, and "black-box" nature render traditional approaches insufficient. ISO/PAS 8800 addresses this issue by promoting a multi-layered approach: verification at the component level, validation at the system level, and comprehensive operational checks. Simulation, scenario replay, and robustness tests become indispensable tools, allowing developers to safely examine dangerous or rare conditions that cannot be physically reproduced.

In practice, to validate an artificial intelligence for lane keeping, for example, you cannot simply drive a car through every imaginable snowstorm, type of tunnel glare, or every faded road marking. Instead, you can create a virtual testing environment that exposes the system to millions of edge cases overnight. This is the strength of simulation. ISO/PAS 8800 makes simulation an essential part of the assurance chain.

What is crucial is that this responsibility does not end at the factory gates. AI operates in a dynamic world with constantly changing conditions. Paragraph 14 of the standard therefore mandates continuous safety monitoring during operation. By collecting field data, detecting anomalies, and managing controlled over-the-air updates, it ensures that the system remains safe throughout its entire operational life. The vehicle being delivered today will face conditions that are unimaginable today. The only way to ensure safety is to view commissioning as the start of an ongoing safety dialogue.

The Unsung Hero: the Software Foundation

AI models often rely on conventional software, which forms the true foundation for safety around them. Artificial intelligence operates within deterministic C and C++ software that ensures safe behavior. This software validates inputs, checks their plausibility, restricts outputs, enforces timing and control limits, and triggers fallback mechanisms if the AI behaves unexpectedly or with insufficient reliability. These software safety barriers prevent uncertainty from becoming a danger. If these safety barriers fail, even a well-trained model can contribute to an unsafe outcome. Therefore, the rigorous testing of conventional software in AI-powered vehicles remains indispensable. Static analysis, enforcement of coding standards, automated unit tests, structural code coverage, integration tests, requirements traceability, and CI/CD-based verification all play a direct role in proving that the safety framework around the AI functions effectively.

AI is like a telemetry system in a high-performance race car: it analyzes tire grip, predicts the optimal racing line, and advises the driver on the perfect braking point. But if the roll cage is weak, the brake lines are corroded, or the fire suppression system fails, no control system can save the vehicle. Even the most advanced vehicle intelligence is only as reliable as the hardware and software ensuring the vehicle stays on the road.

What Development Teams Should Consider

For teams developing or evaluating AI-powered vehicle systems, clear operational boundaries for the AI should be explicitly defined. Datasets should be treated as controlled technical resources. Additionally, it is essential to test behavior in meaningful scenarios and edge cases. Developing deterministic software safety precautions is also beneficial. Furthermore, traceability between hazards, requirements, datasets, tests, and safety mechanisms must be ensured. Beyond that, the system should be monitored even after commissioning. Only these measures make artificial intelligence something that can be trusted on public roads.

No Complete Safety, but a Structured Framework

The ISO/PAS 8800 standard does not promise to make AI completely safe—this goal is unattainable. Instead, it provides a structured framework for minimizing AI risks. AI safety is not a property of a single model but of the entire system in which the model is embedded. At the core of this system is well-tested, reliable software. It is the foundation upon which all trust in automated vehicles must be built. Models are only as safe as their safety precautions. 

*Ricardo Camacho is Director of Product Strategy, Embedded & Safety Compliance at Parasoft