Physical AI When AI Agents Prepare 3D Worlds for Reality

Related Vendors

NVIDIA GmbH

Taiwan Machine Tool & Accessory Builders' Association (TMBA)

SIMTOS (Seoul International Machine Tool Show)

For robots and autonomous systems to function reliably, they must be trained in realistic virtual environments. AI agents analyze 3D scenes, add physical properties, and prepare digital worlds for use in the real world.

Realistic 3D models alone are not enough for the development of robots, factories, and autonomous systems. In order to simulate, test, and train such systems, 3D assets require additional information—such as their structure, materials, scale, sensors, and physical properties.Processing this data is often complex and time-consuming. Nvidia aims to simplify the process with new Omniverse libraries in the Agent Toolkit. These software components provide AI agents with tools they can use to analyze 3D scenes, create workflows, flag errors, and prepare assets for simulations. This should enable developers to augment existing applications and workflows with physical AI capabilities without having to completely replace their established tools. 

The NVIDIA Agent Toolkit, with Omniverse libraries, integrates AI agents into the 3D tools that developers already use, helping to create simulatable worlds where robots, factories, and autonomous systems can be trained and tested long before they enter the real world.

Jensen Huang, founder and CEO of NVIDIA

Key Features of the Omniverse Libraries

Among the libraries' most important features are RTX-based sensor simulation, GPU-accelerated physics, and the preparation of 3D assets for simulations.

• Nvidia RTX Sensor Simulation:
  The Ovrtx library generates simulated output from 3D scenes for cameras, lidar, radar, and other sensor systems. Developers and AI agents can use it to test how physical AI systems perceive and react to virtual environments.
• Physical Behavior: 
  Ovphysx simulates physical properties such as collisions, mass, friction, and motion using GPU-accelerated physics. This allows teams to first test how objects and systems interact with each other in a simulation before conducting real-world tests.
• Simulation-ready 3D objects: 
  The CAD-to-SimReady features support the conversion of CAD data into OpenUSD-based SimReady assets. In the process, the 3D objects are assigned the properties required for physical AI, virtual testing, and simulation-based development processes.
   

Faster Preparation of Simulation-Ready Environments

The new features are designed to help developers move more quickly from individual 3D models to complete, simulation-ready environments. AI agents can take on repetitive tasks, check scenes, and identify missing or incorrect asset properties. However, human oversight remains crucial. The quality of a simulation continues to depend on the quality of the underlying data, the physical parameters used, and verification by the developers.
 

Integration with 3D and CAD Applications

Software vendors such as PTC are integrating the Omniverse libraries into their applications and workflows, such as the cloud-based CAD and product data management (PDM) platform Onshape. The platform uses Open-USD and Ovrtx to connect cloud-native design workflows with physical simulations. This ensures that product data remains linked to existing CAD, PDM, collaboration, and simulation processes. Developers and designers can thus integrate simulations more closely into their established workflows, rather than having to transfer product data between separate systems.
 

Libraries are Freely Available

The Omniverse libraries are freely available on GitHub. Also available on GitHub is a new blueprint that demonstrates how the libraries can be integrated into Blender. This allows AI agents to perform tasks such as scene verification and the preparation of simulation content even within a widely used 3D application.

AI Enters the Shop Floor

Physical AI in Manufacturing: Eight Examples