Artificial Intelligence FEV and Microsoft Are Working on Multimodal Speech, Text, and Gesture Interactions

Related Vendors

Taiwan Machine Tool & Accessory Builders' Association (TMBA)

SIMTOS (Seoul International Machine Tool Show)

FEV and Microsoft aim to integrate generative AI functions directly into vehicles—based on GPU-accelerated computing power from Nvidia and AI model microservices. The goal of the collaboration is to enable multimodal speech, text, and gesture interactions directly in the vehicle—independent of a permanent internet connection.

The focus of the collaboration is the use of so-called Small Language Models (SLM), such as Microsoft's Phi-4-mini-instruct in Microsoft Foundry, which is based on NVIDIA DRIVE AGX accelerated computing power. The solution enables functions like configuring the dashboard or individual vehicle profiles via voice command. At the same time, the system serves as a local AI safeguard for cloud-based Large Language Models (LLMs).

Economic Scaling of Software-Defined Vehicle Functions

Embedded Small Language Models enhance the intelligent functions and responsiveness of modern vehicles. Since inference takes place directly in the vehicle, key functions remain available even with limited or no internet connection. Additionally, embedded SLMs enable a reduction in backend and infrastructure costs, as cloud-supported LLMs can be supplemented or partially replaced depending on the use case. This helps economically scale software-defined vehicle functions.

Key Application Fields for Embedded GenAI

As part of the collaboration, FEV is exploring several application fields with high series potential:

• Automated and autonomous driving (SAE Levels 3 to 5): Multimodal GenAI models enhance the recognition of objects, traffic situations, and driving paths, especially in complex urban environments and edge cases.
• Driver and occupant monitoring: Embedded GenAI improves the detection of fatigue, distraction, or unusual behavior and increases the robustness of safety-critical functions through local availability—also as a backup for cloud-based systems.
• Personalized vehicle and HMI configuration: Vehicle functions and user interfaces can be intuitively adjusted via voice commands, for example, for different driver profiles or usage scenarios—without reliance on external cloud infrastructure.

Multimodal System Architecture on Nvidia Platforms

The underlying architecture is designed as a multimodal system and processes speech, text, and visual information. To achieve high performance in the constrained environment of an embedded vehicle system, FEV used synthetically generated data curated with Nvidia NeMo during the fine-tuning process to optimize the Phi-4-mini-instruct model. The resulting AI is then integrated and deployed on Nvidia Drive AGX, enabling further enhancement of model performance. The AI functions are operated as modular software services within the vehicle.

Dashboard Configurator as a Technology Demonstrator

Building on these technologies, the development service provider has created a dashboard configurator as a technology demonstrator to showcase the potential of intelligent vehicle interfaces. Using natural voice commands, a locally deployed SLM dynamically updates the dashboard—reducing dependency on a continuous cloud connection while enabling fast and responsive performance.

In the future, FEV plans to gradually complement or replace cloud-based AI functions with models running locally in the vehicle.