Rethinking On-Device AI: Challenges and Realities for Automotive and Robotics Workflows

Introduction to On-Device AI in Automation

Large language models (LLMs) and vision-language models (VLMs) are increasingly considered for use beyond traditional data centers. In automotive and robotics sectors, there is growing interest in running AI agents directly on vehicles or robots. This approach promises benefits such as lower latency, increased reliability, and the ability to function without constant cloud connectivity. However, deploying these sophisticated AI systems on edge devices presents several challenges that affect automation and workflow efficiency.

Popular Assumptions about Edge AI in Vehicles and Robots

Many developers believe that embedding conversational AI and multimodal perception directly on vehicles or robots will seamlessly enhance automation workflows. The assumption is that local processing eliminates delays and dependence on networks, enabling real-time decision-making and improved autonomy. While this is an appealing vision, it overlooks key technical and operational constraints that limit current feasibility.

Hardware Constraints and Model Complexity

LLMs and VLMs typically require substantial computational power and memory. Automotive and robotic platforms have strict size, weight, and power limitations that make integrating high-performance AI accelerators difficult. Although some specialized hardware exists, balancing model size, energy consumption, and thermal management remains a significant challenge. These hardware constraints often necessitate model simplifications that can reduce the accuracy or capabilities critical for complex automation tasks.

Latency and Reliability: Not Just About Proximity

Running AI models locally is often promoted as a solution to latency issues. However, real-time responsiveness depends not only on where computation occurs but also on model efficiency and system integration. Large models may still require significant processing time, which can be problematic for safety-critical automotive or robotic functions. Additionally, local hardware failures or software bugs could compromise reliability, potentially more so than managed cloud services with redundant systems.

Offline Operation: Benefits and Trade-offs

Operating without cloud connectivity is advantageous in environments with poor or no network access. Yet, offline deployment complicates model updates and maintenance. Unlike cloud-hosted AI, edge models require physical or wireless updates, which can be logistically complex and costly. Moreover, offline models may lack access to the latest data or improvements, potentially degrading performance over time and impacting workflow consistency.

Workflow Integration and Automation Complexity

Integrating on-device AI into existing automotive and robotics workflows is not straightforward. These workflows often rely on coordinated systems across cloud and edge components. Introducing large AI models locally demands rethinking data pipelines, error handling, and user interaction models. Without careful design, on-device AI could introduce new points of failure or complicate automation processes rather than streamline them.

Conclusion: A Balanced Perspective on Edge AI Deployment

The drive to embed LLMs and VLMs directly into vehicles and robots reflects a desire for more autonomous and responsive automation workflows. However, this approach must be evaluated with a clear understanding of current hardware limitations, latency and reliability factors, offline operation challenges, and workflow integration complexities. Rather than assuming on-device AI is an immediate solution, stakeholders should carefully assess where and how these technologies best fit within broader automation strategies.