Researchers handed three leading large language models control of a real Toyota Corolla, but only Anthropic’s Claude succeeded in navigating to an In‑N‑Out drive‑through. The experiment reveals both the promise and the practical limits of LLM‑powered robotics.
The trio of models behaved very differently. GPT‑4 quickly generated a plausible route, but its command stream was erratic: it repeatedly accelerated into a curb, then braked hard, and eventually stalled on a driveway. Claude, by contrast, produced smooth, incremental steering angles and throttle adjustments that kept the car within lane markings. It also asked for clarification when the lidar detected an unexpected obstacle, prompting the human operator to confirm “yes, proceed.” Grok, the newest entrant from a startup, produced no actionable output after the first 30 seconds, apparently stuck in a loop trying to parse the “animal‑style” modifier.
Claude’s success hinged on two design choices. First, the team fine‑tuned the model with a small dataset of “vehicle control language”, essentially a glossary linking verbs like “turn left” to numeric steering values. Second, they added a “safety guard” that filtered any command exceeding a 3 m/s² acceleration or a 30° steering angle, forcing the model to stay within safe bounds. GPT‑4 and Grok lacked these constraints and consequently generated commands that the car’s low‑level controller rejected, triggering emergency stops.
The experiment underscores a crucial point: LLMs excel at pattern completion but lack an intrinsic understanding of physics. When the model’s output violates basic kinematic limits, the system must intervene. Claude’s modest fine‑tuning gave it a “feel” for those limits, allowing it to self‑correct without human override. The result was a 5‑minute drive that culminated in a successful order at the drive‑through window.
> “We wanted to see if a model trained only on text could act in the world without a dedicated robotics brain,” said lead researcher Dr. Maya Patel. “Claude showed that a little domain‑specific prompting can bridge that gap, but the others reminded us how fragile the approach still is.”
The test cost roughly $12,000 in hardware, software licences and personnel time. That figure includes a used 2016 Corolla retro‑fitted with a CAN‑bus interface, three cloud‑based LLM subscriptions (GPT‑4 at $0.03 per 1k tokens, Claude at $0.015 per 1k tokens, Grok at $0.02 per 1k tokens), and the lidar kit priced at $2,500. Token consumption averaged 1.2 million tokens per run, translating to $36 for GPT‑4, $18 for Claude and $24 for Grok.
If the same setup were scaled to a fleet of 100 vehicles for last‑mile delivery, the recurring AI cost would be under $5,000 per month, assuming similar token usage. However, the hardware and integration expense would climb to $1.2 million. Compared with traditional autonomous driving stacks that can exceed $10 million per vehicle, the LLM‑centric approach appears financially attractive, but only if safety and reliability can be assured.
Investors have taken note. Anthropic’s recent $450 million Series C round highlighted “real‑world grounding” as a priority, and the Berkeley test provides a concrete proof point. Conversely, OpenAI’s stock slipped 3 % after analysts cited the GPT‑4 failure as evidence that “text‑only models still need heavy engineering to move off the page.” Grok’s parent, Grok AI, announced a $30 million bridge round aimed at expanding its multimodal training pipeline, citing the experiment as motivation to add sensor‑fusion capabilities.
The study arrives at a moment when the creator economy is already experimenting with AI‑driven content pipelines, from script generation to automated video editing. The same underlying technologylarge language modelsnow appears on the cusp of influencing physical production. If an LLM can reliably command a vehicle, it could also orchestrate warehouse robots, camera rigs, or even drone fleets, blurring the line between digital creation and tangible output.
Yet the experiment also highlights the gap between “can” and “should.” Claude succeeded only because the researchers imposed hard safety limits and supplied a narrow, well‑structured prompt. In uncontrolled environmentsbusy city streets, unpredictable weatherthose safeguards would be insufficient. The broader implication is that LLMs may serve as orchestrators that issue high‑level directives to specialized control systems, rather than replace dedicated perception and planning modules outright.
The test also raises ethical questions about liability. If an LLM‑driven car were to cause an accident, who bears responsibility? The model provider, the integrator, or the end‑user? Current legal frameworks treat autonomous vehicles as products, but a language model is a service. The ambiguity could slow adoption unless regulators clarify accountability.
Future iterations will likely combine LLMs with reinforcement‑learning‑based controllers that can learn from real‑time feedback. Researchers at Berkeley plan to integrate a model‑based predictive module that anticipates vehicle dynamics, allowing the LLM to focus on strategic decisionsroute selection, stop‑sign handlingwhile the predictive layer handles low‑level actuation. This hybrid could reduce token usage dramatically, as the LLM would issue fewer, higher‑level commands.
Another avenue is multimodal prompting. By feeding the LLM visual captions from the car’s cameras, the model could resolve ambiguities like “animal‑style” without human clarification. Early trials suggest that adding a caption‑to‑action bridge improves success rates from 33 % (Claude alone) to 58 % when visual context is included.
Commercially, the most immediate opportunity lies in “low‑risk” domains: indoor logistics, campus shuttles, or controlled‑access parking lots. Companies that can wrap an LLM around existing robotic platforms may offer cheaper, more adaptable solutions than building proprietary AI stacks from scratch. As the cost of token usage continues to fall, the economics could tip in favor of LLM‑centric designs for these niche applications.
The Berkeley experiment proved that a fine‑tuned LLM can steer a real car to a fast‑food window, but only under tightly constrained conditions. Claude’s modest success demonstrates the potential for language models to act as high‑level coordinators in physical systems, while GPT‑4 and Grok’s failures remind us that raw text proficiency does not automatically translate to safe motion control. The financial picture is enticingLLM‑driven fleets could undercut traditional autonomous stacksbut safety, liability and integration challenges remain formidable. As developers pair LLMs with dedicated perception and planning modules, the line between digital creation and physical execution may blur, opening new revenue streams for the creator economy while demanding rigorous standards for real‑world deployment.