GPT-6 Astra has gained the ability to drive a car
summary
DrivingBench evaluates whether frontier Large Language Models can control a physical vehicle by giving them direct access to a Toyota Corolla's steering, acceleration, and braking systems. The project tests these models on a fixed cone course to see if they can navigate a physical environment in real-time. Results show a significant gap in performance, with only one model achieving a full completion of the course. This experiment demonstrates the current limitations of AI in handling high-stakes, physical world interactions.
key points
- The benchmark uses a physical Toyota Corolla and a fixed cone course to measure progress, distance, and finish time for various AI models.
- GPT-6 Astra achieved the highest success rate, completing the course in 5:22, while other models like Claude and Grok struggled significantly.
- Performance is tracked via a 'best of three' attempt system within a single chat context, allowing models to reflect and adjust their driving strategy.
who should read it
AI researchers and automotive engineers interested in the intersection of LLMs and physical robotics. It is also useful for those tracking the progress of frontier models in real-world application tests.
discussion positive
consensus
Commenters generally agree that while LLMs show promise as a benchmark for high-level reasoning, they are currently impractical for real-time driving due to extreme latency issues. Most believe that any viable autonomous system must process controls locally to ensure safety and rapid reaction times. There is a general consensus that a hybrid approach, combining a high-level reasoning model with a low-level, fast-acting control model, is the most plausible path forward.
top insight
The distinction between perception-to-reaction latency and actuation frequency is critical; while humans may have slower overall reaction times, the continuous, high-frequency adjustments required to maintain vehicle stability (around 20Hz) are what make cloud-based LLMs fundamentally unsuitable for direct control. This explains why human drivers can function despite slow strategic reactions, whereas a car's physical control loop cannot.
Generated by an LLM from the article and thread. Check the sources before relying on it.