Why I'm still bearish on LLMs after Navier-Stokes

summary

Large Language Models cannot act as fully autonomous replacements for knowledge workers because they require constant human oversight to prevent reward hacking and failures. Meaningful autonomy is only possible when tasks have rigorous, formal specifications, but creating such specifications is often more expensive and difficult than doing the work itself. Consequently, most LLMs will function as helpful but unreliable interns rather than independent agents. This structural limitation suggests that the massive valuations of frontier AI labs are based on an unrealistic narrative of imminent superintelligence.

key points

1. The ability of LLMs to solve complex mathematical theorems or find security vulnerabilities is a best-case scenario because those fields have rigorous, audited verifiers that prevent hallucinations. Most real-world knowledge work lacks this formal structure, making LLMs prone to failure when faced with small changes in task requirements.
2. Human review is an inadequate substitute for formal specification because it does not scale and is susceptible to sophisticated errors or malicious injections. Because human attention is the bottleneck in the production loop, the dream of a self-driving datacenter of AI geniuses is physically and cognitively impossible.
3. Only a few types of businesses can truly benefit from autonomous LLMs: those that can tolerate cheap failures, those with narrowly defined repetitive tasks, or those already invested in expensive formal validation like chip design. Most other firms will find more value in cheap, open-source models used as assistants rather than expensive frontier models used as agents.

who should read it

Business leaders and tech professionals who are evaluating the actual utility of AI agents versus the marketing hype surrounding frontier models.

discussion skeptical