I tested 10 model/harness combinations on the same Three.js task
summary
The author tests various AI models and harness configurations to determine which combination produces the best results for creating a complex Three.js sci-fi hangar scene. By measuring metrics like duration, token usage, cost, and success rate, the author identifies efficiency and quality trade-offs. The data reveals how different model families and harness setups affect the final output and resource consumption.
key points
- Performance metrics are tracked across multiple models including GLM, Qwen, Luna, Sol, and Astra, with varying durations and token counts.
- Cost analysis is provided using OpenRouter-equivalent rates, showing significant disparities in total cost based on the model used.
- The effectiveness of each combination is measured by successful tool calls, browser rendering, and manual screenshot checks.
who should read it
Developers and AI researchers interested in benchmarking LLM performance for complex coding tasks and 3D graphics generation.
discussion positive
consensus
Commenters generally agree that while AI models can generate impressive Three.js scenes, they often fall short of production quality by relying on outdated libraries, ignoring advanced rendering techniques like tone mapping, and producing inefficient code. There is a strong consensus that the 'harness' or environment the model operates in significantly impacts the final output quality, with some models performing drastically better in specific frameworks. Overall, sentiment is mixed; users are amazed by the capabilities but frustrated by the 'demo wall' and the lack of professional-grade optimization.
top insight
The most valuable insight is that asking for a 'Three.js scene' leads to throwaway prototype results because that is not how professional game development is handled. To get production-quality results, users must guide models toward a professional asset pipeline, including independent model creation, mesh optimization, and a coherent renderer architecture, rather than treating the scene as a single one-shot prompt.
Generated by an LLM from the article and thread. Check the sources before relying on it.