Show HN: Mini-AGI – Dynamic continual learning model trained on 8GB VRAM
summary
mini-AGI is a byte-level language model designed to enable continual learning on consumer-grade hardware with as little as 8 GB of VRAM. It avoids catastrophic forgetting by using a mixture-of-experts architecture where only a small subset of weights is loaded into VRAM at any time, with the rest stored on disk. By employing a reduced learning rate for the core model components and a dynamic expert pool, it can learn from a continuous stream of data without erasing previous knowledge.
key points
- The model utilizes a paging system that stores experts on disk and loads only the necessary working set into VRAM, allowing the parameter count to be limited by disk space rather than GPU memory.
- To prevent catastrophic forgetting, the system employs a 'trunk' learning rate that is significantly lower than the experts' rate, ensuring that core architectural updates do not overwrite specialized knowledge.
- The architecture features adaptive depth through a halting head, meaning the model spends more computation on difficult characters and less on easy ones, optimizing efficiency.
who should read it
Developers and AI researchers interested in small-scale, efficient language models and the implementation of continual learning on limited hardware.
discussion skeptical
consensus
Most commenters are highly skeptical of the project's claims, particularly the use of the term 'AGI' and the assertion that it solves catastrophic forgetting. They argue the model produces gibberish, lacks rigorous benchmarks, and relies on a simplistic learning rate trick that may only delay rather than prevent forgetting. A small minority finds the organic approach and the author's effort to build a local model inspiring.
top insight
A professor of continual learning points out that restricting the learning rate of the 'trunk' to prevent forgetting effectively stops the model from acquiring new global knowledge. This creates a stability-plasticity trade-off where the model preserves old data but lacks the capacity for the generalization and merging of memories required for true continual learning.
Generated by an LLM from the article and thread. Check the sources before relying on it.