PSSA: A non-transformer language model written from scratch in Rust
PSSA is a 1.5M-parameter language model built from scratch in Rust using a selective state-space core and hyperbolic memory bank, claiming higher efficiency than transformers.
read it It provides a detailed architectural breakdown of a custom non-transformer language model, complete with equations, design rationale, and comparative benchmarks against a transformer baseline.
summary
The author introduces PSSA, a small language model avoiding the transformer architecture in favor of a selective state-space model, a hyperbolic memory bank, and plastic weight updates consolidated via ridge regression. Implemented entirely in Rust without ML frameworks, it achieves lower cross-entropy loss and faster CPU generation than a parameter-matched transformer on WikiText-103. While positioned as a research prototype at a 1.5M-parameter scale, the design highlights alternative routes to linear sequence scaling. The comment thread features lively debate on architecture lineage, skepticism over the framing, and clarification from the author on its relation to Neural Turing Machines.
key points
- PSSA combines a diagonal state-space recurrence with a 512-slot episodic memory bank in hyperbolic Poincare space to maintain linear scaling with sequence length.
- On a 12.7M-token WikiText-103 corpus, a 1.5M-parameter PSSA model achieved a held-out cross-entropy of 3.997 compared to the transformer's 4.429.
- Generation speed reaches 12x faster than a transformer on the same CPU due to a fixed-size recurrent state that avoids re-reading the full context history.
- Fast plastic weights are periodically folded back into base transition matrices using closed-form ridge regression.
who should read it
Machine learning researchers and engineers interested in state-space models, alternative LLM architectures, and building deep learning systems from scratch without heavy frameworks.
discussion mixed
consensus
Commenters generally recognize PSSA as part of the broader wave of alternative linear-scaling architectures like state-space models and Neural Turing Machines, while critiquing the 'written in Rust' subtitle as clickbait.
pushback
Some users expressed skepticism over the novelty of the design and questioned whether parts of context were AI-generated without fully working mechanics.
That's the closest prior work, yes, and the memory half sits squarely in that lineage. Three differences worth naming. Reads are content-addressed in the Poincare ball rather than by cosine similarity in R^n, which is what lets a bounded top-4 read cover a hierarchy of contexts instead of a flat neighborhood. There is no controller emitting explicit read and write heads; writes are novelty-gated with a refractory counter that rate-limits overwriting the same slot, so repeated contradictory updates do less damage. And the fast weights do not stay external forever, they get consolidated into …
The author steps in to clarify how PSSA differs from Neural Turing Machines through Poincare-ball content addressing and ridge-regression weight consolidation.
Generated by an LLM from the article and thread. Check the sources before relying on it.