How to win a beer with high-dimensional statistics
An author used iterative search to find ten random-looking words whose embeddings form a geometric circle, demonstrating that high-dimensional representation geometry can yield spurious patterns.
summary is enough A quick, entertaining read that illustrates how easily high-dimensional data visualization can produce misleading geometric artifacts.
summary
A researcher bet a colleague that a seemingly meaningful circular pattern found in word embeddings could be artificially replicated with unrelated words. Using an iterative optimization search over a 25,000-word vocabulary, he successfully found ten words that project into a clear circle and match a circulant Gram matrix. This highlights the danger of spurious geometric patterns in high-dimensional data analysis and automated interpretability research.
key points
- Given a 25,000-word vocabulary, there are roughly 3 x 10^37 possible combinations of 10 words.
- An iterative search optimizing for a target circulant Gram matrix successfully generated a spurious circular PCA plot from random-looking words.
- This demonstrates that low-dimensional projections of high-dimensional embeddings can easily trick researchers into seeing structure where none exists.
who should read it
Machine learning researchers and data scientists interested in embedding geometry and interpretability pitfalls.
discussion warm
consensus
Commenters appreciate the practical use of high-dimensional math for bar bets and note that high-dimensional data naturally concentrates on thin spherical shells due to the curse of dimensionality.
After seeing the claim, I knew it would work because of a few intuitive things:
1. Things that are the result of a very large number of small perturbations follow a Gaussian distribution.
1a. Word embeddings are the result of millions/billions/trillions of small perturbations (i.e. gradient descent)
2. As you increase the dimension, d, of a multivariate Gaussian distribution, the actual mass of the data is increasingly concentrated in the thin spherical shell. For MVN distributions with standardized marginal dimensions, this is on the sphere of radius \sqrt{d}.
In fact when I searched …
It explains the underlying mathematical mechanism, linking the phenomenon to the concentration of mass in multivariate Gaussian distributions on thin spherical shells.
Generated by an LLM from the article and thread. Check the sources before relying on it.