Extracting Steering Vectors from J space
summary
The author investigates whether steering vectors for large language models can be derived by inverting the Jacobian lens, which maps internal activations to vocabulary tokens. While this method successfully steers simple behaviors like all-caps output, it fails to reliably produce complex behaviors such as refusal. The core limitation is that complex behaviors cannot be fully represented by a small set of tokens, making the resulting vectors brittle and prone to hallucinations.
key points
- By inverting the Jacobian lens for pairs of uppercase and lowercase tokens, the author created a steering vector that successfully forces the model to output text in all caps.
- Attempts to steer the model toward refusal behavior using sets of refusal-related tokens resulted in high hallucination rates and unreliable performance compared to traditional steering baselines.
- The research suggests that while simple token-based mappings in the J space can identify some behavioral directions, complex cognitive behaviors are not fully captured by the linear mapping to the vocabulary.
who should read it
AI researchers and developers interested in LLM interpretability, activation steering, and the internal mechanisms of model activations.
discussion positive
consensus
Commenters generally agree that while steering vectors can be used for commercial purposes like targeted advertising, this approach risks degrading model output quality and increasing hallucinations. There is a consensus that the overhead for applying these lenses is computationally cheap compared to inference, making the technical implementation feasible. However, a divide persists regarding whether such steering constitutes 'thinking' or merely a manipulation of outputs.
top insight
The idea of integrating steering vectors into an adtech business model allows model owners to monetize prompts by selling bids to advertisers for product placement, effectively creating a native AI version of SEO. This approach is viewed as a more controlled and trustworthy method for the model owner to capture value compared to the current industry practice of poisoning training data to influence AI mentions.
Generated by an LLM from the article and thread. Check the sources before relying on it.