A Tiny Interpretable Codec for Analogy Structure in Qwen Hidden States

Abstract

This report describes a small experiment in extracting a human-readable semantic coordinate system from the hidden states of a frozen language model. A linear encoder maps Qwen2.5 activations for king, queen, man, and woman into a two-dimensional space where the classic analogy relation holds with high cosine similarity, the four classes separate, and learned gender and royalty directions are nearly orthogonal. Held-out vocabulary tests show partial generalization, while a pseudoinverse intervention provides small directional evidence for write access. The result is a compact proof of concept for building small, testable semantic interfaces around model internals.