# Centers, Not Scores: KV Compression from the Pre-RoPE Geometry of Attention
### Paper reading group companion
> Mao, Lin, Huang, Xie, Fu, Zhuang, Han & Chen (MIT / NVIDIA / ZJU), 2026 preprint (arXiv:2604.04921).
> A KV-cache compression method that estimates which cached tokens matter *before* a query
> ever arrives — by reading the stable geometry of query/key vectors in the **pre-RoPE** space
> instead of watching recent attention scores.
---
## Contents
1. [TL;DR](#1-tldr)
2. [Glossary](#2-glossary)
3. [Helpful clarifications](#3-helpful-clarifications)
4. [Help with diagram interpretation](#4-help-with-diagram-interpretation)
5. [Honest caveats](#5-honest-caveats)
6. [Discussion prompts](#6-discussion-prompts)
7. [Key papers referenced](#7-key-papers-referenced)
8. [Explore in more depth](#8-explore-in-more-depth)
- [8.1 RoPE in depth: geometry, integration, and what this paper changes](#81-rope-in-depth-geometry-integration-and-what-this-paper-changes)
- [The positional-encoding menu](#the-positional-encoding-menu)
- [RoPE mechanics](#rope-mechanics)
- [Periodicity and the distance-scale spectrum](#periodicity-and-the-distance-scale-spectrum)
- [Where positions enter: original Transformer vs. modern models](#where-positions-enter-original-transformer-vs-modern-models)
- [How RoPE is wired into a transformer runner](#how-rope-is-wired-into-a-transformer-runner)
- [What TriAttention changes about the runner](#what-triattention-changes-about-the-runner)
- [8.2 The KV-cache compression landscape](#82-the-kv-cache-compression-landscape)
- [8.3 Attention sinks and the structure of attention](#83-attention-sinks-and-the-structure-of-attention)
- [8.4 Long reasoning and test-time compute](#84-long-reasoning-and-test-time-compute)
- [8.5 Directional statistics and Fourier views of attention](#85-directional-statistics-and-fourier-views-of-attention)
- [8.6 Attention heads and head-sharing architectures](#86-attention-heads-and-head-sharing-architectures)
- [A field guide to head types](#a-field-guide-to-head-types)
- [How heads are shared: MHA, MQA, GQA, MLA](#how-heads-are-shared-mha-mqa-gqa-mla)
- [In practice: trained-in, not bolted on](#in-practice-trained-in-not-bolted-on)
- [8.7 The argument, geometrically: one step at a time](#87-the-argument-geometrically-one-step-at-a-time)
- [Step 0 — taming high dimensions](#step-0--the-move-that-tames-high-dimensions)
- [Steps 1–6 — the argument](#step-1--qk-are-highly-concentrated-around-fixed-centers)
- [Step 7 — from the picture to the scoring function](#step-7--from-the-picture-to-the-scoring-function-the-second-paragraph)
- [8.8 Offline vs. online: the two-phase pipeline](#88-offline-vs-online-the-two-phase-pipeline)
- [8.9 The complex-number representation: from dot product to Equations 15 and 16](#89-the-complex-number-representation-from-dot-product-to-equations-15-and-16)
- [8.10 The constant-Q/K approximation: deriving the trigonometric series (Equation 17)](#810-the-constant-qk-approximation-deriving-the-trigonometric-series-equation-17)
- [8.11 Is the reasoning trace worth caching? Faithfulness, "reasoning theater," and cross-turn memory](#811-is-the-reasoning-trace-worth-caching-faithfulness-reasoning-theater-and-cross-turn-memory)
---
## 1. TL;DR
**The problem.** Reasoning LLMs emit chain-of-thought sequences tens of thousands of tokens long,
and the **KV cache** grows in lockstep, so memory — not compute — becomes the binding constraint on
long generation. The standard fix is **KV-cache compression**: keep only the tokens a future query
will actually attend to, and evict the rest. But how do you know which tokens matter *before* the
relevant query exists? The dominant family of methods estimates importance from **attention scores
of recent queries** — and those queries are **post-RoPE**, meaning they have been rotated by their
position. Because the rotation is position-dependent, only the most recent handful of queries carry
an "up-to-date" orientation, so the usable observation window is tiny (prior work finds it peaks
around ~25 queries and then *declines*). A key that happens to receive little attention during that
short window can be evicted permanently — even if it becomes essential thousands of tokens later.
That failure mode is especially bad for **retrieval heads**, whose target tokens can lie dormant for
a long time, and in reasoning it severs the chain of thought.
**What the paper does.** It moves the importance estimate out of the unstable post-RoPE space and
into the **pre-RoPE** space — the query/key vectors *before* positional rotation is applied — where
it reports a striking regularity and builds a method on top of it:
1. **Q/K concentration (the observation).** Across the great majority of attention heads, the
pre-RoPE Q and K vectors are tightly clustered around a fixed, non-zero **center** that is stable
across token positions *and* across input content. Quantified by the **Mean Resultant Length**
*R* (a directional-statistics measure; *R* → 1 = perfectly concentrated), roughly **90% of heads
in Qwen3-8B have *R* > 0.95**, and the value barely moves across math/code/chat domains. Crucially,
this is observed in **pre-RoPE space** — *before* any positional rotation — so position can't be
creating the similarity; and because the clustering also holds across prompts and positions, the
prompt isn't creating it either. By elimination, the stability is **model-intrinsic** (a property of
the learned weights), not a coincidence of the prompt. (The "because" in the paper's own phrasing is
really *"and therefore we can conclude"* — unpacked in §3.)
2. **A trigonometric series (the mechanism).** When Q/K can be replaced by their centers, the RoPE
attention logit collapses into a function of the **Q-K distance Δ alone** — a trigonometric
series whose coefficients are set by the centers. That series traces an **attention-vs-distance
curve** with peaks at particular distances, so the centers *predict* which key positions a head
will favor (nearby keys, far-away "sink" keys, etc.). The authors verify this by reconstructing
real attention from the centers and measuring a **reconstruction correlation** that averages
above 0.5 across three model families.
3. **TriAttention (the method).** Score every cached key by combining two terms: a
**trigonometric-series score** S_trig that uses the Q center as a stand-in for future queries to
estimate expected attention at each distance, and a **norm-based score** S_norm that captures the
leftover importance the center can't explain. The two are blended *adaptively per frequency band*
using concentration (1 − R_f), so the trig series dominates where concentration is high and the
norm term picks up the slack where it isn't. Keys are scored against many future offsets, then the
top-B are retained.
**Why it matters.** On AIME25 with 32K-token generation, TriAttention matches Full-Attention
accuracy while delivering **2.5× higher throughput** *or* **10.7× KV-memory reduction**, where
leading baselines (R-KV, SnapKV) land at roughly half the accuracy at the same efficiency. At a
fixed budget it nearly doubles R-KV on the AIME benchmarks, and on a purpose-built **memory-retention
(recursive DFS) benchmark** it tracks Full Attention up to high recursion depth while R-KV collapses.
The conceptual payoff is the bigger story: importance can be read off a **stable, model-intrinsic
geometry** rather than a noisy, position-dependent observation window. **Caveats:** the headline
results are concentrated on mathematical-reasoning benchmarks (general-task results live in the
appendix); the method leans on the prior pre-RoPE / frequency-analysis line of work (see §5); and the
reconstruction correlation that grounds the whole story is *moderate* (means ~0.5), not near-perfect.
---
## 2. Glossary
Terms are grouped by theme rather than alphabetized, so the list reads roughly in the order the
ideas appear in the paper — it can double as a quick warm-up for newer members.
Terms marked **[paper]** are this paper's own framing or coinage rather than field-general
vocabulary — useful to flag so the borrowed concepts aren't confused with the authors' specific
definitions.
### Positional-encoding concepts
**KV cache** — The stored key and value vectors for every already-processed token. Because a causal
model's earlier representations never change, K/V are cached and reused instead of recomputed at each
decoding step. Its size grows linearly with sequence length, which is the bottleneck this paper
attacks.
**RoPE (Rotary Position Embedding)** — The dominant positional scheme in modern LLMs. It splits each
Q/K vector into 2-D subspaces and *rotates* each subspace by an angle proportional to the token's
position, so the attention dot product ends up depending only on the *relative* offset between two
tokens. See §8.1.
**Frequency band *f*** — One of the d/2 2-D subspaces RoPE rotates. Band *f* rotates at frequency
$\omega_f=\theta^{-2f/d}$ (θ = 10000 typically), so different bands "spin" at very different rates — a
**geometric** progression of frequencies, not the harmonic one of a classical Fourier series.
**Pre-RoPE vs. post-RoPE** **[paper-emphasized]** — *Pre-RoPE* = the Q/K vectors **before** the
positional rotation is applied; *post-RoPE* = after. The entire method hinges on the fact that
pre-RoPE vectors carry no positional rotation, so their geometry is stable across positions, whereas
post-RoPE vectors are smeared out into position-dependent arcs.
**Dominant frequency band** **[paper]** — For each head, the band(s) contributing most to the
attention logit, ranked by $C_f=\mathbb{E}[\lVert q_f\rVert]\cdot\mathbb{E}[\lVert k_f\rVert]$. The paper visualizes Q/K concentration in the
2-D plane of these dominant bands (typically the top 2). Borrows the band-selection idea from
EliteKV.
### Concentration concepts
**Q/K concentration** **[paper]** — The central empirical claim: in pre-RoPE space, a head's Q
vectors (and K vectors) cluster tightly around a single fixed direction/center, stable across
positions and contents. This is what makes a *center* a meaningful summary of the head's behavior.
**Center (E[q_f], E[k_f])** **[paper]** — The mean pre-RoPE Q (or K) vector in a frequency band,
computed once, offline, from calibration data. The method uses the Q center as a proxy for *all
future queries*.
**Mean Resultant Length (MRL), *R*** — A standard directional-statistics measure of how tightly a
set of vectors points the same way: *R* = ‖E[q]‖ / E[‖q‖]. *R* → 1 means perfect concentration
(all vectors aligned); *R* → 0 means uniform dispersion. Computed per band as R_f. Doubles as the
adaptive-weighting knob in the scoring function.
### Attention-vs-distance concepts
**Trigonometric series** **[paper-central]** — When Q/K are replaced by their (constant) centers, the
attention logit becomes Σ_f [a_f cos(ω_f Δ) + b_f sin(ω_f Δ)], a sum of sinusoids in the Q-K distance
Δ whose coefficients are fixed by the centers. The frequencies are RoPE's (geometric), so it's a
*synthesis* analog of Fourier, not a literal Fourier series.
**Q-K distance Δ** — $\Delta=p_q-p_k$, the gap in positions between a query and a key. The trig series
turns "which key is important?" into "what attention does this head assign at distance Δ?"
**Phase difference φ_f** — arg(q_f) − arg(k_f), the angular offset between a Q and K component in band
*f*. Together with the band amplitudes it sets where the attention-vs-distance curve peaks.
**Distance preference** **[paper]** — The fact that a head's attention-vs-distance curve peaks at
particular Δ, so the head systematically favors keys at certain distances (nearby, or far-away
sinks). Encoded entirely in the centers.
**Attention sink** — A token — typically among the first positions (often the initial/BOS token) — that
absorbs a large share of attention while contributing almost nothing to the output, because its *value*
vector has near-zero norm. It acts as a "no-op" place for softmax to dump probability mass when no real
key is a good match (softmax must sum to 1). In this paper's framing a *sink head* is just the special
case where the centers produce a curve peaking at very large Δ (the earliest tokens), and the small
value-norm is precisely why the S_norm term is needed. See §8.3 and Demo 5.
**Reconstruction correlation r̄** **[paper]** — The validation metric: for each query, the Pearson
correlation between its *actual* attention logits (over log-spaced distances) and the curve
*predicted* from the centers, averaged over queries/heads. r̄ = 0.72 for the first head of the first
layer; per-model means ~0.51–0.56.
**Pearson correlation coefficient (r)** — A standard statistic measuring how well two quantities move
together *linearly*, on a scale from **−1 to +1**: r = +1 is a perfect straight-line match, r = 0 is
no linear relationship, r = −1 is a perfect inverse. It is scale- and offset-invariant — it compares
the *shapes* of two curves, not their absolute magnitudes — which is why the paper can use it to ask
"does the predicted attention-vs-distance curve rise and fall *in step with* the real one?" without
the two needing the same units. Rules of thumb often read |r| ≈ 0.5 as a *moderate* association and
|r| ≳ 0.7 as *strong*, so the paper's r̄ ≈ 0.5 means the center-based prediction tracks real attention
noticeably but imperfectly (hence the norm-term fallback and the §5 caveat). Note r captures only
*linear* agreement: a real but non-linear relationship can show a modest r, and correlation alone is
not causation — see the §6 discussion prompt.
### Method terms
**KV budget (B)** — The fixed number of tokens the cache is allowed to retain. Compression aggression
is set by B (the paper's default is 2048; some settings use 512–4096).
**S_trig (trigonometric-series score)** **[paper]** — Scores a cached key by the expected attention
the Q center would give it at its distance Δ: Σ_f ‖E[q_f]‖·‖k_f‖·cos(ω_f Δ + φ_f). Captures distance
preference.
**S_norm (norm-based score)** **[paper]** — A complementary term, $\sum_f (1-R_f)\,\mathbb{E}[\lVert q_f\rVert]\,\lVert k_f\rVert$,
weighting each band by how *un*-concentrated it is. Algebraically equals Σ_f (E[‖q_f‖] − ‖E[q_f‖])·‖k_f‖
— i.e. it scores by the *spread* around the center, not the center itself.
**Adaptive weighting via concentration** **[paper]** — The (1 − R_f) factor: where a band is highly
concentrated (R_f high), the trig series is trusted and S_norm contributes little; where concentration
is weak, the norm term is preserved. Final score S = S_trig + S_norm.
**Future offsets D** **[paper]** — Because a key may be queried from *any* later position, each key is
scored at many offsets and averaged: D = {1, 2, 4, …, 2¹⁶}, **geometrically spaced** so near distances
(which are numerous) get denser sampling.
**Window-based pruning** — To avoid scoring every key at every step, pruning is triggered once per
**128-token window**; when the cache exceeds B, keys are scored and the top-B retained. (Same cadence
as R-KV.)
**Query head / KV head** — Terms from the head-sharing architectures (§8.6). An attention head splits
into a **query** projection and a **key/value (KV)** projection. In classic Multi-Head Attention every
**query head** has its own **KV head** (its own K and V). To shrink the cache, modern models let several
query heads *share* one KV head: **MQA** shares a single KV head across all query heads, and **GQA**
(Grouped-Query Attention) splits the query heads into G groups, each group sharing one KV head. So "G
query heads per KV head" just means G query projections read from the same cached K/V — which is why a
single cached key receives G differently-scaled scores and needs the reconciliation below. Full
taxonomy (MHA / MQA / GQA / MLA) is in §8.6.
**Normalize-then-aggregate (GQA handling)** **[paper]** — In Grouped-Query Attention each KV head is
shared by G query heads (see *Query head / KV head* above, and §8.6), giving each key G differently-scaled
scores. The fix: z-score each head's scores, then take the **max** across the G heads, so a key survives
if *any* sharing query head deems it important.
**z-score (standardization)** — A way to put numbers from different sources on a common scale by
re-expressing each value as *how many standard deviations it sits from its group's mean*:
z = (x − μ) / σ, where μ is the mean and σ the standard deviation of the group. The result is centered
at 0 with unit spread, so z = +2 means "two standard deviations above average" regardless of the
original units. Here it matters because the G query heads sharing a KV head produce scores on different
scales; z-scoring each head's scores first makes them comparable, so the **max** across heads reflects
*relative* importance within each head rather than just whichever head happens to output the largest raw
numbers.
**Offline calibration vs. online pruning** **[paper]** — TriAttention runs in **two phases**. *Offline*
(once, before deployment) a **calibration** pass over a corpus estimates each head's per-band centers
(E[q_f], E[k_f]), concentrations R_f, and norm statistics — pure statistics collection over forward
passes, with **no weight updates**, so it is *not* retraining. Those fixed numbers become the parameters
of S_trig/S_norm. *Online* (per request, during decode) those statistics are used to score cached keys
and prune the cache in 128-token windows, with no further calibration. §5.2.3 reports the offline
statistics are robust even to *mismatched* calibration data. See §8.8 for the full pipeline.
### Evaluation terms
**AIME 24 / AIME 25** — Competition-level math benchmarks (30 problems each) requiring long multi-step
reasoning. The paper's headline numbers.
**MATH 500** — A 500-problem subset of the MATH dataset spanning diverse mathematical reasoning.
**Recursive State Query / memory-retention benchmark** **[paper]** — A custom DFS-simulation task: the
model must traverse a graph, retain intermediate stack states, and backtrack. Deeper recursion = more
intermediate states to keep = more memory pressure, making it a direct probe of whether pruning
discards critical mid-sequence information. Scored by **stack exact match**.
**Retrieval head** — An attention head whose job is to fetch a specific earlier token when it becomes
relevant. These are the heads most damaged by importance estimates drawn from a short observation window, because their target
can stay dormant for a long time. They're one of several specialized head *types* (alongside
**induction heads** and **streaming/sink heads**) — see the dig-deeper section §8.6 for the taxonomy
and why it matters for KV compression.
---
## 3. Helpful clarifications
Several things in the paper are easy to misread. Each is worth getting straight before the findings.
### "Pre-RoPE concentration" is about *direction*, measured per frequency band
The claim is **not** "the raw Q/K vectors are all nearly equal." It's that, **within a single
frequency band's 2-D plane**, a head's pre-RoPE Q (and K) vectors point in nearly the same *direction*
— tightly clustered around one center, regardless of token or prompt. That's why the metric is the
**Mean Resultant Length** (a *directional* statistic) and why the figures plot Q/K in the complex plane
of a head's *dominant* bands rather than the full vector. (Why this implies the stability is
*model-intrinsic* is the next clarification.) *(Demo 1 in the interactive companion lets you watch a
head's pre-RoPE cloud tighten or disperse and reports its Mean Resultant Length R against the 0.95
threshold.)*
### "Intrinsic" is an inference from ruling out alternatives — not a mechanism
The paper writes that *because* the pre-RoPE vectors are untouched by positional rotation, the
stability is intrinsic. That **because** is misleading: being in pre-RoPE space doesn't *cause* the
clustering. The clustering is produced by the model's weights either way. Pre-RoPE space is simply the
**vantage point** from which you can *conclude* that the cause is the weights and not something else.
The reasoning is one of elimination:
- **Tight clustering in *post*-RoPE space would be ambiguous.** RoPE rotates vectors by position, so
tokens at similar positions are rotated to similar orientations. Clustering you saw there might just
reflect having sampled tokens from similar positions — an artifact that could evaporate under a
different prompt or position mix. Position is a live alternative explanation you cannot rule out.
- **Observing tight clustering in *pre*-RoPE space removes that alternative.** No positional rotation
has been applied, so position cannot be what is aligning the vectors. And because the clustering
also persists *across many prompts and positions*, the input content can't be the explanation
either.
- **What remains is the only thing fixed and prompt-independent: the learned weight matrices** that map
token embeddings into Q and K. That transformation doesn't change from prompt to prompt, which is
why the same center reappears every time. *That* is what "model-intrinsic" means.
So the accurate phrasing is closer to: *"we can conclude the stability is intrinsic because we observe
it in pre-RoPE space — where position cannot be the explanation — and yet the clustering still holds
across prompts."* The paper's "because" should be read as **"and therefore we can conclude"**: an
epistemological claim about what the evidence *rules out*, not a mechanistic claim about what *produces*
the clustering. (This is also why r̄ matters in §6: ruling out position and prompt as explanations for
the *clustering* is not the same as proving the clustering *causes* the distance preference — that
second causal step is what the moderate r̄ ≈ 0.5 is asked to support.)
### Q/K concentration is the *cause*; attention sinks are one *consequence* — don't conflate them
A natural misreading is "this is just the attention-sink phenomenon restated." It isn't. (An **attention
sink** is a token — usually among the first in the sequence — that soaks up a large share of attention
while contributing almost nothing to the output; see §8.3 for the full treatment.) Concentration
is a statement about pre-RoPE geometry (the centers exist and are stable). Whether a head ends up
being a *sink* head, a *local* head, or something else is determined by **where its
attention-vs-distance curve peaks**, which the centers set. Sinks are simply the special case where
the centers produce a curve that peaks at very large Δ (the earliest tokens). So sinks fall out of the
framework as one point on a spectrum — they aren't the premise.
### The "trigonometric series" is a Fourier *analogy*, not a Fourier series
RoPE's frequencies follow a **geometric** progression ($\omega_f=\theta^{-2f/d}$), whereas a classical Fourier
series uses a **harmonic** one (ω_n = n·ω_0). So calling the attention-vs-distance curve a "Fourier
reconstruction" is loose. The right reading: the centers act like fixed Fourier-style *coefficients*
that the model learned, and different centers "synthesize" different distance-preference curves — but
over RoPE's particular (non-harmonic) frequency grid. Worth flagging so no one over-claims a clean
spectral decomposition.
### What S_norm actually measures (the (1 − R_f) algebra is the point)
S_norm looks like a plain "norm of the key" term, but the weighting is the substance. Because
$R_f=\lVert\mathbb{E}[q_f]\rVert/\mathbb{E}[\lVert q_f\rVert]$, the factor (1 − R_f)·E[‖q_f‖] collapses to **$\mathbb{E}[\lVert q_f\rVert]-\lVert\mathbb{E}[q_f]\rVert$** — the
gap between the *average length* of the query in that band and the *length of the average* query.
That gap is exactly the part of the query that **doesn't** sit at the center, i.e. the spread the trig
series throws away. So S_norm isn't a separate heuristic bolted on; it's the principled complement:
"score by the center (S_trig), then add back what the center couldn't represent (S_norm), in
proportion to how much got thrown away."
### Calibration is offline, one-time, and input-independent
The Q centers and expected norms (E[q_f], E[‖q_f‖]) are **statistics computed once from a calibration
corpus**, not quantities recomputed from the live prompt. This is why the method can score a key
*before* the relevant query exists — it never needs the actual future query, only the head's typical
query. The ablations argue these statistics are **model-intrinsic**: accuracy is stable across
calibration sizes (50k–960k tokens) and even across wildly different calibration data (Google-homepage
HTML scores about the same as curated chat data), and calibrating on *coding* data transfers to *math*
benchmarks. Treat calibration as characterizing the model, not the task.
### A key is scored at *many* distances on purpose
Don't read S̃(k) as "score this key at its current distance." A cached key will be looked at from every
future position, so its importance is the *average* of S over a fan of future offsets D = {1, 2, 4, …,
2¹⁶}. The **geometric spacing** matters (see the ablation: geometric beats linear by ~17 points),
because near-distance behavior is where the curve is busiest and needs denser sampling.
### "Matches — and sometimes exceeds — Full Attention" needs a careful reading
On a few settings TriAttention slightly *beats* the uncompressed model (e.g. AIME25 at budget 4096;
the DFS benchmark at depths 8 and 12). The paper's interpretation is that **Full Attention's cache
carries redundancy** under these workloads, so well-chosen pruning acts a little like denoising. Read
this as "pruning isn't always pure loss," not as "compression creates information." It's a modest,
workload-specific effect, and worth probing rather than taking at face value (see §6).
---
## 4. Help with diagram interpretation
Reading aids for the paper's denser figures. Each note says what to look *for*, not just what's
plotted.
### Figure 2 — Q/K concentration and its consequences (the conceptual keystone)
This four-panel figure carries the paper's whole premise; read the panels as a chain of reasoning, not
four separate plots.
- **(A) Pre-RoPE Q/K.** Q (and K) points form a *tight blob* away from the origin — high MRL
(R_Q = R_K = 1.00 in the example). The tell: **three different input sequences are overlaid and you
can't tell them apart** — that's the "stable across content" claim made visual. Also notice the **Q
and K blobs point in different directions** (here one vertical, one horizontal): that gap is the Q–K
relative phase φ_f, which sets where the band's distance curve sits — documented structure, not noise
(details and citations in §8.9).
- **(B) Post-RoPE Q/K.** The *same* vectors, now rotated by position, smear into **arcs**. This is
the visual argument for *why* post-RoPE observation is unstable: position has dispersed a clean
cluster into a spread-out arc, so a "recent query's orientation" is a moving target.
- **(C) Concentration distribution.** A histogram of R across *all* heads, piled up near 1.0. The
point isn't any single head — it's that concentration is **prevalent**, not cherry-picked.
- **(D) Attention reconstruction.** The predicted attention-vs-distance curve (from centers) laid over
the actual logits, with Pearson r = 0.72. **Read the agreement, but also the scatter** — this is the
evidence that centers *predict* attention, and the moderate-not-perfect fit is honest about how far
the approximation goes.
The mental throughline: A says the centers exist and are stable → B says that's *only* true pre-RoPE →
C says it's everywhere → D says the centers actually predict attention. If A or D doesn't convince you,
the method doesn't stand.
### Figure 1 — the two trade-off curves (what "2.5× / 10.7×" means)
Two panels, both with **accuracy on the y-axis** and an efficiency quantity on the x-axis. Don't read
them as "higher line wins" — read them at a *fixed accuracy*:
- **(A) Throughput vs. accuracy.** Find the horizontal line at Full Attention's accuracy (40.8%), then
compare *how far right* each method reaches at that height. TriAttention sits ~2.5× further along the
throughput axis than Full Attention — same accuracy, more tokens/second.
- **(B) KV memory vs. accuracy.** Same trick on the memory axis: at matched accuracy, TriAttention
needs ~1/10.7 of the cache. The star markers are the Full-Attention reference points.
The thing to internalize: the claim is always *"at equal accuracy, cheaper,"* not *"more accurate."*
### Figure 3 — reconstruction-correlation distributions (the honesty check)
Three histograms (DS-Qwen3-8B, DS-Qwen-7B, DS-Llama-8B) of **per-head reconstruction correlation r̄**,
with the mean as a red dashed line. What to look for: all three are **right-skewed with means above
0.5** (0.51–0.56), and a healthy mass sits in the 0.6–0.9 range. But also notice the **left tail** —
plenty of heads fall below 0.5. The fair takeaway is "the centers predict attention well for *most*
heads across *several* architectures," which is exactly why the method *also* carries the norm-based
fallback for the heads where the trig series is weak.
### Figure 5 — results vs. budget (A–C) and the memory-retention benchmark (D)
Panels A–C (MATH500, AIME24, AIME25) plot **accuracy vs. KV budget** for Full Attention, R-KV, and
TriAttention. Read three things: (1) the **gap to the Full-Attention dashed line** shrinks as budget
grows; (2) TriAttention's lead over R-KV is **widest at low-to-mid budgets** (the regime that
actually matters for compression); (3) on MATH500 TriAttention **meets the Full-Attention line around
budget 1024** and can edge above it.
**Panel D is the subtle one — read it carefully.** The x-axis is **DFS recursion depth** (memory
pressure), the y-axis accuracy. The story isn't "TriAttention is highest everywhere"; it's the
*shape*: TriAttention hugs Full Attention up to about depth 16–18 (even nosing ahead at 8 and 12),
while **R-KV falls off a cliff at depth 16** (≈61% → ≈31%). That cliff is the visual of catastrophic
intermediate-state loss — exactly the retrieval-head failure the method was designed to avoid. The
honest note: beyond depth ~18 TriAttention also begins to lag Full Attention, so it preserves *most*,
not *all*, of the essential state.
### Figure 4 / Figure B — the method schematic (and its real-attention twin)
Figure 4 is the pipeline cartoon; Figure B is the same pipeline drawn over **real attention maps**.
Read left to right: **offline calibration** → **S_trig** (the diagonal-ish distance-preference
structure) → **+ S_norm** (which flags the far-but-low-norm earliest token the trig term would
wrongly keep) → **prune to top-B**. The single most instructive detail is the worked example: S_trig
correctly demotes nearby keys for a distance-preferring head, while S_norm catches a *far* key that
S_trig would have over-credited because its norm is tiny. That's the "two complementary signals"
argument in one picture.
---
## 5. Honest caveats
A few limitations worth keeping in view (the paper is candid about several in its own
Limitations appendix).
- **The evidence base skews to math reasoning.** The headline numbers are AIME24/25 and MATH500 on a
handful of reasoning-tuned models. General-task evidence (LongBench, RULER) and the cross-architecture
MLA check live in the appendices, on fewer models. The claim "more reliable signal than
observation-based methods" is best supported *in the reasoning regime the paper targets*; treat
the breadth as appendix-grade for now.
- **Reconstruction is moderate, and the causal claim rides on a correlation.** The whole argument that
"concentration *causes* distance preference" is validated by a reconstruction *correlation* whose
means are ~0.5 with long left tails (Figure 3). That's good evidence the centers are *predictive*,
but it's weaker than a mechanistic/causal demonstration, and a non-trivial fraction of heads
reconstruct poorly. The method's norm-fallback is essentially an admission of this.
- **An offline calibration step is required.** Unlike purely online methods, TriAttention needs a
one-time calibration pass to estimate centers and norms. The ablations argue it's cheap and robust
(model-intrinsic, data-insensitive), but it's still an extra dependency and an assumption that the
deployment distribution resembles calibration closely enough.
- **Throughput/memory numbers are setup-specific.** "2.5× / 10.7× / 6.3×" are measured at maximum
batch size on A100-80GB (H100 for GPT-OSS), at particular budgets chosen to *match* Full-Attention
accuracy. They're legitimate but configuration-dependent; the OpenClaw single-RTX-4090 deployment
(Appendix J) is a compelling demo but a single anecdote.
- **The memory-retention benchmark is a proxy.** The DFS Recursive State Query task is a clever,
controllable stressor for intermediate-state retention, but it's synthetic and uniform by design.
How well "survives DFS depth 16" maps onto real long-reasoning robustness is an open question.
**"We turn to the pre-RoPE space" is a genuine reframing — but it stands on a recent lineage, not
empty ground.** It's worth getting the provenance straight to separate what's borrowed
from what's contributed:
- **RoPE-as-relative and its frequency structure** are foundational (RoFormer, and the analysis line
on *what makes RoPE useful*). The decomposition of the logit into per-band sinusoids is standard
RoPE algebra, not new here.
- **Dominant-frequency-band selection** (ranking bands by their contribution to the logit) is taken
from the EliteKV-style frequency-analysis work the paper cites.
- **Norms as an importance signal** is the VATP / "attention is not only a weight" thread; S_norm is a
refinement of that idea, not its origin.
- **Window-based pruning and the reasoning-model evaluation protocol** follow R-KV directly.
**So what is plausibly novel?** Not RoPE algebra or norm-based scoring in isolation — the
*combination and the observation*: (1) the empirical **Q/K concentration** phenomenon and its
stability across positions/contents/domains/architectures; (2) using the **Q center as a proxy for
future queries** so importance can be scored *before* the query arrives; (3) the
**concentration-weighted blend** of a center-derived trig score with a spread-derived norm score; and
(4) the **memory-retention (DFS) benchmark** as a retrieval-stress test. A fair one-line framing: *the
RoPE math and the norm idea are inherited; the concentration observation and the
score-before-the-query mechanism are the contribution.* As a reviewer you'd reasonably ask the authors
to soften any language implying they discovered pre-RoPE frequency analysis.
---
## 6. Discussion prompts
*(More prompts to be added.)*
- **Is "exceeds Full Attention" real, and if so, what is it?** On a few settings, pruning *beats* the
uncompressed model. The paper waves at "Full Attention has redundancy." Is that a genuine
denoising/regularization effect, run-to-run noise on 30-problem benchmarks (AIME has only 30
problems, sampled 8×), or an artifact of the budget chosen? What experiment would settle it — and
does the effect survive on MATH500's 500 problems?
- **Does center-based scoring actually protect retrieval heads, or just survive a proxy?** The pitch is
that stable centers avoid the short-observation-window failure that evicts dormant retrieval targets. The
evidence is the DFS benchmark. Is DFS a fair stand-in for real retrieval-head behavior? Would a
needle-in-a-haystack-style retrieval test, or a direct measurement on identified retrieval heads (à
la Wu et al.), strengthen or complicate the claim?
- **How robust is the offline-calibration assumption at the edges?** The ablations show calibration is
data-insensitive on *English text / code*. But what about a serving distribution far from
calibration — a different language, a heavily structured/agentic format, or a domain with unusual
positional statistics? Since the centers are fixed at calibration time, is there a failure mode where
a head's *deployment* center drifts from its *calibration* center, and how would you detect it
online?
- **Where does this sit relative to quantization and low-rank methods?** TriAttention is an *eviction*
method (keep top-B tokens). A lot of recent KV work instead *quantizes* or *low-rank-compresses* the
cache without dropping tokens. Are these complementary (evict-then-quantize) or competing? Could the
concentration centers also inform a *non-eviction* compression?
- **Is r̄ ≈ 0.5 enough to carry a causal story?** The paper says concentration *causes* distance
preference and validates with reconstruction correlation. Given the moderate means and long left
tails, how much of the method's success is the trig term vs. the norm fallback doing quiet work on
the low-r heads? The ablation (removing S_trig is catastrophic) argues the trig term matters — but
does that pin down *causation*, or just that the term is *useful*?
- **What are positional embeddings actually encoding — a specific *other position*, or a *fuzzy*
relationship?** RoPE makes a head’s logit a smooth function of the distance Δ (the trig series of
§8.1/§8.10), so “position” enters as a *graded preference over offsets* — a soft “I tend to look
about this far back” — not a hard pointer to “the token at position *k*.” Is that the right primitive
for language? One view: precise positional addressing is what **retrieval heads** appear to do (fetch
*that* specific earlier token), and a fuzzy distance-preference is a poor stand-in — exactly the
failure mode TriAttention worries about. The opposing view: most of what attention does is *semantic*
(match by content), and position is only a soft prior layered on top, so a fuzzy relationship is not
just adequate but desirable for generalizing across lengths. Where on that spectrum does a given head
sit — and does TriAttention’s center-based, distance-only score quietly assume the *fuzzy* end (safe
for distance-preferring heads, dangerous for sharp retrieval heads)? (Ties to §8.6 on head types and
the retrieval-head reading in §8.3.)
- **Is the reasoning trace even worth caching faithfully?** Recent "reasoning theater" work suggests much
of a chain-of-thought is performative — the model is often committed to its answer early and coasts
through the rest. If so, is a distance/attention scorer keeping the *load-bearing* tokens or just the
ones that produced *this* answer, and what should persist into the next turn — the verbatim trace, a
summary, or nothing? (Full treatment in §8.11.)
---
## 7. Key papers referenced
A short, opinionated map of the work this paper builds on. Each entry links to the live arXiv page
where the ID is verified, summarizes the key finding, and notes the connection to TriAttention.
> *Sourcing note:* every arXiv ID below has been verified against arXiv (title, authors, and venue
> checked). Where a work appeared at a conference, the venue is given alongside the arXiv link.
### The positional-encoding foundation
**RoFormer: Enhanced Transformer with Rotary Position Embedding** — Su et al. (2021). ✓
[arxiv.org/abs/2104.09864](https://arxiv.org/abs/2104.09864)
> Introduces RoPE: encode absolute position by rotating Q/K, so the attention dot product depends only
> on relative offset. Every equation in TriAttention's §2–3 is RoPE algebra; the "pre-RoPE vs.
> post-RoPE" distinction the whole method rests on is defined relative to this rotation.
### What it improves on — post-RoPE / attention-score compression
**H₂O: Heavy-Hitter Oracle for Efficient Generative Inference** — Zhang et al. (2023). ✓
[arxiv.org/abs/2306.14048](https://arxiv.org/abs/2306.14048)
> Accumulates attention scores across decoding to keep "heavy-hitter" tokens. The canonical
> attention-score-based eviction method, and the archetype of the post-RoPE, observation-window
> approach TriAttention argues is unstable.
**SnapKV: LLM Knows What You Are Looking for Before Generation** — Li et al. (2024). ✓
[arxiv.org/abs/2404.14469](https://arxiv.org/abs/2404.14469)
> Uses attention in a local "observation window" at the end of the prompt to predict which tokens
> matter. A main baseline; its observation-window premise is exactly the "tiny, position-dependent
> window" TriAttention sets out to replace.
**Efficient Streaming Language Models with Attention Sinks (StreamingLLM)** — Xiao et al. (2024). ✓
[arxiv.org/abs/2309.17453](https://arxiv.org/abs/2309.17453)
> The heuristic anchor: keep a few initial "sink" tokens plus a sliding recent window. TriAttention
> reframes sinks as just one point on the distance-preference spectrum its centers can predict (see
> §3) rather than a special-cased rule.
**R-KV: Redundancy-aware KV Cache Compression for Reasoning Models** — Cai et al., NeurIPS 2025,
[arxiv.org/abs/2505.24133](https://arxiv.org/abs/2505.24133).
> The primary head-to-head baseline. R-KV scores cached tokens *on the fly during decoding* on two
> axes — **importance** (attention from recent queries) and **non-redundancy** (it explicitly drops the
> repetitive self-checking and re-derivation that bloats chain-of-thought), keeping only the
> informative, diverse tokens; it reports near-full accuracy at ~10% cache. It is **training-free and
> plug-and-play** for any autoregressive LLM. TriAttention borrows its 128-token windowed pruning
> cadence but replaces the recent-query importance score with the center-based trigonometric score.
**Scissorhands** — Liu et al., NeurIPS 2023, [arxiv.org/abs/2305.17118](https://arxiv.org/abs/2305.17118).
> Builds on a "persistence of importance" hypothesis, using historical attention to guide eviction.
> Another instance of the historical-attention family TriAttention contrasts itself with.
**Quest** — Tang et al., ICML 2024, [arxiv.org/abs/2406.10774](https://arxiv.org/abs/2406.10774).
> Query-aware sparsity for long-context inference. Part of the broader attention-based selection
> landscape surveyed in §2.
**PyramidKV** — Cai et al., 2024, [arxiv.org/abs/2406.02069](https://arxiv.org/abs/2406.02069).
> Allocates KV budget non-uniformly across layers (pyramidal funneling). A LongBench baseline in the
> appendix.
### The norm-based thread
**Attention Score is not All You Need… Value Also Matters (VATP)** — Guo et al. (EMNLP 2024). ✓
[arxiv.org/abs/2406.12335](https://arxiv.org/abs/2406.12335)
> Observes that attention sinks get huge attention but near-zero value norms, so attention-only
> importance is misleading; adds value-norm information. The intellectual ancestor of TriAttention's
> **S_norm**, generalized here to per-band, concentration-weighted norms.
### The pre-RoPE / frequency-analysis lineage
**EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection…** — Zhou et al., 2025,
[arxiv.org/abs/2503.01586](https://arxiv.org/abs/2503.01586).
> Selects RoPE frequency bands by their contribution to attention. TriAttention borrows this
> **dominant-frequency-band** machinery (the C_f ranking in Appendix B.7) to decide which 2-D planes
> to analyze. (EliteKV also notes that prior methods store keys *before* rotation and re-apply RoPE at
> decode — the same un-rotation trick §8.1 describes.)
**Round and Round We Go! What Makes Rotary Positional Encodings Useful?** — Barbero et al., ICLR 2025,
[arxiv.org/abs/2410.06205](https://arxiv.org/abs/2410.06205).
> A mechanistic look at *why* RoPE works: studying Gemma, they find high RoPE frequencies build robust
> positional attention patterns while the lowest frequencies carry semantic content — the empirical
> backing for §8.1's distance-scale stratification and for treating per-band centers as the carriers of
> distance preference.
**Expected Attention: KV Cache Compression by Estimating Attention from Future Queries** — Devoto et
al., 2025, [arxiv.org/abs/2510.00636](https://arxiv.org/abs/2510.00636).
> The closest cousin in spirit: a training-free method that *estimates* (rather than observes) how
> future queries will attend, computing expected attention in closed form from activation statistics.
> A natural point of comparison for "score before the query arrives."
### Retrieval heads — the failure mode being protected
**Retrieval Head Mechanistically Explains Long-Context Factuality** — Wu et al., ICLR 2025,
[arxiv.org/abs/2404.15574](https://arxiv.org/abs/2404.15574).
> Identifies specific heads responsible for fetching distant tokens. Motivates *why* eviction driven by a short observation window is
> dangerous: a retrieval head's target can stay dormant and get evicted before it's needed.
### Infrastructure the method plugs into
**GQA: Training Generalized Multi-Query Transformer Models** — Ainslie et al. (EMNLP 2023). ✓
[arxiv.org/abs/2305.13245](https://arxiv.org/abs/2305.13245)
> Shared KV heads across query heads — the architecture that forces TriAttention's "normalize-then-
> aggregate" scoring fix (§4.3).
**Directional Statistics** — Mardia & Jupp (Wiley, 1999).
> The source of the Mean Resultant Length used to quantify concentration. Worth a look for the
> §8.5 deep dive on the math.
---
## 8. Explore in more depth
A running collection of topics that go beyond this single paper, toward a real (non-anthropomorphic)
understanding of how attention and KV memory actually work. Each sub-topic is self-contained — a short
framing followed by a curated reading list — so they can be read in any order or pulled out for a
future session.
*Topics:*
- **8.1 — RoPE in depth: geometry, integration, and what this paper changes** *(below)*
- **8.2 — The KV-cache compression landscape** *(below)*
- **8.3 — Attention sinks and the structure of attention** *(below)*
- **8.4 — Long reasoning and test-time compute** *(below)*
- **8.5 — Directional statistics and Fourier views of attention** *(below)*
- **8.6 — Attention heads and head-sharing architectures (MHA/MQA/GQA/MLA)** *(below)*
- **8.7 — The argument, geometrically: one step at a time** *(below)*
- **8.8 — Offline vs. online: the two-phase pipeline** *(below)*
- **8.9 — The complex-number representation: from dot product to Equations 15 and 16** *(below)*
- **8.10 — The constant-Q/K approximation: deriving the trigonometric series (Equation 17)** *(below)*
- **8.11 — Is the reasoning trace worth caching? Faithfulness, "reasoning theater," and cross-turn memory** *(below)*
- *More to come (placeholder for topics you add next).*
---
### 8.1 RoPE in depth: geometry, integration, and what this paper changes
> This paper *is* a RoPE-geometry paper — every equation in §2–3 is RoPE algebra, and the central
> "pre-RoPE vs. post-RoPE" distinction is defined relative to RoPE's rotation. So this thread goes
> deeper than the others: first a quick map of the positional-encoding menu (so RoPE has context),
> then RoPE's mechanics with pictures, then **how RoPE is actually wired into a transformer runner**,
> and finally **what TriAttention changes about that wiring** (the short answer: surprisingly little,
> and that's a feature).
>
> *Note on rendering:* the diagrams below are inline **SVG** (geometry) and **Mermaid** (flow charts).
> They render in markdown viewers that support those (GitHub renders Mermaid; the Claude artifact
> viewer renders both). If a viewer shows raw code instead, the captions and prose carry the same
> content — ask and I can export any of them as standalone images.
#### The positional-encoding menu
The core problem every scheme here solves: self-attention is **order-invariant** (it's a set
operation), so without some positional signal a transformer literally cannot tell "A then B" from
"B then A." How you inject that signal strongly affects length/depth generalization. The menu, most
classical to most recent:
- **Absolute position embeddings (APE)** — add a position-indexed vector to each token embedding,
either *sinusoidal* (fixed; Vaswani et al. 2017, [arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762))
or *learned* (a trainable vector per slot; GPT/BERT). Simple and effective in-distribution, but
generalizes poorly past the trained length.
- **Relative position representations** — encode the *distance* between query and key rather than
absolute slots, so "two apart" is reused everywhere. Shaw et al. 2018
([arxiv.org/abs/1803.02155](https://arxiv.org/abs/1803.02155)); the **T5** bias is a popular
bucketed variant. Better length behavior than APE, at some attention-time cost.
- **RoPE — Rotary Position Embedding** — Su et al. 2021 ([arxiv.org/abs/2104.09864](https://arxiv.org/abs/2104.09864)).
Rotates each query/key vector by an angle proportional to its absolute position; because the dot
product then depends only on the *relative* offset, you get relative-position behavior with no extra
parameters and a drop-in fit to standard attention. **Now the de-facto standard in open LLMs**
(LLaMA, Mistral, Qwen, Gemma). Extending it past the trained context needs tricks (position
interpolation, NTK/YaRN scaling). *This is the scheme TriAttention is built on, and it's why the
paper exists.*
- **ALiBi — Attention with Linear Biases** — Press et al. 2021 ([arxiv.org/abs/2108.12409](https://arxiv.org/abs/2108.12409)).
Adds *no* embedding; instead biases each attention score by a penalty linear in query–key distance
("train short, test long"). Cheap and a strong length-extrapolator.
- **NoPE — No Positional Encoding** — Kazemnejad et al. 2023 ([arxiv.org/abs/2305.19466](https://arxiv.org/abs/2305.19466)).
Literally no positional signal at all. The counterintuitive part: a **causal** decoder can still
infer order, because the causal mask itself breaks the symmetry. Because there's no length-specific
machinery to overfit, it can generalize to unseen lengths better than APE.
- **PaTH — Position encoding via Accumulating Householder Transformations** — Yang et al. 2025
([arxiv.org/abs/2505.16381](https://arxiv.org/abs/2505.16381)). Makes the transformation between two
positions **data-dependent** (built from a product of content-dependent reflections), so the
encoding depends on the actual *path* of tokens between positions, not just the distance. Extra
expressivity for state-tracking; strong length extrapolation.
*(The menu is adapted from the recurrent-depth companion's positional-embeddings section, trimmed to
general background. All arXiv IDs above have been verified — sinusoidal/APE 1706.03762, Shaw relative
1803.02155, RoPE 2104.09864, ALiBi 2108.12409, NoPE/Kazemnejad 2305.19466, PaTH/Yang 2505.16381.)*
#### RoPE mechanics
RoPE splits each d-dimensional Q/K vector into **d/2 two-dimensional subspaces** ("frequency bands"),
indexed by f. Band f is rotated by an angle **ω_f · p**, where p is the token's position and
$\omega_f=\theta^{-2f/d}$ (θ = 10000 typically). So the *only* thing position does is **spin each 2-D slice of
the vector around the origin** — by a band-specific rate, the same way for Q and K.
Two consequences matter for this paper:
**1. Different bands spin at very different rates (a geometric progression).** Low-f bands rotate fast
(short wavelength → resolves nearby positions); high-f bands barely move (long wavelength → carries
long-range structure). At a *single* position, the bands sit at wildly different angles.
This is why TriAttention talks about **dominant frequency bands**: a head's attention logit is a sum
over bands, and only a few bands carry most of the weight, so those are the ones worth analyzing.
**2. The dot product depends only on the *distance* $\Delta=p_q-p_k$.** Rotating q by ω_f·p_q and k by
ω_f·p_k and taking the dot product leaves an angle of ω_f·(p_q − p_k) — the absolute positions cancel,
the relative one survives. That single fact is TriAttention's Equation 2: the logit is
$\sum_f \lVert q_f\rVert\lVert k_f\rVert\cos(\omega_f\Delta+\varphi_f)$.
The three equations behind this — the ones the paper proves in its appendix and that §8.10 derives
step by step — are worth having in one place, because they are what makes the “centers ⇒ a distance
curve” claim concrete:
$$
\begin{aligned}
\operatorname{logit}(\Delta)
&= \sum_f \lVert q_f\rVert\,\lVert k_f\rVert\,\cos(\omega_f\Delta+\varphi_f)
&& \text{(Eq. 2: per-band cosines)}\\
&= \sum_f |q_f|\,|k_f|\,\cos(\omega_f\Delta+\varphi_f)
&& \text{(Eq. 16: magnitude--phase form)}\\
&= \sum_f \big[\,a_f\cos(\omega_f\Delta)+b_f\sin(\omega_f\Delta)\,\big]
&& \text{(Eq. 17: linear in cos/sin)}
\end{aligned}
$$
$$
\varphi_f=\arg(q_f)-\arg(k_f),\qquad
a_f=|q_f|\,|k_f|\cos\varphi_f,\qquad
b_f=-|q_f|\,|k_f|\sin\varphi_f
$$
Read top to bottom: the logit is a **sum of cosines, one per frequency band**, each with its own rate
ω_f, amplitude |q_f||k_f|, and phase φ_f (Eq. 2/16); expanding cos(ω_fΔ + φ_f) turns every term into a
fixed `a_f·cos + b_f·sin` (Eq. 17), so once the centers fix the amplitudes and phases the entire
attention-vs-distance curve is determined. That linear-in-(cos, sin) form is exactly what licenses
substituting a *center* for the live query/key, and it is the algebra the method’s appendix leans on.
(Demo 6 in the interactive companion lets you turn φ_f and watch the curve move; §8.9–§8.10 derive
these three lines from the complex-number form.)
Here's the crux the paper builds on. The vectors being rotated — **pre-RoPE** q_f, k_f — turn out to
be tightly clustered around a fixed center for most heads. Apply the position-dependent rotation and
that clean cluster smears into a position-dependent **arc** (this is exactly Figure 2 A→B):
The takeaway: **all the instability post-RoPE methods suffer from is just from RoPE's rotation.** Strip the
rotation off and the geometry is stable — which is why the centers are a reliable signal.
#### Periodicity and the distance-scale spectrum
A natural question: since each band is a rotation, isn't RoPE *periodic*? Yes — within a single band.
Band *f* advances its angle by ω_f·Δ, so it returns to where it started every **period of 2π/ω_f
tokens**. With $\omega_f=\theta^{-2f/d}$ and θ = 10000, that period runs from about **2π ≈ 6 tokens** in the
fastest band (index f = 0, ω = 1) up to about **2π·θ ≈ 63,000 tokens** in the slowest, highest-index
band. So RoPE lays out a whole **spectrum of wavelengths**, geometrically spaced — and that spectrum is
exactly the **stratification** intuition: RoPE implicitly maps *different embedding dimensions to
different distance scales*. *(Demo 2 in the interactive companion shows this spectrum as six bands spinning at their own rates; Demo 3 sums them into the attention-vs-distance curve.)*
The one thing to pin down is the direction of the indexing, which is easy to flip. With
$\omega_f=\theta^{-2f/d}$:
- **Low-index bands (small f) → high frequency → short period (~6 tokens).** They wrap many times
across a long context, so they resolve *fine, local* offsets but can't tell "100 back" from "106
back" (same phase). These behave *locally/relatively*.
- **High-index bands (large f) → low frequency → long period (tens of thousands of tokens).** Within a
normal context they don't complete even one cycle, so their angle is nearly monotone in position —
they carry *coarse, long-range* distance, almost like an absolute-position signal.
So the period is **shortest for small f and longest for large f** — periodicity grows *with* the band
index (equivalently, with *smaller* frequency). If you were picturing *f as the frequency itself*, then
yes: smaller frequency ⇒ longer period, same conclusion. Either way the substance is identical: a clean
low-to-high mapping from embedding dimension to the distance scale it encodes.
Two consequences worth carrying:
- **Why it isn't globally periodic.** The per-band periods are geometrically spaced and mutually
incommensurate, so the *joint* pattern across all bands effectively never repeats within any
practical context — which is what lets RoPE keep distinguishing far-apart positions even though every
individual band wraps.
- **Why context-extension tricks target specific bands.** Position-interpolation / NTK / YaRN rescaling
focuses on the long-wavelength (high-index) bands, because those are the ones whose single period
approaches or exceeds the trained context — the dimensions that "run out of room" first.
This stratification is also why TriAttention's **dominant frequency bands** matter (different heads lean
on different distance scales), and it's the structure being summed in §8.7's trigonometric series: the
geometrically-spaced frequencies *are* this spectrum, and a head's centers decide which scales reinforce
into a peak. For the explicit treatment, see Hong et al. (2024), *On the token distance modeling ability
of higher RoPE attention dimension* (cited in the paper), and Barbero et al. (2025) for the mechanistic
version.
#### Where positions enter: original Transformer vs. modern models
Worth stating outright, because it trips people up: **where position is injected moved.** In the
original Transformer (Vaswani et al. 2017), a *positional encoding is added to the token embeddings at
the input* — once, before layer 1, on both the encoder and decoder input stacks — and the attention
blocks themselves are position-agnostic. Modern decoder-only LLMs do the opposite: **nothing is added
at the input**; instead **RoPE is applied to Q and K inside every attention layer**, so position is
re-injected at each layer and enters *only* through the attention dot product. There is no separate
"positional decoding" stage — in a decoder-only model position enters once per layer, via RoPE, and
nowhere else. And the same code path that rotates Q and K then writes the **rotated** key into the KV
cache — which is exactly the post-RoPE-key fact TriAttention has to work around (next subsection).
A third variant matters here because TriAttention validates on it (Appendix I): **DeepSeek-V3-style
MLA** uses a *decoupled* RoPE. Most of the key/value is compressed into a **position-free latent**, and
position is carried by a **separate, small RoPE branch** — so in MLA part of the key never passes
through RoPE at all. (This is also why §8.6's "head-sharing" axis and this one are linked: MLA is both
a head-sharing scheme and a positional-encoding scheme.)
*Is “position enters only through attention” special to decoder-only models?* No — it is a property
of the **scheme**, not of being a decoder. Absolute schemes (sinusoidal or learned APE) add a
position vector at the **input embedding**, and they do so identically for encoder-only models
(BERT), for decoders, and for the encoder *and* decoder stacks of the original encoder–decoder
Transformer; in all of those the self-attention is itself position-agnostic. Relative/rotary schemes
(RoPE, ALiBi, T5’s relative bias) instead act **inside the attention scores**, again regardless of
whether the block is an encoder or a decoder — T5, an encoder–decoder, injects position purely as a
relative bias inside each stack’s self-attention. The one place position is usually *absent* is
**cross-attention** (the decoder attending to encoder outputs): there the decoder’s queries match
encoder keys by content, and most architectures add no positional term to that sub-layer. So the
honest statement is that modern models inject position only through attention *because they use
RoPE/relative schemes everywhere — encoder, decoder, or both* — not because the model is decoder-only.
```mermaid
flowchart TB
subgraph OT["Original Transformer, 2017"]
direction TB
ot1["token embeddings"] --> ot2["ADD positional encoding, once, at the input"]
ot2 --> ot3["attention layers, no position inside"]
ot3 --> ot4["output"]
end
subgraph MD["Modern decoder-only with RoPE: Llama, Qwen, Mistral, Gemma"]
direction TB
md1["token embeddings, no positional add"] --> md2["each layer: project to q, k, v"]
md2 --> md3["apply RoPE to q and k; position enters HERE, every layer"]
md3 --> md4["attention; the cache stores the post-RoPE key"]
md4 --> md5["output"]
end
subgraph MLA["DeepSeek-V3 MLA: decoupled RoPE"]
direction TB
mla1["token embeddings"] --> mla2["compress k, v into a position-free latent"]
mla1 --> mla3["small separate RoPE branch carries position"]
mla2 --> mla4["attention = content path with no RoPE, plus the RoPE branch"]
mla3 --> mla4
mla4 --> mla5["output"]
end
```
**Real diagrams from actual models** (linked rather than reproduced — these are copyrighted paper/blog
figures):
- **Original Transformer** — Vaswani et al., [arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762).
Figure 1 shows the positional encoding added at the input on both stacks.
- **RoPE in attention** — RoFormer, Su et al., [arxiv.org/abs/2104.09864](https://arxiv.org/abs/2104.09864).
- **DeepSeek-V3 (MLA, decoupled RoPE)** — technical report,
[arxiv.org/html/2412.19437v1](https://arxiv.org/html/2412.19437v1). The attention figure with the
separate RoPE branch — *the most directly relevant real diagram for this paper*, since it's the
architecture class tested in Appendix I.
- **Cross-model comparison** — Sebastian Raschka, "The Big LLM Architecture Comparison" (2025),
[sebastianraschka.com/blog/2025/the-big-llm-architecture-comparison.html](https://sebastianraschka.com/blog/2025/the-big-llm-architecture-comparison.html).
Llama, Qwen, DeepSeek-V3, Gemma side by side, marking where attention/RoPE sit — the best single
resource for seeing "same skeleton, RoPE inside attention."
- **Code-level confirmation** — the HuggingFace
[`LlamaAttention` forward](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py)
applies RoPE to the query and key and *then* writes the rotated key to the cache; walkthroughs like
[LearnOpenCV's RoPE guide](https://learnopencv.com/rope-position-embeddings/) and
[`llama-nuts-and-bolts`](https://github.com/adalkiran/llama-nuts-and-bolts/blob/main/docs/10-ROPE-ROTARY-POSITIONAL-EMBEDDINGS.md)
trace the same path.
*A note on closed models:* frontier **closed** models (GPT-4/5, Claude, Gemini) don't publish
architecture diagrams at this level — the open frontier-class models above are the realistic source,
and they all share the RoPE-in-attention skeleton, so they're a faithful stand-in.
#### How RoPE is wired into a transformer runner
In a standard decoder layer, RoPE is applied **after** the Q/K projections and **before** attention.
Crucially for KV caching: the rotation is applied to keys *once, at the position they're produced*,
and the **post-RoPE keys are what get cached** — you don't want to re-rotate the whole cache every
step. (Values are not rotated; RoPE touches only Q and K.)
```mermaid
flowchart LR
X["token hidden state"] --> WQ["W_q"]
X --> WK["W_k"]
X --> WV["W_v"]
WQ --> QPRE["q pre-RoPE"] --> RQ["rotate band f by w_f times p"] --> QPOST["q post-RoPE"]
WK --> KPRE["k pre-RoPE"] --> RK["rotate band f by w_f times p"] --> KPOST["k post-RoPE"]
KPOST --> CACHE[("KV cache: post-RoPE k and v")]
WV --> CACHE
QPOST --> ATT["attention: softmax of q dot kT over sqrt d, times v"]
CACHE --> ATT --> OUT["layer output"]
```
A few integration facts worth having on hand, because they frame what TriAttention can
and can't touch:
- **In practice the rotation is the `rotate_half` trick.** The paper’s picture pairs *adjacent*
dimensions — (x₀,x₁), (x₂,x₃), … — and rotates each 2-D pair by its band angle. Real implementations
(HuggingFace, GPT-NeoX) use an algebraically identical but vectorized layout: they split the head
dimension into two contiguous **halves**, pair dimension *i* with dimension *i + d/2*, and compute
the whole rotation as two elementwise vector ops, `x·cos + rotate_half(x)·sin`, where `rotate_half(x)`
concatenates the negated second half with the first half, `[−x_{d/2:}, x_{:d/2}]`. That one line
applies all d/2 of the 2-D rotations at once (no Python loop over pairs). It is also why **converting a model into the HuggingFace
format permutes the columns of its Q and K projection weights** (W_q, W_k) into this half-split order:
the stored weights are reordered once, at conversion time, so the two-halves `rotate_half` lands the
exact rotation the original interleaved-pairs convention produced. ("Checkpoint" here just means the
saved weight file — e.g. a Llama model in HF format.) It is the *same* set of
per-band rotations — only the bookkeeping of which two coordinates form each pair is reindexed
(i ↔ i + d/2 instead of 2f ↔ 2f+1). So TriAttention’s per-band 2-D view is the conceptual form;
runners use this fused half-split form. (HF source: `apply_rotary_pos_emb` / `rotate_half` in
`transformers/models/llama/modeling_llama.py`; EleutherAI’s GPT-NeoX notes and RoFormer §3.4.2
derive the equivalence.)
- **The cache stores post-RoPE keys.** This is the single most relevant fact for TriAttention (next
section): the thing sitting in memory is *rotated*, but the method wants *pre-rotation* geometry.
- **FlashAttention sits downstream of RoPE.** *FlashAttention* (Dao et al. 2022) is the standard way
attention is actually evaluated on a GPU today: an **IO-aware, exact** kernel that tiles Q/K/V and
fuses the score → softmax → value-weighting steps so the full n×n score matrix is **never written to
GPU memory** — it is recomputed block-by-block in fast on-chip SRAM. It is not a different *kind* of
attention (same softmax(QKᵀ)·V, same numbers), just a far more memory-efficient way to compute it,
and it underlies essentially every modern LLM-serving stack (PyTorch SDPA, vLLM, TensorRT-LLM). Two
things follow for TriAttention. First, the kernel runs *after* RoPE — it consumes already-rotated
Q/K and never sees positions — so a method that only changes *which* tokens are cached is orthogonal
to it: a shorter cache just means fewer K/V blocks to stream, which any kernel handles. Second, and
the part the paper leans on: a method that needs the **full, materialized O(n²) attention matrix** —
e.g. H₂O, which reads each token’s accumulated attention mass to decide evictions — is asking for the
very thing FlashAttention refuses to produce, so it cannot use the fused kernel and pays an
efficiency penalty. Eviction by a *predicted* score (TriAttention) needs no such matrix, so it
composes with FlashAttention cleanly.
#### What TriAttention changes about the runner
Less than you'd expect — and the paper is clear it's an **eviction** method, so the attention
computation itself is untouched. What it adds is (a) a one-time **offline calibration** to get the
pre-RoPE Q centers and expected norms, and (b) a **periodic scoring-and-pruning pass** (every 128
tokens) that shrinks the cache to the top-B keys. Standard RoPE attention runs before and after,
unmodified.
```mermaid
flowchart TD
CAL["calibration corpus, offline and one-time"] --> CENT["pre-RoPE Q centers and expected query norms, per band"]
KPOST["cached post-RoPE key, at known position p_k"] --> NORM["norm term: key norm usable directly, since rotation preserves length"]
KPOST --> UNROT["un-rotate by w_f times p_k to recover the pre-RoPE angle for the phase term"]
CENT --> SCORE["score key: trig-series term plus norm term, blended per band by concentration 1 minus R_f"]
NORM --> SCORE
UNROT --> SCORE
SCORE --> AVG["average the score over future offsets 1, 2, 4, up to 2 to the 16"]
AVG --> PRUNE{"every 128 tokens: is the cache over budget B?"}
PRUNE -- yes --> TOPB["keep the top-B keys, evict the rest"]
PRUNE -- no --> KEEP["leave the cache unchanged"]
TOPB --> ATT2["standard RoPE attention on the smaller cache, kernel unchanged"]
KEEP --> ATT2
```
The one genuinely interesting wrinkle — and the paper does **not** spell out the kernel-level
integration, so this is the natural reading rather than a quoted recipe — is the **pre-RoPE/post-RoPE
mismatch**. The scoring function wants pre-RoPE keys (the phase term uses arg(k_f)), but the cache
holds post-RoPE keys. Two facts make this cheap to reconcile:
- **The query side needs nothing at inference.** The Q "center" E[q_f] is a *fixed offline statistic*,
so TriAttention never needs the live, rotated query at all — it uses the head's *typical* query as
the proxy. No per-step pre-RoPE query recovery is required.
- **On the key side, only the angle needs recovering, and the norm comes for free.** Because RoPE is
an orthogonal rotation, it **preserves length**, so ‖k_f‖ (used by S_norm) is identical pre- and
post-RoPE — you read it straight off the cached key. Only the *phase* needs the pre-rotation angle,
and since the key's position p_k is known, you recover it by rotating the cached key **back** by
−ω_f·p_k. So "go to pre-RoPE space" is, operationally, a known per-position un-rotation applied only
when scoring (every 128 tokens), not a second cache of pre-RoPE keys.
Net effect on the runner: the attention path, the KV-cache format, and the FlashAttention kernel are
all unchanged; TriAttention bolts on an offline calibration table and a lightweight periodic scorer
that reads the existing post-RoPE cache (un-rotating angles as needed) and trims it. That minimal
footprint is exactly why it can drop into an existing reasoning-model server — and why the OpenClaw /
single-RTX-4090 demo (Appendix J) is plausible.
#### Why this matters for TriAttention
Everything the paper claims is downstream of one geometric fact: **position enters attention purely as
a rotation, and the rotation is the only thing making post-RoPE observation unstable.** Read RoPE this
way and the method almost writes itself — un-rotate (or, better, work with the rotation-invariant
center), and you recover a stable signal that predicts attention before any query arrives. If a group
member is fuzzy on RoPE, the rotation pictures above are the thing to get solid; the rest of the paper
is bookkeeping on top of them.
#### Reading list
- **RoFormer / RoPE** — Su et al. (2021), [arxiv.org/abs/2104.09864](https://arxiv.org/abs/2104.09864).
The original; §3 has the rotation formula and the relative-position proof (TriAttention's Eq. 1–2).
- **What makes RoPE useful** — Barbero et al., ICLR 2025,
[arxiv.org/abs/2410.06205](https://arxiv.org/abs/2410.06205). How different frequency bands carry
different roles (high frequencies → positional patterns, low → semantics) — the backdrop for
"dominant bands."
- **Context-extension tricks** — Position Interpolation, Chen et al. 2023,
[arxiv.org/abs/2306.15595](https://arxiv.org/abs/2306.15595); YaRN, Peng et al. 2023,
[arxiv.org/abs/2309.00071](https://arxiv.org/abs/2309.00071). Both push RoPE past its trained context
by rescaling the long-wavelength frequency bands — concrete payoff of the §8.1 stratification view.
- **On the token distance modeling ability of higher RoPE attention dimension** — Hong et al., EMNLP
2024 Findings, [arxiv.org/abs/2410.08703](https://arxiv.org/abs/2410.08703). Dimension-level evidence
that higher (lower-frequency) RoPE dimensions carry long-range distance.
- **HuggingFace RoPE / `rotate_half` implementation** — worth a code read to see the
vectorized half-split form that real runners use versus the per-band 2-D form in the paper.
---
### 8.2 The KV-cache compression landscape
> TriAttention is one method in a crowded field. Mapping the field shows what's genuinely
> different here. The main axes: **what you drop** (whole tokens via eviction, vs. precision via
> quantization, vs. rank via low-rank projection) and **how you decide** (heuristic rules, observed
> attention scores, value/key norms, or — here — a *predicted* importance from stable statistics).
>
> *(Interactive: Demo 4 scores and prunes a 256-key cache; Demo 7 shows how eviction reshapes the matmul, including the accuracy-vs-budget cliff.)*
- **Heuristic / fixed-pattern:** StreamingLLM — Xiao et al. (2024),
[arxiv.org/abs/2309.17453](https://arxiv.org/abs/2309.17453). Sinks + sliding window; the simplest
baseline.
- **Attention-score eviction:** H₂O — Zhang et al. (2023),
[arxiv.org/abs/2306.14048](https://arxiv.org/abs/2306.14048); SnapKV — Li et al. (2024),
[arxiv.org/abs/2404.14469](https://arxiv.org/abs/2404.14469). The mainstream TriAttention critiques.
- **Norm-aware:** VATP — Guo et al. (EMNLP 2024),
[arxiv.org/abs/2406.12335](https://arxiv.org/abs/2406.12335). Value norms as a corrective to
attention-only scoring; the seed of S_norm.
- **Budget-allocation / frequency-selection:** PyramidKV (Cai et al. 2024,
[arxiv.org/abs/2406.02069](https://arxiv.org/abs/2406.02069)) and EliteKV (Zhou et al. 2025,
[arxiv.org/abs/2503.01586](https://arxiv.org/abs/2503.01586)), which decide *where* to spend a fixed
budget (across layers, across frequency bands) rather than *which tokens* to keep.
- **Reasoning-specific:** R-KV — Cai et al., NeurIPS 2025,
[arxiv.org/abs/2505.24133](https://arxiv.org/abs/2505.24133), which scores tokens for both importance
and **non-redundancy** (pruning the repetitive self-checking that bloats long chains of thought); the
closest baseline.
- **Surveys** — the paper's §2 points to KV-cache management/efficiency surveys (Li et al. 2024; Shi et
al. 2024) for a broad orientation.
**Training-free by construction — can you layer these onto an existing open-weight model?** Yes, and
this is the practical headline. Every method in *this* section (TriAttention, R-KV, H₂O, SnapKV,
StreamingLLM, Quest, quantization) is an **inference-time policy that changes only what sits in the KV
cache, not the weights** — so it drops onto any pretrained open-weight model (Llama, Qwen, DeepSeek,
GPT-OSS, …) with **no retraining and no fine-tuning**. R-KV's authors describe it exactly this way —
"training-free, plug-and-play for any autoregressive LLM" — and it's why TriAttention can be evaluated
across four different model families off the shelf. The one asterisk for TriAttention specifically: it
needs a **one-time offline calibration pass** to estimate the Q/K centers, but that is *statistics
collection over a forward pass, not weight training* — no gradients, no checkpoint changes — and §5.2.3
shows the statistics are robust enough that even mismatched calibration data works. Contrast this
sharply with the **head-sharing architectures** of §8.6 (MHA/MQA/GQA/MLA): those *are* baked into the
weights and can't be toggled on a finished model without at least conversion + uptraining. The clean
mental model: head-sharing is a *training-time* lever on cache size; the methods here are *runtime*
levers that compose on top of it.
---
### 8.3 Attention sinks and the structure of attention
> Sinks are where this paper's framework meets one of the most-studied empirical regularities in
> transformers. The classic observation: the first few tokens absorb a disproportionate share of
> attention regardless of content, and removing them hurts. TriAttention's contribution to this thread
> is conceptual — sinks aren't a special rule, they're the centers producing a distance-preference
> curve that peaks at the largest distances.
>
> *(Demo 5 in the companion shows why a sink can win attention weight yet contribute almost nothing — attention × value-norm.)*
- **StreamingLLM / attention sinks** — Xiao et al. (2024),
[arxiv.org/abs/2309.17453](https://arxiv.org/abs/2309.17453). The paper that named the phenomenon and
exploited it.
- **When/why attention sinks emerge** — Gu et al., ICLR 2025 (Spotlight),
[arxiv.org/abs/2410.10781](https://arxiv.org/abs/2410.10781). An empirical study of the conditions
(optimization, data, loss, architecture) under which sinks appear — useful for asking whether the
center-based view predicts *which* heads become sinks.
- **Value norms and sinks** — VATP (above) is also the key reference here: sinks get huge attention but
tiny value norms, which is *why* a norm term is needed alongside any attention/distance term.
#### Why high attention + near-zero value norm? (the "no-op" question)
A sharp question worth raising: VATP's observation is that a token can win the **QK**
competition (large query·key dot product → large attention *weight*) yet contribute almost nothing to
the output because its **V** (the value vector that actually gets summed) has near-zero norm. "Value
norms" here is exactly the V of QKV. If a token is going to be attended to, why would the model learn
to pair that with an empty value — isn't that wasted capacity?
The literature's answer is that this is a *functional* mechanism, not waste — it's the **attention
sink as a learned "no-op."** The mechanism:
- **Softmax forces the attention weights over keys to sum to 1.** A head often wants to attend to
*nothing* in particular (no relevant token at this step), but softmax won't let it output a
near-zero attention vector. So the head dumps the leftover probability mass onto a cheap,
content-free token — usually the first/BOS position — and pairs that token with a **near-zero
value**, so the dumped mass adds ≈0 to the output. The high-attention/low-value token is effectively
a learned `/dev/null` that lets the head "abstain."
- **These tokens carry large, input-independent activations that act as fixed biases.** *Massive
Activations* (Sun et al., COLM 2024, [arxiv.org/abs/2402.17762](https://arxiv.org/abs/2402.17762))
shows a handful of activations are ~constant regardless of input, drive the attention concentration
onto the sink token, and function as implicit bias terms — i.e., the sink is doing a job, not idling.
- **Give the model a dedicated scratch slot and the artifact moves there cleanly.** *Vision
Transformers Need Registers* (Darcet et al., ICLR 2024,
[arxiv.org/abs/2309.16588](https://arxiv.org/abs/2309.16588)) adds explicit "register" tokens; the
model relocates exactly this behavior into them and downstream features improve — evidence the model
*wants* a no-op/aggregation slot. Some recent LLMs bake in a **learnable sink token** for the same
reason.
- **Sinks emerge predictably from training.** Gu et al. (above) show they appear after enough
optimization on enough data and are encouraged by e.g. weight decay — a learned equilibrium, not
random parameter waste.
So the resolution to your intuition: it *isn't* wasted parameters — it's a cheap, learned solution to
a real constraint (softmax normalization), letting heads represent "attend to nothing meaningful."
And it's reclaimable by design: **softmax-off-by-one / "quiet attention"** (let the weights sum to
*less* than 1), explicit **register tokens**, or **learnable sink tokens** all reduce or relocate the
phenomenon. For TriAttention this is precisely why a distance/attention signal alone is insufficient
and the **norm term** is needed — a sink key scores high on S_trig (it's at a "preferred" distance for
a sink head) but should be demoted because its value contributes nothing. Good discussion territory:
*is the sink a necessary feature of softmax attention, or an artifact we should engineer away — and if
we engineer it away, does TriAttention still need its norm term?*
---
### 8.4 Long reasoning and test-time compute
> The whole motivation: reasoning models spend tokens to think, chains of thought run to tens of
> thousands of tokens, and the KV cache — not FLOPs — becomes the wall. This thread is the "why does
> this problem exist" backdrop.
>
> *(Demo 8 in the companion plots when the KV cache overtakes the model weights as context grows.)*
- **Chain-of-thought prompting** — Wei et al., NeurIPS 2022,
[arxiv.org/abs/2201.11903](https://arxiv.org/abs/2201.11903). The origin of explicit step-by-step
reasoning that inflates sequence length.
- **DeepSeek-R1** — DeepSeek-AI, *Nature* 645:633–638 (2025),
[arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948). The reasoning-model line whose distilled
variants (DS-Qwen, DS-Llama) are TriAttention's test models; explains why long, KV-heavy generations
are now the common case.
- **FlashAttention-2 / -3** — Dao 2023, [arxiv.org/abs/2307.08691](https://arxiv.org/abs/2307.08691) /
Shah et al., NeurIPS 2024, [arxiv.org/abs/2407.08608](https://arxiv.org/abs/2407.08608). The
exact-attention kernels TriAttention runs on; relevant because some baselines (e.g. H₂O's O(n²)
materialization) *can't* use them, which is part of the efficiency story.
- **The framing question:** is the right lever for cheaper long reasoning *compressing
the cache* (this paper), *shortening the reasoning* (efficiency-of-CoT work), or *internalizing it*
(latent/recurrent reasoning)? A good cross-paper discussion alongside the recurrent-depth
material.
---
### 8.5 Directional statistics and Fourier views of attention
> The paper reaches for two pieces of math that are worth a short primer: **directional statistics**
> (to quantify "how concentrated is a cloud of vectors") and a **Fourier-style synthesis** view (to
> turn centers into an attention-vs-distance curve). Both are standard tools used lightly here, and
> understanding them sharpens what the paper is and isn't claiming.
- **Directional statistics** — Mardia & Jupp (1999). The source of the **Mean Resultant Length**
*R* = ‖E[q]‖ / E[‖q‖]. The natural next concept is the **von Mises (–Fisher) distribution**, the
directional analog of a Gaussian, whose concentration parameter is closely related to *R* — a useful
lens for "how tight is this head's center?" (Demo 1 in the interactive companion lets you tighten or
scatter a von Mises cloud and watch R respond.)
- **Fourier is an *analogy* here, not a tool.** The key subtlety from §3: classical Fourier uses a
*harmonic* frequency progression (ωₙ = n·ω₀); RoPE uses a *geometric* one ($\omega_f=\theta^{-2f/d}$). So the
attention-vs-distance curve is a **weighted sum of cosines** — a *synthesis* that merely resembles
Fourier. Nothing functionally Fourier is happening: no Fourier **transform** is computed, the
frequencies are not a harmonic/orthogonal basis, and no signal is decomposed into a spectrum and
reconstructed from it. The coefficients are the **learned centers**, not transform outputs. Read
"Fourier" as descriptive shorthand for "a sum of sinusoids" and infer no spectral machinery from it.
(Demo 3 in the companion plots this synthesis; Demo 6 shows how the per-band phase φ_f shifts where
the curve peaks.) Worth pairing with the RoPE frequency-analysis references in §8.1.
- **Spectral / frequency analyses of attention** — EliteKV (Zhou et al. 2025,
[arxiv.org/abs/2503.01586](https://arxiv.org/abs/2503.01586)) treats RoPE bands as a spectrum to
select from; Hong et al. (EMNLP 2024, [arxiv.org/abs/2410.08703](https://arxiv.org/abs/2410.08703))
and Barbero et al. (ICLR 2025, [arxiv.org/abs/2410.06205](https://arxiv.org/abs/2410.06205)) study
which bands carry which roles. Reading these alongside the directional-statistics view connects
"which bands dominate" (spectral) with "how concentrated each band is" (directional).
---
### 8.6 Attention heads and head-sharing architectures
> "Retrieval heads" run through this paper's motivation, so it's worth knowing that attention heads are
> not interchangeable: mechanistic-interpretability work finds that individual heads specialize into
> recurring functional *types*, and that a small subset does the heavy lifting for long-context recall.
> That's exactly why eviction driven by a short observation window is dangerous (it starves those heads) and why several
> KV-compression methods now treat heads differently. A second, related axis is how heads *share* their
> K/V — the lever that sets KV-cache size in the first place.
#### A field guide to head types
These are behavioral roles found by ablation/interpretability studies, not architectural distinctions:
every head is the same matmul, but they learn different jobs.
- **Retrieval heads** — a small fraction of heads that locate and "copy forward" a specific earlier
token when it becomes relevant; ablating them sharply degrades long-context factuality and
needle-in-a-haystack retrieval, while ablating other heads barely matters. Wu et al.,
[arxiv.org/abs/2404.15574](https://arxiv.org/abs/2404.15574). **These are the heads TriAttention is
really trying to protect** — their targets lie dormant, so a short observation window evicts the key
before the head ever needs it.
- **Streaming / sink heads** — the complement: heads that mostly attend to recent tokens plus a few
initial "sink" tokens, and are fine with a small fixed cache. The retrieval-vs-streaming dichotomy is
made explicit by **DuoAttention** (Xiao et al., ICLR 2025,
[arxiv.org/abs/2410.10819](https://arxiv.org/abs/2410.10819)), which keeps a full cache only for
retrieval heads and a constant-length cache for streaming heads. Directly comparable to TriAttention's
framing — and from an overlapping author group.
- **Induction heads** — heads implementing the pattern "…[A][B]…[A] → [B]" (copy what followed the last
time you saw this token); argued to underlie much of in-context learning. Olsson et al. 2022,
[arxiv.org/abs/2209.11895](https://arxiv.org/abs/2209.11895) /
[transformer-circuits.pub](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html).
Conceptually upstream of retrieval heads — both are "go find the relevant earlier token" machinery.
- **Previous-token, name-mover / copy, and positional heads** — finer-grained roles from circuit
analysis, e.g. the name-mover heads in the IOI circuit (Wang et al., ICLR 2023,
[arxiv.org/abs/2211.00593](https://arxiv.org/abs/2211.00593)). Worth knowing the vocabulary exists;
not load-bearing for this paper.
The takeaway: "important tokens" is **head-relative**. A token that's noise for a
streaming head can be the whole game for a retrieval head — part of why TriAttention scores keys *per
head* and, under GQA, has to reconcile scores across the heads sharing a KV (§4.3).
#### Do any heads do *semantic* work?
Worth asking, because the heads above are mostly *positional/relational* — they locate or copy a token
by where it sits or what it matches, not by what it *means*. A few documented head types sit further
toward genuine abstraction:
- **Induction heads, and their fuzzier cousins.** Plain induction ("…[A][B]…[A] → [B]") is largely a
*copy-by-match* mechanism keyed on token identity — closer to pattern-matching than to meaning. But
"soft"/semantic induction generalizes the match to *similar* tokens rather than identical ones, which
begins to look semantic (Olsson et al. 2022, above).
- **Successor heads** — Gould et al. (ICLR 2024,
[arxiv.org/abs/2312.09230](https://arxiv.org/abs/2312.09230)). Heads that **increment an ordinal**
(Monday→Tuesday, January→February, one→two) through a single abstract "successor" direction shared
across number/month/day vocabularies. That shared, vocabulary-independent representation is real
*semantic* structure, not a positional copy.
- **Function-vector heads** — Todd et al. (ICLR 2024,
[arxiv.org/abs/2310.15213](https://arxiv.org/abs/2310.15213)). A small set of heads that, after
in-context demonstrations, **transport a compact vector encoding the demonstrated input→output
function**; inject that vector elsewhere and it triggers the task with no examples present. This is
the closest documented thing to "the model builds an internal *function* from context and re-applies
it."
This maps onto a natural intuition: that in-context learning might **synthesize a small function inside
the model** — an "ab → b" operator that *applies* a learned map to its argument — and the
function-vector results make a concrete version of that real. One caveat on the **monad** framing,
though: a monad in the functional-programming sense is about *sequencing* computations inside a context
(bind/return), whereas "ab → b" is closer to plain **function application / a learned combinator** than
to monadic bind. So the analogy points at something real (first-class, reusable internal functions) but
the specific category-theory label is a loose fit, not a mechanism anyone has demonstrated. (Adjacent
evidence that heads manipulate *meaning*, not just position: Merullo et al. 2023,
[arxiv.org/abs/2305.16130](https://arxiv.org/abs/2305.16130), find word2vec-style vector *arithmetic*
implementing simple relations.)
For TriAttention this is a useful boundary: its centers summarize a head's *distance* behavior, so the
more a head's real job is *semantic* (successor / function-vector style) rather than
distance-or-retrieval, the less a distance-only score can say about it — such heads must lean on the
norm-term fallback, if they are well-described by the framework at all.
#### How heads are shared: MHA, MQA, GQA, MLA
A separate, architectural axis — how many K/V projections the heads share — and the single biggest
lever on KV-cache size, so it frames every compression method. First, the term: a **KV head** is one
stored key/value projection (the K and V vectors that get cached and read at attention time), and a
**query head** is one query projection; letting several query heads read from the *same* KV head is
exactly what shrinks the cache. The four schemes differ only in that sharing ratio:
- **MHA (Multi-Head Attention)** — every query head has its own K and V. Maximum expressivity, maximum
cache.
- **MQA (Multi-Query Attention)** — all query heads share **one** K/V. Smallest cache, some quality
loss. Shazeer, [arxiv.org/abs/1911.02150](https://arxiv.org/abs/1911.02150).
- **GQA (Grouped-Query Attention)** — query heads are split into **G groups**, each sharing a K/V; the
middle ground used by most current open models. Ainslie et al.,
[arxiv.org/abs/2305.13245](https://arxiv.org/abs/2305.13245). This is why TriAttention needs its
**normalize-then-aggregate** step: each cached key gets G differently-scaled scores (one per sharing
query head), so it z-scores per head and takes the max (§4.3).
- **MLA (Multi-head Latent Attention)** — DeepSeek-V2/V3's scheme: compress K/V into a low-rank latent
(tiny cache) with a separate decoupled-RoPE branch for position (see the three-way "Where positions enter" flow diagram in §8.1 — its third branch traces MLA's decoupled-RoPE path).
TriAttention checks that its Q/K-concentration phenomenon still holds here — Appendix I / Table G
reports ~96.6% of MLA heads with R > 0.95, even stronger than GQA — which matters because MLA is
increasingly common in frontier-class open models.
#### In practice: trained-in, not bolted on
A common point of confusion: these four are **architectural choices baked in at pretraining**, not
inference-time cache policies. The number of KV heads (and, for MLA, the latent dimension) is part of
the model definition — it fixes the *shape* of the K/V projection weights — so a model is *born* MHA,
MQA, GQA, or MLA. The cache savings are a structural consequence: with fewer KV heads you simply
project, store, and load fewer K/V tensors. At attention time, MQA/GQA **broadcast** the shared KV
heads back across their query group (in code the config differs only in `num_key_value_heads`, and the
forward pass does a `repeat_kv`-style expansion); MLA instead caches a small **latent** and, at
inference, "absorbs" its up-projection weights into the query/output projections so attention runs
directly against the latent (an MQA-like compute pattern). None of this is something you toggle on a
finished model for free.
But "trained-in" doesn't mean "only from scratch" — there's a cheap **conversion** path, which is
where the post-hoc/trained-in line genuinely blurs:
- **MHA → MQA/GQA (uptraining).** The GQA paper's actual recipe converts an existing MHA checkpoint by
**mean-pooling** each group's K/V heads, then **uptraining for only ~5% of the original pretraining
compute** so the model adapts to the new structure; mean-pooling beats selecting one head or random
init. So GQA is trained-in, but it can be retrofitted onto a trained MHA model cheaply rather than
pretrained anew. Ainslie et al., [arxiv.org/abs/2305.13245](https://arxiv.org/abs/2305.13245).
- **GQA/MHA → MLA (recent conversions).** Newer work converts already-trained models into MLA:
**TransMLA** ([arxiv.org/abs/2502.07864](https://arxiv.org/abs/2502.07864)) argues MLA is strictly
more expressive than GQA at equal cache size and gives a largely training-free GQA→MLA conversion
(performance mostly restored after a few billion tokens of fine-tuning); the concurrent **MHA2MLA**
does similar from MHA. These are "architecture surgery plus a little retraining," not runtime
switches.
**The clean way to hold it:** head-sharing shrinks the cache by storing *fewer K/V heads* — a property
of the *weights*, decided in pretraining (or via conversion + uptraining). KV-cache *compression* —
TriAttention, R-KV, H₂O, SnapKV, quantization — shrinks the cache by keeping *fewer tokens* (or fewer
bits) at **inference time, with no weight changes**. The two axes are orthogonal and **compose**:
TriAttention is evaluated on a GQA model (Qwen3-8B) and an MLA model (GLM-4.7-Flash), running *on top
of* whatever head-sharing the model was trained with. That's precisely why §4.3's GQA handling and
Appendix I's MLA check exist — the method has to coexist with the architectural choice, not replace it.
#### Why head structure matters for TriAttention
Both axes feed the method directly. The **type** axis is the *motivation*: the entire case against
post-RoPE observation windows is that they fail precisely on retrieval heads. The **sharing** axis is
an *implementation constraint*: GQA forces the cross-head score reconciliation, and MLA is the
architecture the generality claim is tested against. A good discussion question: TriAttention scores
every head with the same machinery — would an explicitly *head-type-aware* budget (full cache for
retrieval heads à la DuoAttention, aggressive pruning for streaming heads) stack with it, or does the
center-based score already capture that distinction implicitly?
#### Reading list
- **Retrieval Head Mechanistically Explains Long-Context Factuality** — Wu et al.,
[arxiv.org/abs/2404.15574](https://arxiv.org/abs/2404.15574).
- **DuoAttention** — Xiao et al., [arxiv.org/abs/2410.10819](https://arxiv.org/abs/2410.10819)
(retrieval vs. streaming heads).
- **In-context Learning and Induction Heads** — Olsson et al.,
[arxiv.org/abs/2209.11895](https://arxiv.org/abs/2209.11895).
- **Successor Heads** — Gould et al., [arxiv.org/abs/2312.09230](https://arxiv.org/abs/2312.09230)
(abstract incrementation — a genuinely *semantic* head type).
- **Function Vectors in Large Language Models** — Todd et al.,
[arxiv.org/abs/2310.15213](https://arxiv.org/abs/2310.15213) (heads that transport an
in-context-learned input→output function).
- **MQA** — Shazeer, [arxiv.org/abs/1911.02150](https://arxiv.org/abs/1911.02150); **GQA** — Ainslie
et al., [arxiv.org/abs/2305.13245](https://arxiv.org/abs/2305.13245); **MLA** — DeepSeek-V2,
[arxiv.org/abs/2405.04434](https://arxiv.org/abs/2405.04434), and DeepSeek-V3,
[arxiv.org/html/2412.19437v1](https://arxiv.org/html/2412.19437v1).
- **Converting trained models** — TransMLA (GQA → MLA),
[arxiv.org/abs/2502.07864](https://arxiv.org/abs/2502.07864); plus the GQA paper's own MHA → GQA
mean-pool-and-uptrain recipe (above).
- Broader interpretability backdrop: the **Transformer Circuits** thread,
[transformer-circuits.pub](https://transformer-circuits.pub).
---
### 8.7 The argument, geometrically: one step at a time
> This section unpacks the two paragraphs that carry the paper's core logic: first the *argument*
> (concentration → approximate by centers → substitute into RoPE → the logit depends only on distance →
> a trigonometric series / attention-vs-distance curve → peaks are "distance preference" → verified
> against real attention), then the *design move* that turns that picture into the scoring function.
> Each step gets a concrete geometric picture — including how the "high-dimensional" part stays
> manageable.
The paper compresses these moves into a few sentences, and each one is geometric — the geometry is what
makes the claims feel inevitable rather than magical. We'll walk them in order, and (this is where
people get stuck) show that nothing here requires visualizing a 128-dimensional space: RoPE hands us a
decomposition that turns the whole thing into a stack of ordinary 2-D pictures that we add up.
*(The interactive companion animates these steps: Demo 1 is Step 1 (the R "needle"), Demo 6 is the
per-band phase φ_f of Step 3, and Demo 3 is the summed attention-vs-distance curve the later steps
build toward.)*
#### Step 0 — The move that tames high dimensions
A query or key vector lives in **ℝ^d** (often d = 128 per head). You can't picture that, and you don't
have to. RoPE splits the vector into **d/2 two-dimensional planes** (the "frequency bands"), and **the
only thing position does is rotate each plane** — band *f* turns at its own rate ω_f. The attention
logit between a query and a key is then just a **sum, over those planes, of a 2-D dot product**. So
every statement in the argument is really a statement about *one 2-D plane*, repeated d/2 times and
summed. "High-dimensional intuition" here = "many little 2-D pictures, added together." Hold onto that;
it's the whole trick.
#### Step 1 — "Q/K are highly concentrated around fixed centers"
Plane by plane: collect a head's query vectors across many tokens and prompts, drop them into one band's
2-D plane, and they form a **tight little cloud around a single direction** (same for keys). "Tight" is
the **Mean Resultant Length** *R*, which has a clean reading: lay the arrows tip-to-tail and see how far
you get.
When the arrows point the same way, the tip-to-tail sum is a long resultant whose length nearly equals
the individual lengths, so the ratio *R* → 1. When they scatter, the sum curls back on itself and the
resultant collapses toward zero, so *R* → 0. The paper's finding is that for the vast majority of heads
*R* is close to 1 — the cloud really is a tight needle, stable across positions and content.
#### Step 2 — "Approximate each Q/K by its center"
If the cloud is a needle, you lose almost nothing by replacing every arrow with the **one center
arrow**. Geometrically: collapse the cloud to its mean direction. This is the load-bearing
simplification — it turns *variable* vectors (which differ token to token) into *constants*.
Worth flagging what gets discarded: the little bit of *spread* around the center. That residue is
exactly what the method's **norm-based score** later mops up, and the per-band weight (1 − R_f) is
literally "how much spread did we ignore in this plane." High R → tiny spread → trust the center; lower
R → keep the norm correction. Step 2 and the two-part scoring function are the same idea seen twice.
#### Step 3 — "Substitute into the RoPE formula → the logit depends only on distance"
The geometry of a single plane. In plane *f*, the query-center and key-center are two fixed arrows.
RoPE rotates the query's arrow by ω_f·p_q and the key's by ω_f·p_k. The 2-D dot product between them is
their lengths times the cosine of the **angle between them** — and after both rotations that angle is
just *(their fixed pre-RoPE angle gap)* **plus** *ω_f·(p_q − p_k)*. The absolute positions cancel; only
the **distance $\Delta=p_q-p_k$** survives. Because the arrows are now constants (Step 2), the *only*
moving part left is that relative rotation ω_f·Δ.
The blue/orange arrows are the query- and key-centers in three example planes; the gray arc is the
position-driven rotation, large in fast bands and tiny in slow ones. The box on the right is the whole
logit: add up the per-plane cosines. Slide the key further away (increase Δ) and every plane's angle
advances by ω_f·Δ, so each cosine swings — and the total swings with it.
#### Step 4 — "...which is a trigonometric series — an attention-vs-distance curve"
Put Step 3 on a graph with **Δ on the horizontal axis**. Each plane contributes a cosine that
oscillates at its own fixed rate ω_f; fast bands wiggle quickly, slow bands drift. The attention logit
at distance Δ is the **weighted sum of all these cosines**, with weights and starting phases set by the
centers. That sum is one curve: attention as a function of distance. Where the band cosines happen to
**line up (crest together)**, the curve peaks — here, at a moderate Q–K distance.
The shaded band marks the distance where all three cosines crest together; that constructive alignment
is the peak in the summed curve below. This is the **Fourier-synthesis analogy** made literal: the
frequencies are fixed by RoPE (and they're *geometric*, not the harmonic ladder of a textbook Fourier
series), while the **centers supply the coefficients**. Different centers synthesize a different curve —
slide the peak left for a head that favors nearby keys, or far right for an "attention sink."
#### Step 5 — "Peaks at specific distances = distance preference; the centers decide which"
A peak means the planes' cosines reinforce each other **constructively** at that Δ, so a key sitting
that far from the query earns a big logit; troughs are destructive cancellation. So "this head prefers
keys at distance X" is just "the band cosines line up at X," and *which* X falls straight out of the
centers (their lengths set the amplitudes, their angle gaps set the phases). Crucially, nothing about
the actual incoming query is needed — the head's *typical* query (its center) already fixes the curve.
That is the leap that lets the method score a key **before** the query that will read it ever exists.
#### Step 6 — "Verified: keys at those distances really do get more attention"
The last move of the argument is empirical and closes the loop. If concentration truly *causes* the
distance preference, the curve predicted from centers should match the head's *real* attention over Δ.
The paper measures this as the **reconstruction correlation** r̄ and finds it meaningfully positive
across three model families — geometrically, the predicted and measured curves rise and fall together.
Keep §5's honesty in view: the correlation is *moderate* (means around 0.5, with a long tail of
poorly-reconstructed heads), which is exactly why the method retains the norm-based fallback for the
planes where the center-only picture is weak.
#### Step 7 — From the picture to the scoring function (the second paragraph)
The follow-on paragraph is where the geometry becomes the method, and it reads cleanly once Steps 0–6
are in hand:
- *"Use the Q center together with the trigonometric series to evaluate importance differences among
keys."* Each cached key has a known position, hence a known Δ to a future query. Drop that key onto
the **attention-vs-distance curve** (built from the Q center): its height is the attention it can
expect. Keys landing on peaks are worth keeping; keys in troughs are evictable. That is **S_trig**.
- *"For the minority of heads where Q is less concentrated, incorporate Q/K norms as complementary
signals."* When a head's cloud isn't a tight needle (low *R*, Step 1), the center is a poor summary
and the curve is unreliable — so lean on the **spread/norm** information instead (Step 2's discarded
residue). That is **S_norm**.
- *"Automatically balance these two components using a Q/K concentration metric."* The blend is the
per-band weight **(1 − R_f)**: where concentration is high, trust the curve; where it's low, let the
norm term carry more. One knob, read straight off the geometry.
So the whole method is: *build each head's distance-preference curve from its centers, score every
cached key by where it lands on that curve, correct with norms where the centers are untrustworthy, and
keep the top-scoring keys.* Every later engineering detail (future-offset averaging, 128-token pruning
windows, GQA reconciliation) sits on top of this one geometric picture.
#### The whole thing in one breath
Split the high-dimensional vector into 2-D planes (Step 0); in each plane the head's vectors huddle
around a fixed center (Steps 1–2); RoPE rotates the planes so the dot product depends only on distance
(Step 3); summing the planes' cosines over distance draws a fixed attention-vs-distance curve whose
shape the centers fully determine (Steps 4–5); that predicted curve matches real attention well enough
to act on (Step 6); so you score each cached key by where it lands on the curve and prune the rest
(Step 7).
---
### 8.8 Offline vs. online: the two-phase pipeline
A point that's easy to miss on a first read, and worth making explicit: TriAttention is **not** a single
runtime algorithm. It has a one-time **offline** phase and a per-request **online** phase, and almost
everything that makes the method *cheap and query-agnostic* lives in that split.
```mermaid
flowchart LR
subgraph OFF["OFFLINE · once per model · statistics only, no weight updates"]
direction TB
a1["calibration corpus"] --> a2["forward passes: collect pre-RoPE q, k per head, per band f"]
a2 --> a3["per head/band: centers E[q_f], E[k_f], magnitudes + phase φ_f, concentration R_f, norm stats"]
a3 --> a4["derive constants: a_f, b_f (trig coeffs) (1−R_f) weights dominant bands per head"]
a4 --> a5["freeze TABLE T per head: a_f, b_f, R_f, norms, chosen bands"]
end
subgraph ON["ONLINE · every request, during decode"]
direction TB
b1["tokens stream; KV cache grows"] --> b2["every 128 tokens, if cache over budget B"]
b2 --> b3["compute S_trig + S_norm S_trig from a_f, b_f S_norm from (1−R_f), norms"]
b3 --> b4["score = S_trig + S_norm, averaged over future offsets D"]
b4 --> b5["GQA normalize-then-aggregate; keep top-B + recent window, evict the rest"]
b5 --> b1
end
a5 -->|frozen table T| b3
```
**Preparation recipe — what you do once, per model:**
1. **Choose a calibration corpus.** A modest, generic text sample is enough; §5.2.3 reports the result
is robust even when this data doesn't match the eventual serving workload.
2. **Collect pre-RoPE Q/K.** Run forward passes and, for every attention head and every frequency band
f, record the *pre-RoPE* query and key vectors (the 2-D pair per band) across the sampled tokens.
3. **Compute per-head/band statistics.** From those vectors get the centers E[q_f] and E[k_f] (the mean
pre-RoPE directions) — and hence the magnitudes |q_f|, |k_f| and the phase φ_f = arg(q_f) − arg(k_f)
— plus the concentration R_f and the norm statistics.
4. **Derive the scorer constants.** Convert the statistics into the numbers the curve actually uses: the
trig-series coefficients a_f = |q_f||k_f|·cos φ_f and b_f = −|q_f||k_f|·sin φ_f (§8.10), the (1 − R_f)
weights for S_norm, and each head's **dominant bands** (which f to keep).
5. **Freeze the table.** Store a small per-head table T = { a_f, b_f, R_f, norm stats, chosen bands }.
No weights are updated — the model itself is unchanged.
That table is everything inference needs. At decode time the online phase simply evaluates S_trig (from
a_f, b_f) + S_norm (from the (1 − R_f) weights and norm stats) over the future offsets D, applies the
GQA normalize-then-aggregate step, and prunes — it never recomputes a statistic. Steps 1–3 are the
"analyze the heads offline" part of your reading; step 4 is the "generate the constants" part; step 5
is the table those constants live in.
**What happens offline (once).** A **calibration** pass runs the model over a modest corpus and records,
for each head and each frequency band, the pre-RoPE query/key **centers** (the mean directions), the
**concentration** R_f, and **norm** statistics. That's the entire offline cost — *statistics collection
over ordinary forward passes*. Crucially there are **no gradient steps and no weight changes**: the
model's parameters are untouched. The output is a small table of fixed numbers that becomes the
parameters of the scoring function (S_trig, S_norm, and the (1 − R_f) weights). The paper's §5.2.3
ablation reports this is robust even when the calibration data does not match the eventual workload —
the centers are a stable property of the model, not of the prompt.
**What happens online (every request).** As tokens stream during decode, the cache grows; once per
**128-token window**, if the cache exceeds the budget B, each cached key is scored *using the frozen
offline centers* — no recomputation of statistics, no look at future queries — averaged over the future
offsets D, and the top-B keys (plus the recent window) are kept while the rest are evicted. This is the
only per-request work, and it's deliberately light.
**Why the split matters.**
- **It's what makes scoring query-agnostic.** Because the centers are fixed offline, the online phase
can score a key for a query that *doesn't exist yet* (§8.7 Step 5). A purely online method must wait
to observe attention; TriAttention reads it off a precomputed curve.
- **Offline ≠ training.** This is the sharp distinction from §8.2 and §8.6: calibration is *not*
retraining (no weight updates), so the method still layers onto any open-weight model — it just adds
a cheap, one-time statistics pass rather than being fully zero-setup.
- **It's a different cost profile from the baselines.** R-KV and SnapKV are essentially **fully online**
(they score from observed/recent attention with no calibration step); Expected Attention also avoids
calibration by computing expected attention in closed form at runtime. TriAttention trades a small
offline pass for cheaper, query-independent online scoring. A good discussion question:
*when is a one-time offline calibration worth it versus a purely online scorer — does it pay off most
for repeated deployment on a fixed model, and how sensitive is it to distribution shift between
calibration and serving?*
The mental model to carry: **offline decides the *shape* of each head's distance-preference curve;
online just drops cached keys onto that curve and prunes.**
---
### 8.9 The complex-number representation: from dot product to Equations 15 and 16
> This section answers two questions the paper leaves implicit: *why do Figure 2(A)'s axes say "Re" and
> "Im"?* and *where does the cosine in the trigonometric series actually come from?* The answer is one
> continuous chain starting from the definition of a dot product. (Notation: `z̄` means the complex
> conjugate of `z`; we write it `conj(z)` inside the boxed steps. Indices are written `x[2f]`.)
#### What is an attention logit?
In a transformer, attention computes a score between a query vector **q** and a key vector **k**. This
score — the "logit" — determines how much attention the query pays to that key. At the most basic level
it is just a dot product:
$$\operatorname{logit}=q\cdot k=\sum_i q_i\,k_i$$
Higher dot product = greater alignment = more attention. Nothing more is happening yet.
#### RoPE splits the vector into 2-D chunks
RoPE operates on a d-dimensional vector by slicing it into d/2 pairs of coordinates. Each pair
(x[2f], x[2f+1]) forms a little 2-D subspace, indexed by the band f = 0, 1, …, d/2−1.
Instead of carrying pairs of real numbers, it is mathematically cleaner to represent each 2-D pair as a
single complex number:
$$z_f=x[2f]+i\,x[2f+1]\qquad(\text{a complex number},\ \in\mathbb{C})$$
This is purely notational convenience — a complex number `a + ib` is a 2-D vector `(a, b)` in disguise.
The full query q ∈ ℝ^d is therefore re-expressed as d/2 complex numbers q_f ∈ ℂ, one per band.
**This is why Figure 2(A) has "Re" and "Im" axes.** The paper is plotting q_f and k_f in the complex
plane for a single band f: the real part is dimension 2f, the imaginary part is dimension 2f+1. It is
the same picture as Demo 2's spotlight (which labels those axes "dim 2f" and "dim 2f+1") — just using
the standard complex-number vocabulary instead.
**Why one blob looks vertical and the other horizontal.** Each blob lies along its center's *direction*
in that band, and the Q and K centers point different ways: in the displayed head the Q center sits near
the imaginary axis (Q cloud vertical) and the K center near the real axis (horizontal) — i.e., the two
centers are roughly **orthogonal**, a relative phase φ_f = arg(q_f) − arg(k_f) ≈ 90°. Only that
*relative* angle is meaningful (the absolute orientation is an arbitrary consequence of which dimension
is labelled "real"), and it is exactly the constant phase that sets where the band's cosine sits: from
§8.10 the band contributes |q_f|·|k_f|·cos(ω_f·Δ + φ_f), so φ_f ≈ 0 gives a maximum at Δ = 0 (attend to
self/nearby), φ_f ≈ 180° gives a *minimum* at Δ = 0 (mass pushed outward), and φ_f ≈ 90° (this panel)
makes the contribution ~0 at Δ = 0 and sinusoidal in distance.
This Q/K directional offset is a documented phenomenon, not a plotting accident. *Frayed RoPE: A
Geometric Perspective* ([arxiv.org/abs/2603.18017](https://arxiv.org/abs/2603.18017)) finds pre-RoPE Q
and K forming tight clusters in *separated / opposing* directions with the origin between them, and ties
that separation (with small sink-token norm) to the attention-sink bias of §8.3. The NeurIPS 2024 study
*What RoPE Can Tell Us* ([openreview.net/forum?id=e5Mv7iWfVW](https://openreview.net/forum?id=e5Mv7iWfVW);
code: [github.com/Ytchen981/RoPE_investigate](https://github.com/Ytchen981/RoPE_investigate)) shows the
angle between a band's Q and K *weight* vectors governs head behavior: near-aligned (non-orthogonal)
pairs lock onto a fixed relative position and are input-insensitive (syntactic), while near-orthogonal
pairs stay position-flexible (semantic). So the orientation gap encodes real structure — but only φ_f
matters (the absolute orientation is arbitrary), and since the TriAttention figure shows one example
head, whether *this* near-orthogonality is typical the figure alone can't establish.
#### What does the dot product look like in complex notation?
For two 2-D real vectors (a, b) and (c, d), their dot product is a·c + b·d. In complex notation, with
z₁ = a + ib and z₂ = c + id, the same quantity is Re(z₁ · z̄₂), where z̄₂ = c − id is the complex
conjugate. Verification:
$$z_1\cdot\overline{z_2}=(a+ib)(c-id)=\underbrace{(ac+bd)}_{\text{Re}}+\,i\,\underbrace{(bc-ad)}_{\text{Im (discarded)}}$$
Taking the real part gives a·c + b·d. ✓ The imaginary part (b·c − a·d) is discarded — it is an
artefact of the complex representation that carries no meaning for the dot product (it is the 2-D cross
product, a signed area, not a similarity measure).
So the real part of z₁·conj(z₂) is *exactly* the dot product of the corresponding 2-D real vectors.
#### The full dot product across all bands
The complete dot product between q and k is the sum of contributions from all d/2 bands:
$$q\cdot k=\sum_f \operatorname{Re}\!\big(q_f\,\overline{k_f}\big)$$
#### After RoPE rotation
RoPE rotates each band by a position-dependent angle. For a token at position p, band f is rotated by
ω_f·p — which in complex notation is multiplication by e^(i·ω_f·p):
$$\tilde q_f=q_f\,e^{\,i\,\omega_f p_q},\qquad \tilde k_f=k_f\,e^{\,i\,\omega_f p_k}$$
The attention logit between the rotated query and key is then:
$$\operatorname{logit}=\sum_f \operatorname{Re}\!\big(\tilde q_f\,\overline{\tilde k_f}\big)\qquad\text{(Equation 15)}$$
The Re(·) is not an extra operation being imposed — it is literally what a dot product *is* when
written in complex notation.
#### Why does this simplify so nicely?
Expand the per-band term:
$$\begin{aligned}
\operatorname{Re}\!\big(\tilde q_f\,\overline{\tilde k_f}\big)
&= \operatorname{Re}\!\big(q_f\,e^{i\omega_f p_q}\cdot\overline{k_f\,e^{i\omega_f p_k}}\big)\\
&= \operatorname{Re}\!\big(q_f\,\overline{k_f}\;e^{\,i\,\omega_f (p_q-p_k)}\big)
\end{aligned}$$
The two absolute positions cancel; only the **relative distance** $\Delta=p_q-p_k$ survives. Writing
q_f·conj(k_f) in polar form as |q_f|·|k_f|·e^(i·φ_f) and taking the real part gives:
$$\operatorname{Re}\!\big(q_f\,\overline{k_f}\;e^{\,i\,\omega_f\Delta}\big)=|q_f|\,|k_f|\,\cos(\omega_f\Delta+\varphi_f)\qquad\text{(summand of Equation 16)}$$
where φ_f = arg(q_f) − arg(k_f) is the angle between the query and key centers in band f. This is the
**trigonometric series** whose sum is the attention logit as a function of distance.
#### The whole chain in one thread
$$\begin{aligned}
& q\cdot k && \text{(dot product)}\\
\rightarrow\;& \sum_f \operatorname{Re}\!\big(q_f\,\overline{k_f}\big) && \text{(complex notation)}\\
\rightarrow\;& \sum_f \operatorname{Re}\!\big(q_f\,\overline{k_f}\;e^{\,i\,\omega_f\Delta}\big) && \text{(after RoPE: positions collapse to }\Delta\text{)}\\
\rightarrow\;& \sum_f |q_f|\,|k_f|\,\cos(\omega_f\Delta+\varphi_f) && \text{(trigonometric series)}
\end{aligned}$$
The Re/Im notation in Figure 2(A), the complex exponential in Equation 15, and the cosine series in
Equation 16 are all the same object viewed at different stages of this single derivation. The paper
presents them as three separate results; the chain above shows they are one continuous thread.
---
### 8.10 The constant-Q/K approximation: deriving the trigonometric series (Equation 17)
> §8.9 ended at Equation 16 — the per-band logit `|q_f|·|k_f|·cos(ω_f·Δ + φ_f)`. This section takes the
> one further step the method actually relies on: under the paper's "constant Q/K" approximation, that
> expression collapses into a fixed trigonometric series in the distance Δ (Equation 17), with all the
> Q/K information baked into constant coefficients. It is the formal version of §8.7 Steps 4–5.
#### Why would we even treat Q/K as constant?
At first this looks like a wild assumption — surely every token has a different query and key? It does,
but remember we are in **pre-RoPE space**. The paper's central empirical observation (§3.1, Figure
2(A)) is that the *pre-RoPE* query and key vectors of real LLMs cluster very tightly around a fixed
mean: the Mean Resultant Length R ≈ 1 for most heads (at their dominant band), so the vectors nearly
all point the same way with similar magnitude.
So "constant Q/K" is **not** claiming all tokens are identical. It is a first-order approximation:
replace the tight cloud of vectors with its center point and ignore the small spread. (That ignored
spread is exactly what the norm term S_norm, weighted by (1 − R_f), later mops up — see §8.3 and the
glossary.) The math then tells you what attention behavior the center implies, and it matches real
attention reasonably well — reconstruction correlations land around r ≈ 0.5–0.9 across heads, with the
honest caveat from §5 and §8.7 Step 6 that the mean is only moderate and there is a poorly-reconstructed
tail.
#### The setup
From Equation 16, the per-band logit is:
$$|q_f|\,|k_f|\,\cos(\omega_f\Delta+\varphi_f)\qquad\text{where}\quad \varphi_f=\arg(q_f)-\arg(k_f)$$
When Q/K are treated as constant, |q_f|, |k_f|, and φ_f are all fixed numbers — they do not vary with
position. The only thing that varies across different keys is $\Delta=p_q-p_k$, the distance.
#### The angle-addition formula
Trigonometry's angle-addition identity:
$$\cos(A+B)=\cos A\,\cos B-\sin A\,\sin B$$
Applied to cos(ω_f·Δ + φ_f), with A = ω_f·Δ and B = φ_f:
$$\cos(\omega_f\Delta+\varphi_f)=\cos(\omega_f\Delta)\cos\varphi_f-\sin(\omega_f\Delta)\sin\varphi_f$$
Substitute back and absorb the fixed quantities into coefficients:
$$\begin{aligned}
|q_f|\,|k_f|\,\cos(\omega_f\Delta+\varphi_f) &= a_f\cos(\omega_f\Delta)+b_f\sin(\omega_f\Delta)\\
\text{with}\quad a_f &= |q_f|\,|k_f|\cos\varphi_f\\
b_f &= -|q_f|\,|k_f|\sin\varphi_f
\end{aligned}$$
The point of this step is to **separate what depends on Δ from what depends on Q/K**. Before the
identity, φ_f was tangled inside the cosine together with Δ. After it, Δ appears only inside pure
sinusoids, and everything about Q/K is isolated in the constant coefficients a_f, b_f.
#### Summing over all frequency bands
Summing over all bands f gives Equation 17:
$$\operatorname{logit}(\Delta)\approx\sum_f\big[\,a_f\cos(\omega_f\Delta)+b_f\sin(\omega_f\Delta)\,\big]$$
This is a **trigonometric series in Δ alone**. The coefficients a_f, b_f are fixed constants determined
entirely by the Q/K centers, computed once **offline** during calibration (§8.8). At inference time,
scoring a key at distance Δ is just evaluating this series — no actual attention computation needed.
#### Why this is analogous to Fourier synthesis
A classical Fourier series builds an arbitrary periodic function from sinusoids at **harmonically**
related frequencies ω, 2ω, 3ω, … with chosen coefficients. Here the structure is the same, with two
differences:
- the frequencies $\omega_f=\theta^{-2f/d}$ follow a **geometric** progression set by RoPE, not a harmonic one
(the §8.5 caveat — it is a synthesis over a non-harmonic grid, not a true Fourier series);
- the **Q/K centers play the role of the coefficients** — they shape which distances get high attention
and which get low.
A head whose centers produce a curve peaking at small Δ behaves like **local attention**; one peaking
at large Δ behaves like an **attention sink** (§8.3). The centers determine the shape; the trigonometric
series makes that shape explicit and computable. This is exactly the picture Demo 3 animates — the bands
summing into the attention-vs-distance curve — and the formal basis for §8.7 Steps 4–5.
---
### 8.11 Is the reasoning trace worth caching? Faithfulness, "reasoning theater," and cross-turn memory
> **Beyond the paper.** TriAttention makes no claims about reasoning *faithfulness*; this section connects
> its setting — compressing the KV cache of long reasoning traces — to a separate, fast-growing literature
> on whether those traces mean what they appear to. It is deliberately speculative, flagged as open
> questions rather than findings, but the link is concrete: the redundancy of reasoning traces is the very
> thing that makes methods like this one work.
#### The empirical premise: much of a reasoning trace is redundant or "performative"
A line of recent work argues that long chains-of-thought are often not faithful records of the model's
computation. Models can be confident in their final answer very early and then keep emitting step-by-step
text without revealing that commitment — **performative** CoT — and probes reading the model's internal
state can detect this and exit the trace early with large token savings (*Reasoning Theater*,
[arxiv.org/abs/2603.05488](https://arxiv.org/abs/2603.05488)); relatedly, trace *length* turns out to be
only brittly correlated with problem difficulty rather than a reliable measure of "thinking effort"
(Palod et al., [arxiv.org/abs/2509.07339](https://arxiv.org/abs/2509.07339)). Earlier results show CoT
explanations can be plausible yet **post-hoc** rather than causal (Turpin et al.,
[arxiv.org/abs/2305.04388](https://arxiv.org/abs/2305.04388); Lanham et al.,
[arxiv.org/abs/2307.13702](https://arxiv.org/abs/2307.13702)); a 2026 survey frames
much CoT as post-hoc rationalization stemming from a mismatch between parallel transformer computation and
sequential text ([arxiv.org/abs/2601.14270](https://arxiv.org/abs/2601.14270)); and reasoning models used
as judges can be fooled by *fake* CoT — "reasoning theater bias"
([arxiv.org/abs/2507.13758](https://arxiv.org/abs/2507.13758)). The nuance worth keeping is *not* "reasoning
is fake," but "models reason on the hard steps and coast on the easy ones," so a trace is a mixture of
load-bearing and decorative tokens.
#### Why this is exactly what makes reasoning-model KV compression work
This connects straight back to §8.2 and the **R-KV** baseline. R-KV's own motivation is that over half of a
reasoning model's tokens contribute little, and that naive attention-based eviction fails because repetitive
self-checks attend heavily to *themselves*
([arxiv.org/abs/2505.24133](https://arxiv.org/abs/2505.24133)) — the same self-attention pathology behind
the attention sinks of §8.3. Others find pruning redundant reasoning can even *raise* accuracy (Think Clearly,
[arxiv.org/abs/2507.08806](https://arxiv.org/abs/2507.08806)) and that compression is best allocated by
"thought" importance (ThinKV, [arxiv.org/abs/2510.01290](https://arxiv.org/abs/2510.01290)). So
*performativity* and *compressibility* are two views of one fact: if much of the trace isn't doing causal
work, much of its KV cache is safe to drop. TriAttention's distance-preference scoring is one more way to
decide *which* tokens those are.
The same pattern shows up from the opposite direction in the recurrent-depth literature: looping a
reasoning model *too many* times **degrades** accuracy — it reaches the right answer and then talks
itself back out of it ("overthinking") — so more reasoning compute is not monotonically better
(recurrent-depth study, [arxiv.org/abs/2604.07822](https://arxiv.org/abs/2604.07822)). Whether the
surplus steps are emitted as text (a long chain-of-thought) or spent as internal iterations
(recurrence), past a point they add confusion rather than signal — the same reason much of a trace's
KV cache is safe to drop.
#### Cross-turn memory: what (if anything) to keep from a reasoning trace
The harder, less-studied question is what should survive into *later* turns. The two cases pull in opposite
directions:
- **Invalid / performative traces are a pollution risk.** If a trace is post-hoc rationalization, carrying it
verbatim into the next turn — or into a long-term memory store — risks treating confident-sounding but
non-causal "reasons" as established fact. This compounds a known multi-turn failure: extra reasoning tokens
don't help models with multi-turn underspecification, and longer reasoning responses introduce more
*assumptions* that derail later turns by blurring what the user asked versus what the model itself
previously assumed (*LLMs Get Lost in Multi-Turn*,
[arxiv.org/abs/2505.06120](https://arxiv.org/abs/2505.06120)). Under this case, aggressively pruning or
fully discarding the trace before the next turn is likely *protective*, not merely cheap.
- **Valid traces may matter for their *structure*, not their *text*.** When a trace did real work, what a
future turn plausibly needs is the *outcome* of that work — which subgoals were settled, which branches were
ruled out, which intermediate result was reached — not the verbatim token stream. That argues for
**abstracting or summarizing** a valid trace into memory rather than caching it token-for-token, and it
rationalizes the common production choice to drop the raw reasoning trace between turns and keep only the
final answer. It also exposes a tension with KV compression: a distance- or attention-based scorer keeps the
tokens that mattered *for producing this answer*, which is not necessarily the set that would matter *for a
future, different question*.
This is where the topic touches **memory systems and long context** directly. A verbatim reasoning trace left
in context is both expensive (the problem this paper attacks) and potentially *misleading* on the next turn; a
summarized trace is cheap and safer but risks "summary drift" and the loss of a detail a later turn needed.
Neither the faithfulness literature nor the KV-compression literature currently tells you, for a given trace,
which regime you are in.
#### The open gap
Conspicuously absent from both literatures: **faithfulness-aware** compression. Today's reasoning-model KV
methods (R-KV, ThinKV, and by extension a TriAttention-style scorer) decide what to keep using *attention or
distance* signals — proxies for "what this answer used," not "what was causally load-bearing" or "what a
future turn will need." A scorer that *knew* a span was performative could drop it with confidence; one that
knew a span was load-bearing could protect it across turns. Whether the probe-based performativity signals
from the faithfulness work can be turned into a compression or memory-eviction signal is, as far as we can
tell, unexplored — and is the most interesting thing this paper's machinery gestures at without addressing.
#### Discussion prompts
- If half a reasoning trace is performative, is a distance-preference scorer (this paper) accidentally a
*performativity* detector — or is it keeping the wrong tokens, the ones that produced *this* answer rather
than the ones a *future* turn needs?
- Should a system summarize a valid reasoning trace into memory rather than cache it verbatim — and what would
a "faithfulness-aware" eviction signal even look like?
- Does keeping a possibly-unfaithful reasoning trace in context help or *hurt* the next turn, given the
multi-turn "assumption pollution" finding?
- **Is *token-level* attention even the right granularity for a reasoning *conclusion*?** A conclusion a
later step needs to reuse is rarely carried by a single token — and certainly not by tokens spaced at
RoPE's fixed geometric offsets; it is a property of a *span* ("having established X, …"). A
distance-preference score over individual keys may therefore be structurally mismatched to what reuse
needs: arguably the model wants to **summarize a conclusion and then refer to the summary** — a key
that points at a *region* of context rather than at one token. That is essentially the mechanism
behind **landmark tokens** (one retrievable token per block; Mohtashami & Jaggi,
[arxiv.org/abs/2305.16300](https://arxiv.org/abs/2305.16300)), **gist tokens** (Mu et al.,
[arxiv.org/abs/2304.08467](https://arxiv.org/abs/2304.08467)), and **AutoCompressor summary vectors**
(Chevalier et al., [arxiv.org/abs/2305.14788](https://arxiv.org/abs/2305.14788)). Open question:
should reasoning-trace compression operate on *spans/summaries* — keep or evict a whole reasoned
conclusion via a single learned anchor — rather than scoring individual cached keys? And would a
center/distance signal like TriAttention's even apply at that granularity, or does it presuppose the
token-level view it inherits from RoPE?
#### Further reading
- **Faithfulness / "reasoning theater":** Reasoning Theater, [2603.05488](https://arxiv.org/abs/2603.05488);
Turpin et al., [2305.04388](https://arxiv.org/abs/2305.04388); Lanham et al. (measuring CoT faithfulness),
[2307.13702](https://arxiv.org/abs/2307.13702); "Reasoning models don't always say what they think,"
[2505.05410](https://arxiv.org/abs/2505.05410); Palod et al. (CoT length vs. difficulty),
[2509.07339](https://arxiv.org/abs/2509.07339); Kambhampati et al. (stop anthropomorphizing intermediate
tokens), [2504.09762](https://arxiv.org/abs/2504.09762); reasoning-theater bias / THEATER,
[2507.13758](https://arxiv.org/abs/2507.13758); mechanisms survey,
[2601.14270](https://arxiv.org/abs/2601.14270).
- **Reasoning-trace KV compression:** R-KV, [2505.24133](https://arxiv.org/abs/2505.24133); Think Clearly,
[2507.08806](https://arxiv.org/abs/2507.08806); ThinKV, [2510.01290](https://arxiv.org/abs/2510.01290).
- **Multi-turn / memory:** LLMs Get Lost in Multi-Turn, [2505.06120](https://arxiv.org/abs/2505.06120).
- **Summary / span-level keys (the granularity question):** Landmark Attention,
[2305.16300](https://arxiv.org/abs/2305.16300); Gist Tokens, [2304.08467](https://arxiv.org/abs/2304.08467);
AutoCompressors, [2305.14788](https://arxiv.org/abs/2305.14788).