Visual paper explainer · training-free sparse attention

Sol-Attn

Sparsifying online attention accelerates the attention operation inside a video or image diffusion transformer: it evaluates important token blocks exactly, represents the rest cheaply, and combines both paths in one approximately global softmax.

Paper Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification Authors Haopeng Li et al. Date July 2026 technical report Source arXiv:2607.24027

Start with one long video

TokenA compact vector representing a small region of a video latent. A long video can produce tens of thousands of tokens.
Block / tileA hardware-sized group of tokens processed together. The paper writes its token count as B.
Proxy scoreA cheap estimate of how strongly one query block relates to one key block, computed from their pooled representatives.
Online softmaxA numerically stable way to merge attention blocks while retaining only a running maximum, denominator and value accumulator.
Task

Generate an image or video

A diffusion model repeatedly denoises a large set of visual tokens. Dense attention lets every token exchange information with every other token.

Input → output

75K noisy tokens → cleaner tokens

At each denoising step, attention transforms the current visual-token sequence into contextualized features used for the next prediction.

Bottleneck

Quadratic pair count

Doubling sequence length roughly quadruples dense query–key comparisons, and makes attention dominate long-video inference.

Running example. One query block faces 16 key/value blocks. Instead of computing all 16 exactly, SOL-Attn will dynamically choose a few exact blocks and approximate every other block with one pooled key and one summed value.
All query blocksfull-resolution Q
×
All key blocksfull-resolution K
→
Every QK and PV tileaccurate, but expensive

Source: paper Introduction and Section 2. The 75K-token example is illustrative of the long-video regime, not a benchmark input fixed by the method.

The complete idea in one pass

Scroll horizontally to inspect the complete pipeline.

Pool Q, K and V into blockscheap representatives
→
Stream proxy scorescompare each score with a query-specific threshold
→
Above thresholdload original K,V and compute exact token attention
Below thresholdreuse pooled K and summed V as an approximate contribution
→
One online-softmax stateexact and approximate terms share the running maximum, denominator and numerator

Formalized from Sections 3.1–3.2, Equations (4), (9), (10), and (11), and Algorithm 1.

Not ordinary pruning

Unselected blocks still matter

SOL-Attn does not simply erase low-scoring blocks. It retains their aggregate probability mass and value contribution at block resolution.

Not VSA

No learned coarse/fine gate

VSA trains separate coarse and fine outputs and combines them with learned gates. SOL-Attn is training-free and merges both paths into one online-softmax calculation.

Dynamic threshold: one knob, query-specific budgets

Equation (4) · selection rule

Standardize before deciding

τi=μi+βσi,Si={j:s^ij>τi}.\tau_{i}=\mu_{i}+\beta\sigma_{i},\qquad\mathcal{S}_{i}=\{j:\widehat{s}_{ij}>\tau_{i}\}.
Read it as: query block i keeps key block j for exact attention when its proxy score is more than beta standard deviations above that query’s mean.
τi\tau_i
raw threshold for query block i
μi\mu_i
mean proxy score for query i
β\beta
shared sparsity control
σi\sigma_i
proxy-score standard deviation
Si\mathcal S_i
selected exact blocks
j,ij, i
key-block and query-block indices
s^ij\widehat s_{ij}
pooled block proxy score
>,=,+,:,{}, comma and period\gt,=,+,:,\{\},\text{ comma and period}
comparison, definition, addition, set condition and punctuation

Running example: with beta 1.04, a roughly Gaussian score row retains about its upper 15%—but this query may keep more or fewer than the average.

Visual mapping: the slider below controls beta; green blocks satisfy the set condition and violet blocks take the approximate path.

Source: Section 3.1, Equation (4).

Equation (5) · efficient moments

No complete proxy map is needed

μi=Qˉi(1N∑j=1NKˉj)⊤,σi2=Qˉi(1N∑j=1NKˉj⊤Kˉj)Qˉi⊤−μi2.\mu_{i}=\bar{\boldsymbol{Q}}_{i}\left(\frac{1}{N}\sum\nolimits_{j=1}^{N}\bar{\boldsymbol{K}}_{j}\right)^{\top},\qquad \sigma_{i}^{2}=\bar{\boldsymbol{Q}}_{i}\left(\frac{1}{N}\sum\nolimits_{j=1}^{N}\bar{\boldsymbol{K}}_{j}^{\top}\bar{\boldsymbol{K}}_{j}\right)\bar{\boldsymbol{Q}}_{i}^{\top}-\mu_{i}^{2}.
Read it as: summarize pooled keys once with first and second moments, then project those summaries with each pooled query to obtain its threshold statistics.
Qˉi\bar{\boldsymbol Q}_i
mean query vector in block i
Kˉj\bar{\boldsymbol K}_j
mean key vector in block j
NN
number of key blocks
⊤\top
transpose
μi,σi2\mu_i,\sigma_i^2
mean and variance
∑j=1N\sum_{j=1}^{N}
sum over pooled key blocks
i,j,1,2i,j,1,2
indices and powers
=,−,1N,( ), comma and period=,-,\frac{1}{N},(\,),\text{ comma and period}
operators, grouping and punctuation

Running example: the 16 pooled keys provide one mean and one second-moment summary; every query block gets its own mu and sigma from those summaries.

Visual mapping: this is the threshold-computation stage before the proxy-score stream in the overview.

Source: Section 3.1, Equation (5). The paper also gives a cheaper diagonal estimator in Equation (15).

Try the beta threshold on 16 illustrative proxy scores

This teaching control applies Equation (4) to fixed standardized scores. It is not a reproduced benchmark.

1.04 σ
Gaussian target15.0%expected exact density
This score row—selected exact blocks
Raw threshold—mu + beta sigma

Green = exact path; violet = approximate correction. The fixed scores are an illustrative running example; the selection rule is Equation (4).

Approximate correction: why the value is summed

Equation (9) · denominator

Every approximated block still represents B tokens

Di:=∑j∈UiB exp⁡(QiKˉj⊤)⏟Approx.+∑j∈SiRowSum⁡(exp⁡(QiKj⊤))⏟Exact∈RB×1.\boldsymbol{D}_{i}:=\underbrace{\sum_{j\in\mathcal U_i}B\,\exp(\boldsymbol Q_i\bar{\boldsymbol K}_j^{\top})}_{\text{Approx.}}+\underbrace{\sum_{j\in\mathcal S_i}\operatorname{RowSum}(\exp(\boldsymbol Q_i\boldsymbol K_j^{\top}))}_{\text{Exact}}\in\mathbb R^{B\times1}.
Read it as: an unselected block contributes one pooled logit multiplied by B, while a selected block contributes the exact sum over its token logits.
Di\boldsymbol D_i
softmax denominator for query block i
Ui,Si\mathcal U_i,\mathcal S_i
unselected and selected block sets
BB
tokens per block
Qi\boldsymbol Q_i
query-token matrix
Kˉj,Kj\bar{\boldsymbol K}_j,\boldsymbol K_j
pooled and original keys
RowSum⁡,exp⁡\operatorname{RowSum},\exp
row reduction and exponential
RB×1\mathbb R^{B\times1}
output domain and shape
j,i,⊤,:=,+,∈,( ), comma and periodj,i,\top,:=,+,\in,(\,),\text{ comma and period}
indices, operators, grouping and punctuation

Running example: if each block contains 64 tokens, one approximate proxy logit represents 64 denominator terms—not one.

Visual mapping: violet blocks contribute the first term; green blocks contribute the second.

Source: Section 3.2, Equation (9). Source-only color-box commands are omitted visually; all mathematical terms are preserved.

Equation (10) · numerator

The summed value preserves the block’s aggregate contribution

Ni:=∑j∈Uiexp⁡(QiKˉj⊤)V^j⏟Approx.+∑j∈Siexp⁡(QiKj⊤)Vj⏟Exact∈RB×d.\boldsymbol{N}_{i}:=\underbrace{\sum_{j\in\mathcal U_i}\exp(\boldsymbol Q_i\bar{\boldsymbol K}_j^{\top})\widehat{\boldsymbol V}_j}_{\text{Approx.}}+\underbrace{\sum_{j\in\mathcal S_i}\exp(\boldsymbol Q_i\boldsymbol K_j^{\top})\boldsymbol V_j}_{\text{Exact}}\in\mathbb R^{B\times d}.
Read it as: after approximating all keys in an unselected block by one pooled key, their common exponential weight factors out, leaving the sum of their values.
Ni\boldsymbol N_i
unnormalized value numerator
Ui,Si\mathcal U_i,\mathcal S_i
unselected and selected block sets
Qi\boldsymbol Q_i
query-token matrix
Kˉj,Kj\bar{\boldsymbol K}_j,\boldsymbol K_j
pooled and exact keys
V^j,Vj\widehat{\boldsymbol V}_j,\boldsymbol V_j
summed and original values
exp⁡\exp
unnormalized attention weight
RB×d\mathbb R^{B\times d}
output domain and shape
j,i,⊤,:=,+,∈,( ), comma and periodj,i,\top,:=,+,\in,(\,),\text{ comma and period}
indices, operators, grouping and punctuation

Running example: each violet block supplies one pooled-key score and its sum of 64 value vectors; each green block supplies its original 64 keys and values.

Visual mapping: both colored paths terminate at the same dark online-softmax box in the overview.

Source: Section 3.2, Equation (10). Source-only color-box commands are omitted visually; all mathematical terms are preserved.

Key consequence. If only one approximate block existed, its numerator divided by its denominator would be the block’s mean value. When many blocks compete, the factor B correctly preserves how much softmax mass that block represents.

How the fused kernel executes it

Equation (11) · proxy reuse

The approximation tile also performs routing

S~i(t):=Qi(Kˉ(t))⊤,Mean⁡(S~i(t))=Qˉi(Kˉ(t))⊤=s^i(t).\widetilde{\boldsymbol S}_i^{(t)}:=\boldsymbol Q_i(\bar{\boldsymbol K}^{(t)})^{\top},\qquad\operatorname{Mean}(\widetilde{\boldsymbol S}_i^{(t)})=\bar{\boldsymbol Q}_i(\bar{\boldsymbol K}^{(t)})^{\top}=\widehat{\boldsymbol s}_i^{(t)}.
Read it as: compute token-to-pooled-block scores once; averaging each column gives exactly the proxy score needed for routing, so the same tile drives both decisions and approximate correction.
S~i(t)\widetilde{\boldsymbol S}_i^{(t)}
token-to-block score tile
Qi,Qˉi\boldsymbol Q_i,\bar{\boldsymbol Q}_i
query tokens and pooled query
Kˉ(t)\bar{\boldsymbol K}^{(t)}
chunk t of pooled keys
s^i(t)\widehat{\boldsymbol s}_i^{(t)}
chunk of proxy scores
Mean⁡\operatorname{Mean}
column mean over query tokens
i,ti,t
query-block and chunk indices
⊤\top
transpose
:=,=,( ), comma and period:=,=,(\,),\text{ comma and period}
definition, equality, grouping and punctuation

Running example: process the 16 pooled keys in chunks; as each proxy-score chunk appears, route its green blocks immediately and never store a complete 16-entry map.

Visual mapping: Step 2 below uses one score tile for both the threshold decision and the approximate path.

Source: Section 3.2, Equation (11), Figure 4, and Algorithm 1.

Source: Algorithm 1 and Section 3.2 hardware-aligned implementation.

Place it among nearby methods

Ordinary block sparse

Exact or dropped

A router selects blocks for exact token attention. Every unselected block contributes zero, so error grows quickly at aggressive sparsity.

VSA

Learned coarse + fine

Pooled cube attention routes fixed top-K cubes and supplies a coarse global output; a learned gate combines it with fine sparse attention. Training or adaptation is required.

SOL-Attn

Exact or approximate

A standardized threshold gives a dynamic block count. Both routes update one online-softmax state, with no learned router or gate.

What the paper measured

Wan 2.1 T2VText-to-video generation at 720p and 81 frames.
HunyuanVideo T2VText-to-video generation at 720p and 129 frames.
End-to-end speedup ↑Dense FA3 wall time divided by method wall time; higher is faster.
VBench AVG ↑Paper-reported aggregate video-generation quality; it does not measure dense-output equivalence by itself.
Wan 2.1 · Table 12.02×end-to-end speedup at 85.12% sparsity
Wan 2.1 · Table 176.13VBench AVG; dense FA3 reports 75.90
HunyuanVideo · Table 12.12×end-to-end speedup at 85.80% sparsity
Kernel · Figure 5a5.41×versus FA3 at 128K tokens and 90% sparsity
Interpret carefully. The method is approximate. Similar quality scores do not mean token-for-token equivalence to dense attention, and the speedup grows with sequence length and sparsity because routing and fixed overhead must be amortized.

Limits and open questions

Approximation error remains

Replacing every key in an unselected block by its mean is accurate only when within-block key variation has a limited effect on attention.

Beta is still a deployment knob

No calibration dataset is mandatory, but the Gaussian density mapping is approximate. Quality and actual density should be validated on the target workload.

Long sequences benefit most

At moderate lengths, proxy preparation, thresholding and irregular exact dispatch can consume the saved time.