A model is a learned program that transforms tokens, small pieces of text or image input, through successive layers. Its learned numbers are parameters. During generation, a KV cache keeps reusable representations of earlier tokens so the model does not rebuild all prior context state for every new token.
An agent is software that repeatedly asks a model to inspect information and choose the next step. Its existing input is the prompt. Prefill processes that prompt; decode produces the answer one token at a time. Producing each next token from earlier tokens is autoregressive generation.
Teaching example: a sampled 100,000-token coding-agent session. It is illustrative, not a workload measurement from the report.
A Transformer is a model built from repeated blocks. Its depth is the number of layers traversed. Causal means a position can use only earlier or current positions. A hidden state is its internal vector for a token. To encode is to turn input into those internal representations.
Attention compares a current query with stored keys, then mixes their associated values. A useful key receives more influence on the output.
Global attention can reach the whole earlier context. Sliding-window attention (SWA) reaches only a recent fixed window. Sparse attention reads a selected subset instead of every old position.
Causal Encoder-Decoder (CED) keeps most prompt work in the lower half. Compressed Sparse Attention 2 (CSA2) compresses and shares the far-reaching memory and can reuse its selection decisions.
Sources: report Abstract, Introduction, and Sections 2.1-2.3.
In the qualitative baseline, each layer derives its own global key-value entries from its own hidden states. A longer prompt therefore multiplies both prompt computation and cache storage across layers.
Each bar is one layer; height variation is only visual grouping.
Interpretive baseline based on report Sections 2.2-2.3. This does not assign a generic Transformer a made-up byte count.
Every prompt token runs through every layer even when most of the context is old tool history.
Independent global caches repeat long-context state and must be stored, moved, and sometimes persisted for later reuse.
The 40-layer language backbone is split into a 20-layer causal encoder and a 20-layer decoder. The key is phase-dependent: most prompt tokens stop after the encoder, but every newly generated token still traverses both halves.
Prefill: all prompt tokens build final encoder states. The first decoder layer, paper layer 21 and zero-indexed config layer 20, projects the shared global cache from those states; only the recent window is replayed through decoder layers for their local SWA state.
Final causal-encoder hidden states for the prompt.
Layer-dependent maps in the report's general CED rule.
Global KV entries and compression weights for l > L/2.
Source: report Sections 2.2 and 4.2.1, Equation (1), plus official `kv_source_layer_ids`, `compress_ratios`, and reference `Compressor`. The equation's general Z branch is not materialized by the released ratio-1 decoder source.
The equation extends horizontally.
Coding-agent example: the 100K-token prompt reaches H20. Equation (1) gives the general Cl/Zl rule; the released ratio-1 decoder source uses only the unpooled main-KV projection because no multi-token compression gate is needed.
Visual mapping: the three generic boxes above map HL/2 through WlKV and WlZ into Cl and Zl; the green callout then narrows that rule to the released implementation.
Source: report Section 2.2, Equation (1).
A compression ratio says how many source positions become one stored entry. That compact internal vector is a latent. An indexer is a lightweight scorer that chooses the Top-K most relevant positions. Quantization stores numbers at lower precision: FP4 uses roughly four bits per value, while FP8 uses roughly eight.
Encoder ratio 2 turns each non-overlapping token pair into one main-KV entry. Decoder ratio 1 keeps one entry per position.
Many layers borrow main KV and indexer K; some also reuse the latest Top-K indices.
This is complementary to CSA2. Sliding-window KV stays FP8 because the authors found it more sensitive.
Source: report Sections 2.3 and 4.2.1. Unlike the earlier CSA, CSA2 removes overlap between neighboring compression groups, removes absolute positional embeddings from compression, and projects indexer K from main KV.
A candidate pool is a bounded shortlist of positions that later indexers may rescore. Every CSA2 mode still computes a current-layer main query and local SWA KV.
Full mode publishes main KV and indexer K, computes an indexer query, and selects fresh Top-K positions.
Source: report Section 2.3.1 and Figure 4. Green means current computation, amber means shared cache, and coral means reused indices; the diagram is redrawn rather than copied.
Cross-layer index reuse removes many indexer calls, but the remaining Reindex layers would still score the whole context. The decoder's first Full layer therefore creates one shared candidate pool for deeper indexers.
For each decoder query separately, zero-indexed config layer 20 scores the full causally visible context. Its score output then splits into two paths.
The sampled context strip extends horizontally.
Expose every position in the winning blocks: at most 16,384 candidates.
Source: report Section 2.3.2 and Figure 5. Candidate restriction is applied per query, was introduced during post-training, and is used identically in training and inference. The 48 marks are a sampled visual, not one mark per context position.
The layer schedule extends horizontally.
Layer strip continues to the right.
Layer strip continues to the right.
Source: report Section 4.2.1 and official `config.json`. All tile numbers are zero-based config indices: KV sources 2, 8, 14, 20; index sources 2, 8, 14, 20, 24, 28, 32, 36; Top-K 512; SWA window 128.
A persistent cache stores reusable prefix state beyond one active request, typically in host memory or on solid-state drives. The report separates long-tail global reuse from minutes-scale SWA reuse.
Approximate: exact recovery would replay the most recent L x 128 prompt tokens. Suffix state can therefore depend on the cache-hit position.
Approximate: it is not mathematically identical to a full decoder pass over the prompt.
Source: report Sections 3.2.1-3.2.2. The quality impact is author-reported as negligible; Section 6 still names replay reconstruction as an incompletely characterized robustness boundary.
A mixture of experts (MoE) stores many feed-forward subnetworks but routes each token through only a few; the weights used for that token are its active parameters. Engram is hashed conditional memory. DSpark is a speculative draft-and-verify module. mHC, or multi-head hyper-connections, mixes several residual streams between blocks.
A 32-layer vision encoder uses 2D rotary positions. A 3 x 3 pixel-unshuffle reduces visual-token count ninefold before a two-layer projector maps features into the language width.
Every Transformer block has one shared expert and 384 routed experts; six routed experts are selected per token.
Two modules at zero-indexed layers 1 and 14 retrieve hashed 2-, 3-, and 4-token patterns using eight hash heads.
Main KV uses one E4M3 scale per 16 FP4 channels after rotary position encoding. Local SWA KV remains FP8.
Three small Transformer blocks produce five base draft positions in parallel; a Markov head models draft dependencies and a confidence head helps schedule verification length.
Each block consumes the preceding block's input-mixing coefficients, letting deployment fuse residual update, input mixing, coefficient prediction, normalization, and conversion.
Sources: report Sections 2.1.1 and 2.4. Components are shown separately because none is another name for CED or CSA2.
The input-mixing pass must wait until Al has been predicted from all of Xl.
Input mixing can proceed while the same residual traversal accumulates the coefficients for the next block.
The equation extends horizontally.
Agent example: at one token position, the current four residual streams must be fully reduced before Al is ready to mix the block input.
Visual mapping: the left comparison branch places coefficient prediction before current input mixing.
Source: report Section 2.4.1, Equation (2).
The equation extends horizontally.
Agent example: the current token's residual tile can immediately use Al-1 while the kernel predicts Al for the following block.
Visual mapping: the right comparison branch shows ready Al-1 entering the same pass that predicts the next coefficients.
Source: report Section 2.4.1, Equation (6).
The report's weighted single-token decode FLOPs count one BF16 operation as 1, one FP8 operation as 0.5, and one FP4 operation as 0.25. This is an author-defined compute proxy, not measured wall-clock latency.
Parameters active per token during prefill / decode. Stored backbone size is 552B, plus 196B Engram parameters.
Architecture setupAbout one quarter of V4-Flash at equal sequence length, attributed jointly to cross-layer reuse and FP4 caching.
Author-reportedV4-Flash's footprint under identical workloads after removing SWA KV from persistent storage and compressing global KV.
Author-reportedFrom 4K to 1M context, Figure 2 reports only about one-quarter more weighted single-token decode FLOPs.
Author-reported proxyInference kernels per layer during prefill / decode after the authors' kernel fusion.
Implementation claimMultimodal pretraining from 64K sparse context, extended to 1M at 34T tokens. Text-only to multimodal token ratio is 7:1.
Training setupSources: report Abstract, Figures 1-2, Sections 3.2 and 4.2. As of September 10, 2026, these are first-party release results rather than independent replication.
Visuals on this page are original explanatory redrawings. No report figure is embedded or copied.