Mamba-2 to GDN2 and ReplaySSM: a visual guide to memory and decoding

01 / Start with the job

A language-model layer must mix now with before

A model reads a sequence one token at a time. At each position, one internal layer must combine the current token with useful information from earlier positions before the model predicts what comes next.

Minimal language-model vocabulary.

A transformer is a neural-network design built from repeated processing blocks; each block contains layers such as attention and feed-forward computation. Learned parameters are the numbers adjusted during training. A latent representation is an internal numeric description rather than readable text. To decode means to turn an internal representation into an output such as the next-token probabilities.

The job

Choose useful history

Attention is a sequence-mixing operation: it scores which earlier pieces are relevant to the current piece and combines information from them.

The alternative

Carry one summary forward

A recurrence updates one summary at every token. A model built around such repeated state updates is a recurrent neural network (RNN).

The pressure

Long histories are expensive

Keeping every earlier item preserves detail, but storage and comparison work grow as the sequence grows. Compressing history controls cost, but makes editing and retrieval harder.

Three names that otherwise make the papers look harder than they are.

A recurrent state is the carried summary. A state-space model (SSM) updates that hidden state from an input and reads an output from it. Structured state space duality (SSD) is Mamba-2's bridge showing that a restricted SSM can also be viewed as a structured form of masked attention and computed efficiently in chunks.

Running example

One fact changes while the stream continues

Ada -> room 31 weather: rain Bo -> room 12 Ada -> room 47 Ada's room?

The desired answer is the newest association, room 47. A useful memory update should revise Ada without needlessly damaging Bo or other stored information.

Interpretive teaching example. The papers evaluate related associative-recall and retrieval behavior, but do not use this exact token stream.

Query, key, value: A query describes what the current token wants; a key describes what a stored item matches; a value is the information returned.
Head and channel: A model uses several parallel memory units called heads. A channel is one numeric coordinate inside a key or value vector.
Gate and decay: A gate is a learned number, usually between 0 and 1, controlling how much information passes. Decay multiplies old memory by such numbers, making selected content fade.
Outer product: An outer product multiplies every key coordinate by every value coordinate, producing a matrix that stores their association.
Delta rule: The delta rule writes an error, or delta: target value minus the value currently retrieved. It corrects an association instead of only adding another copy.
KV cache: A key-value cache (KV cache) stores past keys and values so autoregressive generation does not recompute them.
Linear attention: Linear attention reorganizes attention-like computation around a fixed state so work grows linearly, rather than quadratically, with sequence length. The rest of the neural network is still nonlinear.
Hybrid model: A hybrid model mixes recurrent fixed-state layers with some local or global attention layers, combining bounded memory with more direct access to selected tokens.

02 / Two storage strategies

Keep every note, or keep one evolving summary

The architectural choice is easiest to see as a storage decision. Full causal attention retains past key-value pairs. Recurrent linear attention folds each new pair into a bounded matrix.

Full causal attention

Explicit history

Each earlier token keeps a key and value. The current query scores all available keys, then forms a weighted mixture of their values.

t1key 1value 1

t2key 2value 2

t3key 3value 3

t4key 4value 4

Training score workgrows as T^2

Generation cachegrows as T

Recurrent linear attention

Compressed history

Each pair updates one state matrix. A query reads the matrix directly; earlier keys and values are no longer individually available.

S
one matrix per head
size does not grow with T

Sequence workgrows as T

Generation stateconstant in T

Constant size is not free memory.

Many associations must share the same finite matrix. Similar keys can interfere, and obsolete associations can remain unless the update rule knows what to remove.

Sources: Mamba-2 Sections 2.2 and 4.2; GDN2 Introduction and Section 2. The complexity statements describe scaling in sequence length T.

03 / The bridge

From a state-space recurrence to writable memory

Mamba-2 begins from the ordinary selective state-space view: update a hidden state from the current input, then read an output. SSD restricts the state transition enough to reveal an equivalent structured-attention view and a hardware-friendly chunk algorithm.

Mamba-2 paper, Equations 2a-2b

The selective SSM recurrence

Equation continues horizontally.

\begin{aligned}h_t &= A_t h_{t-1}+B_t x_t\\y_t &= C_t^\top h_t\end{aligned}

Read it as: transform the previous hidden state, inject the current input, then read the new state into an output.

h_t-1, h_t
Hidden state before and after token t

x_t, y_t
Input and output at token t

A_t
Input-dependent state transition

B_t
Input-to-state map

C_t^T
Transposed state-to-output map

=, +, subscripts
Equality, addition, and time indices; neither source line ends in punctuation

Ada example: the input containing the new room fact changes the carried hidden state; a later query token reads an answer from it.

Visual mapping: the previous-state, current-input, and output roles become the matrix, write pair, and read vector in the next card.

Source: Mamba-2 Section 2.1, Equations (2a) and (2b). Original Mamba-2 notation.

Mamba-2 paper, Equation 14

The linear-attention reassociation

Equation continues horizontally.

Y=Q\cdot\mathsf{cumsum}(K^{\top}V)

Read it as: accumulate key-value outer products first, then let each query read that running matrix. Reordering the multiplications avoids all token-pair comparisons.

Q, K, V
All query, key, and value vectors stacked by time

K^TV
Key-value outer products after transposing K

$cumsum$
Causal cumulative sum over time

Y
All output vectors

dot
Matrix multiplication shown by the centered dot

=, ( )
Equality and grouping; the source equation has no terminal punctuation

Ada example: the fact's key-value outer product enters the running sum; the later Ada query reads from the sum.

Visual mapping: the center matrix below is the running `cumsum(K^T V)` state, and Q is the blue read vector.

Source: Mamba-2 Section 4.2, Equation (14). Original Mamba-2 notation.

Fixed-size associative memory

Write with an outer product, read with a query

More of the memory diagram is to the right.

Key 𝒌_t
[name-like features]

Value 𝒗_t
[room-like features]

State matrix 𝐒_t

key1/value1

key1/value2

key2/value1

key2/value2

𝐒_t = 𝐒_t-1 + 𝒌_t𝒗_t^T

Query 𝒒_t
[who is Ada?]

Output 𝒐_t
𝐒_t^T𝒒_t

Visual legend: blue marks key/query directions, green marks persistent state, and amber marks values and outputs. The write uses both 𝒌_t and 𝒗_t; the read uses both 𝐒_t and 𝒒_t.

Source: GDN2 Section 2.1, Equation (1), using its key-by-value state orientation.

GDN2 paper, Equation 1

The additive memory baseline

Equation continues horizontally.

{\mathbf{S}}_{t} ={\mathbf{S}}_{t-1}+{\bm{k}}_{t}{\bm{v}}_{t}^{\top},\qquad {\bm{o}}_{t} ={\mathbf{S}}_{t}^{\top}{\bm{q}}_{t}.

Read it as: add a key-value association to memory, then retrieve by multiplying the memory's transpose by a query.

𝐒_t-1, 𝐒_t
Previous and updated state matrices

𝒌_t
Current key vector

𝒗_t
Current value vector

𝒒_t, 𝒐_t
Current query and retrieved output

T
Transpose; turns a column vector or matrix around

=, +, comma, period
Equality, additive write, and separator; Equation 1 ends with a period

Ada example: the new Ada-room pair is added to the same matrix that already contains the old pair, which can leave stale content.

Visual mapping: 𝒌 and 𝒗 enter the green matrix together; 𝒒 enters through the separate blue read path and produces 𝒐.

Source: GDN2 Section 2.1, Equation (1). This paper's shared comparison orientation is used from here onward.

Notation checkpoint.

This page uses the GDN2 paper's orientation: 𝐒 is key-channels by value-channels, a write is 𝒌𝒗^T, and a read is 𝐒^T𝒒. The original GDN paper stores the transpose, so it writes 𝒗𝒌^T and reads 𝐒𝒒. Mamba-2's earlier symbols play corresponding roles: hidden state h becomes matrix 𝐒, B is key-like, input x is value-like, and C is query-like.

04 / The edit problem

Adding is easy; replacing is the hard part

With additive memory, a repeated key adds another value instead of explicitly replacing the old association. The delta rule asks the memory what it currently predicts for the key, subtracts that prediction from the target, and writes only the remaining error.

GDN2 paper, Equation 5

The delta rule writes the residual

Equation continues horizontally.

{\mathbf{S}}_{t} ={\mathbf{S}}_{t-1}+\beta_{t}{\bm{k}}_{t}({\bm{v}}_{t}-{\mathbf{S}}_{t-1}^{\top}{\bm{k}}_{t})^{\top}=({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{\top}){\mathbf{S}}_{t-1}+\beta_{t}{\bm{k}}_{t}{\bm{v}}_{t}^{\top}.

Read it as: current prediction is 𝐒_t-1^T𝒌_t; target minus prediction is the error; beta controls how far the update moves toward the target.

𝐒_t-1, 𝐒_t
Previous and corrected state matrices

𝒌_t, 𝒗_t
Current key and target value vectors

β_t
Scalar update strength

𝐒_t-1^T𝒌_t
Value currently retrieved by the key

𝐈
Identity matrix; leaves a vector unchanged

=, +, -, ( ), T, period
Equivalent forms, addition, subtraction, grouping, and transpose; Equation 5 ends with a period

Ada example: if the state already predicts room 31 for Ada, the update writes the difference between room 47 and that old prediction instead of blindly adding 47.

Visual mapping: the GDN and KDA ladder branches show beta controlling both the red erase and amber write portions of this residual update.

Source: GDN2 Section 2.2, Equation (5), restating the DeltaNet update in the page's shared orientation.

05 / Four memory rules

Each step adds one finer control

The models differ less in what they store than in how precisely they can modify the same bounded state. The selected branch changes the update, gate granularity, and remaining limitation together.

Memory-rule comparison

From one forget knob to separate erase and write vectors

Mamba-2 / GDN2 Eq. 4

𝐒_t = α_t𝐒_t-1 + 𝒌_t𝒗_t^T

New control: one learned scalar α_t decides how much of the entire state survives before an additive write.

Decay

Erase

Write

Remaining issue: forgetting is global within a head, and the write does not subtract what the key already retrieves.

Mamba-2 selected: scalar decay and additive write.

Sources: GDN2 Section 2, Equations (4), (6), and (7), plus Section 3.1, Equation (10). KDA's branch is equivalent to the original Kimi Linear Equation (1), transcribed into GDN2's shared comparison orientation.

GDN2 paper, Equation 4

Mamba-2 as decayed memory

Equation continues horizontally.

{\mathbf{S}}_{t}=\alpha_{t}{\mathbf{S}}_{t-1}+{\bm{k}}_{t}{\bm{v}}_{t}^{\top},\qquad\alpha_{t}\in(0,1].

Read it as: multiply all previous state rows by one scalar decay, then add the new key-value association.

𝐒_t-1, 𝐒_t
Previous and new state matrices

α_t
Data-dependent scalar decay in the interval (0, 1]

𝒌_t𝒗_t^T
New key-value outer-product write

t, t-1
Current and previous token indices

=, +, comma
Equality, additive write, and qualifier separator

( ], period
Open/closed interval; Equation 4 ends with a period

Ada example: all earlier facts fade by the same alpha before room 47 is added.

Visual mapping: the Mamba-2 branch shows one amber scalar spanning all four decay channels.

Source: GDN2 Section 2.2, Equation (4), a restatement of Mamba-2 up to parameterization.

GDN2 paper, Equation 6

GDN: decay plus correction

Equation continues horizontally.

{\mathbf{S}}_{t}=\alpha_{t}({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{\top}){\mathbf{S}}_{t-1}+\beta_{t}{\bm{k}}_{t}{\bm{v}}_{t}^{\top}.

Read it as: scalar alpha forgets globally; scalar beta controls a key-targeted erase and the matching write.

𝐒_t-1, 𝐒_t
Previous and updated state

α_t
Scalar state-decay gate

β_t
Shared scalar erase/write strength

𝐈 - β_t𝒌_t𝒌_t^T
Key-targeted erase transform

𝒌_t𝒗_t^T
Key-value write

=, -, +, ( ), T, period
Equality, erase, write, grouping, and transpose; Equation 6 ends with a period

Ada example: GDN first fades the state, removes the value currently associated with Ada's key direction, and writes toward room 47.

Visual mapping: the GDN branch shows one decay scalar plus one beta scalar shared by red erase and amber write.

Source: GDN2 Section 2.2, Equation (6), transposed from the original GDN paper's Equation (10).

GDN2 paper, Equation 7

KDA: decay by key channel

Equation continues horizontally.

{\mathbf{S}}_{t}=({\mathbf{I}}-\beta_{t}{\bm{k}}_{t}{\bm{k}}_{t}^{\top}){\mathbf{D}}_{t}{\mathbf{S}}_{t-1}+\beta_{t}{\bm{k}}_{t}{\bm{v}}_{t}^{\top}.

Read it as: diagonal matrix 𝐃 lets each key channel decay differently before the same scalar-beta delta update.

𝐃_t
Diagonal matrix of channel-wise decay values α_t

𝐈
Identity matrix

β_t
Shared scalar active-edit strength

𝒌_t, 𝒗_t
Current key and value vectors

𝐒_t-1, 𝐒_t
Previous and new states

=, -, +, ( ), T, period
Equality, erase, write, grouping, and transpose; Equation 7 ends with a period

Ada example: some key features can retain old facts while others fade faster before Ada's association is corrected.

Visual mapping: the KDA branch shows four violet decay cells but one scalar beta spanning erase and write.

Source: GDN2 Section 2.2, Equation (7). Equivalent original formulation: Kimi Linear Section 3, Equation (1).

06 / GDN2 under a microscope

Decay, inspect, erase, and write are separate decisions

GDN2 keeps KDA's channel-wise decay but replaces the single beta with two learned vectors. The erase vector lives on key channels; the write vector lives on value channels.

Causal update flow

Every input enters the operation it actually controls

state key/query erase/read target/write

More of the update flow is to the right.

Formal symbol map: 𝐃 and 𝐒_t-1 jointly produce decayed state; 𝒃 and 𝒌 jointly produce 𝒆; 𝐰 and 𝒗 jointly produce 𝒛; decayed state and 𝒆 jointly produce 𝒓; 𝒛, 𝒓, and 𝒌 jointly update 𝐒; 𝐒 and 𝒒 jointly produce 𝒐. The blue curved key path supplies the write direction independently of the erase path.

Sources: GDN2 Section 2.1, Equation (1), and Section 3.1, Equations (8)-(9). Diagram topology follows the decomposed recurrence; spacing and colors are explanatory.

GDN2 paper, Equations 8-9

The update as five small operations

Equation continues horizontally.

\begin{aligned}{\bm{e}}_{t} &={\bm{b}}_{t}\odot{\bm{k}}_{t}, & {\bm{z}}_{t} &=\mathbf{w}_{t}\odot{\bm{v}}_{t},\\\bar{{\mathbf{S}}}_{t} &={\mathbf{D}}_{t}{\mathbf{S}}_{t-1}, & {\bm{r}}_{t} &=\bar{{\mathbf{S}}}_{t}^{\top}{\bm{e}}_{t}, & {\mathbf{S}}_{t} &=\bar{{\mathbf{S}}}_{t}+{\bm{k}}_{t}({\bm{z}}_{t}-{\bm{r}}_{t})^{\top}.\end{aligned}

Read it as: gate the key into an erase direction, gate the value into a target, decay old state, read selected old content, then write target minus read.

𝒆_t, 𝒃_t, 𝒌_t
Erase direction, erase gate, and key

𝒛_t, 𝐰_t, 𝒗_t
Gated target, write gate, and value

𝐒̄_t
Decayed state, written with a bar over bold 𝐒

𝐃_t, 𝐒_t-1
Diagonal decay matrix and previous state

𝒓_t
Old content read through erase direction

𝐒_t
Updated state

⊙
Element-wise multiplication

T
Transpose

=, +, -, comma, period
Definitions and residual write; Equation 8 ends with a comma and Equation 9 ends with a period

Ada example: the erase vector chooses which Ada-like key features inspect old state, while the write vector independently chooses which new value features enter.

Visual mapping: red nodes form 𝒆 and 𝒓; amber nodes form 𝒛; both converge on the green updated state.

Source: GDN2 Section 3.1, Equations (8) and (9).

GDN2 paper, Equation 10

The compact Gated Delta Rule-2

Equation continues horizontally.

\boxed{{\mathbf{S}}_{t}=\bigl({\mathbf{I}}-{\bm{k}}_{t}({\bm{b}}_{t}\odot{\bm{k}}_{t})^{\top}\bigr){\mathbf{D}}_{t}{\mathbf{S}}_{t-1}+{\bm{k}}_{t}(\mathbf{w}_{t}\odot{\bm{v}}_{t})^{\top}}

Read it as: decay old state, subtract selected old content along the key, and add a separately gated value along that same key.

𝐒_t-1, 𝐒_t
Previous and updated state matrices

𝐈
Identity matrix

𝐃_t
Channel-wise decay matrix

𝒌_t
Key and write direction

𝒃_t
Channel-wise key-side erase gate

𝐰_t
Channel-wise value-side write gate

𝒗_t
Candidate value

⊙, T
Element-wise product and transpose

=, -, +, ( )
Definition, erase, write, and grouping; source has no terminal punctuation

Ada example: GDN2 need not use one beta value for both removing the old association and committing the new value.

Visual mapping: the ladder's GDN2 branch shows independent violet cells on decay, erase, and write rows.

Source: GDN2 Section 3.1, Equation (10), named Gated Delta Rule-2.

GDN2 paper, Equations 11-12

The token generates its own controls

Equation continues horizontally.

\begin{aligned}{\bm{b}}_{t} &=\sigma(\mathbf{W}_{b}{\bm{x}}_{t}), & \mathbf{w}_{t} &=\sigma(\mathbf{W}_{w}{\bm{x}}_{t}).\\\bm{g}_{t}&=-\exp(\mathbf{a})\odot\operatorname{softplus}(\mathbf{W}_{f}{\bm{x}}_{t}+\bm{\delta}), & {\bm{\alpha}}_{t}&=\exp(\bm{g}_{t}).\end{aligned}

Read it as: learned projections of the current token produce erase, write, and decay controls. Users do not set these gates during generation.

𝒙_t
Current token representation

𝐖_b, 𝐖_w, 𝐖_f
Learned projection matrices

𝒃_t, 𝐰_t
Erase and write gate vectors

σ
Sigmoid, mapping gate values between 0 and 1

𝒈_t, 𝜶_t (bold α)
Log-decay vector and positive decay vector

𝐚, 𝜹 (bold δ)
Learned decay-rate and bias parameters

exp, $softplus$
Exponential and smooth positive functions

⊙
Element-wise product

=, -, +, ( ), comma, periods
Definitions and operators; Equation 11 ends with a period and Equation 12 ends with a period

Ada example: the representation of the correction token determines how strongly each feature erases, writes, and decays.

Visual mapping: the lab displays the resulting gate values, not the learned projection matrices that produced them.

Source: GDN2 Section 3.1, Equations (11) and (12).

07 / Hypothetical matrix lab

Watch the same three writes pass through each rule

This two-key-channel by two-value-channel example uses unit-length keys, invented teaching values, and invented gate values. It calculates the recurrences exactly, but it is not a trained model run or paper measurement.

State starts at zero

Selected update

State before this token

rows = key channels; columns = value channels

State after decay

after decay; before erase and write

Inputs and controls

key 𝒌[1.000, 0.000]

query 𝒒same vector as key

value 𝒗[0.200, 0.800]

decay0.920

erasenone

writeadditive

old readnot used

State after this token

query output: [0.200, 0.800]

Unrelated-key probe after Ada's revision: available at step 3

Equations implemented: GDN2 paper Eqs. (4), (6), (7), and (10). Inputs and gates are invented for explanation. Runtime values are checked against independently precomputed fixtures; a mismatch raises a browser error.

08 / Same rule, two execution views

Parallel chunks for training, one-token steps for decoding

The recurrence describes token-by-token behavior. The papers also derive algebraically equivalent chunkwise algorithms so training can use dense matrix multiplications instead of a slow serial loop. Standard autoregressive decoding still advances causally, one accepted token at a time, but the next section shows that the state need not be stored after every step.

More of the execution comparison is to the right.

Training / prefill

Work inside chunks in parallel

tokens
1-64tokens
65-128tokens
129-192

Dense operations handle tokens within each chunk. A compact state summarizes one chunk for the next.

Autoregressive decoding

Update once per new token

S0->S1->S2->S3

The fixed state is updated recurrently. No history-sized KV cache is required for these layers.

Conceptual summary from Mamba-2 Section 6, GDN Section 3.3, Kimi Linear Section 3.1, and GDN2 Section 3.3. WY/UT derivations and kernel details are intentionally excluded.

The recurrence is not the whole language model

Mamba-2

SSD token mixer

SSD sits inside a block with input projections, short convolution, normalization, an output gate, a residual path, and model heads.

Gated DeltaNet

Recurrent or hybrid

The paper evaluates pure recurrent stacks and hybrids that add sliding-window attention or Mamba-2 layers.

Kimi Linear

3 KDA : 1 global latent-attention layer

KDA is the recurrent mechanism. Kimi Linear is the larger hybrid architecture, alternating three KDA layers with one Multi-Head Latent Attention (MLA) layer.

Gated DeltaNet-2

Recurrent or sliding-window hybrid

The paper evaluates a recurrent-only stack and a hybrid that pairs GDN2 with sliding-window attention (SWA) and multilayer perceptrons.

Sources: Mamba-2 Section 7 and Figure 6; GDN Section 3.4; Kimi Linear Sections 1 and 5.2; GDN2 Sections 3.4-3.5.

09 / Same rule, a different storage schedule

ReplaySSM caches recent inputs instead of storing state every step

A recurrence specifies the answer a layer must compute. It does not require the implementation to materialize and save the entire new state matrix after every token. ReplaySSM exploits that freedom during inference.

The bottleneck

High-bandwidth memory traffic

High-bandwidth memory (HBM) is the GPU's large external memory. A kernel is memory-bound when moving numbers to and from HBM, rather than arithmetic, limits its speed. A recurrent layer normally loads and stores a large state matrix each token.

The replacement

A small buffer and occasional flush

A buffer temporarily holds recent state-building inputs. A flush folds those inputs into one new checkpoint state, writes that matrix once, and clears the buffer.

Why rollback matters

Draft, verify, keep or reject

Speculative decoding proposes several future tokens with a cheap draft path, then verifies them with the main model. A ring buffer stores entries in reusable circular slots, so rejected drafts can be discarded by moving a pointer instead of copying matrices.

Interactive Mamba-2 standard decoding

Eight tokens, one teaching buffer of four slots

Move the token step. This diagram counts full-state HBM operations; it does not estimate wall-clock speed.

Decoded token Token 1 of 8

Baseline recurrence

Eagerly store state

every token

Load the previous matrix, update it, read the output, then write the full matrix back.

Load S0 full state from HBM

Update, read, store write S1

state loads1

state writes1

current outputo1

ReplaySSM for Mamba-2

Keep a checkpoint and inputs

periodic store

For Mamba-2, load one checkpoint plus recent inputs and use the output-only route. Save a full state only when the standard-decoding buffer fills.

Load checkpoint S0 loaded, rarely written

Output-only route append token 1; no state store

Recent input buffer after this step

t1emptyemptyempty

checkpoint loads1

checkpoint writes0

small appends1

Model invariant: baseline output o1 = ReplaySSM output o1, apart from floating-point rounding.

Token 1 is retained as a small input record beside checkpoint S0. ReplaySSM computes o1 without storing S1.

Source-derived Mamba-2 teaching schedule from ReplaySSM Sections 3-5. GDN follows the same checkpoint cadence but reconstructs state rather than using this output-only route. The four-slot capacity is illustrative; the post reports best standard-decoding buffers of 8 for Nemotron-3 and 16 for Qwen3.5 in its tested settings. Speculative decoding uses a separate early-flush schedule explained below.

The algebraic opening: output does not require a materialized state.

For the post's Mamba-2 recurrence, transposed into this page's key-by-value state orientation, let a decay old state, Delta scale a new write, and c combine that write scale with all later decays.

State-and-output route

Build the matrix, then read it

a-bar_t = Π_r=1..t a_r, c_j,t = Δ_j Π_r=j+1..t a_r S_t = a-bar_t S₀ + Σ_j=1..t c_j,t k_j v_j^T o_t = S_t^T q_t

This route produces both the current state and output. ReplaySSM uses it when a flush must create the next checkpoint.

Output-only route

Associate first; skip the matrix

(k_j v_j^T)^T q_t = v_j (k_j^T q_t) o_t = a-bar_t (S₀^T q_t) + Σ_j=1..t c_j,t v_j (k_j^T q_t)

This route forms each key-query similarity first and mixes values directly. It returns the same output without constructing the full new matrix on ordinary Mamba-2 steps.

S0, St
The checkpoint state and the state after token t

k, v, q
Write key, written value, and current read query

a, Delta
Per-token decay and Mamba-2 write scale

abar
Product of decays since the checkpoint

c(j,t)
Write j's Delta times all decays after j through t

Pi, Sigma, T
Product, sum, and matrix transpose

Exact orientation used by the source post

The post stores values by keys, the transpose of this page's comparison convention:

S_t = a-bar_t S₀ + Σ_j=1..t s_j,t (v_j k_j^T) y_t = a-bar_t (S₀ q_t) + Σ_j=1..t s_j,t v_j (k_j^T q_t)

Here the post defines s to include Delta and subsequent decay. Transposing the state gives the two route cards above; the computation is unchanged.

Source: ReplaySSM Sections 3 and 5.1. The post's equations are unnumbered, so this section does not invent equation IDs or add them to the numbered paper ledger.

1 / Less state traffic

Remove most full-state stores

baseline

ReplaySSM

With 4-byte state entries, loading and storing a d by n state costs a dominant 8dn bytes per head. ReplaySSM still loads the checkpoint but avoids its store on most steps, leaving 4dn plus smaller input-buffer traffic.

2 / Cheap rollback

Reject entries, not state snapshots

d1 keepd2 keepd3 rejectd4 reject

The ring pointer advances past accepted drafts and ignores rejected entries. No full state snapshot is restored for each rejected token.

In the post's speculative schedule, a window of T drafts flushes early when h + 2T > L, preserving room for the next full window. This is not the four-token standard schedule above.

3 / Parallel verification

The route depends on the recurrence

Mamba-2: masked key-query products and value mixing become dense matrix multiplications across drafts.

GDN: each correction depends on earlier drafts, so one small triangular solve resolves those corrections before dense matrix multiplications produce outputs.

Both avoid a separate full-state write for every draft, but GDN does not use Mamba-2's simple output-only path.

What was actually implemented.

The authors demonstrate ReplaySSM for Mamba-2 and Gated DeltaNet in a public vLLM research fork. They list GDN2 support as future work; the post does not evaluate Kimi Delta Attention. ReplaySSM changes inference execution, not model weights, training, memory quality, or benchmark accuracy.

Reported systems results

All values below are author-reported in the June 2026 post, not independently replicated here.

Standard decode

End-to-end per-step speedup is baseline step time divided by ReplaySSM step time for the whole model. Above 1 means ReplaySSM is faster.

Speculative decode

Throughput speedup is ReplaySSM output tokens per second divided by ordinary non-speculative decoding throughput.

Fixed HBM budget

Request capacity compares the maximum concurrent requests supported by ReplaySSM and the baseline speculative path at the same memory budget.

Batch size 256 1.20-1.48x

standard-decoding speedup

Across evaluated Nemotron-3 Mamba-2 and Qwen3.5 GDN models from 4B to 550B.

Largest tested batches 1.87-1.96x

throughput over standard decode

Large NVFP4 models on B300, using model multi-token-prediction drafters and a four-token speculative window.

Speculative window 4 3.0-3.3x

maximum concurrent requests

Versus the baseline speculative path under a fixed HBM budget.

Source: ReplaySSM Section 6 and Figures 6-11. Standard tests use H100 or B300, FP32 SSM states, BF16 or NVFP4 models, CUDA Graph, and buffers of 8 or 16. Speculative throughput uses GSM8K prompts, temperature 0, window 4, and sweeps serving batch size up to 512. These measurements isolate serving efficiency, not model quality.

10 / What the GDN2 paper measures

Matched small-scale evidence favors finer editing

GDN2 compares recurrent-only versions of these rules at 1.3B parameters, trained with the same recipe on 100B FineWeb-Edu tokens. FineWeb-Edu is a filtered educational web-text corpus. These are author-reported preprint results, not independent replication.

Common-suite average

Average accuracy is the percentage correct averaged over LAMBADA next-word accuracy and eight listed common-sense reasoning tasks. Higher is better.

MK-NIAH-1

Multi-key needle-in-a-haystack from RULER tests whether a model retrieves a requested association among distractors. Retrieval accuracy is percent correct; higher is better.

Real-world recall average

The recall average is mean accuracy over six extraction and question-answering tasks listed in GDN2 Table 4. It is not an average of the neighboring synthetic columns.

Language and common-sense average

Matched recurrent-only models, percent accuracy.

Mamba-251.82

GDN52.07

KDA52.28

GDN253.11

Source: GDN2 Table 2. Axis is truncated to the displayed 0-60 range through proportional bar lengths; read exact labels.

Multi-key retrieval at 4K

MK-NIAH-1, recurrent-only, percent accuracy.

Mamba-221.4

GDN27.8

KDA28.0

GDN237.8

Source: GDN2 Table 3. Axis uses a displayed 0-40 range; exact values are printed.

Does each new channel-wise gate matter?

GDN2 recurrent-only gate-structure ablation. The parameter count is held fixed while one gate is averaged across channels.

More table columns are to the right.

Variant	Channel-wise erase	Channel-wise write	Recall average
write-only channel structure	no	yes	28.92
erase-only channel structure	yes	no	29.51
full GDN2	yes	yes	29.88

Sources: GDN2 Table 5 for ablation values; Table 4 for the six-task recall-average definition. The erase-only variant recovers more of the full result than write-only at this scale.

Kimi Linear is a different-scale systems result.

The Kimi Linear paper's 48B-total / 3B-active hybrid is not on the matched charts above. At 1M context it reports a measured 2.3x batch-one decoding speed over MLA. It also estimates an up-to-6.3x larger-batch decoding speedup from reduced memory use, with associated time-per-output-token values of 1.84 ms versus 11.48 ms. The former is an author-reported measurement; the latter is a theoretical estimate in the paper.

11 / Limits

What these equations do not guarantee

Finite state stays finite

More precise gates improve editing, but a fixed matrix still compresses an unbounded stream. Information can collide or disappear.

Retrieval is not understanding

Needle tasks isolate memory behavior. Better scores do not by themselves establish reasoning, factuality, or robust long-document understanding.

Gates are predictions

The model must learn good decay, erase, and write values from data. Extra control only helps when those predictions are useful.

Generation remains causal

ReplaySSM can delay state materialization and verify a drafted window in parallel, but accepted outputs still depend on earlier accepted tokens. It does not make future tokens independent.

Hybrids remain common

Kimi Linear and GDN2 hybrids retain attention layers because direct local or global token access complements compressed memory.

GDN2 is a preprint

The May 2026 study is controlled at 1.3B parameters and 100B tokens. Larger-scale, independent replication remains open.

One-sentence takeaway:

The model lineage gives fixed-size memory finer control over what fades, erases, and writes; ReplaySSM shows that the same recurrence can be served by briefly retaining inputs and materializing state only when needed.

12 / Primary sources

Papers, systems article, and released implementations

Mamba-2: Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, May 2024. Official code.
Gated DeltaNet: Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule, ICLR 2025. Official code.
Kimi Linear: Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture, October 2025 preprint. Official release.
Gated DeltaNet-2: Ali Hatamizadeh, Yejin Choi, and Jan Kautz. Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention, May 2026 preprint. Official code.
ReplaySSM: Ze-Wei Liou and Tri Dao. ReplaySSM: Cache SSM Inputs, Not State, June 15, 2026 systems article. Public vLLM research fork.

Scope boundary

This explainer teaches the recurrent memory-update lineage and ReplaySSM's decoding idea. It intentionally excludes the full semiseparable-matrix proof, continuous-time discretization, WY/UT derivations, gate-aware backward pass, full Triton kernel listings, Mamba-1, Mamba-3, and exhaustive benchmarks.