MiniMax H3 — one transformer that denoises video and audio together

Raw video is enormous and mostly redundant. So a separate small network — an autoencoder, here called a VAE — is trained first, and only to do one thing: squash video into a much smaller grid of numbers (encode) and expand it back to pixels (decode). That small grid is the latent. H3's visual VAE shrinks each frame 16× in width and height, shrinks the frame rate 4×, and describes every surviving cell with 24 numbers ("channels") instead of 3 RGB values.

The big 33 B model never sees a pixel. It works entirely in latent space. Pixels exist only at the very start and the very end.

A transformer does not consume grids; it consumes a flat list of vectors, called tokens. So the latent grid has to be chopped into a list. Patchify is that chop: H3 takes every non-overlapping 1 × 2 × 2 block (1 time-step × 2 latent rows × 2 latent columns), and glues those 4 cells' 24 channels into a single vector of 24 × 1 × 2 × 2 = 96 numbers. That 96-number vector is one token.

Net effect: 16× from the VAE, times another 2× from patchify, means 32× spatial shrink from pixels to tokens. Worked example below.

To generate, H3 fills the target tokens with pure random noise and then runs the transformer N times. Each run predicts "which direction is less noisy?" (a velocity), and a small non-learned routine called the scheduler takes one step in that direction. After N steps the noise has become a video. This is flow matching, a modern form of diffusion.

The transformer is the only learned part of that loop, and it is run from scratch every step — there is no carry-over cache between steps, unlike a chatbot generating words.

These two words get used loosely and they mean different things. A latent frame is one time-slice of the latent grid — a whole two-dimensional sheet of cells, 44 × 80 in the example above. Patchified 1 × 2 × 2, that single sheet becomes 22 × 40 = 880 tokens, every one of which carries the same time coordinate and differs only in height and width.

So the 27,280 figure is 31 latent frames × 880 tokens each. Audio works differently: one audio latent frame is a single 32-number vector, which is exactly one token — two, counting stereo.

Take the finished video's latents, call them x₀. Take pure random noise the same shape. Now blend them on a straight line. H3 writes that blend as x_t = t·x₀ + (1 − t)·noise.

So t = 1 is the clean video and t = 0 is pure noise. (Careful: most diffusion papers use the opposite convention, where t = 0 is clean. H3 flips it.) The noise fraction is σ = 1 − t. Generation starts at σ = 1 and walks down to σ = 0. That is the entire meaning of "timestep" here — a mixing ratio, nothing more.

A transformer's attention is permutation-invariant: shuffle the tokens and it computes the identical answer, just shuffled back. Position in the list means nothing. So on its own the model cannot tell which video token is frame 1 versus frame 30, or which snippet of sound belongs with which picture. Every token is just a vector of 5376 numbers.

And the two streams tick at different rates — video produces 6 latent frames per second, audio produces 40. If each were numbered by its own index, "video 3" and "audio 3" would be entirely different moments: 0.5 s versus 0.075 s. There would be no way to express "these two happen at the same time", which is precisely the thing a video-with-sound model has to know.

A convolution has one stride, not a different one per position. Uneven groupings like 3, 3, 3, 3, 5 cannot be produced by a strided conv stack at all; you would need a different architecture entirely, with learned or variable-width pooling.

And 4 = 2 × 2 is precisely what allows the reduction to be built from two stride-2 stages (assert time_stride in [1, 2] in the source). 3, 3, 3, 3, 5 factors into nothing usable.

A causal downsampler with ratio r maps T frames to (T−1)/r + 1 latents. That formula has a property plain T/r does not: T = 1 gives exactly 1 latent. T/r would give a quarter of a latent, which means nothing.

H3 cannot function without T = 1 working — reference images, first/last-frame conditioning and joint image+video training all feed single stills through this same encoder. The lopsided first latent is the price of that, and it is a price worth paying.

Text tokens get t = 0, 1, 2, … (their index) and h = w = 0. So on the time axis they behave exactly like ordinary 1-D language-model RoPE, and the two spatial axes contribute nothing. No separate text-position machinery was needed: the 3-D scheme contains the 1-D one.

The clock unit is chosen to be one audio latent = 1/40 second. A video frame at 24 fps lasts 1/24 second, so in clock units it is (1/24) ÷ (1/40) = 40/24 = 5/3. That is the whole derivation, and that constant — _ROPE_FRAME_RESCALE — is the only place the model's sense of time is set.

Two things confirm the separation. rope_theta and rope_freq_dim live in transformer/config.json and would be byte-identical for 30 fps video or 48 kHz audio — only the packing constants would move. And rope.inv_freq is computed, not loaded: it is not a stored tensor, let alone a learned one. Nothing in the 33 B of weights encodes a time unit.

The chain end to end: _ROPE_FRAME_RESCALE → _temporal_position_grid() → position_ids[:,0] → rope() → rotate Q, K.

The spatial grid is normalised by √(height × width) and centred on [0, 32), so a 768p frame and a 2K frame produce the same coordinate range — only the sampling density between the endpoints changes. Aspect ratio is preserved: a 16:9 frame spans wider in w than in h, both centred on the same midpoint.

That is how one set of weights generates at several resolutions. The model never learns "position 40 of 80"; it learns positions on a fixed unit canvas. It also explains why In-Context Regeneration can feed a 768p result back in as conditioning for a 2K pass — both live on the same coordinate system.

A familiar way to separate two kinds of token is a segment embedding: a small learned table, x = x + segment_embed[0 or 1], added once at the input. H3 does nothing of the sort.

Every transformer normalizes activations before each sublayer, then applies a learned per-channel scale and shift: y = norm(x) · γ + β. Those two are learned constants — identical for every input the model will ever see.

AdaLN makes them computed instead of constant: y = norm(x) · (1 + scale) + shift, where scale and shift come out of a small network fed some conditioning signal. In diffusion that signal is the noise level.

So the same weights behave one way at σ = 0.9 and another at σ = 0.1 — one network acting like many, with the noise level turning the dial. Which is exactly what the task needs: at high noise a block should be settling coarse global structure, at low noise sharpening fine texture.

In a standard DiT there is one noise level for the whole image, so AdaLN is computed once per forward pass and applied uniformly. H3 instead builds a table indexed by (noise level, modality), and every row of the sequence looks up its own entry: adaln_indices = timestep_indices × 3 + token_tags.

That per-row lookup is precisely what lets a clean reference frame and a fully noised target frame sit in the same sequence and be processed correctly by the same weights — the mechanism §08 depends on.

In a chatbot, the prompt is encoded once and reused for every generated word. That does not work here. Text rows inherit the video timestep, which moves every step, so their AdaLN modulation — and therefore their contribution to attention — changes every step. And because attention is bidirectional, information flows from the changing target rows back into the reference rows after the very first block.

Net: the only reusable region is block 0's keys and values for the reference rows. Everything else is recomputed N times. That is the structural reason video diffusion is so much more expensive per output than text generation.