Skip to content
Same bits at 1 or 64 rows: a prompt fed one row per step and 64 rows per step gives the same hash of its logits.

Same bits at 1 or 64 rows: FP4 MoE kernels on Blackwell

, 8 min read, By Jaber Jaber

Research

RightNow AI is a YC-backed GPU research lab. We run Inkling 2 Small, a 594B parameter mixture of experts trained in NVFP4, in its own numbers across 8 B300 GPUs: the NVFP4 experts of layers 0 to 39 on Blackwell's block-scaled FP4 tensor cores, and the MXFP8 experts of layers 40 to 47 on its block-scaled FP8 ones.

Feeding a prompt 64 positions per step or one position per step gives bit for bit the same logits at every position we checked, and the next-token log-probabilities stay within 0.025 nats of vLLM's on average, on the same GPUs.

CheckResult
Same logits, 64 rows per step against 1 row per step, full model on 8 GPUs0 of 599 rows differ, bit for bit
Next-token log-probability difference from vLLM, same model and GPUs0.025 nats mean, 0.13 at p95
Top-1 agreement with vLLM where its top token leads by 0.1 nats or more, 582 of 595 rows98.6%
w13 expert GEMM, one GPU, one token, rebuilt against our first Blackwell kernel5.1x faster on NVFP4, 7.5x on MXFP8, same bits
Kernel test suites on one B300, 8 ranks emulated10 of 10 pass
Measured 2026-11-21 on B300 GPUs, reference vLLM nightly at tensor parallel 8, temperature 0

Every kernel here was written by our AI kernel engine: coding agents that write CUDA, run our test harness and keep or revert each change.

A kernel was kept only after it passed its gates: bit for bit equal to the kernel it replaced where the arithmetic is unchanged, and batch invariant within a stated tolerance of the reference math where we changed the arithmetic on purpose.

Why the same bits matter

Most inference engines are close, not exact. A token's result depends on what else shares its step: the batch changes how sums are split, and floating point addition is not associative. So "lossless" usually means an accuracy score that moved within noise.

We hold the stricter line that Thinking Machines named batch invariance: a row's result may not depend on the batch at all. We keep it through NVFP4 and MXFP8 block-scaled tensor cores, experts split across 8 GPUs, and every exchange between them.

  • Every reduction has a fixed shape. Sums run in fixed chunks and balanced trees, sums across GPUs fold in rank order in FP32, and the build turns off automatic contraction, so every fused multiply-add is one we wrote on purpose.

  • Every property has a broken twin. Each bit-identity property is paired with a deliberately broken variant that must be caught. Where a break cannot move the bits, such as reordering sums that are exact, the log says so instead of counting it.

  • The proof runs on the real model. On 8 B300 GPUs, the full 48 layer model fed 64 positions per step and 1 position per step gives identical logits on all 599 rows we compared.

Kernel 1: FP4 experts split across GPUs, exactly

Inkling's experts in layers 0 to 39 are stored in NVFP4: 4 bit E2M1 values, an E4M3 scale per block of 16 and an FP32 scale per expert. The activations are quantized at run time the same way, with an FP32 scale per token computed from the maximum over the whole row.

Layers 40 to 47 use MXFP8, whose power of two scales need no row maximum. The serving engines we studied run the NVFP4 experts with expert parallelism, whole experts per GPU, because splitting an expert's intermediate columns across GPUs splits the row whose maximum the second GEMM's quantizer needs.

We split the experts anyway, one eighth of every expert's columns per GPU, and exchange the row maximum. A maximum does not depend on the order it is taken in, so the exchange is exact.

  • The exchange gives the same bits in all 8 rank orders we tested, with the 8 ranks emulated on one B300, and an order-dependent combine is caught.

  • The quantizer matches the reference's four over six rule bit for bit, before and after the exchange.

  • Every GPU reads an equal slice of every active expert, so no GPU reads more expert bytes than the others. The price is one more exchange per FP4 layer.

A second form replaces the maximum exchange with an exchange of the activation itself, and splits w2 by output rows instead of by K. Every GPU then holds the full row and quantizes it on its own, and the closing all-reduce of FP32 partials becomes an all-gather of BF16 slices. Both forms make two exchanges per FP4 layer.

  • Each GPU sends 3.2 times fewer bytes per FP4 layer, counted from the layouts: 40 KB against 129 KB at one row, and 0.32 MB against 1.03 MB at an 8 row verify step. On the 8 MXFP8 layers it adds one exchange.

  • It rounds where SGLang's implementation of the model rounds, each expert's output to BF16 before the sum. Against an FP64 model of those rounding points it shows zero error on the FP4 layer we tested and 5.1e-4 on the FP8 layer, where the first form sits at 2.6e-3 and 2.5e-3.

  • Split across 8 ranks emulated on one B300, against one launch computing all 8 slices, 0 of 16 rows differ, and a withheld slice is caught.

Kernel 2: the FP4 tensor core GEMM, rebuilt for one token

Both kernels run tcgen05.mma with kind::mxf4nvf4.block_scale.block16 for NVFP4 and kind::mxf8f6f4.block_scale for MXFP8: 128 weight rows as M, the tokens swapped into N, FP32 accumulators in TMEM and the block scales copied into TMEM with tcgen05.cp.

At one token per step, the w13 GEMM on each GPU streams 11.9 MB of FP4 weights and scales per layer, and w2 another 6.0 MB. Our first Blackwell kernel gave each expert tile one CTA and walked K in stages it never overlapped: at one token, 40 CTAs on 148 SMs, at 114 GB/s.

The rebuilt kernel splits the fixed K chunks across CTAs and pipelines every stage's loads under the MMAs. Each chunk is still the same chain of block-scaled MMAs in the same K order from a zeroed accumulator, and the last CTA to arrive combines the chunk partials in the same fixed tree, so every output keeps its bits.

GEMM, one GPU's share, one tokenFirst kernelRebuiltSpeedup
w13, NVFP4, 11.9 MB104.4 µs20.5 µs5.1x
w13, MXFP8, 21.9 MB184.3 µs24.6 µs7.5x
w2, first form, NVFP4, 6.0 MB14.3 µs14.3 µs1.0x
w2, second form, NVFP455.8 µs16.9 µs3.3x
w2, second form, MXFP888.0 µs19.1 µs4.6x
One GPU's launch on a B300, L2 flushed before each run, median of 9, measured 2026-11-21

The first form's w2 row confirms the diagnosis: that GEMM already launched 288 CTAs, so it had enough work in flight, and the rebuild leaves it unchanged.

At one token the rebuilt w13 reads 583 GB/s. At 8 tokens per step, with 40 distinct experts per layer, it reads 1.39 TB/s on NVFP4 and 1.84 TB/s on MXFP8 per GPU, against the 6.85 TB/s we measured for a B300's memory, so this kernel still has headroom.

Against the first kernel, on layers 0 and 40 at batches of 1 to 16, 24, 32 and up to 64 rows and over three CUDA graph replays, every intermediate is bit for bit equal: 0 of 512 rows differ.

A partial read before it lands is caught, and so is a reordered chunk tree on the FP8 layer. On the FP4 layer the FP32 accumulators equal an FP64 sum exactly, so no order can show.

Kernel 3: a depth mix in one round

Inkling mixes the outputs of earlier layers before every attention and every expert block: a softmax over up to nine earlier outputs, then a normalization. Done directly, each mix needs two reductions across the GPU.

We keep the Gram matrix of the earlier outputs, computed once when each block closes, so the mix's norm comes from that matrix and one new column. Each mix needs one reduction instead of two: 2 rounds per layer instead of 4, counted, not yet timed.

The value path stays bit for bit the direct one and only the norm changes, so this kernel is batch invariant but not bitwise the direct mix.

In our tests the norm lands within 1.6e-7 relative of the direct one, against a gate of 2^-16, with a compensated sum and a fallback to the direct second round for any row where cancellation could exceed it.

We also tried folding the expert router into the same reduction. On the real routers of layers 0 and 40, with inputs built from the model's own embeddings, its best variant changes the top 8 expert set of 1.1% and 1.6% of tokens, so it stays out: a change that moves routing is not lossless.

Where this goes

Today our engine runs a decode step as separate launches under CUDA graphs, and at one row without the draft a step takes about 16 ms on 8 B300 GPUs.

Nsight puts 5.5 ms of it in the w13 GEMM of our first kernel, which the rebuilt kernel cuts about fivefold in isolation but which is not in the engine yet, and 3.3 ms in the exchanges between GPUs.

The next piece is one persistent decode program per GPU: every layer in one resident kernel, with the exchanges inside it.

Our byte and latency budget for one verify step reads about 7.8 GB per GPU across 154 exchanges, with Inkling's own speculative draft at 3.35 tokens per step and an assumed 45 distinct experts per layer, which we have not measured yet.

On that budget the program targets about 1,900 tokens per second per user, a projection with a range of about 1,630 to 2,160. For scale, vLLM with the same draft runs this model at 666 tokens per second per user on our own 8 B300 node. The next post reports our measured number, bit for bit.

How we checked

  • Every kernel runs against an FP64 twin of the reference math, stage by stage, with a stated tolerance.

  • Every bit-identity property is paired with a deliberately broken control that must be caught.

  • The full model is compared with vLLM's own log-probabilities at 595 teacher-forced positions over 4 prompts. vLLM run twice agrees with itself exactly, so the remaining difference is between the two engines' arithmetic, not run to run noise.

Conditions

Our engine and the vLLM reference ran on one cloud node of 8 NVIDIA B300 SXM6 GPUs, driver 620.70.06, with peer to peer access between all 8 GPUs; the kernel suites and GEMM timings ran on one of its GPUs. Our kernels are built with CUDA 13.0 and -fmad=false.

Checkpoint thinkingmachines/Inkling-2-Small at revision 885d78bf. Reference: vLLM nightly image sha256:c17e9d88, tensor parallel 8, temperature 0. vLLM's speed with the draft (2026-11-21) and the 6.85 TB/s memory bandwidth (2026-11-20) were measured on our own 8 B300 node; everything else on 2026-11-21.

Get started

$1 free for a new account.

Get an API key
rightnow
RightNow open models
Billed per token.
import OpenAI from "openai";const client = new OpenAI({baseURL: "https://api.rightnowai.co/v1",   apiKey: process.env.RIGHTNOW_API_KEY,      });const response = await client.chat.completions.create({model: "MODEL_ID",                         messages: [{ role: "user", content: "Hello" }],max_tokens: 16384,});