
LLMs Can Now Write GPU Kernels That Beat torch.compile
, 3 min read, By Jaber Jaber
ResearchEditorial note: This retained article previously presented exact latency, speedup and agent results without a dated measurement record. Those claims have been withdrawn. The diagrams below are illustrative. This article does not announce an available Forge product.
The historical title describes the question that motivated this research note. It should not be read as a measured claim about RightNow or an available service. No numerical performance comparison remains in this article.
What a generated kernel has to prove
A generated kernel has separate correctness and performance obligations. Producing code that compiles does not establish that its outputs match a reference. Matching outputs on a convenient example does not establish correctness across the inputs a workload accepts.
Likewise, an isolated timing does not establish a useful improvement. The baseline, evaluation method and tested workload all belong beside a performance claim. A result without that context is too easy to misread.
KernelBench: Can LLMs Write Efficient GPU Kernels? studies generated kernels through functional checks and comparisons with a declared baseline. Its source date is attached to the evaluation concepts below. We do not reproduce numerical results from the paper here.
| Check | What to inspect |
|---|---|
| Correctness | Compare outputs against the reference. |
| Performance | Measure against the declared baseline. |
| Protocol | Record the task, environment and evaluation method. |
Concepts only. No numerical results reproduced.
The distinction matters when reading any demonstration. A candidate that looks plausible can still be incorrect. A candidate that passes a check can still fail on inputs outside that check. The evaluation protocol is part of the result.
An illustrative search structure
The Forge diagrams in this retained note describe a possible search structure. They do not document a measured deployment, a particular agent count, a published speedup, or a guaranteed optimum.
Forge architecture, illustrative
- Generate a candidate kernel.
- Compile and compare its outputs.
- Measure a correct candidate against the baseline.
- Keep a verified improvement or reject it.
Generating several candidates can explore different implementations. Each candidate still needs the same correctness checks and measurement discipline. Retaining several ideas does not by itself establish that a search is more efficient or that its final kernel is suitable for production.
- Candidate rejected
- Incorrect output or no verified improvement.
- Candidate retained
- Correct under the stated checks and faster under the stated protocol.
- Unresolved candidate
- Needs further validation before any performance claim.
A rejected candidate can be useful evidence about the search. Record why it failed, including incorrect outputs, unsupported inputs and measurement errors. Reporting only the retained candidate hides the work and failures required to find it.
The withdrawn comparison
The old latency chart and its associated model results had no dated measurement record in this repository. We have removed the figures rather than attaching an unrelated paper as their source. The paper above does not substantiate a separate Forge measurement.
The Forge latency comparison has been withdrawn because its figures have no dated measurement record.
The same correction applies to the former claims about exact agent counts, search optima, retrieval collection sizes and economic savings. They are not retained as measurements or converted into illustrative benchmark values.
How to assess a future result
Ask for the immutable source record, the date, the precise workload and the declared baseline. Read the correctness checks before the speed comparison. Check whether failed runs, preparation work and repeated attempts are included in the reported result.
Keep the comparison narrow. A result on a particular operation does not establish the behavior of an entire application. A change in hardware, software, shapes or measurement method can change the outcome.
The research question remains useful: how can generated code be evaluated so that a measured improvement is reproducible and meaningful? The answer needs records. A schematic is an explanation of a possible method, not a substitute for evidence.