vLLM vs SGLang vs TensorRT-LLM: a serving benchmark

June 20, 2026, 8 min read, By Jaber Jaber

Engineering

Key takeaways

There is no single fastest engine. The winner changes with the operating point.
TensorRT-LLM had the lowest first-token latency at moderate load (235 ms vs vLLM 514 ms at concurrency 32).
vLLM reached the highest throughput at saturation (5,333 tok/s on one H100) and the lowest cost.
The lowest published derived GPU cost is vLLM at fp8 on H100, $0.158 per 1 million output tokens.
The published bf16 cost rows put L40S at about twice the H100 cost per output token.

Everything below is scoped to what we tested: Llama-3.1-8B-Instruct, one H100 80GB and one L40S 48GB, bf16 and fp8. We do not generalize past it. Versions, as of June 20 2026: vLLM 0.23.0, SGLang 0.5.13, TensorRT-LLM 1.2.1. These are self-reported results; the harness and raw data are not publicly available. Published unique-prompt throughput and latency rows cover H100 bf16. FP8 and L40S appear only in derived cost results.

The short version

There is no single fastest engine. The winner changes with how you load it.

If you care about	Use	On this benchmark
Lowest first-token latency at moderate load	TensorRT-LLM	235 ms TTFT at c32 vs vLLM 514 ms
Highest throughput at saturation	vLLM	5,333 tok/s on one H100
Lowest cost per token	vLLM at fp8	$0.158 per 1 million tokens
Shared-prefix caching	a big win for both	throughput at a 90% hit rate, cache on vs cache off: 3.13x for vLLM (2,855 vs 910 tok/s), 2.84x for SGLang (2,488 vs 875 tok/s)

Decision guide

The rule for these configs: TensorRT-LLM for latency-bound serving below its throughput ceiling, vLLM for the highest throughput and the lowest cost. vLLM was ahead of SGLang at every concurrency: by 1.8 to 9.0 percent on throughput, and SGLang's time to first token was 6.2 to 26.2 percent higher.

Two things we are explicit about
TensorRT-LLM 1.2 removed the TensorRT backend; PyTorch is the only execution backend in the tested 1.2.1 release. Our single-prefix cache test is not the many-distinct-prefix case SGLang's RadixAttention is designed for, so we ran that separately too. These results apply only to the tested configurations.

We ran every engine on the same weights, same GPU, and same request stream. The harness and raw data are not publicly available, so these results cannot currently be independently reproduced from this publication. In the published bf16 cost rows, L40S costs about twice as much per output token as H100.

Why we ran this

Hugging Face put TGI into maintenance mode in December 2025 and archived the repository on March 21, 2026, so a lot of teams are re-choosing a serving engine right now. The comparisons you can find are point-in-time snapshots on someone else's workload. So we ran it ourselves. This article records our results and their disclosed conditions; it does not provide a public reproduction harness.

How we measured

One client drives all three engines with identical request streams, so the timing definitions are the same everywhere. TTFT is the time to the first streamed token. Throughput is output tokens per second. We warm up, then time three repeats per operating point and chart the mean. Raw repeats and confidence intervals are not publicly available. Prefix caching is off for the unique-prompt sweeps so nothing gets a free cache hit.

Throughput vs concurrency

We swept concurrency from 1 to 256 at 1,024 input and 256 output tokens. At batch 1 the three engines are within four percent of each other. As load rises, TensorRT-LLM is competitive or best through concurrency 64, then flattens. vLLM keeps scaling and reaches the highest peak, 5,333 tok/s.

concurrency	vLLM	SGLang	TensorRT-LLM
1	158	153	152
8	1096	1005	1027
32	2921	2730	2978
64	4040	3898	4029
128	4943	4816	4774
256	5333	5235	4813

Output throughput vs concurrency, Llama-3.1-8B bf16, one H100, output tok/s

The latency vs throughput frontier

The throughput table hides the more useful picture. Plot first-token latency against throughput and the tradeoff is clear. TensorRT-LLM holds the lowest TTFT across the whole mid range and keeps that edge up to about 4,000 tok/s, then hits its ceiling and TTFT climbs steeply. vLLM owns the high-throughput end: it reaches 5,333 tok/s at a lower TTFT than the other two get near that rate.

engine	output tok/s	TTFT p50 (ms)
vLLM	158	35
vLLM	1096	177
vLLM	2921	514
vLLM	4040	790
vLLM	4943	1658
vLLM	5333	2221
SGLang	153	41
SGLang	1005	221
SGLang	2730	597
SGLang	3898	997
SGLang	4816	1761
SGLang	5235	2543
TensorRT-LLM	152	32
TensorRT-LLM	1027	56
TensorRT-LLM	2978	235
TensorRT-LLM	4029	504
TensorRT-LLM	4774	1691
TensorRT-LLM	4813	2848

Latency vs throughput frontier (each point is a concurrency level)

So the frontier has two owners. TensorRT-LLM for latency-sensitive serving below its ceiling, vLLM for maximum throughput. That is the whole point: the right engine is a function of your operating point, not a constant.

TensorRT-LLM: the backend in version 1.2.1

Read this before you weigh the throughput numbers. We ran TensorRT-LLM on its PyTorch backend, which loads a Hugging Face checkpoint and serves it with no ahead-of-time engine build. PyTorch is the only execution backend as of TensorRT-LLM 1.2.

NVIDIA's TensorRT-LLM 1.2 release notes document removal of the TensorRT backend and its engine-build CLI. The 4,813 tok/s result is the highest throughput we reported for TensorRT-LLM 1.2.1 in this sweep. It does not establish a ceiling for other workloads or configurations. TensorRT-LLM had the lowest first-token latency of the three across the mid range; among the configurations we ran, vLLM reached the highest throughput.

Cost per token

We turn throughput into dollars with one formula: cost per 1 million tokens equals the GPU hourly price divided by sustained tokens per hour. The H100 is $3.95 an hour and the L40S is $1.95 an hour, rented, observed June 20 2026.

engine	bf16 / H100	fp8 / H100	bf16 / L40S
vLLM	0.206	0.158	0.425
SGLang	0.21	0.17	0.438
TensorRT-LLM	0.228	not measured	not measured

Cost per 1 million output tokens at each config's saturation (lower is better), $ / 1 million tokens

On H100, vLLM's reported derived cost is $0.158 per 1 million output tokens at fp8, versus $0.206 at bf16. The published fp8 results provide no throughput or latency rows. They do not establish a speedup.

The published vLLM bf16 cost rows are $0.425 per 1 million output tokens on L40S and $0.206 on H100. The L40S cost is roughly twice as high at the recorded rates and saturation points. No L40S throughput or latency rows are published, so these results do not support a speed comparison.

Shared-prefix workloads

RAG and agent traffic reuse a long system prompt across requests, so a prefix cache should help. We ran this two ways, because the shape of the reuse matters.

One shared prefix

First the simple case: one 2,048-token prefix sent to a fraction of requests, with that fraction varied from 0 to 90 percent, caching on and off. At zero hits, cache-on already exceeds cache-off: 999 vs 910 tok/s for vLLM (9.7%) and 937 vs 879 for SGLang (6.5%). This gap is unexplained by the published data and is not evidence of cache reuse. Hardware and precision for this sweep are not published. The decision guide's ratios compare cache on with cache off at a 90% hit rate, for the same engine.

prefix-cache hit rate (%)	vLLM (cache on)	vLLM (cache off)	SGLang (cache on)	SGLang (cache off)
0	999	910	937	879
50	1559	912	1528	873
90	2855	910	2488	875

Single shared prefix: throughput vs hit rate, caching on vs off (c32), output tok/s

With caching on, vLLM was ahead of SGLang at every hit rate: by 6.6 percent at zero hits, 2.0 percent at 50 percent and 14.7 percent at 90 percent. We do not read a winner into that gap. A single shared prefix is the easy case that any block-level cache handles well. It is not the workload SGLang's RadixAttention is built for.

Many distinct prefixes, the RadixAttention case

RadixAttention is designed for many distinct prefixes held in a tree at once, the shape you get from branching conversations and a pool of system prompts. So we ran that: 256 requests spread across a growing number of distinct 2,048-token prefixes, 8 then 32 then 128 of them, both engines with caching on.

number of distinct prefixes	vLLM	SGLang
8	2677	2493
32	2277	2086
128	1467	1376

Many distinct prefixes: throughput vs number of prefixes (c32, both caches on), output tok/s

RadixAttention worked. SGLang's logs show the prefixes served straight from its tree cache. But vLLM's prefix cache stayed ahead on throughput by 6.6 to 9.1 percent at every prefix count, with lower TTFT. So on Llama-3.1-8B on one H100 we did not reproduce a RadixAttention throughput win, even on the workload it targets. Both engines cache shared prefixes well; vLLM was 6.6 to 9.1 percent faster, in line with its general edge on this hardware. RadixAttention may pull ahead with longer prefixes, deeper trees, or heavier eviction pressure than we tested. On our test it did not, and we are not going to claim otherwise.

Evidence availability

The harness, raw CSVs, and confidence intervals are not publicly available. The tables preserve our self-reported measurements and disclosed conditions, but they are not a complete reproduction package. Updated September 20, 2026: the previous open-source and reproduction claims have been withdrawn.

Updated 2026-11-13: the sentences that summarize the tables now state the figures the tables give, and the hardware is described as rented without naming the provider. No measured row, derived cost, date or condition changed.

FAQ

vLLM or SGLang? On plain unique-prompt traffic vLLM was ahead at every concurrency in our tests: by 1.8 to 9.0 percent on throughput, with SGLang's time to first token 6.2 to 26.2 percent higher. vLLM's derived cost was also lower in each published configuration. Use vLLM as the default; reach for SGLang when its programmable frontend fits your workload.

Is TensorRT-LLM worth it? For latency-sensitive serving below its throughput ceiling, yes. It held the lowest time to first token across the mid range, 235 ms at concurrency 32 versus vLLM's 514 ms. We ran the PyTorch backend, the only execution backend in TensorRT-LLM 1.2.1.

Which GPU should I use? For reported derived GPU cost on Llama-3.1-8B, H100 is lower in the published bf16 rows. With vLLM, the figures are $0.206 per 1 million output tokens on H100 and $0.425 on L40S. No L40S throughput or latency rows are published, so these results do not support a speed comparison.

Method notes and caveats

One H100 80GB and one L40S 48GB, rented, single GPU, tensor-parallel size 1.
Same Llama-3.1-8B-Instruct weights and same request stream across engines. Three timed repeats. Raw repeats and confidence intervals are not publicly available.
Latency here is TTFT and per-request percentiles, not goodput at a fixed SLO. Whether 514 ms first-token at high load is acceptable depends on your use case; an SLO-goodput sweep is future work.
Numbers are self-reported on our harness and are not cross-calibrated against an external suite like MLPerf. The harness and raw data are not publicly available, so this publication does not support independent reproduction.
TensorRT-LLM 1.2.1 ran the PyTorch backend, its only execution backend. NVIDIA's TensorRT-LLM 1.2 release notes document removal of the TensorRT backend and engine-build CLI.
The L40S sweep stopped at concurrency 128 (its saturation), the H100 at 256.
Engines move weekly. These numbers are vLLM 0.23.0, SGLang 0.5.13, TensorRT-LLM 1.2.1, as of June 20 2026.