LLM evaluation

How to Compare vLLM and SGLang at Your Expected Load

A fair serving test with a defined request mix, tail-latency limit, cache policy, and worked load decision.

For a live assistant, choose the server that meets your response-time target at your expected peak request rate. Test both servers with the same model, hardware, prompts, and cache policy. The best choice can change for a batch job or a workload with many repeated prefixes. This is a method, not a BuildAI benchmark result.

Set the limit from the product

Ask how long a person can wait before the first words appear. In this example, the product team allows 1.5 seconds for 95% of requests at a peak of 12 requests per second. “95%” means that at most one request in twenty can take longer. The team also requires fewer than 1% failed requests. These are example product requirements, not general targets.

Make a workload sheet from recent traffic: for example, 60% short requests (300 input, 150 output tokens), 30% medium (1,000 and 300), and 10% long (4,000 and 500). Record the actual length distribution rather than relying on those example values. Use arrival rates such as 6, 9, 12, and 15 requests per second; a fixed number of concurrent users does not describe the same traffic. Separate streaming from non-streaming requests.

Run the same test on both servers

Pin the model revision, tokenizer, GPU, quantization, output limits, and software versions. Warm each server, repeat each load point, and report time to first token (TTFT), total response time, output rate, failures, and queue growth. Both vLLM's benchmark guide and SGLang's serving benchmark guide describe rate-controlled serving tests and latency measures. Their tools can test compatible endpoints; check each version's supported flags before a run. Run a second test with realistic repeated prefixes if your traffic has them, and report cache hits separately.

Worked example: a live assistant

Suppose repeated trials produce the illustrative table below. Each cell is the 95th-percentile TTFT in seconds; the product limit is 1.5. Assume failures stay below 1% through 12 requests per second, but rise above it at 15.

Offered loadvLLM TTFTSGLang TTFTMeets 1.5 s limit?
6 req/s0.7 s0.8 sBoth
9 req/s1.1 s1.0 sBoth
12 req/s1.8 s1.3 sSGLang only
15 req/s2.7 s2.2 sNeither; failures also exceed limit
Illustrative load curve in table form. The 12 req/s row decides this example; the latency limit is 1.5 seconds at p95.

At the required 12 requests per second, SGLang passes and vLLM does not. That does not make SGLang generally faster. If the expected peak is only 9 requests per second, both pass and cost or operating needs can decide. If this is an overnight batch job, total throughput may matter more than first-token delay.

Before a launch decision

Run enough requests at each load to estimate a tail value: with 20 requests, p95 is effectively one slow request. Repeat on different days and inspect the spread. If the result sits near 1.5 seconds, collect more traffic before calling it a pass. Test a held-out traffic trace after tuning so neither server is picked on the same requests used to change settings. Record the request mix, rate, cache-hit share, metric distribution, failures, and versions.

Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.

Sources

  1. vLLM serving benchmark and metric definitions
  2. SGLang serving benchmark guide
← All articlesOur article standard →