LLM evaluation
How to Compare vLLM and SGLang at Your Expected Load
A fair serving test with a defined request mix, tail-latency limit, cache policy, and worked load decision.
For a live assistant, choose the server that meets your response-time target at your expected peak request rate. Test both servers with the same model, hardware, prompts, and cache policy. The best choice can change for a batch job or a workload with many repeated prefixes. This is a method, not a BuildAI benchmark result.
Set the limit from the product
Ask how long a person can wait before the first words appear. In this example, the product team allows 1.5 seconds for 95% of requests at a peak of 12 requests per second. “95%” means that at most one request in twenty can take longer. The team also requires fewer than 1% failed requests. These are example product requirements, not general targets.
Make a workload sheet from recent traffic: for example, 60% short requests (300 input, 150 output tokens), 30% medium (1,000 and 300), and 10% long (4,000 and 500). Record the actual length distribution rather than relying on those example values. Use arrival rates such as 6, 9, 12, and 15 requests per second; a fixed number of concurrent users does not describe the same traffic. Separate streaming from non-streaming requests.
Run the same test on both servers
Pin the model revision, tokenizer, GPU, quantization, output limits, and software versions. Warm each server, repeat each load point, and report time to first token (TTFT), total response time, output rate, failures, and queue growth. Both vLLM's benchmark guide and SGLang's serving benchmark guide describe rate-controlled serving tests and latency measures. Their tools can test compatible endpoints; check each version's supported flags before a run. Run a second test with realistic repeated prefixes if your traffic has them, and report cache hits separately.
Worked example: a live assistant
Suppose repeated trials produce the illustrative table below. Each cell is the 95th-percentile TTFT in seconds; the product limit is 1.5. Assume failures stay below 1% through 12 requests per second, but rise above it at 15.
| Offered load | vLLM TTFT | SGLang TTFT | Meets 1.5 s limit? |
|---|---|---|---|
| 6 req/s | 0.7 s | 0.8 s | Both |
| 9 req/s | 1.1 s | 1.0 s | Both |
| 12 req/s | 1.8 s | 1.3 s | SGLang only |
| 15 req/s | 2.7 s | 2.2 s | Neither; failures also exceed limit |
At the required 12 requests per second, SGLang passes and vLLM does not. That does not make SGLang generally faster. If the expected peak is only 9 requests per second, both pass and cost or operating needs can decide. If this is an overnight batch job, total throughput may matter more than first-token delay.
Before a launch decision
Run enough requests at each load to estimate a tail value: with 20 requests, p95 is effectively one slow request. Repeat on different days and inspect the spread. If the result sits near 1.5 seconds, collect more traffic before calling it a pass. Test a held-out traffic trace after tuning so neither server is picked on the same requests used to change settings. Record the request mix, rate, cache-hit share, metric distribution, failures, and versions.
Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.