Continuous vs Dynamic Batching
Comparing scheduling algorithms for LLM inference servers.
Verdict: Never use traditional Dynamic Batching for modern LLMs; always use an inference server that supports Continuous Batching (like vLLM or TGI) to achieve up to 20x higher throughput.
The Short Answer
When hosting an LLM, you want to process multiple user requests at the same time to maximize GPU usage. Dynamic Batching groups requests together but forces the GPU to wait until the longest request in the batch finishes generating before accepting new requests. Continuous Batching (or iteration-level batching) evaluates the batch after every single token is generated. If a request finishes early, it is instantly ejected, and a new request is slotted into the GPU in its place on the very next token.
Where They Differ
| Feature | Dynamic Batching | Continuous Batching |
|---|---|---|
| Scheduling Level | Request-level (Wait for full answer) | Token-level (Evaluate every iteration) |
| Throughput | Low (GPU sits mostly idle) | Up to 20x higher |
| Latency | High (Requests queue behind long generations) | Low |
| Frameworks | Legacy systems | vLLM, Text Generation Inference (TGI), TensorRT-LLM |
Why Dynamic Batching Fails for LLMs
In classic machine learning (like image classification), every request takes the exact same amount of time. Grouping 32 images into a batch is perfect. However, LLMs generate variable-length text. Request A might generate 5 tokens ("Yes, I can help.") while Request B generates 500 tokens (an entire essay). If they are dynamically batched together, Request A finishes in 200 milliseconds, but the GPU slot for Request A remains locked and empty for the next 10 seconds while it waits for Request B to finish the essay.
Why Continuous Batching is the Standard
Continuous batching solves this by interrupting the process after every single token.
- The GPU generates Token 1 for Request A and Request B.
- The GPU generates Token 5 for Request A and Request B. Request A is finished.
- The server immediately returns Request A to the user.
- The server pulls Request C from the waiting queue and inserts it into the empty slot.
- The GPU generates Token 6 for Request B, and Token 1 for Request C.
This completely eliminates "GPU bubbling" (idle time).
What People Get Wrong
People often try to host custom LLMs using basic Python scripts or naive Hugging Face pipeline() implementations. These do not support continuous batching. If you put a naive script into production, it will handle maybe 2 requests per second before crashing. By simply serving the exact same model weights using vLLM (an open-source continuous batching server), the same GPU can suddenly handle 40+ requests per second.