Core ML

Dynamic Batching

GPUs are terrible at processing one request at a time, but great at processing 50 at once. When users send requests via an API at random times, the server forces the first user to wait a few milliseconds, gathers 49 more requests from other users, and sends them to the GPU together.

When user requests hit the server, instead of sending them instantly to the GPU, a queue pauses them for a fraction of a second. Once the queue reaches the optimal batch size (or a timeout is hit), it unleashes them all to the GPU as a single matrix.

Why Does This Exist?

In batch-vs-realtime-inference, we learned that GPUs are designed for Throughput (doing massive amounts of math simultaneously), not Latency (doing one tiny math operation instantly).

If you have a Real-Time REST API, you are usually processing one request at a time. This causes terrible GPU utilization. If a GPU runs a forward pass on a single sentence, 95% of its internal cores sit completely idle. Because of this, processing 1 sentence takes 20 milliseconds, but processing 50 sentences at the exact same time might also take 20 milliseconds!

To save money and maximize throughput in a real-time system, we use Dynamic Batching. It tricks the GPU into doing Batch Inference while still providing the user with Real-Time latency.

Think of It Like This

Imagine an elevator in a 100-story skyscraper (the GPU).

If a single person presses the call button, gets in, and rides to the 80th floor, the elevator is fundamentally underutilized. It costs the building the same amount of electricity whether 1 person is in it or 20 people are in it.

Dynamic Batching is like programming the elevator doors to stay open for exactly 5 seconds after the first person steps in. The first person is slightly annoyed they have to wait, but during those 5 seconds, 10 more people walk into the elevator. The elevator then takes all 11 people up at once. You sacrificed a tiny bit of latency for a massive gain in throughput.

How It Works

Dynamic batching is implemented inside an Inference Engine (like Triton Inference Server or TorchServe) that sits between your web server and the GPU.

The Queue: User A sends a request at T=0ms. Instead of sending it to the GPU, the Inference Engine puts it in a holding queue.
The Waiting Window: The engine waits for a predefined window (e.g., 10ms). During this time, Users B, C, and D send requests.
The Execution: At T=10ms, the engine takes all 4 requests, pads them to be the same length, combines them into a single mathematical matrix (a Batch), and sends it to the GPU.
The Response: The GPU processes the massive matrix in one shot. The engine splits the answers back apart and sends them to Users A, B, C, and D.

The Two Triggers

The queue releases the batch to the GPU based on two rules. Whichever happens first triggers the execution:

Max Batch Size Hit: E.g., The queue hits exactly 32 requests. Don't wait anymore; send it to the GPU instantly.
Max Timeout Hit: E.g., The queue has waited 10 milliseconds. Even if there are only 3 requests, send it to the GPU so the users don't have to wait too long.

Show Me the Code

You don't write dynamic batching logic manually in Python (doing so is usually buggy and slow). You configure it in your Inference Engine. Here is a standard configuration file for Nvidia's Triton Inference Server.

1# config.pbtxt (Triton Configuration)2name: "my_resnet_model"3platform: "onnxruntime_onnx"4max_batch_size: 1285
6# Enable Dynamic Batching!7dynamic_batching {8  # If you wait this long, send the batch immediately9  max_queue_delay_microseconds: 5000  # 5 milliseconds10  11  # Optional: If you hit these exact sizes, send the batch immediately12  preferred_batch_size: [ 16, 32, 64 ]13}

Watch Out For

The Padding Problem in NLP. Dynamic Batching works perfectly for images because all images can easily be resized to $224 \times 224$ . However, in Natural Language Processing, User A might send a 5-word sentence, and User B might send a 500-word essay. To put them in the same matrix, you have to pad User A's sentence with 495 empty <PAD> tokens. The GPU still has to do the math on those 495 empty tokens! If your sequence lengths are highly variable, standard dynamic batching wastes massive amounts of GPU compute on padding. This flaw is what led to the invention of Continuous Batching for LLMs.

The Quick Version

GPUs are drastically under-utilized when serving one request at a time.
Dynamic Batching intentionally delays the first user's request for a few milliseconds to collect more incoming requests.
It combines multiple requests into a single batch, drastically increasing GPU throughput at the cost of a tiny bit of latency.
It is triggered by either hitting a Max Batch Size or a Max Timeout.
It works poorly for Large Language Models because of the need to pad sequences of different lengths.