Continuous Batching
A highly specialized scheduling algorithm that dynamically injects entirely new requests into a running inference batch exactly as older requests finish up.
Think of It Like This
Like a busy restaurant seating a new party at a freshly cleared table immediately, rather than waiting for the entire dining room to finish eating first.
Traditional batching waits for every single sequence in a batch to completely finish generating before starting a new batch, wasting massive amounts of compute while waiting for the longest response. Continuous batching operates at the iteration level, swapping completed sequences out for new queries on the fly. This drastically improves GPU utilization and throughput.