In his book Inference Engineering, expert Philip Kiely outlines strategies for optimizing AI models after training. The focus is on balancing performance, latency, and infrastructure costs to ensure a product remains viable.
The inference process consists of two stages: prefill (reading the prompt) and decode (generating the response). Each stage has distinct bottlenecks, with memory bandwidth often limiting the decoding phase.
Key optimization techniques include batching, which groups requests to improve throughput, and prefix caching, which stores repeated prompt parts to save computation time. Quantization, such as FP8 or MXFP8, reduces memory usage by simplifying numerical precision.
Speculative decoding uses smaller models to draft responses, which are then verified by a larger model. This approach reduces resource consumption but is most effective for predictable tasks.
Infrastructure management involves configuring GPU clusters to handle specific workloads. Automated scaling is recommended to manage traffic spikes, using metrics like request volume and GPU utilization to trigger capacity adjustments.
Beyond text, optimization extends to image and audio processing. Techniques include reducing image resolution, processing audio in parallel segments, and streaming responses through a multi-model pipeline to minimize user wait times.
