Inference optimization
Engineering improvements to the systems that serve a trained model, making it faster or cheaper to run without changing the model itself.
Inference optimization covers a range of techniques — from custom GPU kernels to batching strategies to quantization — that reduce the cost or latency of running a trained model in production. Unlike training improvements that change what the model knows, inference optimization changes how efficiently the same model is served. As AI models become commoditized, inference efficiency is an increasingly important competitive differentiator.
Also known as
serving optimization, inference efficiency