llm-d 1.0 adds traffic controls and rolling upgrades for distributed AI inference

The llm-d project announced version 1.0 on October 20, adding operational controls to its open-source system for serving large language models across a cluster. IBM, Red Hat, Google Cloud, CoreWeave and NVIDIA founded the project in 2025, and it is now part of the Cloud Native Computing Foundation's ecosystem.

The new release targets a problem faced by teams running many AI applications on the same infrastructure: one burst of automated requests can compete with interactive traffic for limited inference capacity. Version 1.0 introduces priority- and fairness-based flow control intended to allocate that capacity across workloads with different response-time needs.

Operators also gain a highly available router with standby capacity and support for updates while requests are being served. A version-one configuration API is intended to provide a more consistent interface for production deployments.

For AI agents that return repeatedly to the same instructions or documents, llm-d includes routing designed to reuse previously processed context. The broader system can work with model-serving engines such as vLLM and SGLang, while handling tasks including cache management and separating prompt processing from token generation across resources.

The project says its community includes more than 500 contributors across over 25 organizations. IBM describes lower serving costs in particular tests, but those workload-specific results do not establish what savings an individual deployment will achieve.