5 posts tagged with "Updates"

When we introduced co-operative time-slicing in llm-d, we made a claim: if RL phases become schedulable units, independent jobs can share accelerators with near-zero waste. Today we're backing that claim with a measured, end-to-end proof.

OpenRL, an open source, Kubernetes-native, Tinker-compatible, self-hosted fine-tuning service built on the llm-d time-slicing stack, runs supervised fine-tuning (SFT) and full-parameter reinforcement learning for multiple tenants concurrently on the same GPUs.

In Reinforcement Learning (RL) post-training for Large Language Models using algorithms like GRPO, optimizing the ratio of generator (sampler) to trainer throughput is the single largest driver of infrastructure Total Cost of Ownership (TCO). Because typical RL post-training loops alternate sequentially between generation and optimization phases, GPU and TPU clusters sit completely idle for 40% to 60% of their lifecycle.

Today, llm-d introduces a new well-lit path for Co-operative Time-Slicing: the Snapshot Agent. Rather than forcing physical hardware to wait on upstream phases or sit idle during blocking operations, this platform-level capability dynamically interleaves independent RL jobs onto shared hardware blocks, driving aggregate accelerator duty cycles from the 40% baseline up to 70%+ efficiency without altering underlying model convergence or accuracy.

Key Takeaways​

• Near Zero Idle Accelerators: Co-operative time-slicing multiplexes concurrent RL jobs onto shared GPU and TPU hardware, reclaiming stranded compute capacity and boosting duty cycles from ~40% up to 70%+.
• No Model Degradation: Because active jobs retain exclusive access to physical accelerators during their compute windows, there is no loss in token generation throughput or training step convergence.
• Low Code Changes: A lightweight two-call client API (acquire and yield) integrates seamlessly into existing RL loops, while the Snapshot Agent automatically handles CUDA context evacuation and restoration to host DRAM under the hood.

The llm-d project provides a series of “well-lit paths” - tested, benchmarked solutions for deploying large language models in production. Our first path, Intelligent Inference Scheduling, established a baseline for AI-aware routing by balancing both cluster load and prefix-cache affinities. The default configuration for that path uses an approximate method for the latter, predicting cache locality based on request traffic.

This blog illuminates a more advanced and powerful path: precise prefix-cache aware scheduling.

We take a deep dive into the next generation of this feature, which moves beyond prediction and gives the scheduler direct introspection into distributed vLLM caches. This precision is key to maximizing cache hit rates and achieving a new level of performance and maximizing cost-efficiency in your distributed deployments.

The llm-d project lays out clear, “well-lit” paths for anyone to adopt the leading inference optimizations within their existing deployment framework - Kubernetes. These are tested approaches designed to make complex deployments easier and more efficient. In this post, we explore the first of these paths: intelligent inference scheduling. Unlike basic round-robin load balancing, this method takes the unique demands of LLMs into account, leading to better performance across the board: higher throughput, lower latency, and efficient use of resources.

Why Intelligent Inference Is Needed for LLM Inference​

Deploying large language models (LLMs) on Kubernetes has become the norm, but LLM inference workloads behave very differently from standard microservices. Traditional patterns like uniform replicas paired with round-robin load balancing assume each request uses the same amount of resources and finishes in roughly the same time. In contrast, LLM requests can vary wildly in token count and compute needs, making simple load-spread strategies prone to bottlenecks and imbalanced traffic.

Hey everyone! We've been making great progress with the llm-d project, and I wanted to share some important updates and opportunities to get involved.

Help Shape the Future of the llm-d Project​

To guide the future development of the llm-d project, we need to understand the real-world challenges, configurations, and performance needs of our community. We've created a short survey to gather insight into how you serve Large Language Models, from the hardware you use to the features you need most.

This anonymous, vendor-agnostic survey will take approximately 5 minutes to complete. Your input will directly influence the project's roadmap and priorities. The aggregated results will be shared with the llm-d-contributors mailing list to benefit the entire community.

Take the 5-Minute Survey​

Your participation is invaluable. Please take a few minutes to complete the survey. We encourage you to share it with other users or proxy their needs in your response to ensure our direction reflects the community's diverse requirements.