2 posts tagged with "rl"

When we introduced co-operative time-slicing in llm-d, we made a claim: if RL phases become schedulable units, independent jobs can share accelerators with near-zero waste. Today we're backing that claim with a measured, end-to-end proof.

OpenRL, an open source, Kubernetes-native, Tinker-compatible, self-hosted fine-tuning service built on the llm-d time-slicing stack, runs supervised fine-tuning (SFT) and full-parameter reinforcement learning for multiple tenants concurrently on the same GPUs.

In Reinforcement Learning (RL) post-training for Large Language Models using algorithms like GRPO, optimizing the ratio of generator (sampler) to trainer throughput is the single largest driver of infrastructure Total Cost of Ownership (TCO). Because typical RL post-training loops alternate sequentially between generation and optimization phases, GPU and TPU clusters sit completely idle for 40% to 60% of their lifecycle.

Today, llm-d introduces a new well-lit path for Co-operative Time-Slicing: the Snapshot Agent. Rather than forcing physical hardware to wait on upstream phases or sit idle during blocking operations, this platform-level capability dynamically interleaves independent RL jobs onto shared hardware blocks, driving aggregate accelerator duty cycles from the 40% baseline up to 70%+ efficiency without altering underlying model convergence or accuracy.

Key Takeaways​

• Near Zero Idle Accelerators: Co-operative time-slicing multiplexes concurrent RL jobs onto shared GPU and TPU hardware, reclaiming stranded compute capacity and boosting duty cycles from ~40% up to 70%+.
• No Model Degradation: Because active jobs retain exclusive access to physical accelerators during their compute windows, there is no loss in token generation throughput or training step convergence.
• Low Code Changes: A lightweight two-call client API (acquire and yield) integrates seamlessly into existing RL loops, while the Snapshot Agent automatically handles CUDA context evacuation and restoration to host DRAM under the hood.