Ai2 says a budget-based GPU scheduler cut debug-job waits from hours to seconds

Single-source item: every figure below comes from Ai2's own engineering post. We found no independent measurement.

Ai2's AI Infrastructure team says that replacing its priority-based GPU scheduler cut the 90th-percentile queue wait for short "debug" jobs from 2 hours to 30 seconds, according to a post dated October 9, 2026.

Ai2 runs thousands of H100, B200 and B300 GPUs for about 150 internal researchers, and says demand runs two to three times supply. Its old scheduler let teams mark work as high priority or non-preemptible at no cost. Ai2 says that led to priority inflation, idle jobs parked to hold GPUs ("squatting"), and on-call engineers negotiating with owners of long-running jobs.

The replacement gives managers GPU-time budgets, allocated through the research hierarchy. A fair-share scheduler tracks usage over a sliding window (7 days by default). Each job declares a minimum runtime, capped at 8 hours, during which it cannot be preempted; afterwards it can be requeued. Ai2 says fair-share scheduling is not new, tracing it to Hadoop's scheduler, Slurm's Fair Tree and YARN, and that its contribution is using manager-set budgets as inputs. Those projects' own documentation describes ranking sibling accounts so one account's children all outrank another's, and guaranteed minimum queue shares with spare capacity split among other applications.

- Debug-job p90 wait: 2 hours to 30 seconds. A pre-rollout simulation on hand-built test data had predicted 6 hours to 5 minutes.
- On its largest H100 cluster, median wait fell from 5 minutes to 24 seconds, and p90 from 2.8 to 1.8 hours.
- Teams received 98% of the GPU hours they were owed over 30 days, and cluster occupancy held at 98%.
- Repairs needing a human fell 74%, because unhealthy hosts drain as jobs reach their minimum runtime.

The headline figure rests on a thin base. Ai2 says the baseline had fewer debug workloads, so the measurement has higher variance. It did not publish sample sizes. All results are internal and self-reported, not peer reviewed. Ai2 also reports costs: interactive sessions that once could last a week now become preemptible after the 8-hour cap, and researchers lose state. It is also investigating whether capacity fragmentation will lengthen waits for the largest jobs.

Queue time for small test runs shapes how quickly researchers can iterate. The post suggests that turning scarce compute into a visible budget, rather than a priority contest, can shorten that loop without leaving GPUs idle. Whether the gain holds at other labs, or for the largest training runs, is not shown by this evidence.

Revised 16 October 2026: Guth re-read the original sources and rewrote this article. The earlier version remains available as revision 1.