Constraint-Aware Allocation Boosts GPU Utilization up to 33 Percentage Points Over FIFO Scheduling

Constraint-Aware Allocation Boosts GPU Utilization up to 33 Percentage Points Over FIFO Scheduling Dharma AI details a constraint-aware GPU scheduling framework that optimizes workload placement across a time horizon rather than relying on arrival order or static reservations. Evaluated across seven benchmark scenarios against a FIFO baseline on identical hardware and workloads, the system improved GPU utilization by up to 33 percentage points and increased priority-weighted output by an average of 52%. A constraint-aware allocator outperformed standard FIFO scheduling across seven benchmark scenarios, increasing GPU utilization by up to 33 percentage points and priority-weighted value by up to 105.1%. Replacing static real-time peak GPU reservations with dynamic, penalty-weighted demand curves allows idle trough capacity to be safely utilized by batch workloads. The allocation engine relies on five core constraints and runs via a hot-path heuristic executing in 1 to 15 milliseconds, with an optional full optimization mode for periodic refinement. Accurate scheduling is underpinned by specialized demand forecasters across training, quantization, and inference, avoiding uniform-model estimation errors. A rolling 24-hour horizon commits only the immediate timestep and re-runs every 30 to 60 minutes to absorb forecasting errors without preempting active jobs. Limitations of FIFO Scheduling Under Contention FIFO schedulers introduce major capacity inefficiencies under contention through two coupled behaviors: peak reservations and arrival-order placement. Because real-time inference requires instant availability, FIFO systems traditionally reserve capacity equal to an application's peak daily demand, leaving GPUs idle during demand troughs and unavailable to batch jobs. Furthermore, placing jobs strictly as they arrive ignores priority and future scheduling needs, leading to fragmented capacity that blocks subsequent workloads needing specific contiguous GPU blocks. Mathematical Formulation and Structural Constraints The allocation problem maps GPUs, jobs, and timesteps onto an explicit grid defined by five legal constraints: one job per GPU per timestep, adherence to job demand ranges with inherited running states, contiguous power-of-two GPU blocks for batch jobs, bounded GPU swap caps across consecutive steps for real-time jobs, and zero preemption for running work. The objective function assigns rewards to batch allocations scaled by priority and time-decay weights, while penalizing unmet real-time demand at 5 to 10 times the batch reward weight. This asymmetric penalty enforces latency requirements directly within the optimization rather than relying on an external autoscaler. Architecture, Execution Modes, and Benchmark Results To respect API latency budgets during incoming requests, the system implements a heuristic on the hot path that outputs valid grid allocations by design in 1 to 2 milliseconds for standard tests and 15 milliseconds at 64 GPUs and 30 jobs. A full optimization mode is available for periodic reviews. In benchmarking, a training-heavy scenario jumped from 53.6% to 87.0% utilization with a 105.1% value increase. In a 64-GPU scale test where both schedulers achieved identical 44.9% utilization and completed 27 of 30 jobs, the allocator generated 15.9% more priority-weighted value. An identical-priority test also saw utilization rise from 76.8% to 87.5%, confirming that horizon planning yields gains independent of priority signals. Workload-Specific Forecasting and Horizon Rolling Effective placement requires accurate predictions of workload duration and GPU requirements. Rather than using generic estimators, specialized forecasters evaluate each workload type: training models condition on 22 features covering fine-tuning strategies and loop architectures; quantization tasks are modeled using calibration tiers and specific algorithm profiles; and real-time inference relies on weekly, hourly updated traffic profiles. To prevent optimization boundary failures, the system plans across a 24-hour horizon but only commits the immediate timestep, re-optimizing every 30 to 60 minutes to absorb forecast inaccuracies while holding running jobs stable.

重构调度决策顺序：约束感知GPU分配器实现最高33%利用率提升 Dharma AI构建了一种面向全局约束与优先级的GPU分配器，通过取代传统FIFO调度模式，在不改变硬件和负载的前提下，使GPU利用率最高提升33个百分点，优先级加权产出平均提高52%。 传统FIFO调度因全天静态预留实时推理峰值资源以及按到达顺序排队，在资源争用时造成大量算力浪费。 该分配器将实时推理视作动态需求曲线，结合非抢占式、连续块等5项约束与衰减加权目标函数进行全局规划。 在7种基准测试场景下，利用率从52–85%升至72–88%，优先级加权价值提升24.6%至105.1%，决策延迟仅1至15毫秒。 系统依赖训练、量化与实时推理三种专用预测器提供算力需求，并通过优化24小时、仅提交当前时步滚动重调度吸收预测误差。 FIFO调度在算力争用下的固有缺陷 在缺乏资源争用时FIFO调度与复杂调度器表现无异，但在算力紧张时会产生两重严重浪费。首先是静态预留，由于实时推理不可等待且缺乏波谷释放机制，系统必须按单日最高峰值整天预留GPU，导致低谷期闲置显卡既不能用也无法分配给批处理任务，使基准利用率降至约一半。其次是排队顺序，FIFO仅按请求到达顺序提交容量，忽视任务价值与未来容量形状匹配，导致高优先级和特定形状需求的后续任务因碎片化无法运行。 基于形式化约束与启发式算法的极速决策 调度核心被抽象为一个网格决策问题，由5项硬约束界定：每卡每时步最多承载一任务、继承锁定正在运行的任务、批处理任务需占用2的幂次连续GPU块、实时任务限制时步间切换卡数、已启动任务不可抢占。目标函数对批处理分配赋予带时间衰减的优先级奖励，并对实时推理短缺施加5至10倍于批处理权重的惩罚以保障延迟。为满足线上API请求的低延迟预算，热路径由规则严格对齐约束的启发式算法执行，运行耗时仅1至15毫秒，冷路径则提供形式化优化器用于周期性复核。 七大基准场景的实测效果验证 在覆盖重度争用和不同规模的测试中，分配器均显著优化了指标。8卡16任务的重度训练场景下，利用率由53.6%提高到87.0%，加权价值激增105.1%。在64卡30任务的扩展测试中，两者虽取得完全相同的44.9%利用率和相同完成任务数，但新分配器交付的加权价值高出15.9%，证实单纯占用率指标无法反映算力产出质量。在统一优先级测试中，利用率仍从76.8%升至87.5%，证明收益不仅源于优先级排序，还得益于全局时域内的容量规划。 专用预测器与滚动时间窗口设计 精准调度依赖前置预测。针对异构负载，系统构建了涵盖22个特征且区分10种训练变体的训练预估器，对涵盖不同量化算法的模型量化任务单独建模，并根据历史流量生成动态实时推理周曲线。为避免预测偏差与“世界末日效应”，架构采用优化24小时视界但仅提交当前时步、每30至60分钟重新运行的机制。新一轮计算继承实际运行状态并随时间衰减加权，使预测误差在滚动优化中被吸收而非复合放大。

The previous post argued that utilization, not intelligence, is where the next real constraint in enterprise AI is forming, and it closed by noting that no playbook has emerged yet for what a mature GPU Management practice looks like. This is ours. We built a constraint-aware GPU allocator and benchmarked it against a FIFO scheduler across seven benchmark scenarios. On identical hardware, running identical workloads, GPU utilization rose by as much as 33 percentage points, and priority-weighted output rose in every one of them, by as much as 105%. Nothing about the hardware changed. What changed was the order in which allocation decisions get made. One note on measurement before the numbers start. Every gain below is expressed as improvement over the FIFO result on the same scenario. Utilization is reported in percentage points; value is reported as a percentage increase in priority-weighted output. The decision, stated precisely "Keep the GPUs busy" is not a decision a system can execute. The decision is narrower and much harder: which GPU runs which job, in which timestep, at what priority. Formally it is one binary choice per combination of GPU, job and timestep, and the output is a grid — every GPU, across the whole scheduling horizon, with a job name in each cell or nothing at all. Four workload types compete for that grid: training, real-time inference, batch inference, and quantization. They split into two allocation shapes, and the split is where the difficulty lives. Training, batch inference and quantization are batch-like: once started, each needs a contiguous block of GPUs held without interruption until the job finishes. Real-time inference is the opposite: elastic, driven by a demand curve that changes every timestep, growing and shrinking as traffic does. Two incompatible shapes competing for the same hardware in the same timestep is the core problem. A second heterogeneity sits inside a single type: for the same base model, training jobs range from a few hours to several days, and from one GPU to dozens. What FIFO costs under contention The comparison point throughout is a FIFO-based scheduler: real-time inference served from a fixed reservation, and every other job placed in arrival order, without regard for priority. Under the right conditions, that is a reasonable policy. When the cluster has slack, allocation order costs nothing in utilization, everything fits regardless of sequence, so FIFO and anything more sophisticated fill the same fraction of the pool. Contention is where that ordering cost stops being invisible and starts costing capacity too. It then becomes expensive in two separate ways, and they are worth taking one at a time. The reservation. Real-time inference cannot wait for capacity; the GPUs have to be there the moment traffic needs them. A scheduler that places jobs in arrival order has no mechanism for releasing GPUs during a trough and reclaiming them before the next peak, so the only way to guarantee availability is to take each real-time application's maximum demand for the day and reserve that many GPUs for the whole day. The cost lands in every hour that is not the peak. An application needing six GPUs at midday and two at 4am holds all six for twenty-four hours, and the four idle GPUs are unavailable to any batch job for the entire day. They are not being used, and they are not free either. It is why the baseline sits near half the cluster in the two scenarios where reservation dominates: 51.6% in the mixed control and 53.6% in the training-heavy case. Roughly half a pool, with much of the idle half reserved rather than free. This cost is paid whether the cluster is contended or not — contention only makes it visible. The ordering. Under real contention, which jobs fit at all depends on the order you place them, not just on how much capacity exists. Order is not a tiebreaker applied after the capacity question is settled. Order is a capacity decision. FIFO places each job as it arrives, without weighing what that job is worth and without checking what else still has to fit inside the horizon, so high-priority work waits behind whatever asked first and capacity gets committed in placements that later jobs cannot use. The two compound. The block held for the day's maximum real-time demand is off the table for every batch job in the queue, in every hour, and whatever remains is handed out in the order the requests happened to arrive. It is the GPU equivalent of an airline assigning aircraft to whichever charter called first, then finding nothing left to fly the route that actually pays. And GPUs reserved all day for a peak lasting a couple of hours are the grounded aircraft from the previous piece in the most literal sense: on standby, earning nothing, unavailable to anyone else. [Figure: side-by-side allocation grids — allocator above, FIFO below, same scenario] Across five benchmark scenarios built for genuine contention, the allocator improved both axes at once. Utilization moved from a 52–85% band to a 72–88% band. Priority-weighted value rose between 24.6% and 105.1%, averaging 52%. Every scenario, both metrics, no tradeoff to explain away. The strongest single case was a training-heavy workload on 8 GPUs: utilization went from 53.6% to 87.0%, and value more than doubled, up 105%. Thirty-three points of a fixed, already-depreciating asset, recovered by reclaiming reserved standby capacity and placing the rest in priority order. (This figure reflect a single baseline ordering.) The allocator removes both behaviors. Real-time demand is treated as a curve rather than a ceiling, allocated against demand at each timestep, with batch-like work occupying the troughs, bounded by the cap on how many GPUs a real-time job may swap between consecutive timesteps. And batch-like jobs are placed by priority across the whole horizon instead of in the order they arrived. The rest of this piece is how. Utilization is necessary. Priority is what turns it into value. Utilization measures occupancy: what fraction of available GPU-time is allocated to something. It carries no information about what that something is worth. One scenario pulls the two apart completely, and the gap runs in a direction that is easy to miss. In the scale test, 30 jobs across 64 GPUs, FIFO and the allocator produced identical utilization , 44.9% each, and identical throughput , 27 of 30 jobs completed. The allocator delivered 15.9% more priority-weighted value. Every dashboard reads the same. The cluster produced materially different output. An objective that does not price priority can fill the cluster to exactly the same level, finish exactly as many jobs, and still deliver less. The previous piece argued that occupancy is a poor read on whether a cluster is earning; this is the measured version of that claim. Writing the problem down The alternative is not a longer list of heuristic rules. Some constraints only mean anything globally, and no local rule can express them: contiguous blocks, a budget for how much GPU churn is acceptable across the entire horizon, a guarantee that running work is never preempted. To honor those, the problem has to be written down as one thing. Five constraints define a legal allocation: A GPU serves at most one job per timestep. Every job respects its demand range, and whatever is already running is inherited and held. Batch-like jobs occupy contiguous blocks of GPUs, sized to a power of two. Real-time jobs have a hard cap on how many GPUs they may swap between consecutive timesteps. A job that has started cannot be interrupted. The objective function has two terms. Allocating a GPU to a batch-like job earns a reward equal to its priority multiplied by a time-decay weight. Failing to meet real-time demand incurs a penalty proportional to the size of the shortfall. The relative size of those weights is the entire service-level policy, expressed as one number. The real-time penalty weight is 5 to 10 times greater than the allocation weight. One unit of unmet real-time demand therefore costs what 5 to 10 GPU-timesteps of equal-priority batch work costs. The asymmetry is deliberate, and it means latency obligations are enforced inside the same optimization that places batch work, rather than by a separate autoscaler competing with the scheduler for the same GPUs. It is also what makes the elastic treatment of real-time demand safe. The allocator can hand a GPU to batch work during a trough because underserving real-time demand later is priced so far above whatever that batch work earns — the penalty, not a static reservation, is what protects availability. The time weight decays across the horizon for a reason that only makes sense in an online system: by the next scheduling run, new jobs will have arrived. Capacity used now is worth more than capacity promised later. The allocator that already knows the constraints The formal model defines what a legal, well-scored allocation looks like. Answering an incoming request is a separate job, and it belongs to a separate component. This is NP-hard combinatorial allocation, and the scheduler is re-invoked every time a job arrives, so the decision has to come back in the gap between two API requests. That latency budget is the fixed constraint the architecture is designed around, which is why a heuristic sits on the hot path and the formal model sits behind it as the specification the heuristic is built to satisfy. That heuristic is not a generic greedy allocator. Its rules are the formal model's structural constraints, which means every grid it produces is a legal allocation by construction. Not usually valid. Valid by design. That design, applied across the whole horizon rather than one arrival at a time, is what produces the utilization gain. The allocator sees every queued job before it places any of them, it can hold the free pool in shapes the remaining work can actually occupy, a batch job needing a contiguous block of a given size still has room when its turn comes. Priority decides who gets first claim on that room. FIFO has neither view: it commits capacity to whichever job asked first, and a job that arrives later and needs a specific shape may find nothing left that fits, so it goes unscheduled and the GPU-hours it would have consumed go unclaimed. It runs in 1 to 2 milliseconds on the five contended scenarios, and 15 milliseconds at 64 GPUs and 30 jobs — fast enough to run on every incoming request. The system exposes two modes. Fast mode runs the allocator alone and returns its grid; this is the hot path. Full mode uses that grid as a starting point for the formal model, which attempts to improve on it — suited to periodic review rather than per-request decisions. Results Scenario Utilization Value Value gain Latency Mixed control (8 GPUs, 10 jobs) 51.6% → 72.4% 7,093 → 10,980 +54.8% 1 ms Real-time contention (8 GPUs, 8 jobs) 75.0% → 80.2% 3,233 → 4,029 +24.6% 1 ms Training-heavy (8 GPUs, 16 jobs) 53.6% → 87.0% 8,553 → 17,545 +105.1% 2 ms Large mixed (14 GPUs, 16 jobs) 76.8% → 82.7% 13,977 → 20,101 +43.8% 2 ms Oversubscribed (8 GPUs, 9 jobs) 85.4% → 87.5% 4,311 → 5,760 +33.6% 1 ms Scale test (64 GPUs, 30 jobs) 44.9% → 44.9% 44,233 → 51,248 +15.9% 15 ms Uniform priority (14 GPUs, 16 jobs) 76.8% → 87.5% 25,219 → 31,052 +23.1% 2 ms Utilization improved in every scenario but one, where it tied exactly. Value improved in all seven. The scale test matters because it holds at size: 64 GPUs, 30 jobs, 15 milliseconds, 15.9% more value. The uniform-priority test matters because it addresses the obvious skeptical reading. Override every job to identical priority, so that no priority signal distinguishes any of them, and the allocator still moves utilization from 76.8% to 87.5% and value up 23.1%. The gain is not purely an artifact of ordering by priority. Planning placements across the horizon contributes on its own. None of it works if the demand numbers are wrong Everything above assumes the scheduler knows how many GPU-hours each job needs and how much real-time traffic is coming. Both are predictions, not inputs, and a scheduler is only as good as they are. A single generic estimator does not work, because the four workload types have qualitatively different cost drivers. This is where the specialization argument from the previous piece reconnects: the same logic that makes a task-specific model outperform a generalist applies to the estimators feeding the scheduler. Training is not one workload. It varies along two independent, freely combinable axes. Strategy determines how much of the model is updated (full fine-tuning against parameter-efficient methods like LoRA). Technique determines the optimization objective and the training loop (SFT, DPO, RLHF, RLVR, CPT). The differences are not marginal: LoRA cuts trainable parameters by up to 10,000× and GPU memory roughly 3× against full fine-tuning, on the same base model. DPO removes both the reward model and the sampling loop of RLHF. Estimating from model size alone averages across runs that differ by orders of magnitude, in exactly the two quantities the scheduler decides on, duration and GPU count. Our training forecaster conditions on 22 features, including a categorical variable distinguishing 10 concrete training variants. Quantization is a schedulable job, not a background chore. Quantizing a single large model can consume hours of GPU time on hardware that other work is waiting for. It gets its own forecast, built from calibration tiers by parameter count, with distinct handling per algorithm (bitsandbytes, AWQ, GPTQ) and a safety margin before rounding up to whole GPU-hours. Prior work in this area places quantization outside scheduling scope entirely. Real-time inference is not estimated per job at all. It is forecast as a continuously recalibrated weekly demand profile, rebuilt from hourly traffic history and mapped to GPU counts under the same swap cost the formal model enforces. Forecast and optimizer therefore agree on what churn costs, rather than disagreeing and fighting each other. This forecast is what replaces peak reservation. A per-timestep demand curve is the only thing that lets the scheduler release GPUs during a trough with any confidence that they can be reclaimed before the next peak. This closes back to the ordering argument. Better demand estimates are what make priority-aware placement possible in the first place, you cannot sequence jobs well without knowing what they will consume. Optimize the day, commit the hour The obvious objection to all of this is that forecasts are wrong. What happens then? The failure mode to avoid has a name worth borrowing: the end-of-world effect. An optimizer that cannot see past the end of its horizon makes present decisions that wreck the timesteps immediately outside it, because as far as the model is concerned, nothing exists after the horizon ends. The architecture answers both problems at once. The scheduler optimizes a 24-hour horizon, but commits only the current timestep, and re-runs every 30 to 60 minutes. Run at 9am, and the 9am allocation is real; the plan for 10am through 5pm exists only so that the 9am decision is made by a model that knows a future exists. The real 10am allocation comes from the 10am run, against fresh data. The consequence is that forecast error is absorbed by re-optimization instead of compounding. Each run inherits what is actually running and pins it in place, so successive plans update rather than thrash. There is a secondary payoff. The horizon plan is itself a forecasting product: it surfaces real-time coverage risk and predictable idle windows before they arrive, which is useful whether or not those specific allocations are ever committed. This is also why the time-decay weight exists. By the next run, the workload mix will have changed. What this generalizes to Airlines did not solve utilization by computing an optimal schedule. They solved it by encoding operational discipline into the order things happen (turnaround sequence, maintenance windows, crew rostering) and letting that discipline compound. The same thing happened here. Thirty-three points of utilization in the hardest-packed scenario, and 52% more priority-weighted output on average, on the same hardware, with the same workloads, in a couple of milliseconds, came from encoding what the cluster physically permits into the order decisions get made. Structure beat sophistication. The previous piece argued that specialization and orchestration are two halves of one problem: specialization shrinks what each workload needs, and orchestration decides where the difference goes. Neither lever pays off alone. This is what the second half looks like when it is built. The GPUs were already installed, already committed, already depreciating. The gain was in how we chose to spend them. Further Reading GPU Management: Why Idle GPUs Are the New Grounded Aircraft Newer Models, Same Advantage Why Specialization Is Inevitable Specialization Beats Scale: A Strategic Variable Most AI Procurement Decisions Overlook Text Degeneration: A Production Failure Mode That Most Benchmarks Do Not Track Direct Preference Optimization Beyond Chatbots --- Explore Dharma AI on Hugging Face to try our interactive demos , download our open-source models , and discover how specialized AI systems outperform general-purpose models in real enterprise applications.