When a GPU or batch job arrives, two questions get tangled together. May this job start, given quota and the other tenants? And once it may start, which nodes do the pods land on, and do they all need to start together?
Kueue and Volcano both sit in front of that workload. Kueue is closer to a bouncer at the door: it decides when a job may enter, then the regular staff, kube-scheduler, places the pods. Volcano replaces the floor staff as well. A group only gets in if minMember of the party can be seated together.
The comparison, including a table, is in this guide: https://justrunme.com/guides/gpu-admission-on-kueue/
What Kueue does
Kueue is a job queueing system for admission and quotas, including fair sharing. It does not replace the scheduler. A job stays suspended until Kueue can reserve quota for it in a ClusterQueue. When quota is available, and any configured AdmissionChecks are Ready, Kueue unsuspends the job. From there, kube-scheduler places the pods.
ClusterQueues in the same cohort can borrow unused nominal quota from each other, with optional borrowingLimit and lendingLimit caps. Resource flavors let a queue describe different shapes of capacity, such as different GPU types, if you define them that way. If a pending workload does not fit in the remaining quota, Kueue can preempt already admitted workloads: by priority inside the same queue (withinClusterQueue), and across the cohort to reclaim borrowed quota (reclaimWithinCohort, optionally borrowWithinCohort).
How Kueue approximates all-or-nothing
The guide table lists gang-like behaviour for Kueue as all-or-nothing via waitForPodsReady. Official Kueue documentation is more precise.
By default, Kueue only unsuspends a workload when it can reserve quota for all of its pod sets at once (Partial Admission is an explicit opt-in). That check is about quota totals. On its own, it does not prove that kube-scheduler can place every pod.
waitForPodsReady is a timeout-based safety net. After admission, Kueue watches until all pods are ready. If that does not happen within the timeout, Kueue evicts the workload, releases the quota, and requeues it. Official docs call this a heuristic: it does not prevent partial scheduling. It stops a half-started job from holding resources indefinitely. blockAdmission can be enabled so workloads are admitted one at a time, which avoids two half-scheduled jobs blocking each other.
Official docs also describe Topology Aware Scheduling (TAS). Without it, admission looks at aggregate quota, so a job that wants 8 GPUs can be admitted when 8 GPUs are free in total, then sit pending if those GPUs sit on two nodes with 4 each. TAS is not in the guide table. I have not run a side-by-side of waitForPodsReady against Volcano gang scheduling, so I am not ranking them.
What Volcano does
Volcano replaces the scheduler. Pods and jobs set schedulerName: volcano. Queueing and placement live in one component.
Gang scheduling is native. A PodGroup has minMember: the minimum number of member pods that must be schedulable before any of them start. If the cluster cannot start that many, Volcano starts none. A Volcano Job can set minAvailable; the scheduler then manages the PodGroup. That decision happens at schedule time, not after a readiness timeout.
Multi-tenant sharing uses Queue objects. capability is a hard upper bound on what the queue may use. weight is used by the proportion plugin to compute a deserved share of the cluster, and official docs treat that share as a soft constraint. The scheduler runs preempt and reclaim as separate actions: preempt is priority-based inside a queue, reclaim pulls a queue's deserved share back from other queues that went over.
Comparison
| Kueue | kube-scheduler | Volcano | |
|---|---|---|---|
| Role | Job queueing, quota enforcement, workload admission | Pod placement, node selection | Replaces the scheduler: queue and gang placement |
| Gang scheduling | All-or-nothing via waitForPodsReady (timeout-based heuristic; see above) |
Needs the coscheduling plugin | Native gang via PodGroup minMember
|
| Multi-tenant quota | ClusterQueue with cohort borrowing, resource flavors | Namespace ResourceQuota only | Queue with capability and weight sharing |
| Preemption | Preempts workloads by priority and to reclaim borrowed quota | Priority-based pod preemption | Preempt and reclaim actions per queue |
| When to use | Multi-tenant clusters with quota and fair sharing, keeping the default scheduler | Default placement for all pods; add the coscheduling plugin for pod-level gang without Volcano | When replacing the scheduler is acceptable |
When to pick which
Pick Kueue when you need multi-tenant quotas, priority queues, cohort borrowing, and you want to keep the default scheduler.
Pick Volcano when you can afford to replace the scheduler, and you want gang placement decided inside the scheduler itself, via PodGroup minMember.
If you only need default placement, stay with kube-scheduler. Add the coscheduling plugin if you want pod-level gang and you do not want to run Volcano. I have not compared that plugin's PodGroup behaviour to Volcano's minMember, so I am not calling them equivalent. ResourceQuota can cap a namespace. It will not give you workload admission or cohort borrowing.
One limit remains after you pick Kueue. ClusterQueue answers whether there is capacity. Whether this team may run this GPU flavor, from this registry, with these FinOps labels, is a separate policy check: a Kueue AdmissionCheck, covered in the rest of the guide. That gap is not a reason to choose Volcano.
I am not attaching benchmarks. How these systems behave under contention depends on the cluster, the plugins you enable, and how you set timeouts and preemption.
Top comments (0)