Sharing Kubernetes GPU Infrastructure Across Isolated Tenant Clusters Using KAI Scheduler and vCluster

Sharing Kubernetes GPU Infrastructure Across Isolated Tenant Clusters Using KAI Scheduler and vCluster Organizations often struggle with the balance between cluster isolation and resource sharing across teams. By pairing vCluster with KAI Scheduler, teams can operate dedicated virtual Kubernetes control planes while dynamically sharing underlying physical GPU hardware with defined quotas. Running separate physical clusters per team creates high isolation and resource fragmentation, while multi-tenant single clusters trigger CRD and RBAC conflicts. vCluster provides each team with an isolated virtual control plane and admin autonomy without requiring dedicated hardware nodes. KAI Scheduler manages dynamic, fractional GPU allocations and enforces queue hierarchies with guaranteed quotas and over-quota borrowing. KAI Scheduler handles allocation rather than hardware-level memory boundaries, meaning memory isolation requires application-level limits or hardware features like NVIDIA MIG. Architecture and Core Components The pattern combines vCluster for control plane virtualization with KAI Scheduler for GPU resource management on a shared host cluster. vCluster isolates API servers, custom resource definitions (CRDs), and RBAC per team while using a shared-nodes model. KAI Scheduler operates alongside kube-scheduler to handle topology-aware GPU scheduling and fractional resource assignments for pods targeting it directly. Queue Hierarchy and Workload Deployment Using KAI Scheduler's Queue CRD, administrators define hierarchical budgets such as a parent organization queue and child team queues. Each team queue can specify a guaranteed quota (such as 0.33 of a GPU), upper limits, and weights for bursting into idle capacity. Workloads deployed inside individual vClusters target their team queue via pod labels and annotations, running on shared physical nodes while remaining visible only inside the tenant's virtual cluster. Hardware Isolation and Limitations Under GPU sharing, the GPU time-slices CUDA contexts across pods at kernel boundaries. Because KAI Scheduler governs pod placement and fractional share accounting rather than hardware memory partitioning, applications must manage their own memory limits, or administrators must use hardware-level isolation mechanisms like NVIDIA Multi-Instance GPU (MIG).

利用 KAI Scheduler 与 vCluster 在共享 GPU 架构上运行隔离的多租户 Kubernetes 集群 针对多团队共享 Kubernetes 基础设施时面临的 CRD 冲突、权限重叠与 GPU 配额划分难题，本文介绍了结合开源工具 KAI Scheduler 与 vCluster 的解决方案，在保持底层硬件共享的同时为各团队提供独立的虚拟控制平面。 为每个团队提供专用的 vCluster 虚拟控制平面，具备独立的 API Server、CRD 和 RBAC，避免跨团队环境冲突。 采用 KAI Scheduler 实现拓扑感知与层级队列调度，支持按比例（如 0.33 GPU）切分并动态分配物理 GPU 资源。 演示环境基于单张 NVIDIA L40S GPU，成功支持 NLP、视觉与推荐系统三个团队的工作负载并行运行且互不可见。 KAI Scheduler 的 GPU 共享主要通过时间分片调度资源，应用层需自行控制显存；若需硬件级强显存隔离可配合 NVIDIA MIG 使用。 架构设计与组件分工 该方案基于单个底层集群和共享 GPU 节点构建。vCluster 负责为每个内部团队提供一套独立的控制平面，租户可拥有完整的 cluster-admin 权限并自由管理 CRD，而无需拆分底层服务器；KAI Scheduler 则作为专为 AI 负载设计的调度器，与默认调度器并行运行，通过自定义 Queue 资源按层级定义各团队的最低保障配额（quota）、上限（limit）和溢出权重。 环境部署与工作负载下发 在搭载 MicroK8s 和 NVIDIA GPU Operator 的单卡节点上，部署开启了 gpuSharing 的 KAI Scheduler，并配置组织与子团队的层级队列。创建 vCluster 时禁用属主重写（setOwner: false），以便调度器识别真实的 Pod 属主链。各团队连接自身虚拟集群后，通过指定 kai-scheduler 和 gpu-fraction 注解提交 Pod，底层 GPU 即可实现 330m 等比例的准确切分。 调度隔离特性与硬件限制 实际运行中，不同租户的 Pod 分布在宿主机的独立命名空间中，团队内部仅能查看自身的工作负载。需要说明的是，KAI Scheduler 在软件层面协调 Pod 的 GPU 比例分配，底层由 CUDA 在内核边界通过时间分片切换上下文，并不在硬件层面限制显存用量，租户应用须自行配置内存上限；对于需要硬性隔离的场景，可结合 NVIDIA MIG 硬件切分技术。

Running a dedicated Kubernetes cluster per team often results in more isolation than an organization requires. While one cluster can be successfully shared across many teams, the coordination costs increase as the number of teams grows. Challenges include conflicting CRD versions, overlapping RBAC, and no clean way to carve GPU capacity into team-level budgets. At a certain scale, teams might start asking for their own clusters just to regain autonomy. This post provides a pattern that preserves team autonomy without splitting the hardware. This solution involves a single control plane cluster with a GPU pool, GPU sharing with per-team quotas, and isolated Kubernetes control plane per team including an API server, controller, data store, syncer, and scheduler. This can be achieved using two open source tools: KAI Scheduler and vCluster . Follow along with the tutorial steps and you’ll have three teams running real GPU Kubernetes pods in their own tenant clusters, all sharing a single physical GPU. You’ll also be able to verify that each team only sees their own workloads. Tutorial prerequisites and notes To keep the process reproducible for users that have limited resources, this tutorial uses a cluster with one NVIDIA L40S GPU and three teams sharing fractions of it. This makes the moving parts easy to see and try. The process works the same way on a larger cluster with hundreds of GPU nodes and dozens of teams. You can scale the node pool, the queue hierarchy, and the number of tenant clusters. KAI Scheduler is a robust, efficient, and scalable topology-aware Kubernetes scheduler that was purpose-built for optimizing GPU resource allocation for AI workloads. It’s designed to manage large-scale GPU clusters, including thousands of nodes, and a high throughput of workloads. With KAI Scheduler, you can dynamically allocate GPU resources to workloads. It can run alongside the default kube-scheduler. Any pod with schedulerName: kai-scheduler is handled by KAI Scheduler. Everything else goes through the usual kube-scheduler process. The vCluster Kubernetes platform provisions fully isolated tenant clusters on your infrastructure or directly on bare metal. Each tenant cluster gets its own API server, custom resource definitions (CRDs), and role-based access control (RBAC), indistinguishable from a dedicated Kubernetes cluster, while sharing the underlying nodes and hardware. The virtualized control plane is invisible to tenants: no shared control plane nodes, no in-cluster agent pods, and no lateral path between environments. That makes vCluster a natural fit for GPU infrastructure where teams need their own clean cluster experience without splitting the hardware. The vCluster shared-nodes model is used for this tutorial, so teams share the GPU node while each gets its own isolated control plane, the right fit for trusted internal teams. For untrusted tenants needing node-, network-, and storage-level separation, the same pattern extends to vCluster private nodes. The example in this post uses three teams: NLP Team, Vision Team, and Recommender System Team. The NLP Team wants to install their own CRDs. The Vision Team wants cluster-admin to debug scheduling. The Recommender Team is on a different Kubeflow version. Nobody wants to share a kubectl context and accidentally break one another’s environments. Using vCluster, each team gets their own isolated control plane, RBAC, namespaces, and CRDs. They can each have cluster-admin access. Underneath, all tenant clusters share the same nodes and GPUs. Demo environment This demo runs on an NVIDIA Brev GPU instance on Nebius with: One NVIDIA L40S, 40 vCPUs, 160 GiB RAM, 256 GiB disk (48 GB VRAM) Ubuntu 24.04.4 LTS MicroK8s v1.36.2 – Kubernetes was preconfigured by Brev, including the MicroK8s gpu addon, which pre-installs the NVIDIA GPU Operator into the gpu-operator-resources namespace KAI Scheduler v0.16.4 vCluster CLI 0.35.1 Note: For different setups, the cluster-creation and GPU Operator install steps will differ on GKE/EKS/AKS/vanilla k8s/k3s. Step 3 (KAI Scheduler) onward is identical on any Kubernetes that has the NVIDIA GPU Operator running with Container Device Interface (CDI) enabled. Figure 1. KAI Scheduler and vCluster architecture on a shared Nebius host cluster Step 1: Local tooling Install standalone kubectl and helm to avoid prefixing every command with microk8s . Then wire up a kubeconfig and pin MicroK8s so snap doesn’t auto-upgrade the control plane during the demo. sudo snap refresh --hold microk8s sudo snap install kubectl --classic --channel=1.35/stable sudo snap install helm --classic mkdir -p ~/.kube sudo microk8s config > ~/.kube/config sudo chown $USER:$USER ~/.kube/config chmod 600 ~/.kube/config Next, make sure the required MicroK8s addons are enabled: microk8s enable dns microk8s enable hostpath-storage # vCluster needs PVCs Then verify: kubectl get nodes -o wide kubectl get storageclass NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME brev-8dq0cch1j Ready <none> 57d v1.36.2 10.0.0.20 <none> Ubuntu 24.04.4 LTS 6.11.0-1016-nvidia (amd64) containerd://2.2.3 NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE microk8s-hostpath (default) microk8s.io/hostpath Delete WaitForFirstConsumer false 20m Step 2: Add the Helm repo Add the NVIDIA Helm repo: helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update Step 3: Confirm the GPU Operator kubectl get pods -n gpu-operator-resources NAME READY STATUS RESTARTS AGE gpu-feature-discovery-wpcjm 1/1 Running 0 8m gpu-operator-57d75775c8-npjzz 1/1 Running 0 9m gpu-operator-node-feature-discovery-... 1/1 Running 0 9m nvidia-container-toolkit-daemonset-ft4hb 1/1 Running 0 8m nvidia-cuda-validator-8vqdv 0/1 Completed 0 8m nvidia-device-plugin-daemonset-l6fj6 1/1 Running 0 8m nvidia-operator-validator-cdnmj 1/1 Running 0 8m If you’re using an older GPU Operator, upgrade in place to 26.3.x: helm upgrade gpu-operator nvidia/gpu-operator \ -n gpu-operator-resources \ --version v26.3.3 \ --reset-then-reuse-values If needed, install NVIDIA GPU Operator directly: helm install gpu-operator nvidia/gpu-operator \ -n gpu-operator-resources --create-namespace \ --version v26.3.3 \ --set driver.enabled=false \ --set operator.defaultRuntime=containerd \ --set toolkit.env[0].name=CONTAINERD_CONFIG \ --set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd.toml \ --set toolkit.env[1].name=CONTAINERD_SOCKET \ --set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \ --set-string toolkit.env[2].name=CONTAINERD_SET_AS_DEFAULT \ --set-string toolkit.env[2].value=1 Step 4: Install KAI Scheduler Next, install the KAI Scheduler: helm upgrade -i kai-scheduler \ oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \ -n kai-scheduler --create-namespace \ --version v0.16.4 \ --set "global.gpuSharing=true" Then verify: kubectl get pods -n kai-scheduler NAME READY STATUS RESTARTS AGE admission-57556f949-sp98t 1/1 Running 0 47s binder-66785d8dd9-9frgk 1/1 Running 0 46s kai-operator-6fdf595c4d-292d7 1/1 Running 0 52s kai-scheduler-default-6bb667b767-vq2q4 1/1 Running 0 46s pod-grouper-84dfc7759b-v5qtb 1/1 Running 0 47s podgroup-controller-5878f48dbb-v7wcn 1/1 Running 0 47s queue-controller-7796bb8984-hdn5r 1/1 Running 0 46s Step 5: Define team queues KAI Scheduler uses a Queue CRD to model an org → team hierarchy. This step involves creating one parent ( ml-org , with a total budget of one GPU) and three child queues. Each is guaranteed 0.33 of the GPU and allowed to increase to the full GPU when others are idle. Save the following as create-queues.yaml : apiVersion: scheduling.run.ai/v2 kind: Queue metadata: name: ml-org spec: resources: gpu: { quota: 1, limit: -1, overQuotaWeight: 1 } --- apiVersion: scheduling.run.ai/v2 kind: Queue metadata: name: team-nlp spec: parentQueue: ml-org priority: 100 resources: gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 } --- apiVersion: scheduling.run.ai/v2 kind: Queue metadata: name: team-vision spec: parentQueue: ml-org priority: 100 resources: gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 } --- apiVersion: scheduling.run.ai/v2 kind: Queue metadata: name: team-recommender spec: parentQueue: ml-org priority: 100 resources: gpu: { quota: 0.33, limit: 1, overQuotaWeight: 1 } Here, quota is the guaranteed minimum, limit is the maximum allowed, and overQuotaWeight controls how surplus is split. Next, apply and list: kubectl apply -f create-queues.yaml kubectl get queues queue.scheduling.run.ai/ml-org created queue.scheduling.run.ai/team-nlp created queue.scheduling.run.ai/team-vision created queue.scheduling.run.ai/team-recommender created NAME PRIORITY PARENT CHILDREN DISPLAYNAME default-parent-queue ["default-queue"] default-queue default-parent-queue ml-org ["team-nlp","team-vision","team-recommender"] team-nlp 100 ml-org team-recommender 100 ml-org team-vision 100 ml-org Note that default-parent-queue and default-queue are created automatically by KAI Scheduler on first install. They are the fallback queue for any pod that doesn’t specify one. Step 6: Spin up a vCluster per team Next, install the vCluster CLI : curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" sudo install -m 755 vcluster /usr/local/bin/vcluster rm vcluster vcluster --version vcluster version 0.35.1 Define the vCluster config. The critical setting is setOwner: false . The KAI Scheduler pod-grouper walks ownership chains (Job → Pod, Deployment → ReplicaSet → Pod) to auto-group workloads. Disabling vCluster owner rewriting allows KAI Scheduler to see the real hierarchy. cat > vcluster.yaml <<'EOF' experimental: syncSettings: setOwner: false sync: fromHost: nodes: enabled: true selector: all: true EOF Then create one vCluster per team: vcluster create team-nlp --values vcluster.yaml --connect=false vcluster create team-vision --values vcluster.yaml --connect=false vcluster create team-recommender --values vcluster.yaml --connect=false Verify that all three vClusters are running: vcluster list NAME | NAMESPACE | STATUS | VERSION | CONNECTED | AGE -------------------+---------------------------+---------+---------+-----------+------- team-nlp | vcluster-team-nlp | Running | 0.35.1 | | 102s team-recommender | vcluster-team-recommender | Running | 0.35.1 | | 88s team-vision | vcluster-team-vision | Running | 0.35.1 | | 93s Each team sees the real nodes, including the GPU node: vcluster connect team-nlp -- kubectl get nodes 19:09:22 done vCluster is up and running NAME STATUS ROLES AGE VERSION brev-8dq0cch1j Ready <none> 6m20s v1.36.2 Step 7: Deploy a workload from each team Each team deploys their GPU workload from their own vCluster. The pod spec is simple, with three fields telling KAI Scheduler what to do: vcluster connect team-nlp -- kubectl apply -f - <<'EOF' apiVersion: v1 kind: Pod metadata: name: nlp-sentiment-model labels: kai.scheduler/queue: team-nlp # Which team annotations: gpu-fraction: "0.33" # How much GPU spec: schedulerName: kai-scheduler. # Use KAI, not default tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule containers: - name: nlp-inference image: nvidia/cuda:12.4.0-base-ubuntu22.04 command: ["bash", "-c", "nvidia-smi; sleep infinity"] nodeSelector: nvidia.com/gpu.present: "true" EOF 19:13:34 done vCluster is up and running pod/nlp-sentiment-model created Do the same to deploy from each team, changing the queue name. Now, verify: NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE vcluster-team-nlp nlp-sentiment-model-x-default-x-team-nlp 1/1 Running 0 110s 10.1.171.180 brev-8dq0cch1j vcluster-team-recommender recommender-model-x-default-x-team-recommender 1/1 Running 0 6s 10.1.171.187 brev-8dq0cch1j vcluster-team-vision vision-classifier-model-x-default-x-team-vision 1/1 Running 0 14s 10.1.171.145 brev-8dq0cch1j Three pods, three different vcluster-team-* namespaces, all running on the same physical node brev-8dq0cch1j . Each team sees only their own pod from inside their vCluster: vcluster connect team-nlp -- kubectl get pods -o wide vcluster connect team-vision -- kubectl get pods -o wide vcluster connect team-recommender -- kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nlp-sentiment-model 1/1 Running 0 3m11s 10.1.171.180 brev-8dq0cch1j <none> <none> NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES vision-classifier-model 1/1 Running 0 3m59s 10.1.171.145 brev-8dq0cch1j <none> <none> NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES recommender-model 1/1 Running 0 2m45s 10.1.171.187 brev-8dq0cch1j <none> <none> Finally, the queue confirms the allocation. You can check all three at the same time: kubectl describe queue team-nlp | grep -A4 Status kubectl describe queue team-vision | grep -A4 Status kubectl describe queue team-recommender | grep -A4 Status You will see the same result for all: Status: Allocated: nvidia.com/gpu: 330m Requested: nvidia.com/gpu: 330m Note that KAI Scheduler handles scheduling—which pods land on which GPU and in what proportion. It does not enforce GPU memory isolation at the hardware level when GPU sharing is used. Applications need to respect their memory amount (for example, setting –gpu-memory-utilization in vLLM). Under the hood, the GPU time-slices between CUDA contexts from each pod at kernel boundaries. For hard memory isolation on supported hardware, NVIDIA Multi-Instance GPU (MIG) provides hardware-level partitioning, which can be scheduled by KAI Scheduler as well. Get started running isolated tenant Kubernetes clusters KAI Scheduler decides fairly who gets GPU slices and schedules them as a group using GPU sharing and DRA Driver support, hierarchical queues with guaranteed quotas plus over-quota capabilities, gang scheduling, and topology awareness. Once AI workload scheduling is solved, teams want their own clusters. vCluster gives each team its own isolated Kubernetes control plane without the cost of separate infrastructure. Together, KAI Scheduler and vCluster provide a dedicated cluster experience for three teams with zero waste on a single GPU. The answer isn’t always more GPUs, but better utilization of infrastructure. Ready to get started? Check out KAI Scheduler , vCluster , and NVIDIA GPU Operator on GitHub. Learn more about KAI Scheduler and vCluster integration at KubeCon 2026 North America , November 9-12.