At the KubeCon + CloudNativeCon Japan 2026 conference, experts discussed strategies for GPU resource management in Kubernetes environments. Key topics included multi-tenant isolation, quota management, and bin packing, driven by the rising demand for AI compute.
Meta’s Superintelligence Lab is migrating to Kubernetes using Kueue for scheduling. To handle complex topologies on GB300 systems, they implemented multi-layer Topology-Aware Scheduling (TAS) in Kueue v0.17, allowing for granular resource allocation across data halls and racks.
SNOW, a subsidiary of NAVER, optimized its generative filter infrastructure by using HAMi for software-based GPU partitioning, replacing hardware-based MIG. This approach, combined with KEDA-based autoscaling, reduced their GPU requirements by half.
SoftBank manages over 10,000 GPUs by utilizing vCluster for multi-tenancy. They employ three models: private nodes for dedicated access, shared nodes for resource pooling, and a Shared LLM Service with an AI Gateway that tracks usage via token-based metrics.
The llm-d framework is gaining traction for distributed LLM inference, offering features like prefix-aware routing and P/D-disaggregation. LY Corporation reported significant performance gains in multi-turn scenarios by implementing these techniques alongside SLO-aware autoscaling.
