Karpenter, an autoscaler for Kubernetes clusters, performs node consolidation to optimize costs by repacking pods onto fewer nodes. The process involves two control loops: scaling, which reacts to pending pods, and consolidation, which continuously attempts to minimize the number of active nodes.
Users often observe unexpected node rotation despite stable cluster resource utilization. The consolidation mechanism does not use a utilization threshold; instead, it evaluates whether pods can fit elsewhere once the 'consolidateAfter' timer expires. This leads to continuous node churn when short-lived, resource-intensive jobs interact with long-running pods.
The rotation cycle occurs when Karpenter initiates consolidation on a node, while a new, resource-heavy job requires a fresh node because the terminating node is unavailable. Once the job finishes, Karpenter identifies another underutilized node and repeats the process, creating a continuous loop of node creation and deletion.
Karpenter version 1.14 introduced a 'Balanced' consolidation policy to reduce rotation. Other potential mitigation strategies include implementing utilization thresholds, adjusting node disruption budgets, or improving the scheduler's awareness of recent pod scheduling history to prevent redundant node churn.
