Clusterqueues must prefer borrowing within a cohort before borrowing across cohorts.

[varunsyal]

What happened:

When hierarchical cohorts are used with multiple flavors, a clusterqueue should prefer scheduling in the flavor such that borrowing first happens from other clusterqueues within its parent cohort, before borrowing from clusterqueues of other cohorts. However that does not happen with multiple flavors.
Due to this, even when the cohort has nominal quota in the second flavor, it tends to borrow in the first flavor, making the workloads prone to preemption.

What you expected to happen:

If a cohort's usage is below its nominal quota, it should not try to borrow from another cohort in case of multiple flavors. This leads to unnecessary preemptions when the other cohort needs to reclaim it nominal quota

How to reproduce it (as minimally and precisely as possible):

Example:
Total Size of Cluster:

• Flavor-1 = 10
• Flavor-2 = 10

If we submit 18 workloads each using resources = 1 to clusterqueue cq-p1, we observe that it schedules 10 workloads in Flavor-1 and 8 workloads in Flavor-2. This makes the cohort-A which has a total nominal quota of 18 in its subtree to be borrowing 2 resources from cohort-B, and thus has a positive fair sharing weighted share, even though its total usage is within its total nominal quota.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.32.7
• Kueue version (use git describe --tags --dirty --always): 0.13.4
• Feature Flag enabled: FlavorFungibilityImplicitPreferenceDefault=True

[amy]

/cc

[gabesaba]

kueue/pkg/scheduler/flavorassigner/flavorassigner.go

Line 712 in 50ae954

mode := granularMode{preemptionMode, borrow > 0}

I think that the key is this line. We need to not treat this as a boolean, but as an integer, if we want to find a better assignment in hierarchical case