TrainJob integration: determine the mechanism for identifying the PodSetOverride

[mimowo]

What would you like to be cleaned:

Currently the integration relies on the "kueue.x-k8s.io/override-idx" annotation

Alternatives to consider:

1. A dedicated "manager" field in the PodSpecOverrides, issue opened in Extend the PodSetOverrides structure with "manager" field kubeflow/trainer#2856
2. Introspect the manager recorded by Kubertenes in the "ManagedFields" structure

Why is this needed:

Relying on the annotation is not ideal, because the API is hard to reason about.

A clear API field "manager" would nicely indicate. OTOH, it might be duplicating the responsibility of the ManagedFields structure, so we should also consider this as an option.

[mimowo]

cc @tenzen-y @andreyvelich @kaisoz