[RFC-0011] Extend the tracing logic to advanced use cases

[adri1197]

Part of: #5321

Recently was introduced a new provider type, otel, in the notification-controller. The initial focus was on ensuring support and performing thorough testing across three main use cases:

• <source> -> Kustomization -> KRM (Kubernetes resource manifests).
• OCIRepository -> multiple HelmReleases.
• HelmRepository -> HelmChart (treated as the actual source) -> HelmRelease.

That said, there’s another use case that seems quite common: <source> -> Kustomization -> <any-resource> (including FluxCD’s own resources). Right now, this scenario can lead to separate traces being generated per source applied, depending on what the Kustomization ends up reconciling.

The assumption is that there will always be a source and a Kustomization in the chain. The Kustomization acts as the orchestrator: it decides what gets applied and already carries the top-level source revision as part of its metadata. This opens the door for a neat solution: by appending a commonAnnotation with the source revision (using the event convention event.toolkit.fluxcd.io/), that annotation could automatically propagate to all applied resources. In turn, notification-controller could then pick this up when building spans and linking traces across resources.

Of course, adding/appending the annotation could be something configurable via flags, disabled by default.

I’ve sketched a tentative implementation for this across the following controllers:

• kustomize-controller - internal/controller/kustomization_controller.go.
• notification-controller - internal/notifier/otel.go.

Here's how it may look like with the following scenario a single trace (under the same rootSpan/traceID): GitRepository -> Kustomization -> OCIRepository > collection of HelmRelease.