external-dns unable to generate DNS entries for HTTPRoutes

[timothy-spencer]

Bug Description
If you enable external-dns's gateway sources, it is supposed to look at the HTTPRoute for hostnames and then create the DNS records pointing at the parent gateway. However, it appears as if the gateway parent information is not put into the HTTPRoute, nor is there any information in the Gateway about whether it has accepted the route. The code has a lot going on, but it appears that there is a lot of checking that it does to make sure that things are actually set up properly, and since it fails that, it will not generate DNS entries.

I believe that this is a bug, because I can query the ALB directly, and the HTTPRoute is sending me to the backend and I'm getting the proper content from the app, but this working relationship is not reflected in the status of the HTTPRoute and Gateway resources, and thus external-dns is not able to do it's job.

Then again, if I configured something wrong, please let me know!

Steps to Reproduce

• You should be able to reproduce this by:
  • Setting up external-dns to look for HTTPRoutes.
  • Creating a Gateway and HTTPRoute with hostnames, and probably a backend service of some sort so you can see that you can curl into it.

# Look for CHANGEME in here for stuff that probably needs changing
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gitlab-alb-gw
spec:
  gatewayClassName: alb-gateway
  infrastructure:
    parametersRef:
      group: gateway.k8s.aws
      kind: LoadBalancerConfiguration
      name: gitlab-http-config
  listeners:
  - protocol: HTTPS
    # patch this
    hostname: CHANGEME.DOMAIN
    port: 443
    name: https
    allowedRoutes:
      kinds:
      - kind: HTTPRoute

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: gitlab
spec:
  parentRefs:
  - name: gitlab-alb-gw
    sectionName: https
  hostnames:
    # patch this
    - CHANGEME.DOMAIN
  rules:
  - backendRefs:
    - name: gitlab-gw
      kind: Service
      port: 4430

# XXX From here on is how we get this into the kuma service mesh.
# XXX You would probably want to just make a regular http service instead
---
apiVersion: kuma.io/v1alpha1
kind: MeshGatewayInstance
metadata:
  name: gitlab-gw
  labels:
    kuma.io/mesh: gitlab
spec:
  replicas: 2
  serviceType: LoadBalancer

---
apiVersion: kuma.io/v1alpha1
kind: MeshGateway
mesh: gitlab
metadata:
  name: gitlab-gw
  labels:
    gateway: gitlab-gw
spec:
  selectors:
    - match:
        # patch this
        kuma.io/service: gitlab-gw_NAMESPACE_svc
  conf:
    listeners:
      - port: 4430
        protocol: HTTP
        # patch this
        hostname: CHANGEME
        tags:
          port: http-4430

---
apiVersion: kuma.io/v1alpha1
kind: MeshService
metadata:
  name: webservice
  labels:
    kuma.io/mesh: gitlab
spec:
  selector:
    dataplaneTags:
      app: webservice
  ports:
  - name: webservice
    port: 8080
    targetPort: 8080
    appProtocol: http

• Witnessing external-dns not creating DNS entries.
• If you turn on debug in external-dns, you can see it saying it can't find any hostnames in the HTTPRoute.
• Controller logs/error messages while reproducing the issue:
  • Just this, from external-dns debug logs: time="2025-06-25T23:24:31Z" level=debug msg="No endpoints could be generated from HTTPRoute gitlab2/gitlab"

Expected Behavior
I expected external-dns to be able to see useful status information in the Gateway and HTTPRoute resources so that it can generate DNS entries. You can see some example status in an issue that I found which talks about the importance of this information for their stuff to work.

Actual Behavior
When you turn debug on in external-dns, it just emits this line when trying to find hostnames:

time="2025-06-25T21:53:08Z" level=debug msg="No endpoints could be generated from HTTPRoute gitlab2/gitlab"
time="2025-06-25T21:53:08Z" level=debug msg="No endpoints could be generated from TCPRoute gitlab2/gitlab-shell"

And no DNS entries are created.

Regression
Was the functionality working correctly in a previous version ? [No]

Current Workarounds
No workarounds available

Environment

• AWS Load Balancer controller version: 2.13.2
• Kubernetes version: 1.33
• Using EKS (yes/no), if so version?: Yes, 1.33
• Using Service or Ingress: Gateway API
• AWS region: us-west-2
• How was the aws-load-balancer-controller installed: With argocd's helm integration, with this values file:

createNamespace: true
enableCertManager: true
priorityClassName: system-cluster-critical
serviceAccount:
  create: true
  name: aws-load-balancer-controller-sa
podDisruptionBudget:
  maxUnavailable: "10%"
securityContext:
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault
tolerations:
  - key: "CriticalAddonsOnly"
    operator: "Exists"
  - key: ondemand
    operator: Equal
    value: "true"
    effect: NoSchedule
resources:
  requests:
    cpu: 10m
    memory: 100Mi
  limits:
    memory: 100Mi
controllerConfig:
  featureGates:
    NLBGatewayAPI: true
    ALBGatewayAPI: true

• Current state of the Controller configuration:

Name:                   aws-load-balancer-controller
Namespace:              kube-system
CreationTimestamp:      Fri, 07 Jun 2024 16:34:23 -0700
Labels:                 app.kubernetes.io/instance=aws-load-balancer-controller
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=aws-load-balancer-controller
                        app.kubernetes.io/version=v2.13.2
                        argocd.argoproj.io/instance=aws-load-balancer-controller
                        helm.sh/chart=aws-load-balancer-controller-1.13.2
Annotations:            deployment.kubernetes.io/revision: 16
Selector:               app.kubernetes.io/instance=aws-load-balancer-controller,app.kubernetes.io/name=aws-load-balancer-controller
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=aws-load-balancer-controller
                    app.kubernetes.io/name=aws-load-balancer-controller
  Annotations:      kubectl.kubernetes.io/restartedAt: 2025-06-11T21:48:49Z
                    prometheus.io/port: 8080
                    prometheus.io/scrape: true
  Service Account:  aws-load-balancer-controller-sa
  Containers:
   aws-load-balancer-controller:
    Image:           public.ecr.aws/eks/aws-load-balancer-controller:v2.13.2
    Ports:           9443/TCP, 8080/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      --cluster-name=tspencertest
      --ingress-class=alb
      --aws-region=us-west-2
      --aws-vpc-id=vpc-08291ee586de1ce4e
      --feature-gates=ALBGatewayAPI=true,NLBGatewayAPI=true
    Limits:
      memory:  100Mi
    Requests:
      cpu:        10m
      memory:     100Mi
    Liveness:     http-get http://:61779/healthz delay=30s timeout=10s period=10s #success=1 #failure=2
    Readiness:    http-get http://:61779/readyz delay=10s timeout=10s period=10s #success=1 #failure=2
    Environment:  <none>
    Mounts:
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
  Volumes:
   cert:
    Type:               Secret (a volume populated by a Secret)
    SecretName:         aws-load-balancer-tls
    Optional:           false
  Priority Class Name:  system-cluster-critical
  Node-Selectors:       <none>
  Tolerations:          CriticalAddonsOnly op=Exists
                        ondemand=true:NoSchedule
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  aws-load-balancer-controller-58c96b86df (0/0 replicas created), aws-load-balancer-controller-79bc9f8bd6 (0/0 replicas created), aws-load-balancer-controller-7576bdcf7b (0/0 replicas created), aws-load-balancer-controller-577c6cc4df (0/0 replicas created), aws-load-balancer-controller-6dbcc88dd7 (0/0 replicas created), aws-load-balancer-controller-796c895667 (0/0 replicas created), aws-load-balancer-controller-78589869bf (0/0 replicas created), aws-load-balancer-controller-74f66c9c6f (0/0 replicas created), aws-load-balancer-controller-5b8f876894 (0/0 replicas created), aws-load-balancer-controller-858485f84b (0/0 replicas created)
NewReplicaSet:   aws-load-balancer-controller-8f779fccc (2/2 replicas created)
Events:          <none>

• Current state of the Gateway/HTTPRoute configuration:

Name:         gitlab-alb-gw
Namespace:    gitlab2
Labels:       argocd.argoproj.io/instance=gitlab2-extras
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2025-06-24T19:11:49Z
  Finalizers:
    gateway.k8s.aws/alb
  Generation:        3
  Resource Version:  419928220
  UID:               5ac76c2d-2f65-42c0-ad5f-18939c26cc6a
Spec:
  Gateway Class Name:  alb-gateway
  Infrastructure:
    Parameters Ref:
      Group:  gateway.k8s.aws
      Kind:   LoadBalancerConfiguration
      Name:   gitlab-http-config
  Listeners:
    Allowed Routes:
      Kinds:
        Group:  gateway.networking.k8s.io
        Kind:   HTTPRoute
      Namespaces:
        From:  Same
    Hostname:  gitlab.tspencertest.identitysandbox.gov
    Name:      https
    Port:      443
    Protocol:  HTTPS
Status:
  Addresses:
    Type:   Hostname
    Value:  k8s-gitlab2-gitlabal-ef09106154-242666390.us-west-2.elb.amazonaws.com
  Conditions:
    Last Transition Time:  1970-01-01T00:00:00Z
    Message:               Waiting for controller
    Reason:                Pending
    Status:                Unknown
    Type:                  Accepted
    Last Transition Time:  1970-01-01T00:00:00Z
    Message:               Waiting for controller
    Reason:                Pending
    Status:                Unknown
    Type:                  Programmed
Events:
  Type    Reason                  Age                  From                 Message
  ----    ------                  ----                 ----                 -------
  Normal  SuccessfullyReconciled  46m (x17 over 6h5m)  gateway.k8s.aws/alb  Successfully reconciled

Name:         gitlab
Namespace:    gitlab2
Labels:       argocd.argoproj.io/instance=gitlab2-extras
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
Metadata:
  Creation Timestamp:  2025-06-25T21:24:58Z
  Generation:          2
  Resource Version:    421385619
  UID:                 c5760d52-8dfe-44db-9abf-70bffca9364c
Spec:
  Hostnames:
    gitlab.tspencertest.identitysandbox.gov
  Parent Refs:
    Group:         gateway.networking.k8s.io
    Kind:          Gateway
    Name:          gitlab-alb-gw
    Section Name:  https
  Rules:
    Backend Refs:
      Group:   
      Kind:    Service
      Name:    gitlab-gw
      Port:    4430
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /
Status:
  Parents:
Events:  <none>

Possible Solution (Optional)
I believe that if we get useful status information in the Gateway and HTTPRoute resources that external-dns will be able to generate DNS entries. You can see some example status in an issue that I found which talks about the importance of this information for their stuff to work.

It is also possible that I have something misconfigured, and thus the lack of status information is correct. But I don't think that is true, because if I curl into my endpoint, it gets data from the backend app. I would imagine that if this was misconfigured that the route wouldn't work, and I would get a 404 or something.

Contribution Intention (Optional)

• Yes, I'm willing to submit a PR to fix this issue
• No, I cannot work on a PR at this time

This seems like a fix that somebody unfamiliar with the codebase would have a hard time doing. If it were small, I'd try to do it, but I just don't think I have the time.

[timothy-spencer]

I just noticed that there was a new version of the lb controller, so I upgraded, nuked all of my resources (there was some sort of finalizer problem, but I patched the finalizers away and deleted the ALB/NLBs by hand), and started over, and there are some sort of errors with SA permissions right now, so it might be that there is something wrong with my setup, despite it working before.

I'll try to figure this out and let you know what I find. Thanks!

[timothy-spencer]

This is probably a different problem, but the latest helm chart seems to have removed the ability for it to patch loadbalancerconfigurations in the role.

{"level":"error","ts":"2025-06-26T18:16:15Z","msg":"Reconciler error","controller":"aws-lbc-loadbalancerconfiguration-controller","object":{"name":"gitlab-http-config","namespace":"gitlab2"},"namespace":"gitlab2","name":"gitlab-http-config","reconcileID":"87d46899-913e-4c3a-8471-9c1866e4b65a","error":"loadbalancerconfigurations.gateway.k8s.aws \"gitlab-http-config\" is forbidden: User \"system:serviceaccount:kube-system:aws-load-balancer-controller-sa\" cannot patch resource \"loadbalancerconfigurations\" in API group \"gateway.k8s.aws\" in the namespace \"gitlab2\""}

[zac-nixon]

Hey! Thanks for trying out the Gateway API. Sorry it didn't work as intended. I believe we have fixed the status publication if the latest release.

Example from my own cluster:

  status:
    parents:
    - conditions:
      - lastTransitionTime: "2025-06-24T00:02:20Z"
        message: Route is accepted by Gateway
        observedGeneration: 1
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2025-06-24T00:02:20Z"
        message: ""
        observedGeneration: 1
        reason: Accepted
        status: "True"
        type: ResolvedRefs
      controllerName: gateway.k8s.aws/alb
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: test-gw-alb
        port: 80

I think the finalizer issue should also be fixed in v2.13.3.

Let me investigate these missing permissions, again sorry for the trouble!

[timothy-spencer]

Thanks for your response! FYI, I created #4248 to try to fix the RBAC.

[timothy-spencer]

All right! The latest version (2.13.3) seems to have fixed the problem, once I patched the RBAC problems that are fixed in #4248. I see status, and external-dns says it's creating DNS entries for me. Sorry for not noticing that I wasn't on the latest version right off the bat!

Closing!