Skip to content

l7 documentation, correct some warnings in other docs #4352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

l7 documentation, correct some warnings in other docs #4352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
07771eb
l7 documentation, correct some warnings in other docs
zac-nixon Sep 25, 2025
8fd066d
update examples to L7 gateway
zac-nixon Sep 26, 2025
e0979ec
add invalid backend behavior
zac-nixon Sep 26, 2025
cc753db
add missing fields in lb config documentation
zac-nixon Sep 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 53 additions & 53 deletions docs/deploy/configurations.md
View file
Open in desktop

Large diffs are not rendered by default.

143 changes: 137 additions & 6 deletions docs/guide/gateway/customization.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Customizing your ELB resources

The AWS Load Balancer Controller (LBC) provides sensible defaults for provisioning and managing Elastic Load Balancing (ELB) resources in response to Kubernetes Gateway API objects. However, to accommodate diverse use cases and specific operational requirements, the LBC offers extensive, fine-grained customization capabilities through two Custom Resource Definitions (CRDs): [LoadBalancerConfiguration](../spec/#loadbalancerconfiguration) and [TargetGroupConfiguration](../spec/#targetgroupconfiguration).
The AWS Load Balancer Controller (LBC) provides sensible defaults for provisioning and managing Elastic Load Balancing (ELB) resources in response to Kubernetes Gateway API objects. However, to accommodate diverse use cases and specific operational requirements, the LBC offers extensive, fine-grained customization capabilities through three Custom Resource Definitions (CRDs): [LoadBalancerConfiguration](./spec.md#loadbalancerconfiguration), [TargetGroupConfiguration](./spec.md/#targetgroupconfiguration), and [ListenerRuleConfiguration](./spec.md#listenerruleconfiguration).

![screen showing all LBC Gateway API components](assets/gateway-full.png)

Expand Down Expand Up @@ -28,7 +28,7 @@ This configuration can then be applied by attaching the `LoadBalancerConfigurati
When attached directly to a `Gateway` resource, the specified configuration applies specifically to the Load Balancer provisioned for that individual Gateway.

!!! note
Make sure that the `LoadBalancerConfiguration` must be in same namepace as the `Gateway`.
Make sure that the `LoadBalancerConfiguration` is located in the same namespace as the `Gateway`.

```yaml
apiVersion: gateway.networking.k8s.io/v1
Expand Down Expand Up @@ -66,7 +66,7 @@ spec:

#### Conflict Resolution for `LoadBalancerConfiguration`

It is possible for a `LoadBalancerConfiguration` to be attached to both a `Gateway` and its associated `GatewayClass`. In such scenarios, when identical fields are specified in both configurations, the LBC employs a merging algorithm to resolve conflicts. The precedence of values is determined by the `mergingMode` field, which is exclusively read from the `GatewayClass`'s `LoadBalancerConfiguration`. If `mergingMode` is not explicitly set, the `GatewayClass` configuration implicitly takes higher precedence. For more info on `mergingMode`, refer this [doc](../loadbalancerconfig/#mergingmode)
It is possible for a `LoadBalancerConfiguration` to be attached to both a `Gateway` and its associated `GatewayClass`. In such scenarios, when identical fields are specified in both configurations, the LBC employs a merging algorithm to resolve conflicts. The precedence of values is determined by the `mergingMode` field, which is exclusively read from the `GatewayClass`'s `LoadBalancerConfiguration`. If `mergingMode` is not explicitly set, the `GatewayClass` configuration implicitly takes higher precedence. For more info on `mergingMode`, refer to [the merging mode documentation](./loadbalancerconfig.md#mergingmode).

The following fields exhibit specific merge behaviors:

Expand All @@ -80,7 +80,7 @@ The following fields exhibit specific merge behaviors:

The `TargetGroupConfiguration` CRD enables granular customization of the AWS Target Groups created for Kubernetes Services.

For a comprehensive overview of configurable parameters, please refer the [TargetGroupConfiguration CRD documentation](./targetgroupconfig.md).
For a comprehensive overview of configurable parameters, please refer to the [TargetGroupConfiguration CRD documentation](./targetgroupconfig.md).

**Example: Default Target Group Configuration for a Service**

Expand Down Expand Up @@ -141,9 +141,140 @@ spec:

#### How Default and Route-Specific Configurations Merge

When both `defaultConfiguration` and `routeConfigurations` within a `TargetGroupConfiguration` specify the same field, route-specific configurations take precedence. The controller identifies the most relevant route specification from the list of `routeConfigurations` and merges its `targetGroupProps` with the `defaultConfiguration`'s settings. For detailed information on the route matching logic employed, refer to the [Route Matching section](../targetgroupconfig/#route-matching-logic).
When both `defaultConfiguration` and `routeConfigurations` within a `TargetGroupConfiguration` specify the same field, route-specific configurations take precedence. The controller identifies the most relevant route specification from the list of `routeConfigurations` and merges its `targetGroupProps` with the `defaultConfiguration`'s settings. For detailed information on the route matching logic employed, refer to the [Route Matching section](./targetgroupconfig.md#route-matching-logic).

The following fields exhibit specific merge behaviors:

* **`tags`**: The two tag maps are combined. Any duplicate tag keys will result in the value from the higher-priority (route-specific) configuration being used.
* **`targetGroupAttributes`**: The two attribute lists are combined. Any duplicate attribute keys will result in the attribute value from the higher-priority (route-specific) configuration being applied.
* **`targetGroupAttributes`**: The two attribute lists are combined. Any duplicate attribute keys will result in the attribute value from the higher-priority (route-specific) configuration being applied.

#### Customizing L7 Routing Rules

The `ListenerRuleConfiguration` CRD allows representation of features present in AWS ALB,
that are not represented in the standard Gateway API spec.

An exhaustive list is:

- [Cognito Authentication](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html#cognito-requirements)
- [OIDC Authentication](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html#oidc-requirements)
- [Fixed Response](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/rule-action-types.html#fixed-response-actions)
- [Source IP Conditions](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_SourceIpConditionConfig.html#API_SourceIpConditionConfig_Contents)

For a comprehensive overview of the CRD, please refer to the [ListenerRuleConfiguration CRD documentation](./listenerruleconfig.md).

**Example: Adding source IP routing conditions**
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we provide them with more flexibility here with matchIndexes

  • if user does not specify matchIndexes, this source ip will be applied to all listener rules within same match
  rules:
    - backendRefs:
        - name: echoserver2
          port: 80
      filters:
        - type: ExtensionRef
          extensionRef:
            group: "gateway.k8s.aws"
            kind: "ListenerRuleConfiguration"
            name: "custom-rule-config-source-ip"
      matches:
        - path: # Path Pattern
            type: Exact
            value: /pathExactMatch
          queryParams: # Query String
            - name: "user"
              value: "john"
          method: GET # HTTP Request Method
        - path: # Regex path match
            type: RegularExpression
            value: "/firstRule/some?/users"

for example, here will have 2 rules, and source ip will be applied to all of them


This example adds upon the example found [here](./l7gateway.md#step-by-step-l7-gateway-api-resource-implementation-with-an-example). It adds
a routing rule that only allows requests originating from the range 10.0.0.0/5 to be routed to the backend.

```
# source-ip-condition.yaml
apiVersion: gateway.k8s.aws/v1beta1
kind: ListenerRuleConfiguration
metadata:
name: custom-rule-config-source-ip
namespace: example-ns
spec:
conditions:
- field: source-ip
sourceIPConfig:
values:
- 10.0.0.0/5
---
# updated-http-route.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: my-http-app-route
namespace: example-ns
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-alb-gateway
sectionName: http
- group: gateway.networking.k8s.io
kind: Gateway
name: my-alb-gateway
sectionName: https
rules:
- backendRefs:
- name: <your service>
port: <your service port>
filters:
- type: ExtensionRef
extensionRef:
group: "gateway.k8s.aws"
kind: "ListenerRuleConfiguration"
name: "custom-rule-config-source-ip"
```

To add granular rules, specify the index match:

```
# source-ip-condition.yaml
apiVersion: gateway.k8s.aws/v1beta1
kind: ListenerRuleConfiguration
metadata:
name: custom-rule-config-source-ip
namespace: example-ns
spec:
conditions:
- field: source-ip
matchIndexes: [0,1]
sourceIPConfig:
values:
- 10.0.0.0/5
---
# updated-http-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-app-source-ip
namespace: example-ns
spec:
parentRefs:
- name: my-alb-gateway
port: 90
rules:
- backendRefs:
- name: echoserver
port: 80
filters:
- type: ExtensionRef
extensionRef:
group: "gateway.k8s.aws"
kind: "ListenerRuleConfiguration"
name: "custom-rule-config-source-ip"
matches:
- path: # Path Pattern
type: Exact
value: /pathExactMatch
queryParams: # Query String
- name: "user"
value: "john"
method: GET # HTTP Request Method
- path: # Regex path match
type: RegularExpression
value: "/firstRule/some?/users"
- backendRefs:
- name: echoserver
port: 80
filters:
- type: ExtensionRef
extensionRef:
group: "gateway.k8s.aws"
kind: "ListenerRuleConfiguration"
name: "custom-rule-config-source-ip-2"
matches:
- path: # Path Pattern
type: Exact
value: /secondRulePath
method: POST # HTTP Request Method
- path: # Regex path match
type: RegularExpression
value: "/secondRule/some?/users"
- path:
type: "PathPrefix"
value: "/secondRule"
```
78 changes: 72 additions & 6 deletions docs/guide/gateway/gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Gateway API

!!! warning
- Only very basic (and not conforming) support of the Gateway API spec currently exists. The team is actively trying to close conformance and support gaps.
- The team is actively trying to close conformance and support gaps.
- Using the LBC and Gateway API together is not suggested for production workloads (yet!)


Expand All @@ -11,13 +11,10 @@ Mixing protocol layers, e.g. TCPRoute and HTTPRoute on the same Gateway, is not

## Current Support

!!! warning
- GRPCRoute and HTTPS Listeners for L7 gateways do not currently work. And only basic support is added for HTTPRoute.

The LBC Gateway API implementation supports the following Gateway API routes:

* L4 (NLBGatewayAPI): UDPRoute, TCPRoute, TLSRoute >=v2.13.3
* L7 (ALBGatewayAPI): HTTPRoute (Still work in progress, support matching and filtering is not added yet!)
* L7 (ALBGatewayAPI): HTTPRoute, GRPCRoute >= 2.14.0

## Prerequisites
* LBC >= v2.13.0
Expand Down Expand Up @@ -48,6 +45,13 @@ See [Subnet Discovery](../../deploy/subnet_discovery.md) for details on configur
!!! tip "disable worker node security group rule management"
You can disable the worker node security group rule management using the [LoadBalancerConfiguration CRD](./loadbalancerconfig.md).

## Certificate Discovery for secure listeners

Both L4 and L7 Gateway implementations support static certificate configuration and certificate discovery using Listener hostname.
The caveat is that configuration of TLS certificates can not be done via the `certificateRefs` field of a Gateway Listener,
as the controller only supports certificate references via an ARN. In the future, we may support syncing Kubernetes secrets into ACM.


### Worker node security groups selection
The controller automatically selects the worker node security groups that it modifies to allow inbound traffic using the following rules:

Expand All @@ -64,4 +68,66 @@ The controller automatically selects the worker node security groups that it mod

`${cluster-name}` is the name of the Kubernetes cluster.

If it is possible for multiple security groups with the tag `kubernetes.io/cluster/${cluster-name}` to be on a target ENI, you may use the `--service-target-eni-security-group-tags` flag to specify additional tags that must also match in order for a security group to be used.
If it is possible for multiple security groups with the tag `kubernetes.io/cluster/${cluster-name}` to be on a target ENI, you may use the `--service-target-eni-security-group-tags` flag to specify additional tags that must also match in order for a security group to be used.


## Misconfigured Services

The L4 and L7 gateways handle misconfigured services differently.


### L4

```
# my-tcproute.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: TCPRoute
metadata:
name: my-tcp-app-route
namespace: example-ns
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-tcp-gateway
sectionName: tcp-app # Refers to the specific listener on the Gateway
rules:
- backendRefs:
- name: my-tcp-service # Kubernetes Service
port: 9000
```

When `my-tcp-service` or the configured service port can't be found,
the target group will not be materialized on any NLBs that the route attaches to.


### L7

```
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: my-http-app-route
namespace: example-ns
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-alb-gateway
sectionName: http
- group: gateway.networking.k8s.io
kind: Gateway
name: my-alb-gateway
sectionName: https
rules:
- backendRefs:
- name: my-http-service
port: 9000
```

When `my-http-service` or the configured service port can't be found,
the target group will not be materialized on any ALBs that the route attaches to.
An [503 Fixed Response](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_FixedResponseActionConfig.html)
will be added to any Listener Rules that would have referenced the invalid backend.


9 changes: 5 additions & 4 deletions docs/guide/gateway/l4gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Gateway API for Layer 4 (NLB) Implementation

This section details the **AWS Load Balancer Controller's (LBC)** architecture and operational flow when processing Gateway API resources for Layer 4 traffic.
This section details the **AWS Load Balancer Controller's (LBC)** architecture and operational flow when processing Gateway API resources for Layer 4 traffic utilizing AWS NLB.

### Gateway API Resources and Controller Architecture

Expand All @@ -13,8 +13,9 @@ The LBC instances dedicated to L4 routing monitor the following Gateway API reso
* **`TLSRoute`**: Defines TLS-specific routing rules, enabling secure Layer 4 communication. These routes are satisfied by an **AWS NLB**.
* **`TCPRoute`**: Defines TCP-specific routing rules, facilitating direct TCP traffic management. These routes are satisfied by an **AWS NLB**.
* **`UDPRoute`**: Defines UDP-specific routing rules, facilitating UDP traffic management. These routes are satisfied by an **AWS NLB**.
* **`LoadBalancerConfiguration` (LBC CRD)**: A Custom Resource Definition utilized for fine-grained customization of the provisioned NLB. This CRD can be attached to a `Gateway` or its `GatewayClass`. For more info, please refer [How customization works](../customization)
* **`TargetGroupConfiguration` (LBC CRD)**: A Custom Resource Definition used for service-specific customizations of AWS Target Groups. This CRD is associated with a Kubernetes `Service`. For more info, please refer [How customization works](../customization)
* **`ReferenceGrant`**: Defines cross-namespace access. For more information [see](https://gateway-api.sigs.k8s.io/api-types/referencegrant/)
* **`LoadBalancerConfiguration` (LBC CRD)**: A Custom Resource Definition utilized for fine-grained customization of the provisioned NLB. This CRD can be attached to a `Gateway` or its `GatewayClass`. For more info, please refer [How customization works](customization.md#customizing-the-gateway-load-balancer-using-loadbalancerconfiguration-crd)
* **`TargetGroupConfiguration` (LBC CRD)**: A Custom Resource Definition used for service-specific customizations of AWS Target Groups. This CRD is associated with a Kubernetes `Service`. For more info, please refer [How customization works](customization.md#customizing-services-target-groups-using-targetgroupconfiguration-crd)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i forgot did we explicitly check/validate rule CRD cannot be used in L4?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NLBs don't have Listener Rules (aside from the default action)


### The Reconciliation Loop

Expand Down Expand Up @@ -82,7 +83,7 @@ spec:
* **API Event Detection:** The LBC's L4 controller continuously monitors the Kubernetes API server. Upon detecting the `aws-nlb-gateway-class` (with `controllerName: gateway.k8s.aws/nlb`), the `my-tcp-gateway` (referencing this `GatewayClass`), and `my-tcp-app-route` (referencing `my-tcp-gateway`'s `tcp-app` listener) resources, it recognizes its responsibility to manage these objects and initiates the provisioning of AWS resources.
* **NLB Provisioning:** An **AWS Network Load Balancer (NLB)** is provisioned in AWS for the `my-tcp-gateway` resource with default settings. At this stage, the NLB is active but does not yet have any configured listeners. As soon as the NLB becomes active, the status of the gateway is updated.
* **L4 Listener Materialization:** The controller processes the `my-tcp-app-route` resource. Given that the `TCPRoute` validly references the `my-tcp-gateway` and its `tcp-app` listener, an **NLB Listener** is materialized on the provisioned NLB. This listener will be configured for `TCP` protocol on `port 8080`, as specified in the `Gateway`'s listener definition. A default forward action is subsequently configured on the NLB Listener, directing all incoming traffic on `port 8080` to the newly created Target Group for service `my-tcp-service` in `backendRefs` section of `my-tcp-app-route`.
* **Target Group Creation:** An **AWS Target Group** is created for the Kubernetes Service `my-tcp-service` with default configuration. The Pods associated with `my-tcp-service` are then registered as targets within this new Target Group.
* **Target Group Creation:** An **AWS Target Group** is created for the Kubernetes Service `my-tcp-service` with default configuration. The cluster nodes are then registered as targets within this new Target Group.

### L4 Gateway API Limitations for NLBs
The LBC implementation of the Gateway API for L4 routes, which provisions NLB, introduces specific constraints to align with NLB capabilities. These limitations are enforced during the reconciliation process and are critical for successful L4 traffic management.
Expand Down
Loading
Loading