-
Notifications
You must be signed in to change notification settings - Fork 1.6k
[feat: gw api]Adding docs for L4 routing for gateway api #4232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# This file contains configuration for our reference docs generation. For more | ||
# information about the possible configuration, refer to | ||
# https://github.com/elastic/crd-ref-docs. | ||
|
||
processor: | ||
ignoreTypes: | ||
- "()List$" | ||
# RE2 regular expressions describing type fields that should be excluded from the generated documentation. | ||
ignoreFields: | ||
- "TypeMeta$" | ||
|
||
render: | ||
# Version of Kubernetes to use when generating links to Kubernetes API documentation. | ||
kubernetesVersion: 1.32 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,146 @@ | ||
## Customizing your ELB resources | ||
|
||
The AWS Load Balancer Controller (LBC) provides sensible defaults for provisioning and managing Elastic Load Balancing (ELB) resources in response to Kubernetes Gateway API objects. However, to accommodate diverse use cases and specific operational requirements, the LBC offers extensive, fine-grained customization capabilities through two Custom Resource Definitions (CRDs): [LoadBalancerConfiguration](../spec/#loadbalancerconfiguration) and [TargetGroupConfiguration](../spec/#targetgroupconfiguration). | ||
|
||
 | ||
|
||
### Customizing the Gateway (Load Balancer) using `LoadBalancerConfiguration` CRD | ||
|
||
The `LoadBalancerConfiguration` CRD allows for the detailed customization of the AWS Load Balancer (ALB or NLB) provisioned by the LBC for a given Gateway. | ||
|
||
For a comprehensive list of configurable parameters, please refer to the [LoadBalancerConfiguration CRD documentation](./loadbalancerconfig.md). | ||
|
||
**Example:** To configure your Gateway to provision an `internet-facing` Load Balancer, define the following `LoadBalancerConfiguration` resource: | ||
|
||
```yaml | ||
apiVersion: gateway.k8s.aws/v1beta1 | ||
kind: LoadBalancerConfiguration | ||
metadata: | ||
name: internet-facing-config | ||
namespace: example-ns | ||
spec: | ||
scheme: internet-facing | ||
``` | ||
This configuration can then be applied by attaching the `LoadBalancerConfiguration` resource to either a `Gateway` or a `GatewayClass`. | ||
|
||
**Attaching to a Gateway:** | ||
shraddhabang marked this conversation as resolved.
Show resolved
Hide resolved
|
||
When attached directly to a `Gateway` resource, the specified configuration applies specifically to the Load Balancer provisioned for that individual Gateway. | ||
|
||
```yaml | ||
apiVersion: gateway.networking.k8s.io/v1 | ||
kind: Gateway | ||
metadata: | ||
name: test-gw | ||
namespace: example-ns | ||
spec: | ||
gatewayClassName: nlb-gateway | ||
infrastructure: | ||
parametersRef: | ||
group: gateway.k8s.aws | ||
kind: LoadBalancerConfiguration | ||
name: internet-facing-config # Must be in the same namespace as the Gateway | ||
listeners: | ||
... | ||
``` | ||
|
||
**Attaching to a GatewayClass:** | ||
When attached to a `GatewayClass` resource, the configuration becomes a default for all `Gateway` resources that reference this `GatewayClass`. | ||
|
||
```yaml | ||
apiVersion: gateway.networking.k8s.io/v1 | ||
kind: GatewayClass | ||
metadata: | ||
name: nlb-gateway | ||
spec: | ||
controllerName: gateway.k8s.aws/alb | ||
parametersRef: | ||
group: gateway.k8s.aws | ||
kind: LoadBalancerConfiguration | ||
name: internet-facing-config | ||
namespace: example-ns | ||
``` | ||
|
||
#### Conflict Resolution for `LoadBalancerConfiguration` | ||
|
||
It is possible for a `LoadBalancerConfiguration` to be attached to both a `Gateway` and its associated `GatewayClass`. In such scenarios, when identical fields are specified in both configurations, the LBC employs a merging algorithm to resolve conflicts. The precedence of values is determined by the `mergingMode` field, which is exclusively read from the `GatewayClass`'s `LoadBalancerConfiguration`. If `mergingMode` is not explicitly set, the `GatewayClass` configuration implicitly takes higher precedence. | ||
shraddhabang marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
The following fields exhibit specific merge behaviors: | ||
|
||
* **`tags`**: The tag maps from both configurations are combined. In the event of duplicate tag keys, the value from the higher-priority configuration (as determined by `mergingMode`) will be utilized. | ||
* **`loadBalancerAttributes`**: The attribute lists are combined. For duplicate attribute keys, the value from the higher-priority configuration will prevail. | ||
* **`mergeListenerConfig`**: Listener lists are combined. For duplicate `ProtocolPort` keys, the listener configuration from the higher-priority source will be selected. | ||
|
||
----- | ||
|
||
### Customizing Services (Target Groups) using `TargetGroupConfiguration` CRD | ||
|
||
The `TargetGroupConfiguration` CRD enables granular customization of the AWS Target Groups created for Kubernetes Services. | ||
|
||
For a comprehensive overview of configurable parameters, please refer the [TargetGroupConfiguration CRD documentation](./targetgroupconfig.md). | ||
|
||
**Example: Default Target Group Configuration for a Service** | ||
|
||
To configure the target groups for a specific service (e.g., `my-service`) to use `IP` mode and custom health checks across all routes referencing it, employ the following configuration: | ||
|
||
```yaml | ||
apiVersion: gateway.k8s.aws/v1beta1 | ||
kind: TargetGroupConfiguration | ||
metadata: | ||
name: custom-tg-config | ||
namespace: example-namespace | ||
spec: | ||
targetReference: | ||
name: my-service | ||
defaultConfiguration: | ||
targetType: ip | ||
healthCheckConfig: | ||
healthCheckPath: /health | ||
healthCheckInterval: 30 | ||
healthyThresholdCount: 3 | ||
``` | ||
|
||
Here, `my-service` is referenced within the `targetReference` of `custom-tg-config`. Any target group subsequently created for `my-service` via any route will inherit these default settings. Note that only one `TargetGroupConfiguration` CRD can be declared per service, and it must reside within the same namespace as the service it configures. | ||
|
||
**Example: Route-Specific Target Group Configuration** | ||
|
||
Alternatively, specific target group settings can be applied based on the individual routes referencing a service. This allows for tailored configurations for different traffic flows. | ||
|
||
```yaml | ||
apiVersion: gateway.k8s.aws/v1beta1 | ||
kind: TargetGroupConfiguration | ||
metadata: | ||
name: route-specific-tg-config | ||
namespace: example-ns | ||
spec: | ||
targetReference: | ||
name: my-service | ||
defaultConfiguration: | ||
targetType: ip | ||
routeConfigurations: | ||
- routeIdentifier: | ||
kind: TCPRoute | ||
namespace: example-ns | ||
name: api-route | ||
targetGroupProps: | ||
healthCheckConfig: | ||
healthCheckPath: /api/health | ||
healthCheckProtocol: HTTP | ||
- routeIdentifier: | ||
kind: TCPRoute | ||
namespace: example-ns-2 | ||
name: admin-route | ||
targetGroupProps: | ||
healthCheckConfig: | ||
healthCheckPath: /admin/health | ||
healthCheckInterval: 10 | ||
``` | ||
|
||
#### How Default and Route-Specific Configurations Merge | ||
|
||
When both `defaultConfiguration` and `routeConfigurations` within a `TargetGroupConfiguration` specify the same field, route-specific configurations take precedence. The controller identifies the most relevant route specification from the list of `routeConfigurations` and merges its `targetGroupProps` with the `defaultConfiguration`'s settings. For detailed information on the route matching logic employed, refer to the [Route Matching section](../targetgroupconfig/#route-matching-logic). | ||
|
||
The following fields exhibit specific merge behaviors: | ||
|
||
* **`tags`**: The two tag maps are combined. Any duplicate tag keys will result in the value from the higher-priority (route-specific) configuration being used. | ||
* **`targetGroupAttributes`**: The two attribute lists are combined. Any duplicate attribute keys will result in the attribute value from the higher-priority (route-specific) configuration being applied. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,120 +1,45 @@ | ||
# Gateway API | ||
|
||
!!! warning | ||
- Only very basic (and not conforming) support of the Gateway API spec currently exists. The team is actively trying to close conformance and support gaps. | ||
- Using the LBC and Gateway API together is not suggested for production workloads (yet!) | ||
!!! warning | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is there anyway to make those warning more obvious, like in a red box? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
- Only very basic (and not conforming) support of the Gateway API spec currently exists. The team is actively trying to close conformance and support gaps. | ||
- Using the LBC and Gateway API together is not suggested for production workloads (yet!) | ||
|
||
|
||
The AWS Load Balancer Controller (LBC) supports reconciliation for Kubernetes Gateway API objects. It satisfies | ||
L4 routes (TCPRoute, UDPRoute, TLSRoute) with an AWS NLB. It satisfies L7 routes (HTTPRoute, GRPCRoute) using an AWS ALB. | ||
Mixing protocol layers, e.g. TCPRoute and HTTPRoute on the same Gateway, is not supported. | ||
|
||
|
||
## Current Support | ||
|
||
!!! warning | ||
- GRPCRoute and HTTPS Listeners for L7 gateways do not currently work. And only basic support is added for HTTPRoute. | ||
|
||
The LBC Gateway API implementation supports the following Gateway API routes: | ||
|
||
* L4 (NLBGatewayApi): UDPRoute, TCPRoute, TLSRoute >=v2.13.3 | ||
shraddhabang marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
* L7 (ALBGatewayApi): HTTPRoute | ||
shraddhabang marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
## Prerequisites | ||
* LBC >= v2.13.0 | ||
* For `ip` target type: | ||
* Pods have native AWS VPC networking configured. For more information, see the [Amazon VPC CNI plugin](https://github.com/aws/amazon-vpc-cni-k8s#readme) documentation. | ||
* Installation of Gateway API CRDs | ||
* Standard Gateway API CRDs: `kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml` [REQUIRED] | ||
* Experimental Gateway API CRDs: `kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/experimental-install.yaml` [OPTIONAL: Used for L4 Routes] | ||
* Installation of LBC Gateway API CRDs: | ||
* `kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/refs/heads/main/config/crd/gateway/gateway-crds.yaml` | ||
* Standard Gateway API CRDs: `kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml` [REQUIRED] | ||
* Experimental Gateway API CRDs: `kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/experimental-install.yaml` [OPTIONAL: Used for L4 Routes] | ||
* Installation of LBC Gateway API specific CRDs: `kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/refs/heads/main/config/crd/gateway/gateway-crds.yaml` | ||
|
||
## Configuration | ||
|
||
By default, the LBC will _not_ listen to Gateway API CRDs. To enable support, specify the following feature flag(s) in the LBC deployment: | ||
|
||
``` | ||
- --feature-gates=NLBGatewayAPI=true,ALBGatewayAPI=true | ||
``` | ||
|
||
### Customization | ||
|
||
 | ||
|
||
The LBC tries to use sensible defaults. These defaults are not appropriate for each use-case. | ||
|
||
#### Customizing the Gateway | ||
|
||
For full details on what you can configure: | ||
[LoadBalancerConfiguration CRD](./loadbalancerconfig.md) | ||
|
||
For example, to configure your Gateway to provision an `internet-facing` LoadBalancer you can use this configuration: | ||
|
||
``` | ||
apiVersion: gateway.k8s.aws/v1beta1 | ||
kind: LoadBalancerConfiguration | ||
metadata: | ||
name: internet-facing-config | ||
namespace: echoserver | ||
spec: | ||
scheme: internet-facing | ||
``` | ||
|
||
You can then attach your configuration to either the Gateway or GatewayClass | ||
* `NLBGatewayAPI`: For enabling L4 Routing | ||
* `ALBGatewayAPI`: For enabling L7 Routing | ||
|
||
Gateway | ||
``` | ||
apiVersion: gateway.networking.k8s.io/v1 | ||
kind: Gateway | ||
metadata: | ||
name: test-gw | ||
namespace: echoserver | ||
spec: | ||
gatewayClassName: nlb-gateway | ||
infrastructure: | ||
parametersRef: | ||
group: gateway.k8s.aws | ||
kind: LoadBalancerConfiguration | ||
name: internet-facing-config # Must be in the same namespace as the Gateway | ||
listeners: | ||
... | ||
``` | ||
|
||
GatewayClass | ||
``` | ||
apiVersion: gateway.networking.k8s.io/v1 | ||
kind: GatewayClass | ||
metadata: | ||
name: nlb-gateway | ||
spec: | ||
controllerName: gateway.k8s.aws/alb | ||
parametersRef: | ||
group: gateway.k8s.aws | ||
kind: LoadBalancerConfiguration | ||
name: internet-facing-config | ||
namespace: echoserver | ||
- --feature-gates=NLBGatewayAPI=true,ALBGatewayAPI=true | ||
``` | ||
|
||
|
||
#### Configuration Conflict Resolution | ||
|
||
It is possible to attach a LoadBalancerConfiguration to both the Gateway and GatewayClass resources. When both the | ||
GatewayClass and Gateway LoadBalancerConfiguration specify the same field, the LBC uses a merging algorithm to take one value. | ||
Priority of what value to take is determined by the `mergingMode` field, this field is used to give priority to either the | ||
GatewayClass or Gateway value. When the value is not set, the GatewayClass value takes higher precedence. The `mergingMode` field | ||
is only read from the GatewayClass LoadBalancerConfiguration. | ||
|
||
The following fields have differing merge behavior: | ||
|
||
- `tags`: The two tags maps will be combined. Any duplicate tag keys will use the tag value from the higher priority config. | ||
- `loadBalancerAttributes`: The two attribute lists will be combined. Any duplicate attribute keys will use the attribute value from the higher priority config. | ||
- `mergeListenerConfig`: The two listener lists will be combined. Any duplicate ProtocolPort keys will use the listener config from the higher priority config. | ||
|
||
#### Customizing the Targets | ||
|
||
[Not currently supported] | ||
|
||
|
||
## Protocols | ||
|
||
!!! warning | ||
- TLSRoute, GRPCRoute and HTTPS Listeners do not currently work. | ||
|
||
The LBC Gateway API implementation supports all Gateway API routes: | ||
|
||
L4 (NLB): UDPRoute, TCPRoute, TLSRoute | ||
L7 (ALB): HTTPRoute, GRPCRoute | ||
|
||
|
||
## Subnet tagging requirements | ||
See [Subnet Discovery](../../deploy/subnet_discovery.md) for details on configuring Elastic Load Balancing for public or private placement. | ||
|
||
|
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.