Skip to content

Commit 9ad96d6

Browse files
zac-nixonzac-nixon
committed
implement multicluster support
1 parent 808fcbc commit 9ad96d6

32 files changed

+1904
-96
lines changed

‎apis/elbv2/v1alpha1/targetgroupbinding_types.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,10 @@ type TargetGroupBindingSpec struct {
109109
// targetGroupARN is the Amazon Resource Name (ARN) for the TargetGroup.
110110
TargetGroupARN string `json:"targetGroupARN"`
111111

112+
// MultiClusterTargetGroup Denotes if the TargetGroup is shared among multiple clusters
113+
// +optional
114+
MultiClusterTargetGroup bool `json:"multiClusterTargetGroup,omitempty"`
115+
112116
// targetType is the TargetType of TargetGroup. If unspecified, it will be automatically inferred.
113117
// +optional
114118
TargetType *TargetType `json:"targetType,omitempty"`

‎apis/elbv2/v1beta1/targetgroupbinding_types.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,10 @@ type TargetGroupBindingSpec struct {
127127
// +kubebuilder:validation:MinLength=1
128128
TargetGroupARN string `json:"targetGroupARN"`
129129

130+
// MultiClusterTargetGroup Denotes if the TargetGroup is shared among multiple clusters
131+
// +optional
132+
MultiClusterTargetGroup bool `json:"multiClusterTargetGroup,omitempty"`
133+
130134
// targetType is the TargetType of TargetGroup. If unspecified, it will be automatically inferred.
131135
// +optional
132136
TargetType *TargetType `json:"targetType,omitempty"`

‎config/crd/bases/elbv2.k8s.aws_targetgroupbindings.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,10 @@ spec:
6060
spec:
6161
description: TargetGroupBindingSpec defines the desired state of TargetGroupBinding
6262
properties:
63+
multiClusterTargetGroup:
64+
description: MultiClusterTargetGroup Denotes if the TargetGroup is
65+
shared among multiple clusters
66+
type: boolean
6367
networking:
6468
description: networking provides the networking setup for ELBV2 LoadBalancer
6569
to access targets in TargetGroup.
@@ -233,6 +237,10 @@ spec:
233237
- ipv4
234238
- ipv6
235239
type: string
240+
multiClusterTargetGroup:
241+
description: MultiClusterTargetGroup Denotes if the TargetGroup is
242+
shared among multiple clusters
243+
type: boolean
236244
networking:
237245
description: networking defines the networking rules to allow ELBV2
238246
LoadBalancer to access targets in TargetGroup.

‎config/rbac/role.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,15 @@ kind: ClusterRole
44
metadata:
55
name: controller-role
66
rules:
7+
- apiGroups:
8+
- ""
9+
resources:
10+
- configmaps
11+
verbs:
12+
- create
13+
- delete
14+
- get
15+
- update
716
- apiGroups:
817
- ""
918
resources:

‎controllers/elbv2/targetgroupbinding_controller.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ type targetGroupBindingReconciler struct {
8787
// +kubebuilder:rbac:groups="",resources=nodes,verbs=get;list;watch
8888
// +kubebuilder:rbac:groups="",resources=endpoints,verbs=get;list;watch
8989
// +kubebuilder:rbac:groups="",resources=namespaces,verbs=get;list;watch
90+
// +kubebuilder:rbac:groups="",resources=configmaps,verbs=get;delete;create;update
9091
// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch
9192
// +kubebuilder:rbac:groups="discovery.k8s.io",resources=endpointslices,verbs=get;list;watch
9293

‎docs/guide/ingress/annotations.md‎

Lines changed: 61 additions & 44 deletions
Large diffs are not rendered by default.

‎docs/guide/service/annotations.md‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,8 @@
5252
| [service.beta.kubernetes.io/aws-load-balancer-security-groups](#security-groups) | stringList | | |
5353
| [service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules](#manage-backend-sg-rules) | boolean | true | If `service.beta.kubernetes.io/aws-load-balancer-security-groups` is specified, this must also be explicitly specified otherwise it defaults to `false`. |
5454
| [service.beta.kubernetes.io/aws-load-balancer-inbound-sg-rules-on-private-link-traffic](#update-security-settings) | string | |
55-
| [service.beta.kubernetes.io/aws-load-balancer-listener-attributes.${Protocol}-${Port}](#listener-attributes) | stringMap | |
55+
| [service.beta.kubernetes.io/aws-load-balancer-listener-attributes.${Protocol}-${Port}](#listener-attributes) | stringMap | |
56+
| [service.beta.kubernetes.io/aws-load-balancer-multi-cluster-target-group](#multi-cluster-target-group) | boolean | false | If specified, the controller will only operate on targets that exist within the cluster, ignoring targets from other sources. |
5657

5758
## Traffic Routing
5859
Traffic Routing can be controlled with following annotations:
@@ -289,6 +290,20 @@ for proxy protocol v2 configuration.
289290
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled
290291
```
291292

293+
- <a name="multi-cluster-target-group">`service.beta.kubernetes.io/aws-load-balancer-multi-cluster-target-group`</a> Allows you to share the created Target Group ARN with other Load Balancer Controller managed clusters.
294+
295+
!!!warning ""
296+
This feature does not offer any Deletion Protection. Deleting the service will still delete the Target Group. If you need to support
297+
Target Groups shared with multiple clusters, it's recommended to use an out-of-band Target Group that is not managed by a Load Balancer Controller.
298+
299+
!!!note ""
300+
- It is not recommended to change this value frequently, if ever. The recommended way to set this value is on creation of the service.
301+
302+
!!!example
303+
```
304+
service.beta.kubernetes.io/aws-load-balancer-multi-cluster-target-group: "true"
305+
```
306+
292307
293308
## AWS Resource Tags
294309
The AWS Load Balancer Controller automatically applies following tags to the AWS resources it creates (NLB/TargetGroups/Listener/ListenerRule):

‎docs/guide/targetgroupbinding/targetgroupbinding.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,37 @@ spec:
9292
...
9393
```
9494

95+
## MultiCluster Target Group
96+
TargetGroupBinding CRD supports sharing the same target group ARN among multiple clusters. Setting this flag will ensure the controller only operates on targets within the cluster.
97+
98+
!!!tip ""
99+
The default value is false, meaning that the controller assumes full control over the target group ARN and will deregister any targets that are not found within the cluster.
100+
To set this flag for TGBs managed by the controller use either:
101+
ALB: alb.ingress.kubernetes.io/multi-cluster-target-group: "true"
102+
NLB: service.beta.kubernetes.io/aws-load-balancer-multi-cluster-target-group: "true"
103+
104+
105+
!!!warning ""
106+
It is not recommended to change this value after TGB creation. Changing between shared / not shared might lead to leaked targets.
107+
108+
!!!warning ""
109+
Only use this flag if you intend to share the target group ARN in multiple clusters. This flag will slow down reconciles and put a small additonal load on the kubernetes control plane.
110+
111+
112+
## Sample YAML
113+
```yaml
114+
apiVersion: elbv2.k8s.aws/v1beta1
115+
kind: TargetGroupBinding
116+
metadata:
117+
name: my-tgb
118+
spec:
119+
serviceRef:
120+
name: awesome-service # route traffic to the awesome-service
121+
port: 80
122+
targetGroupARN: <arn-to-targetGroup>
123+
multiClusterTargetGroup: "true"
124+
```
125+
95126

96127
## Reference
97128
See the [reference](./spec.md) for TargetGroupBinding CR

‎docs/guide/use_cases/blue_green/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Multiple target groups can be attached to the same forward action of a listener
1111
In addition to the weighted target group, AWS announced the advanced request routing feature in 2019. Advanced request routing gives developers the ability to write rules (and route traffic) based on standard and custom HTTP headers and methods, the request path, the query string, and the source IP address. This new feature simplifies the application architecture by eliminating the need for a proxy fleet for routing, blocks unwanted traffic at the load balancer, and enables the implementation of A/B testing.
1212

1313
## Overview
14-
The ALB is configured to split traffic using annotations on the ingress resrouces. More specifically, the [ingress annotation](../../../guide/ingress/annotations.md#actions) `alb.ingress.kubernetes.io/actions.${service-name}` configures custom actions on the listener.
14+
The ALB is configured to split traffic using annotations on the ingress resources. More specifically, the [ingress annotation](../../../guide/ingress/annotations.md#actions) `alb.ingress.kubernetes.io/actions.${service-name}` configures custom actions on the listener.
1515

1616
The body of the annotation is a JSON document that identifies an action type, and configures it. The supported [actions](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html#rule-action-types) are `redirect`, `forward`, and `fixed-response`.
1717

‎docs/guide/use_cases/multi_cluster/index.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# MultiCluster Target Groups
2+
3+
The LB controller assumes full control over configured Target Groups. This means when a Target Group is registered with the controller, it will deregister any targets that are not currently in the cluster.
4+
By enabling MultiCluster support, Target Groups can be associated to multiple Kubernetes clusters or support arbitrary targets from other sources.
5+
6+
7+
## Overview
8+
9+
MultiCluster mode is supported in a couple different ways. Be sure that every cluster that is associated with a Target Group has one of these methods
10+
enabled to ensure the Target Group is recognized as MultiCluster enabled! It is recommended to use fresh resources when using MutliCluster mode, there is short period of time
11+
when the feature needs to generate a snapshot of the cluster state in order to support the mode. This data is stored into a ConfigMap which resides in the same namespace
12+
as your load balancer resources. The ConfigMap stores the current snapshot of the managed targets, it can be found under `aws-lbc-targets-$TARGET_GROUP_BINDING_NAME`
13+
14+
15+
ALB, specify this annotation in the ingress or service:
16+
17+
`alb.ingress.kubernetes.io/multi-cluster-target-group: "true"`
18+
19+
NLB, specify this annotation in your service:
20+
21+
`service.beta.kubernetes.io/aws-load-balancer-multi-cluster-target-group: "true"`
22+
23+
24+
For Out-of-Band TargetGroupBindings, specify this field in the spec:
25+
26+
`multiClusterTargetGroup: true`
27+
28+
29+
30+
### Example
31+
32+
We will be setting up an echoserver in two clusters in order to demonstrate MultiCluster mode. See the full echoserver example in the 'Examples' tab.
33+
34+
The following ingress configures the Target Group Binding as MultiCluster. We will take the created Target Group and share it in a second cluster.
35+
36+
```
37+
apiVersion: networking.k8s.io/v1
38+
kind: Ingress
39+
metadata:
40+
name: echoserver
41+
namespace: echoserver
42+
annotations:
43+
alb.ingress.kubernetes.io/multi-cluster-target-group: "true"
44+
alb.ingress.kubernetes.io/scheme: internet-facing
45+
alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
46+
spec:
47+
ingressClassName: alb
48+
rules:
49+
- http:
50+
paths:
51+
- path: /
52+
pathType: Exact
53+
backend:
54+
service:
55+
name: echoserver
56+
port:
57+
number: 80
58+
```
59+
60+
Verify that MultiCluster is enabled by verifying that the created Target Group Binding is marked as MultiCluster.
61+
62+
```
63+
kubectl -n echoserver get targetgroupbinding k8s-echoserv-echoserv-cc0122e143 -o yaml
64+
apiVersion: elbv2.k8s.aws/v1beta1
65+
kind: TargetGroupBinding
66+
metadata:
67+
annotations:
68+
elbv2.k8s.aws/checkpoint: cKay81gadoTtBSg6uVVginqtmCVG-1ApTvYN4YLD37U/_4kBy3Yg64qrXzjvIb2LlC3O__ex1qjozynsqHXmPgo
69+
elbv2.k8s.aws/checkpoint-timestamp: "1729021572"
70+
creationTimestamp: "2024-10-15T19:46:06Z"
71+
finalizers:
72+
- elbv2.k8s.aws/resources
73+
generation: 1
74+
labels:
75+
ingress.k8s.aws/stack-name: echoserver
76+
ingress.k8s.aws/stack-namespace: echoserver
77+
name: k8s-echoserv-echoserv-cc0122e143
78+
namespace: echoserver
79+
resourceVersion: "79121011"
80+
uid: 9ceaa2ea-14bb-44a5-abb0-69c7d2aac52c
81+
spec:
82+
ipAddressType: ipv4
83+
multiClusterTargetGroup: true <<< HERE
84+
networking:
85+
ingress:
86+
- from:
87+
- securityGroup:
88+
groupID: sg-06a2bd7d790ac1d2e
89+
ports:
90+
- port: 32197
91+
protocol: TCP
92+
serviceRef:
93+
name: echoserver
94+
port: 80
95+
targetGroupARN: arn:aws:elasticloadbalancing:us-east-1:565768096483:targetgroup/k8s-echoserv-echoserv-cc0122e143/6816b87346280ee7
96+
targetType: instance
97+
vpcID: vpc-0a7ef5bd8943067a8
98+
```
99+
100+
In another cluster, you can now register that Target Group ARN in a Target Group Binding.
101+
102+
```
103+
apiVersion: elbv2.k8s.aws/v1beta1
104+
kind: TargetGroupBinding
105+
metadata:
106+
name: MyTargetGroupBinding
107+
namespace: echoserver
108+
spec:
109+
serviceRef:
110+
name: echoserver
111+
port: 80
112+
multiClusterTargetGroup: true
113+
targetType: instance
114+
ipAddressType: ipv4
115+
networking:
116+
ingress:
117+
- from:
118+
- securityGroup:
119+
groupID: $SG_FROM_ABOVE
120+
ports:
121+
- port: 32197
122+
protocol: TCP
123+
targetGroupARN: $TG_FROM_ABOVE
124+
```
125+
126+
The configured TargetGroup should have targets from both clusters available to service traffic.
127+
128+

0 commit comments

Comments
 (0)