Skip to content

✨ Add ability to control "EKS Auto Mode" for EKS clusters #5642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

✨ Add ability to control "EKS Auto Mode" for EKS clusters #5642

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions config/crd/bases/controlplane.cluster.x-k8s.io_awsmanagedcontrolplanes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2245,6 +2245,25 @@ spec:
description: AWSManagedControlPlaneSpec defines the desired state of an
Amazon EKS Cluster.
properties:
accessConfig:
description: |-
AccessConfig specifies the EKS cluster access configuration.
It defines the authentication mode and whether to bootstrap the cluster creator
as a cluster-admin.
properties:
authenticationMode:
description: |-
AuthenticationMode mode controls how Kubernetes API authentication is performed:
- CONFIG_MAP — uses only the aws-auth ConfigMap (legacy mode).
- API — uses only EKS Access Entries (aws-auth is ignored).
- API_AND_CONFIG_MAP — enables both Access Entries and aws-auth.
type: string
bootstrapAdminPermissions:
description: |-
Comment on lines +2248 to +2262
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to rebase your changes off https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5578/files -- we're implementing the same changes there.

Rebasing your changes off that would make the PR much smaller and less work down the line -- considering that has a lot of reviews already

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, will do

BootstrapAdminPermissions specifies whether or not the cluster creator IAM principal was set as a cluster
admin access entry during cluster creation time.
type: boolean
type: object
additionalTags:
additionalProperties:
type: string
Expand Down Expand Up @@ -2298,6 +2317,31 @@ spec:
AssociateOIDCProvider can be enabled to automatically create an identity
provider for the controller for use with IAM roles for service accounts
type: boolean
autoMode:
description: |-
AutoMode is the EKS Auto Mode.
allows to create cluster with aws compute, ebs, elb capabilities.
properties:
compute:
description: Compute capability configuration for EKS Auto Mode.
properties:
nodePools:
items:
type: string
type: array
nodeRoleArn:
description: |-
NodeRoleArn the ARN of the IAM Role EKS will assign to EC2 Managed Instances in your EKS
Auto Mode cluster. This value cannot be changed after the compute capability of
EKS Auto Mode is enabled. For more information, see the IAM Reference in the
Amazon EKS User Guide.
type: string
type: object
enabled:
default: false
description: Enabled will enable EKS Auto Mode.
type: boolean
type: object
bastion:
description: Bastion contains options to configure the bastion host.
properties:
Expand Down
45 changes: 45 additions & 0 deletions config/crd/bases/controlplane.cluster.x-k8s.io_awsmanagedcontrolplanetemplates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,25 @@ spec:
description: AWSManagedControlPlaneSpec defines the desired state
of an Amazon EKS Cluster.
properties:
accessConfig:
description: |-
AccessConfig specifies the EKS cluster access configuration.
It defines the authentication mode and whether to bootstrap the cluster creator
as a cluster-admin.
properties:
authenticationMode:
description: |-
AuthenticationMode mode controls how Kubernetes API authentication is performed:
- CONFIG_MAP — uses only the aws-auth ConfigMap (legacy mode).
- API — uses only EKS Access Entries (aws-auth is ignored).
- API_AND_CONFIG_MAP — enables both Access Entries and aws-auth.
type: string
bootstrapAdminPermissions:
description: |-
BootstrapAdminPermissions specifies whether or not the cluster creator IAM principal was set as a cluster
admin access entry during cluster creation time.
type: boolean
type: object
additionalTags:
additionalProperties:
type: string
Expand Down Expand Up @@ -107,6 +126,32 @@ spec:
AssociateOIDCProvider can be enabled to automatically create an identity
provider for the controller for use with IAM roles for service accounts
type: boolean
autoMode:
description: |-
AutoMode is the EKS Auto Mode.
allows to create cluster with aws compute, ebs, elb capabilities.
properties:
compute:
description: Compute capability configuration for EKS
Auto Mode.
properties:
nodePools:
items:
type: string
type: array
nodeRoleArn:
description: |-
NodeRoleArn the ARN of the IAM Role EKS will assign to EC2 Managed Instances in your EKS
Auto Mode cluster. This value cannot be changed after the compute capability of
EKS Auto Mode is enabled. For more information, see the IAM Reference in the
Amazon EKS User Guide.
type: string
type: object
enabled:
default: false
description: Enabled will enable EKS Auto Mode.
type: boolean
type: object
bastion:
description: Bastion contains options to configure the bastion
host.
Expand Down
2 changes: 2 additions & 0 deletions controlplane/eks/api/v1beta1/conversion.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,8 @@ func (r *AWSManagedControlPlane) ConvertTo(dstRaw conversion.Hub) error {
dst.Spec.RolePermissionsBoundary = restored.Spec.RolePermissionsBoundary
dst.Status.Version = restored.Status.Version
dst.Spec.BootstrapSelfManagedAddons = restored.Spec.BootstrapSelfManagedAddons
dst.Spec.AccessConfig = restored.Spec.AccessConfig
dst.Spec.AutoMode = restored.Spec.AutoMode
return nil
}

Expand Down
2 changes: 2 additions & 0 deletions controlplane/eks/api/v1beta1/zz_generated.conversion.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

47 changes: 46 additions & 1 deletion controlplane/eks/api/v1beta2/awsmanagedcontrolplane_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,12 @@ type AWSManagedControlPlaneSpec struct { //nolint: maligned
// +optional
IAMAuthenticatorConfig *IAMAuthenticatorConfig `json:"iamAuthenticatorConfig,omitempty"`

// AccessConfig specifies the EKS cluster access configuration.
// It defines the authentication mode and whether to bootstrap the cluster creator
// as a cluster-admin.
// +optional
AccessConfig AccessConfig `json:"accessConfig,omitempty"`

// Endpoints specifies access to this cluster's control plane endpoints
// +optional
EndpointAccess EndpointAccess `json:"endpointAccess,omitempty"`
Expand Down Expand Up @@ -200,7 +206,10 @@ type AWSManagedControlPlaneSpec struct { //nolint: maligned
// bare EKS cluster without EKS default networking addons
// If you set this value to false when creating a cluster, the default networking add-ons will not be installed
// +kubebuilder:default=true
BootstrapSelfManagedAddons bool `json:"bootstrapSelfManagedAddons,omitempty"`
BootstrapSelfManagedAddons *bool `json:"bootstrapSelfManagedAddons,omitempty"`

// +optional
AutoMode *AutoMode `json:"autoMode,omitempty"`

// RestrictPrivateSubnets indicates that the EKS control plane should only use private subnets.
// +kubebuilder:default=false
Expand All @@ -221,6 +230,42 @@ type KubeProxy struct {
Disable bool `json:"disable,omitempty"`
}

// AccessConfig is the access configuration for the cluster.
type AccessConfig struct {
// AuthenticationMode mode controls how Kubernetes API authentication is performed:
// - CONFIG_MAP — uses only the aws-auth ConfigMap (legacy mode).
// - API — uses only EKS Access Entries (aws-auth is ignored).
// - API_AND_CONFIG_MAP — enables both Access Entries and aws-auth.
AuthenticationMode *string `json:"authenticationMode,omitempty"`

// BootstrapAdminPermissions specifies whether or not the cluster creator IAM principal was set as a cluster
// admin access entry during cluster creation time.
// +optional
BootstrapAdminPermissions *bool `json:"bootstrapAdminPermissions,omitempty"`
}

// AutoMode is the EKS Auto Mode.
// allows to create cluster with aws compute, ebs, elb capabilities.
type AutoMode struct {
// Enabled will enable EKS Auto Mode.
// +kubebuilder:default=false
Enabled bool `json:"enabled,omitempty"`
// Compute capability configuration for EKS Auto Mode.
// +optional
Compute Compute `json:"compute,omitempty"`
}

// Compute allows to run compute capability with EKS AutoMode.
type Compute struct {
NodePools []string `json:"nodePools,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs a comment describing the field.

// NodeRoleArn the ARN of the IAM Role EKS will assign to EC2 Managed Instances in your EKS
// Auto Mode cluster. This value cannot be changed after the compute capability of
// EKS Auto Mode is enabled. For more information, see the IAM Reference in the
// Amazon EKS User Guide.
// +optional
NodeRoleArn *string `json:"nodeRoleArn,omitempty"`
}

// VpcCni specifies configuration related to the VPC CNI.
type VpcCni struct {
// Disable indicates that the Amazon VPC CNI should be disabled. With EKS clusters the
Expand Down
108 changes: 106 additions & 2 deletions controlplane/eks/api/v1beta2/awsmanagedcontrolplane_webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,12 @@ import (
"net"

"github.com/apparentlymart/go-cidr/cidr"
"github.com/aws/aws-sdk-go-v2/aws"
ekstypes "github.com/aws/aws-sdk-go-v2/service/eks/types"
"github.com/pkg/errors"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/apimachinery/pkg/util/validation/field"
"k8s.io/apimachinery/pkg/util/version"
"k8s.io/klog/v2"
Expand Down Expand Up @@ -52,6 +55,9 @@ const (
cidrSizeMin = 16
vpcCniAddon = "vpc-cni"
kubeProxyAddon = "kube-proxy"

autoModeComputeNodePoolSystem = "system"
autoModeComputeNodePoolGeneral = "general-purpose"
)

// SetupWebhookWithManager will setup the webhooks for the AWSManagedControlPlane.
Expand Down Expand Up @@ -102,6 +108,8 @@ func (*awsManagedControlPlaneWebhook) ValidateCreate(_ context.Context, obj runt
allErrs = append(allErrs, r.validateSecondaryCIDR()...)
allErrs = append(allErrs, r.validateEKSAddons()...)
allErrs = append(allErrs, r.validateDisableVPCCNI()...)
allErrs = append(allErrs, r.validateAccessConfig(nil)...)
allErrs = append(allErrs, r.validateAutoMode(nil)...)
allErrs = append(allErrs, r.validateRestrictPrivateSubnets()...)
allErrs = append(allErrs, r.validateKubeProxy()...)
allErrs = append(allErrs, r.Spec.AdditionalTags.Validate()...)
Expand Down Expand Up @@ -142,6 +150,8 @@ func (*awsManagedControlPlaneWebhook) ValidateUpdate(ctx context.Context, oldObj
allErrs = append(allErrs, r.Spec.Bastion.Validate()...)
allErrs = append(allErrs, r.validateIAMAuthConfig()...)
allErrs = append(allErrs, r.validateSecondaryCIDR()...)
allErrs = append(allErrs, r.validateAccessConfig(oldAWSManagedControlplane)...)
allErrs = append(allErrs, r.validateAutoMode(oldAWSManagedControlplane)...)
allErrs = append(allErrs, r.validateEKSAddons()...)
allErrs = append(allErrs, r.validateDisableVPCCNI()...)
allErrs = append(allErrs, r.validateRestrictPrivateSubnets()...)
Expand Down Expand Up @@ -423,6 +433,94 @@ func validateDisableVPCCNI(vpcCni VpcCni, addons *[]Addon, path *field.Path) fie
return allErrs
}

func (r *AWSManagedControlPlane) validateAccessConfig(old *AWSManagedControlPlane) field.ErrorList {
return validateAccessConfig(r.Spec.AccessConfig, old, field.NewPath("spec", "accessConfig"))
}

func validateAccessConfig(accessConfig AccessConfig, old *AWSManagedControlPlane, path *field.Path) field.ErrorList {
var (
allErrs field.ErrorList
authModeOK bool
)

authConfigField := path.Child("authenticationMode")

if accessConfig.AuthenticationMode != nil {
for _, accessMode := range ekstypes.AuthenticationMode("").Values() {
if ekstypes.AuthenticationMode(*accessConfig.AuthenticationMode) == accessMode {
authModeOK = true
}
}

if !authModeOK {
allErrs = append(allErrs, field.Invalid(authConfigField, *accessConfig.AuthenticationMode, "unsupported authenticationMode provided"))
}
}

if old != nil && old.Spec.AccessConfig.AuthenticationMode != nil {
if *old.Spec.AccessConfig.AuthenticationMode != *accessConfig.AuthenticationMode {
if ekstypes.AuthenticationMode(*accessConfig.AuthenticationMode) == ekstypes.AuthenticationModeConfigMap {
allErrs = append(allErrs, field.Invalid(authConfigField, *accessConfig.AuthenticationMode, "cannot switch authenticationMode to legacy CONFIG_MAP mode"))
}
}
}

if accessConfig.BootstrapAdminPermissions != nil && *accessConfig.BootstrapAdminPermissions {
if ekstypes.AuthenticationMode(*accessConfig.AuthenticationMode) == ekstypes.AuthenticationModeConfigMap {
authConfigField := path.Child("bootstrapAdminPermissions")
allErrs = append(allErrs, field.Invalid(authConfigField, *accessConfig.BootstrapAdminPermissions, "authenticationMode CONFIG_MAP has no effect with the bootstrapAdminPermissions parameter"))
}
}

return allErrs
}

func (r *AWSManagedControlPlane) validateAutoMode(old *AWSManagedControlPlane) field.ErrorList {
return validateAutoMode(r.Spec, old, field.NewPath("spec"))
}

func validateAutoMode(spec AWSManagedControlPlaneSpec, old *AWSManagedControlPlane, path *field.Path) field.ErrorList {
var allErrs field.ErrorList

if spec.AutoMode == nil {
return nil
}

if spec.AutoMode.Enabled {
// EKS Auto mode is not compatible with configmap AuthenticationMode.
if *spec.AccessConfig.AuthenticationMode == string(ekstypes.AuthenticationModeConfigMap) {
authConfigField := path.Child("accessConfig", "authenticationMode")
allErrs = append(allErrs, field.Invalid(authConfigField, spec.AccessConfig.AuthenticationMode, "authenticationMode CONFIG_MAP couldn't be used with autoMode"))
}

if old != nil {
// nodeRoleArn cannot be changed after the compute capability of EKS Auto Mode is enabled.
if old.Spec.AutoMode.Compute.NodeRoleArn != spec.AutoMode.Compute.NodeRoleArn {
nodeRoleArnField := path.Child("autoMode", "compute", "nodeRoleArn")
allErrs = append(allErrs, field.Invalid(nodeRoleArnField, spec.AutoMode.Compute.NodeRoleArn, "nodeRoleArn could not be changed"))
}
}

if len(spec.AutoMode.Compute.NodePools) > 0 {
// nodeRoleArn should be always defined with node pools.
if spec.AutoMode.Compute.NodeRoleArn == nil {
nodeRoleArnField := path.Child("autoMode", "compute", "nodeRoleArn")
allErrs = append(allErrs, field.Invalid(nodeRoleArnField, spec.AutoMode.Compute.NodeRoleArn, "nodeRoleArn is required when nodePools specified"))
}

allowedPoolNames := sets.New[string](autoModeComputeNodePoolSystem, autoModeComputeNodePoolGeneral)
for _, poolName := range spec.AutoMode.Compute.NodePools {
nodePoolsField := path.Child("autoMode", "compute", "nodePools")
if !allowedPoolNames.Has(poolName) {
allErrs = append(allErrs, field.Invalid(nodePoolsField, poolName, "nodePools contains an invalid pool"))
}
}
}
}

return allErrs
}

func (r *AWSManagedControlPlane) validateRestrictPrivateSubnets() field.ErrorList {
return validateRestrictPrivateSubnets(r.Spec.RestrictPrivateSubnets, r.Spec.NetworkSpec, r.Spec.EKSClusterName, field.NewPath("spec"))
}
Expand Down Expand Up @@ -568,10 +666,16 @@ func (*awsManagedControlPlaneWebhook) Default(_ context.Context, obj runtime.Obj
}
}

if r.Spec.BootstrapSelfManagedAddons == nil {
r.Spec.BootstrapSelfManagedAddons = aws.Bool(true)
}

if r.Spec.AccessConfig.AuthenticationMode == nil {
r.Spec.AccessConfig.AuthenticationMode = aws.String(string(ekstypes.AuthenticationModeConfigMap))
}

infrav1.SetDefaults_Bastion(&r.Spec.Bastion)
infrav1.SetDefaults_NetworkSpec(&r.Spec.NetworkSpec)

// Set default value for BootstrapSelfManagedAddons
r.Spec.BootstrapSelfManagedAddons = true
return nil
}
Loading