Fix: remove --config option from kubeadm upgrade (#11350) #11352

tmurakam · 2024-07-05T03:57:51Z

We can't mix some options with --config for kubeadm upgrade. The --config on upgrade (with InitConfiguration/ClusterConfiguration) is deprecated, and should be removed.

What type of PR is this?
/kind bug

What this PR does / why we need it:
Fix upgrade issue of kubespray upgrade.
This removes --config option.

Which issue(s) this PR fixes:
Fixes #11350

Special notes for your reviewer:
I have no confidence of side effects of removing the --config options.
Please consider the effects. If it is not acceptable, please reject this PR.

Does this PR introduce a user-facing change?:

Disables reconfiguring the cluster during upgrade (remove --config option from kubeadm upgrade apply)

We can't mix some options with --config for kubeadm upgrade. The --config on upgrade is deprecated, and should be removed.

k8s-ci-robot · 2024-07-05T03:58:00Z

Hi @tmurakam. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tico88612 · 2024-07-05T06:23:28Z

/ok-to-test

tico88612 · 2024-07-05T06:44:21Z

Your PR didn't include that Kubeadm deprecates the --config parameter. I'm attaching a search to find it.

kubernetes/kubeadm#3054
https://github.com/kubernetes/kubernetes/pull/123068/files#diff-657870bffd956e3e3553db7dbb5cb725ea53922da959fcf869c2e31880df72c7L91

However, the documentation I found does not indicate that they are deprecated. For the time being, the v1.30 version can't use the --config parameter, but v1.31 can use UpgradeConfiguration for configuration (only available in apiVersion kubeadm.k8s.io/v1beta4).

related #10935

https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta4/

tmurakam · 2024-07-05T06:53:53Z

@tico88612
You are right. The v1.31 will be able to use --config for UpgradeConfiguration of v1beta4.
I think using --config for InitConfiguration or ClusterConfiguration for upgrade is deprecated.
Updated the description of this PR.

MrFreezeex · 2024-07-05T11:34:55Z

In the issue in the kubeadm project it was said this as well:

you should drop your usage of --config on upgrade to 1.30 and 1.31 for non-v1beta4 APIs and use similar actions to reconfigure a kubeadm cluster:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure/

And I don't think we are applying/editing this cluster wide configmap anywhere which we should be doing if we can't use the local config anymore IIUC

tmurakam · 2024-07-29T14:00:13Z

@MrFreezeex @cyclinder
Could you please proceed with your review of this?

tico88612 · 2024-08-10T13:40:00Z

@yankay @ant31 Do we have a CI for upgrading the cluster?
If this PR is not handled and 2.26 is released in the future, upgrading the cluster will fail.

tico88612 · 2024-08-11T07:02:01Z

/label ci-extended

k8s-ci-robot · 2024-08-11T07:02:03Z

@tico88612: The label(s) /label ci-extended cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label ci-extended

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2024-08-13T00:12:17Z

@tico88612: The label(s) /label ci-extended cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label ci-extended

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tmurakam · 2024-08-18T01:24:48Z

I created a PR #11458 to add a CI job to test cluster upgrading.

tico88612 · 2024-08-18T06:47:19Z

Manual test successful.

/lgtm

yankay · 2024-08-29T01:56:59Z

HI @ErikJiang

Would you please help to review it

yankay · 2024-08-29T02:06:05Z

Thanks @tmurakam
/approve

k8s-ci-robot · 2024-08-29T02:06:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tmurakam, yankay

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [yankay]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…) (kubernetes-sigs#11352) We can't mix some options with --config for kubeadm upgrade. The --config on upgrade is deprecated, and should be removed.

@VannTen

Adds revised support for: - The previously removed `--config` argument for `kubeadm upgrade apply` - Changes to `ClusterConfiguration` as part of the `upgrade-cluster.yml` playbook lifecycle (Fixes kubernetes-sigs#11552) - kubeadm-config `v1beta4` `UpgradeConfiguration` for the `kubeadm upgrade apply` command: [UpgradeConfiguration v1beta4](https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta4/#kubeadm-k8s-io-v1beta4-UpgradeConfiguration). Background: PR kubernetes-sigs#11352 removed the --config flag from `kubeadm upgrade apply` to address the upgrade issues with kubeadm v1.30 identified in kubernetes-sigs#11350. Before this change, kubespray upgrades depended on `kubeadm upgrade apply --config=...` to make ClusterConfiguration changes with the upgrade. However, this reconfiguration was deprecated in `kubeadm upgrade apply` some time ago, and is no longer supported by the `kubeadm upgrade apply` config command. To ensure `ClusterConfiguration` changes are still applied during upgrades in a supportable way, the new solution in this PR reconfigures ClusterConfiguration separately after upgrade with distinct upload-config and control plane static pod rewrite tasks that run immediately after a successful upgrade. See [this comment from @VannTen](kubernetes-sigs#11871 (comment)) for more discussion on why the expectation is to fix reconfiguration as part of the upgrade lifecycle, as well as issue kubernetes-sigs#11552. Additionally, kubeadm v1.31 added back support for `--config`, along with UpgradeConfiguration when using v1beta4. This PR adds support for the `UpgradeConfiguration` in the kubeadm-config file, which is required to fully implement upgrades with `kubeadm.k8s.io/v1beta4`. This addition was omitted from the original v1beta4 implementation in kubernetes-sigs#11674, but it is required to use `--config` correctly during kubeadm upgrades with v1beta4.

@VannTen

Adds revised support for: - The previously removed `--config` argument for `kubeadm upgrade apply` - Changes to `ClusterConfiguration` as part of the `upgrade-cluster.yml` playbook lifecycle (Fixes kubernetes-sigs#11552) - kubeadm-config `v1beta4` `UpgradeConfiguration` for the `kubeadm upgrade apply` command: [UpgradeConfiguration v1beta4](https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta4/#kubeadm-k8s-io-v1beta4-UpgradeConfiguration). kubeadm upgrade apply --config support PR kubernetes-sigs#11352 removed the --config flag from all usages of `kubeadm upgrade apply` to address the upgrade issues with kubeadm v1.30 identified in kubernetes-sigs#11350. This PR enables support for the scenarios in which `--config` can and should still be used with `kubeadm upgrade apply`, with some version specific handling that still avoid kubeadm v1.30's upgrade failures. Control plane reconfiguration during upgrade Before PR kubernetes-sigs#11352, kubespray upgrades depended on `kubeadm upgrade apply --config=...` to make ClusterConfiguration changes during a cluster upgrade. However, this reconfiguration was deprecated from `kubeadm upgrade apply` some time ago, and is [no longer supported](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#additional-information) by the `kubeadm upgrade apply` command. To ensure `ClusterConfiguration` changes are still applied during upgrades in a supportable way, the new solution in this PR reconfigures `ClusterConfiguration` separately with distinct `kubeadm init phase upload-config kubeadm --config=...` and control plane static pod rewrite tasks that run immediately after a successful upgrade. See [this comment from @VannTen](kubernetes-sigs#11871 (comment)) for more discussion on why the expectation is to fix reconfiguration as part of the upgrade lifecycle, as well as issue kubernetes-sigs#11552. This approach is in line with kubeadm's [recommendations for cluster reconfiguration](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure/). kubeadm-config v1beta4 `UpgradeApplyConfiguration` support Additionally, kubeadm v1.31 added back support for `--config` while introducing support for `UpgradeConfiguration` in the kubeadm-config file, which is required to fully implement upgrades with `kubeadm.k8s.io/v1beta4`. UpgradeConfiguration was not added in kubespray's initial v1beta4 implementation (PR kubernetes-sigs#11674), but it is required to use `--config` correctly during kubeadm upgrades with v1beta4. This PR uses UpgradeConfiguration for v1beta4 kubeadm upgrades, while still retaining support for v1beta3.

@VannTen

Adds revised support for: - The previously removed `--config` argument for `kubeadm upgrade apply` - Changes to `ClusterConfiguration` as part of the `upgrade-cluster.yml` playbook lifecycle - kubeadm-config `v1beta4` `UpgradeConfiguration` for the `kubeadm upgrade apply` command: [UpgradeConfiguration v1beta4](https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta4/#kubeadm-k8s-io-v1beta4-UpgradeConfiguration). kubeadm upgrade apply --config support PR kubernetes-sigs#11352 removed the --config flag from all usages of `kubeadm upgrade apply` to address the upgrade issues with kubeadm v1.30 identified in kubernetes-sigs#11350. This PR enables support for the scenarios in which `--config` can and should still be used with `kubeadm upgrade apply`, with some version specific handling that still avoid kubeadm v1.30's upgrade failures. Control plane reconfiguration during upgrade Before PR kubernetes-sigs#11352, kubespray upgrades depended on `kubeadm upgrade apply --config=...` to make ClusterConfiguration changes during a cluster upgrade. However, this reconfiguration was deprecated from `kubeadm upgrade apply` some time ago, and is [no longer supported](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#additional-information) by the `kubeadm upgrade apply` command. To ensure `ClusterConfiguration` changes are still applied during upgrades in a supportable way, the new solution in this PR reconfigures `ClusterConfiguration` separately with distinct `kubeadm init phase upload-config kubeadm --config=...` and control plane static pod rewrite tasks that run immediately after a successful upgrade. See [this comment from @VannTen](kubernetes-sigs#11871 (comment)) for more discussion on why the expectation is to fix reconfiguration as part of the upgrade lifecycle, as well as issue kubernetes-sigs#11552. This approach is in line with kubeadm's [recommendations for cluster reconfiguration](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure/). kubeadm-config v1beta4 `UpgradeApplyConfiguration` support Additionally, kubeadm v1.31 added back support for `--config` while introducing support for `UpgradeConfiguration` in the kubeadm-config file, which is required to fully implement upgrades with `kubeadm.k8s.io/v1beta4`. UpgradeConfiguration was not added in kubespray's initial v1beta4 implementation (PR kubernetes-sigs#11674), but it is required to use `--config` correctly during kubeadm upgrades with v1beta4. This PR uses UpgradeConfiguration for v1beta4 kubeadm upgrades, while still retaining support for v1beta3.

Fix: remove --config option from kubeadm upgrade (kubernetes-sigs#11350)

3ffe9dd

We can't mix some options with --config for kubeadm upgrade. The --config on upgrade is deprecated, and should be removed.

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 5, 2024

k8s-ci-robot requested review from cyclinder and MrFreezeex July 5, 2024 03:57

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 5, 2024

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jul 5, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 5, 2024

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jul 5, 2024

tmurakam mentioned this pull request Aug 18, 2024

[CI] Add a CI job to test cluster upgrading, and fix bug of testcases_run.sh #11458

Merged

k8s-ci-robot assigned tico88612 Aug 18, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 18, 2024

ledroide added a commit to ledroide/kubespray that referenced this pull request Aug 20, 2024

apply kubernetes-sigs#11352

3d4d974

ledroide added a commit to ledroide/kubespray that referenced this pull request Aug 20, 2024

apply kubernetes-sigs#11352

010a323

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 28, 2024

yankay removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 28, 2024

tmurakam mentioned this pull request Aug 28, 2024

Release Proposal v2.26 #11447

Closed

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 29, 2024

k8s-ci-robot merged commit c89ea7e into kubernetes-sigs:master Aug 29, 2024
2 checks passed

tmurakam deleted the feature/11350-fix-upgrade-config branch September 3, 2024 07:36

maxime1907 mentioned this pull request Sep 19, 2024

upgrade: variables modified in kubeadm-config.yaml are not reflected in static manifests anymore #11552

Closed

Hector295 mentioned this pull request Jan 9, 2025

Feature: Adds reconfigure control plane #11871

Closed

chadswen mentioned this pull request Mar 3, 2025

Refactor control plane upgrades with reconfiguration support #12015

Merged

VannTen mentioned this pull request Mar 17, 2025

Unable to upgrade from 1.29.x to 1.30.x with external etcd setup #11500

Closed

Fix: remove --config option from kubeadm upgrade (#11350) #11352

Fix: remove --config option from kubeadm upgrade (#11350) #11352

Uh oh!

Conversation

tmurakam commented Jul 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 5, 2024

Uh oh!

tico88612 commented Jul 5, 2024

Uh oh!

tico88612 commented Jul 5, 2024

Uh oh!

tmurakam commented Jul 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MrFreezeex commented Jul 5, 2024

Uh oh!

tmurakam commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tico88612 commented Aug 10, 2024

Uh oh!

tico88612 commented Aug 11, 2024

Uh oh!

k8s-ci-robot commented Aug 11, 2024

Uh oh!

k8s-ci-robot commented Aug 13, 2024

Uh oh!

tmurakam commented Aug 18, 2024

Uh oh!

tico88612 commented Aug 18, 2024

Uh oh!

yankay commented Aug 29, 2024

Uh oh!

yankay commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Aug 29, 2024

Uh oh!

Uh oh!

Uh oh!

tmurakam commented Jul 5, 2024 •

edited

Loading

tmurakam commented Jul 5, 2024 •

edited

Loading

tmurakam commented Jul 29, 2024 •

edited

Loading

yankay commented Aug 29, 2024 •

edited

Loading