Deprecate support for RedHat 8 family hosts

[VannTen]

RHEL 8 (and derivates, like Almalinux 8, Rockylinux 8, and others) are starting to cause some problems:

• by default, they ship a kernel version which is below the ones recommended by kubeadm : [kubernetes] Support Kubernetes v1.32.0 #11862 (comment)
• the default python version is not supported by ansible-core 2.17 (and thus ansible 10) : Upgrade ansible from 9 to 10 (ansible-core from 2.16 to 2.17) #11519 (comment)
• (minor): they come with an old systemd version (v239) which would need some tweaks with containerd 2.0 for instance : [containerd] Support containerd v2.0.x #11845 (comment)

Another piece of information to consider is that the security support for RHEL8 and clones (alma, rocky) ends in 2029. Given the pace of kubernetes development, I don't think we will be able to support them until that date (cgroupsv2 is not mandated for now, but it might happen,)

So, this is to kick-off a discussion about a timeline to deprecate, and eventually remove, support for RHEL8 family (and also: Opensuse 15 I think ? -> in general OS with kernel < 5.8/10 and python < 3.7)

I see the following extreme possiblities:

• wait until 2029
• deprecate now as part of defaulting to K8s 1.32, remove when 1.31 is removed from our options (== 2.30 I think ?)

There is problably some room between those.

Slack thread on that subject on what other sig cluster-lifecycle projects are doing: https://kubernetes.slack.com/archives/C13J86Z63/p1736414039059959

@ant31 @floryut @yankay @tico88612 @mzaian

Others, feel free to chime in. I'm going to pin this issue for visibility.

[ant31]

Let's drop them and remove from CI.

Eventually we do as kops, have a flag to turn off preflight check with big warning -> close all future issues raised about it.

Not knowing what Kubespray users are using doesn't help.

[VannTen]

Yeah, that's basically why I meant by deprecation.
The removal would be: removing stuff from boostrap-os, preinstall, etc, stuff we know will definitely break on those distribs.

We already have the turn off prefligt, (with a variable) since #11710

[ant31]

let's go for it (removing support).
RHEL 8 and fam., can request to RH and other about bumping kernel in LTS or move on with newer version.

[vdveldet]

The best quote i have found grasping this issue through github PRs:
”we try not to special case distros, just handle Linux as a whole. if distro Foo does not comply with k8s you might want to move to a newer Foo or a different distro.”

[VannTen]

Yeah, I quoted this from neolit123 (one of kubeadm maintainers) in the 1.32 support PR. EDIT: Though kubespray does special case distro (to a point) because we need to install packages, etc. But this is another level.

[ErikJiang]

I believe many users are still using RHEL 8 and its derivatives, so I do not recommend an immediate deprecation. Users typically need time to adapt to such changes. Could we consider providing enough time for users to prepare for migration? During this period, we can maintain support for RHEL 8 as much as possible.

[ant31]

I believe many users are still using RHEL 8

Many user of Kubespray? we don't know.

The Kubespray maintainers are already short, with tones of distro supported, we just can't absorb such a high maintenance cost for others.
By the others I mean both the users of 'old' distro who are not upgrading, and the maintainers of those distro.

So far we've been doing best effort, go as far as possible when it doesn't add too much overload. Now, it hits a road blocker there's no need to continue. We did the same for debian 10 (#11347)

Users typically need time to adapt to such changes

They don't have to adapt, they'll won't be able to upgrade with kubespray or at their own risk (aka not tested).

In Kubernetes there's no LTS strategy

[VannTen]

I believe many users are still using RHEL 8 and its derivatives

Well, maybe ? We don't really have concrete data (getting that would help 👍)

Could we consider providing enough time for users to prepare for migration? During this period, we can maintain support for RHEL 8 as much as possible

That's what we are doing, considering. There are very concrete problems, though. In particular, the EOL of ansible-core 2.16 in 3 months is a bit alarming. We are not going to use an unsupported version for 5 years.

Users typically need time to adapt to such changes

Could you give examples of that ? In my experience (migrating clusters from rhel 7 to rhel 8) the impact of migration on applications was rather minimal (after all, that's the point of containers).

Migrating a cluster is some work, but it's the same magnitude than upgrading (while more involved). (And I'm only talking about live migration without downtime ; because cluster recreation or allowed downtime are certainly easier).

[tico88612]

@ErikJiang Kubernetes v1.32 only “suggests” a new version of the Linux kernel, and there are other ways to get around it. But the awkward thing is that RHEL 8 comes with Python 3.6, which can't be controlled by Ansible 10 (ansible-core 2.17) (which requires Python 3.7 or higher), and I don't think we can afford to stick with Ansible 9 for RHEL 8 for 5 years.