v2.28.1 Containerd 2.0: It's not possible to configure private repository with authentication

[trickyut]

What happened?

It seems that is not possible to configure properly private repository that requires authentication.
In Containerd 2.0 private repo configuration has been moved from config.toml to /etc/containerd/certs.d/registry_name/hosts.toml

Based on this discussion I managed to create proper hosts.toml file https://github.com/containerd/containerd/discussions/6468

server = "https://registry-example.com"
[host."https://registry-example.com"]
  capabilities = ["pull","resolve"]
  skip_verify = false
  [host."https://registry -example.com".header]
   authorization = "Basic base64username:password"

Thanks to this additional host parameter with .header it's possible to authenticate in private registry.
Without it authorization parameter is not accepted

When you take a look at kubespray\roles\container-engine\containerd\templates\hosts.toml.j2 there is no way to configure these additional parameters.

server = "{{ item.server | default("https://" + item.prefix) }}"
{% for mirror in item.mirrors %}
[host."{{ mirror.host }}"]
  capabilities = ["{{ ([ mirror.capabilities ] | flatten ) | join('","') }}"]
  skip_verify = {{ mirror.skip_verify | default('false') | string | lower }}
  override_path = {{ mirror.override_path | default('false') | string | lower }}
{% if mirror.ca is defined %}
  ca = ["{{ ([ mirror.ca ] | flatten ) | join('","') }}"]
{% endif %}
{% if mirror.client is defined %}
  client = [{% for pair in mirror.client %}["{{ pair[0] }}", "{{ pair[1] }}"]{% if not loop.last %},{% endif %}{% endfor %}]
{% endif %}
{% endfor %}

What did you expect to happen?

Adjust hosts.toml.j2 template to support private registry authentication

How can we reproduce it (as minimally and precisely as possible)?

Try to configure private registry for containerd 2.0 with kubespray.

OS

RHEL 9

Version of Ansible

2.16.6

Version of Python

3.11.7

Version of Kubespray (commit)

169a981a4

Network plugin used

cilium

Full inventory with variables

We use AWX with sourced group vars from repo

Command used to invoke ansible

cluster.yaml triggered by AWX

Output of ansible run

none

Anything else we need to know

No response

[trickyut]

Our workaround:

1. Additional authorization parameter to containerd_registries_mirrors

containerd_registries_mirrors:
   - prefix: registry-example.com
     mirrors: 
      - host: https://registry-example.com
        capabilities: ["pull","resolve"]
        skip_verify: false
        authorization: ["Basic base64user:token"]`

2. Modification of kubespray\roles\container-engine\containerd\templates\hosts.toml.j2 by adding additional if statement

server = "{{ item.server | default("https://" + item.prefix) }}"
{% for mirror in item.mirrors %}
[host."{{ mirror.host }}"]
  capabilities = ["{{ ([ mirror.capabilities ] | flatten ) | join('","') }}"]
  skip_verify = {{ mirror.skip_verify | default('false') | string | lower }}
  override_path = {{ mirror.override_path | default('false') | string | lower }}
{% if mirror.authorization is defined %}
  [host."{{ mirror.host }}".header]
   Authorization = {{ mirror.authorization | default('false') | string }}
{% endif %}
{% if mirror.ca is defined %}
  ca = ["{{ ([ mirror.ca ] | flatten ) | join('","') }}"]
{% endif %}
{% if mirror.client is defined %}
  client = [{% for pair in mirror.client %}["{{ pair[0] }}", "{{ pair[1] }}"]{% if not loop.last %},{% endif %}{% endfor %}]
{% endif %}
{% endfor %}

At least for us it solved this issue