`crictl rmi` removes all images with the same image id instead of the specified tag

[nirs]

What happened:

Trying to debug minikube bug with removing images, turns out the minikube image rm uses crictl rmi image:tag under the hood.

So I have 3 images, created using:

$ sudo nerdctl -n k8s.io pull docker.io/kicbase/echo-server:1.0
docker.io/kicbase/echo-server:1.0:                                                resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:42a89d9b22e5307cb88494990d5d929c401339f508c0a7e98a4d8ac52623fc5b: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:ac2c07efdce850736ed8e7b9c3bc15e251d808fc43b9eeeb88797be0def23730:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.4 s                                                                    total:  2.4 Ki (725.0 B/s)                                       

$ sudo nerdctl -n k8s.io image tag docker.io/kicbase/echo-server:1.0 localhost/test-base:latest

$ sudo nerdctl -n k8s.io image tag localhost/test-base:latest localhost/test-tag:latest

$ sudo nerdctl -n k8s.io image ls | grep 127ac38a2bb9
localhost/test-tag                         latest                127ac38a2bb9    8 seconds ago     linux/arm64    4.792MB    1.976MB
localhost/test-base                        latest                127ac38a2bb9    19 seconds ago    linux/arm64    4.792MB    1.976MB
<none>                                     <none>                127ac38a2bb9    31 seconds ago    linux/arm64    4.792MB    1.976MB
kicbase/echo-server                        1.0                   127ac38a2bb9    31 seconds ago    linux/arm64    4.792MB    1.976MB

The purpose of the images is to test minikube image commands. For every test we want to create a new tag and preform an operation on the tag, like deleting it. The other images should bot be affected. This allows running tests in parallel since every test operates on its own image.

The unexpected hehavior - when deleting one tag, all the images with the same image id are deleted:

$ sudo crictl --debug rmi localhost/test-tag:latest
DEBU[0000] Get image connection                         
DEBU[0000] User specified image to be removed: localhost/test-tag:latest 
DEBU[0000] ImageStatusRequest: image:{image:"localhost/test-tag:latest"} 
DEBU[0000] ImageStatusResponse: image:{id:"sha256:ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17"  repo_tags:"docker.io/kicbase/echo-server:1.0"  repo_tags:"localhost/test-base:latest"  repo_tags:"localhost/test-tag:latest"  size:1977221} 
DEBU[0000] RemoveImageRequest: image:{image:"localhost/test-tag:latest"} 
Deleted: docker.io/kicbase/echo-server:1.0
Deleted: localhost/test-base:latest
Deleted: localhost/test-tag:latest

What you expected to happen:

Only the tag "localhost/test-tag:latest" is deleted.

This works as expected with both ctr image rm and nerdcl image rm.

ctr:

$ sudo ctr -n k8s.io image rm localhost/test-tag:latest
localhost/test-tag:latest

$ sudo ctr -n k8s.io image ls | grep 127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6
docker.io/kicbase/echo-server:1.0                                       application/vnd.docker.distribution.manifest.list.v2+json sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6 1.9 MiB   linux/amd64,linux/arm64                                                      io.cri-containerd.image=managed                                 
localhost/test-base:latest                                              application/vnd.docker.distribution.manifest.list.v2+json sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6 1.9 MiB   linux/amd64,linux/arm64                                                      io.cri-containerd.image=managed                                 
sha256:ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17 application/vnd.docker.distribution.manifest.list.v2+json sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6 1.9 MiB   linux/amd64,linux/arm64                                                      io.cri-containerd.image=managed                                 

nerdctl:

$ sudo nerdctl -n k8s.io image rm localhost/test-tag:latest
Untagged: localhost/test-tag:latest@sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6
Deleted: sha256:5d26d12357a043b2446bf19e036f55cee8454728871f8dbe007f3127cd22d649

$ sudo nerdctl -n k8s.io image ls | grep 127ac38a2bb9
localhost/test-base                        latest                127ac38a2bb9    4 minutes ago    linux/arm64    4.792MB    1.976MB
<none>                                     <none>                127ac38a2bb9    4 minutes ago    linux/arm64    4.792MB    1.976MB
kicbase/echo-server                        1.0                   127ac38a2bb9    4 minutes ago    linux/arm64    4.792MB    1.976MB

How to reproduce it (as minimally and precisely as possible):

As explained above. Reproducible with both

• minikube master (using containerd 1.17.23)
• minikube master with containerd v2.1.4 (iso/kicbase: bump to containerd/nerdctl 2.x kubernetes/minikube#21409)

Anything else we need to know?:

Environment:

• Container runtime or hardware configuration:

With containerd 2.1.4:

$ sudo crictl --version
crictl version v1.34.0

$ sudo crictl info     
{
  "cniconfig": {
    "Networks": [
      {
        "Config": {
          "CNIVersion": "0.3.1",
          "Name": "cni-loopback",
          "Plugins": [
            {
              "Network": {
                "ipam": {},
                "type": "loopback"
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      },
      {
        "Config": {
          "CNIVersion": "0.4.0",
          "Name": "bridge",
          "Plugins": [
            {
              "Network": {
                "ipam": {
                  "type": "host-local"
                },
                "type": "bridge"
              },
              "Source": "{\"addIf\":\"true\",\"bridge\":\"bridge\",\"forceAddress\":false,\"hairpinMode\":true,\"ipMasq\":true,\"ipam\":{\"subnet\":\"10.244.0.0/16\",\"type\":\"host-local\"},\"isDefaultGateway\":true,\"type\":\"bridge\"}"
            },
            {
              "Network": {
                "capabilities": {
                  "portMappings": true
                },
                "ipam": {},
                "type": "portmap"
              },
              "Source": "{\"capabilities\":{\"portMappings\":true},\"type\":\"portmap\"}"
            },
            {
              "Network": {
                "ipam": {},
                "type": "firewall"
              },
              "Source": "{\"type\":\"firewall\"}"
            }
          ],
          "Source": "\n{\n  \"cniVersion\": \"0.4.0\",\n  \"name\": \"bridge\",\n  \"plugins\": [\n    {\n      \"type\": \"bridge\",\n      \"bridge\": \"bridge\",\n      \"addIf\": \"true\",\n      \"isDefaultGateway\": true,\n      \"forceAddress\": false,\n      \"ipMasq\": true,\n      \"hairpinMode\": true,\n      \"ipam\": {\n          \"type\": \"host-local\",\n          \"subnet\": \"10.244.0.0/16\"\n      }\n    },\n    {\n      \"type\": \"portmap\",\n      \"capabilities\": {\n          \"portMappings\": true\n      }\n    },\n    {\n       \"type\": \"firewall\"\n    }\n  ]\n}\n"
        },
        "IFName": "eth0"
      }
    ],
    "PluginConfDir": "/etc/cni/net.d",
    "PluginDirs": [
      "/opt/cni/bin"
    ],
    "PluginMaxConfNum": 1,
    "Prefix": "eth"
  },
  "config": {
    "cdiSpecDirs": [
      "/etc/cdi",
      "/var/run/cdi"
    ],
    "cni": {
      "binDir": "",
      "binDirs": [
        "/opt/cni/bin"
      ],
      "confDir": "/etc/cni/net.d",
      "confTemplate": "",
      "ipPref": "",
      "maxConfNum": 1,
      "setupSerially": false,
      "useInternalLoopback": false
    },
    "containerd": {
      "defaultRuntimeName": "runc",
      "ignoreBlockIONotEnabledErrors": false,
      "ignoreRdtNotEnabledErrors": false,
      "runtimes": {
        "runc": {
          "ContainerAnnotations": null,
          "PodAnnotations": null,
          "baseRuntimeSpec": "",
          "cgroupWritable": false,
          "cniConfDir": "",
          "cniMaxConfNum": 0,
          "io_type": "",
          "options": {
            "BinaryName": "",
            "CriuImagePath": "",
            "CriuWorkPath": "",
            "IoGid": 0,
            "IoUid": 0,
            "NoNewKeyring": false,
            "Root": "",
            "ShimCgroup": "",
            "SystemdCgroup": false
          },
          "privileged_without_host_devices": false,
          "privileged_without_host_devices_all_devices_allowed": false,
          "runtimePath": "",
          "runtimeType": "io.containerd.runc.v2",
          "sandboxer": "podsandbox",
          "snapshotter": ""
        }
      }
    },
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "containerdRootDir": "/mnt/vdb1/var/lib/containerd",
    "device_ownership_from_security_context": false,
    "disableApparmor": false,
    "disableHugetlbController": true,
    "disableProcMount": false,
    "drainExecSyncIOTimeout": "0s",
    "enableCDI": true,
    "enableSelinux": false,
    "enableUnprivilegedICMP": true,
    "enableUnprivilegedPorts": true,
    "ignoreDeprecationWarnings": null,
    "ignoreImageDefinedVolumes": false,
    "maxContainerLogLineSize": 16384,
    "netnsMountsUnderStateDir": false,
    "restrictOOMScoreAdj": false,
    "rootDir": "/mnt/vdb1/var/lib/containerd/io.containerd.grpc.v1.cri",
    "selinuxCategoryRange": 1024,
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri",
    "tolerateMissingHugetlbController": true,
    "unsetSeccompProfile": ""
  },
  "features": {
    "supplemental_groups_policy": true
  },
  "golang": "go1.23.11",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK",
  "runtimeHandlers": [
    {
      "features": {
        "recursive_read_only_mounts": true,
        "user_namespaces": true
      }
    },
    {
      "features": {
        "recursive_read_only_mounts": true,
        "user_namespaces": true
      },
      "name": "runc"
    }
  ],
  "status": {
    "conditions": [
      {
        "message": "",
        "reason": "",
        "status": true,
        "type": "RuntimeReady"
      },
      {
        "message": "",
        "reason": "",
        "status": true,
        "type": "NetworkReady"
      },
      {
        "message": "",
        "reason": "",
        "status": true,
        "type": "ContainerdHasNoDeprecationWarnings"
      }
    ]
  }
}

• OS (e.g: cat /etc/os-release):

$ cat /etc/os-release 
NAME=Buildroot
VERSION=2025.02-dirty
ID=buildroot
VERSION_ID=2025.02
PRETTY_NAME="Buildroot 2025.02"

• Kernel (e.g. uname -a):

$ uname -a
Linux minikube 6.6.95 #1 SMP PREEMPT Tue Sep  2 22:29:31 UTC 2025 aarch64 GNU/Linux

• Others:

Note: the minikube image mention comes with crictl 1.28.0. I installed crictl 1.34.0 manually to check if the issue is related to the old version. The behavior is the same.

[nirs]

Reproducible also with lima - much easier to reproduce:

% brew install lima

% limactl --version
limactl version 1.2.1

% limactl start template://k8s
...
INFO[0119] READY. Run `limactl shell k8s` to open the shell. 

% limactl shell k8s

nir@lima-k8s:~$ sudo nerdctl -n k8s.io pull docker.io/kicbase/echo-server:1.0
docker.io/kicbase/echo-server:1.0:                                                resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:127ac38a2bb9537b7f252addff209ea6801edcac8a92c8b1104dacd66a583ed6:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:42a89d9b22e5307cb88494990d5d929c401339f508c0a7e98a4d8ac52623fc5b: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:ac2c07efdce850736ed8e7b9c3bc15e251d808fc43b9eeeb88797be0def23730:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.4 s                                                                    total:  1.9 Mi (567.7 KiB/s)                                     

nir@lima-k8s:~$ sudo nerdctl -n k8s.io image tag docker.io/kicbase/echo-server:1.0 localhost/test-base:latest

nir@lima-k8s:~$ sudo nerdctl -n k8s.io image tag localhost/test-base:latest localhost/test-tag:latest

nir@lima-k8s:~$ sudo nerdctl -n k8s.io image ls | grep 127ac38a2bb9
localhost/test-tag                         latest             127ac38a2bb9    9 seconds ago         linux/arm64    4.792MB    1.976MB
localhost/test-base                        latest             127ac38a2bb9    16 seconds ago        linux/arm64    4.792MB    1.976MB
<none>                                     <none>             127ac38a2bb9    25 seconds ago        linux/arm64    4.792MB    1.976MB
kicbase/echo-server                        1.0                127ac38a2bb9    25 seconds ago        linux/arm64    4.792MB    1.976MB

nir@lima-k8s:~$ sudo crictl --debug rmi localhost/test-tag:latest
DEBU[0000] Get image connection                         
DEBU[0000] User specified image to be removed: localhost/test-tag:latest 
DEBU[0000] ImageStatusRequest: image:{image:"localhost/test-tag:latest"} 
DEBU[0000] ImageStatusResponse: image:{id:"sha256:ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17"  repo_tags:"docker.io/kicbase/echo-server:1.0"  repo_tags:"localhost/test-base:latest"  repo_tags:"localhost/test-tag:latest"  size:1977221} 
DEBU[0000] RemoveImageRequest: image:{image:"localhost/test-tag:latest"} 
Deleted: docker.io/kicbase/echo-server:1.0
Deleted: localhost/test-base:latest
Deleted: localhost/test-tag:latest

$ crictl --version
crictl version v1.34.0

$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

[nirs]

The images before running crictl rmi:

$ sudo crictl inspecti localhost/test-tag:latest | jq .status
{
  "id": "sha256:ce2d2cda2d858fdaea84129deb86d18e5dbf1c548f230b79fdca74cc91729d17",
  "pinned": false,
  "repoDigests": [],
  "repoTags": [
    "docker.io/kicbase/echo-server:1.0",
    "localhost/test-base:latest",
    "localhost/test-tag:latest"
  ],
  "size": "1977221",
  "username": ""
}

[SergeyKanzhelev]

What would be your expected behavior here? Removing the specified repoTag while keeping an image around? How would tooling know if the repoTag was explicitly created/pulled or it was just a side effect of downloading an image from the mirror or something? What would the expected behavior be for multiple labels pointing to the same sha?

[nirs]

What would be your expected behavior here? Removing the specified repoTag while keeping an image around?

@SergeyKanzhelev See the section "What you expected to happen:" in my report.

Basically, removing the tag "localhost/test-tag:latest" specified in the command:

crictl rmi localhost/test-tag:latest

If the image status was:

  "repoTags": [
    "docker.io/kicbase/echo-server:1.0",
    "localhost/test-base:latest",
    "localhost/test-tag:latest"
  ],

After the command the status should be:

  "repoTags": [
    "docker.io/kicbase/echo-server:1.0",
    "localhost/test-base:latest",
  ],

This is how nerdctl image rm and ctr image rm behave.

How would tooling know if the repoTag was explicitly created/pulled or it was just a side effect of downloading an image from the mirror or something?

Why does it matter how the tag was created? I asked to remove this tag, remove it.

What would the expected behavior be for multiple labels pointing to the same sha?

This is exactly this case, remove the tag the user asked to remove.

If this was the last tag, remove the image.

I would check how how docker, nerdctl, and ctr are handling this. There seems to be common behavior that user are expecting.

[nirs]

Another example from docker:

We have one image with 3 tags:

$ docker image ls | grep local/kicbase
local/kicbase                                   latest                               01a7d27dfb0b   2 hours ago    1.31GB
local/kicbase                                   v0.0.47-1756936034-21409             01a7d27dfb0b   2 hours ago    1.31GB
local/kicbase                                   v0.0.47-1756936034-21409-0aa34a444   01a7d27dfb0b   2 hours ago    1.31GB

Removing the first tag:

$ docker image rm local/kicbase:latest 
Untagged: local/kicbase:latest

Second tag:

$ docker image rm local/kicbase:v0.0.47-1756936034-21409-0aa34a444 
Untagged: local/kicbase:v0.0.47-1756936034-21409-0aa34a444

Last tag - image is deleted.

$ docker image rm local/kicbase:v0.0.47-1756936034-21409 
Untagged: local/kicbase:v0.0.47-1756936034-21409
Deleted: sha256:01a7d27dfb0bb933e066413fd386855e2161372d0fad04072ca567548e8e034f

And the same behavior with "docker rmi":

$ docker rmi local/kicbase:latest
Untagged: local/kicbase:latest

$ docker rmi local/kicbase:v0.0.47-1756936034-21409
Untagged: local/kicbase:v0.0.47-1756936034-21409

$ docker rmi local/kicbase:v0.0.47-1756936034-21409-0aa34a444
Untagged: local/kicbase:v0.0.47-1756936034-21409-0aa34a444
Deleted: sha256:01a7d27dfb0bb933e066413fd386855e2161372d0fad04072ca567548e8e034f

[SergeyKanzhelev]

What you posting makes sense. What keeps me un-easy is the following. If image is downloaded from the mirror, I thought it will be automatically tagged with both repos (target and a mirror) without user even asking for it. Quick check shows that I was wrong. At least in a couple examples I tried. If it ever will or there are other cases when it does get auto-marked with multiple identities, than the sequence of pulling and removing an image will keep it in-place as the mirror's image identity was added without user knowing.

k8s operates with references, not names of images. So should not be affected by this, sounds like a reasonable improvement in crictl. Just need to make sure relnotes are reflecting this breaking change (e.g. in the past two crictl for the same image from different "regions" and than a single remove will clean up things. After the change, it will not).