{
  "schemaVersion": 3,
  "dataset": {
    "version": 3,
    "date": "2026-08-13",
    "group": {
      "id": "platform-infrastructure",
      "name": "Platform / Networking / Runtime Infrastructure"
    },
    "repository": {
      "id": "containerd",
      "repo": "containerd/containerd",
      "name": "containerd",
      "keywords": [
        "containerd"
      ]
    },
    "context": {
      "repository": "containerd/containerd",
      "url": "https://github.com/containerd/containerd",
      "description": "An open and reliable container runtime",
      "homepage": "https://containerd.io",
      "language": "Go",
      "topics": [
        "cncf",
        "containerd",
        "containers",
        "cri",
        "docker",
        "hacktoberfest",
        "kubernetes",
        "oci"
      ],
      "license": "Apache-2.0",
      "defaultBranch": "main",
      "stars": 21114,
      "forks": 4059,
      "openIssues": 488,
      "archived": false,
      "collectedAt": "2026-08-13T18:02:01.459351+00:00"
    },
    "news": {
      "repository": "containerd/containerd",
      "collectedAt": "2026-08-13T18:02:01.459351+00:00",
      "latestRelease": {
        "repository": "containerd/containerd",
        "tag": "v2.3.4",
        "title": "containerd 2.3.4",
        "url": "https://github.com/containerd/containerd/releases/tag/v2.3.4",
        "publishedAt": "2026-08-12T17:59:47Z",
        "notes": "Welcome to the v2.3.4 release of containerd!\n\nThe fourth patch release for containerd 2.3 contains various fixes and updates.\n\n### Highlights\n\n#### Container Runtime Interface (CRI)\n\n* Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913))\n* Set default runtimeFeatures.UserNamespacesHostNetwork to true in CRI ([#13914](https://github.com/containerd/containerd/pull/13914))\n* Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868))\n* Support non-UTF-8 binary environment variable values in CRI ([#13454](https://github.com/containerd/containerd/pull/13454))\n* Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13778](https://github.com/containerd/containerd/pull/13778))\n* Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13734](https://github.com/containerd/containerd/pull/13734))\n* Normalize sandbox image references in CRI to resolve images without domain prefixes ([#13759](https://github.com/containerd/containerd/pull/13759))\n\n#### Node Resource Interface (NRI)\n\n* Emit deprecation warnings for plugins using deprecated NRI interfaces ([#13935](https://github.com/containerd/containerd/pull/13935))\n\n#### Runtime\n\n* Enable log scrubbing by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904))\n* Fix memory leak in OOM watcher map when stopping container monitoring ([#13870](https://github.com/containerd/containerd/pull/13870))\n* Avoid orphaning shims on transient errors when loading process IDs ([#13857](https://github.com/containerd/containerd/pull/13857))\n* Fix corruption of binary protobuf shim start responses caused by premature whitespace trimming ([#13803](https://github.com/containerd/containerd/pull/13803))\n\n#### Snapshotters\n\n* Fix EROFS snapshotter dropping lower layers stacked above merged filesystem metadata ([#13876](https://github.com/containerd/containerd/pull/13876))\n\n#### Breaking\n\n* Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913))\n\n#### Deprecations\n\n* Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868))\n\nPlease try out the release binaries and report any issues at\nhttps://github.com/containerd/containerd/issues.\n\n### Contributors\n\n* Samuel Karp\n* Maksym Pavlenko\n* Mike Brown\n* Akihiro Suda\n* Derek McGowan\n* Jordan Liggitt\n* Paweł Gronowski\n* Amir Alavi\n* Andrew Halaney\n* Bing Hongtao\n* Chris Henzie\n* Harshal Patel\n* Krisztian Litkey\n* Phil Estes\n* Wei Fu\n* ningmingxiao\n\n### Changes\n<details><summary>35 commits</summary>\n<p>\n\n  * [`c1f5360ef`](https://github.com/containerd/containerd/commit/c1f5360ef7b9afb748d4119ad7be665ba5144480) Prepare release notes for v2.3.4\n* cri: disable restore in CreateContainer by default ([#13913](https://github.com/containerd/containerd/pull/13913))\n  * [`928c03c68`](https://github.com/containerd/containerd/commit/928c03c6898cad95f795300281fdff0dbc55a32f) cri: disable restore in CreateContainer by default\n* nri,deprecation: record and emit warnings for NRI deprecations. ([#13935](https://github.com/containerd/containerd/pull/13935))\n  * [`5966e2edb`](https://github.com/containerd/containerd/commit/5966e2edb15bb939aa2bb73649fbad73a2b04094) nri,deprecation: emit warnings for old NRI plugins.\n* Set the default of runtimeFeatures.UserNamespacesHostNetwork to true ([#13914](https://github.com/containerd/containerd/pull/13914))\n  * [`ab52c58f1`](https://github.com/containerd/containerd/commit/ab52c58f16ac50bb6724354bf36cbb4da480c2a2) Set the default of runtimeFeatures.UserNamespacesHostNetwork to true\n* Use ScrubLogs by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904))\n  * [`0c255158a`](https://github.com/containerd/containerd/commit/0c255158aa1ecab40aafc42269cfc018a46eb627) ctr: add --scrub-logs flag for Windows\n  * [`1c2b13dc9`](https://github.com/containerd/containerd/commit/1c2b13dc975468197e57321e25fd09163f78a75a) cri/config: use ScrubLogs by default on Windows\n* snapshots/erofs: keep lowers stacked above a merged fsmeta ([#13876](https://github.com/containerd/containerd/pull/13876))\n  * [`0c511e068`](https://github.com/containerd/containerd/commit/0c511e068b7d6dd815418185a235b3ba3347c65a) snapshots/erofs: keep lowers stacked above a merged fsmeta\n* cri: deprecate restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868))\n  * [`dc98141df`](https://github.com/containerd/containerd/commit/dc98141df70d3d72d46125c6f30de0d2de648086) cri: deprecate restore in CreateContainer\n* internal/oom: Fix memory leak by removing watcher from map on Stop ([#13870](https://github.com/containerd/containerd/pull/13870))\n  * [`537d82d54`](https://github.com/containerd/containerd/commit/537d82d545759d4a072d564d4937707af2650163) internal/oom: Fix memory leak by removing watcher from map on Stop\n* shim_load: Consider shim leaked only if we can't find pids ([#13857](https://github.com/containerd/containerd/pull/13857))\n  * [`decf97a9c`](https://github.com/containerd/containerd/commit/decf97a9c7c26f92391d2c862d35832e0a291e80) shim_load: Consider shim leaked only if we can't find pids\n* core/runtime/v2: Drop checkpointctl module dependency ([#13840](https://github.com/containerd/containerd/pull/13840))\n  * [`796f07dc8`](https://github.com/containerd/containerd/commit/796f07dc853d9ae8a3a1278fe47436755a4bc3b7) core/runtime/v2: Drop checkpointctl module dependency\n* Handle []byte envvar value for CRI ([#13454](https://github.com/containerd/containerd/pull/13454))\n  * [`751fddddb`](https://github.com/containerd/containerd/commit/751fddddbb630631e12fd63e5dac4a9530188822) Handle []byte envvar value\n  * [`0bce9060e`](https://github.com/containerd/containerd/commit/0bce9060ef3401e690b541dd6d2008c2abc50b0d) update to v0.36.x kubernetes dependencies\n* fix(cri): introspect OCI runtime features for non-runc runtimes ([#13778](https://github.com/containerd/containerd/pull/13778))\n  * [`61a8f6f45`](https://github.com/containerd/containerd/commit/61a8f6f45e34660d787988e8a7e7a218f556369c) fix(cri): introspect OCI runtime features for non-runc runtimes\n* core/runtime/v2: Preserve protobuf shim response bytes ([#13803](https://github.com/containerd/containerd/pull/13803))\n  * [`1d28017be`](https://github.com/containerd/containerd/commit/1d28017be295005bfa3aee1d47afe86341d391b6) core/runtime/v2: Preserve protobuf shim response bytes\n* Disable checkpoint restore codepath when CRIU is not installed ([#13734](https://github.com/containerd/containerd/pull/13734))\n  * [`374091d67`](https://github.com/containerd/containerd/commit/374091d67c1966969b772594ba96a74f4357bc7a) github/workflows: install criu in node-e2e\n  * [`db03e3968`](https://github.com/containerd/containerd/commit/db03e39685448a1d87025a2ca3a1e4e42713102d) cri: add enable_criu configuration option\n  * [`dacd4c7d0`](https://github.com/containerd/containerd/commit/dacd4c7d00f4a5fd71197e67afdb6355e8536e4f) cri: validate CRIU availability and version early\n* ci: bound Go fuzzing by execution count ([#13785](https://github.com/containerd/containerd/pull/13785))\n  * [`890a9c86c`](https://github.com/containerd/containerd/commit/890a9c86cd9918e53c1ef1b57c6539a943196cd5) ci: bound Go fuzzing by execution count\n* cri: auto-add prefix for pause image ([#13759](https://github.com/containerd/containerd/pull/13759))\n  * [`0b2f1d078`](https://github.com/containerd/containerd/commit/0b2f1d078124b52f4c8030dc686f2d65c1dfc17c) cri: auto-add prefix for pause image\n</p>\n</details>\n\n### Dependency Changes\n\n* **k8s.io/api**                            v0.36.0 -> v0.36.3\n* **k8s.io/apimachinery**                   v0.36.0 -> v0.36.3\n* **k8s.io/client-go**                      v0.36.0 -> v0.36.3\n* **k8s.io/component-base**                 v0.36.0 -> v0.36.3\n* **k8s.io/cri-api**                        v0.36.0 -> v0.36.3\n* **k8s.io/cri-client**                     v0.36.0 -> v0.36.3\n* **k8s.io/cri-streaming**                  v0.36.0 -> v0.36.3\n* **sigs.k8s.io/structured-merge-diff/v6**  v6.3.2 -> v6.3.3\n\nPrevious release can be found at [v2.3.3](https://github.com/containerd/containerd/releases/tag/v2.3.3)\n### Which file should I download?\n* `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`:         ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04).\n* `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`:  Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent.\n\nIn addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases)\nand [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too.\n\nSee also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.\n",
        "highlights": [
          "Highlights",
          "Container Runtime Interface (CRI)",
          "Disable checkpoint restore in CreateContainer by default, requiring the enableexperimentalrestoreviacreate configuration option to enable (#13913)",
          "Set default runtimeFeatures.UserNamespacesHostNetwork to true in CRI (#13914)",
          "Deprecate checkpoint restore in CreateContainer (#13868)",
          "Support non-UTF-8 binary environment variable values in CRI (#13454)"
        ],
        "prerelease": false
      },
      "upcoming": [
        {
          "repository": "containerd/containerd",
          "kind": "milestone",
          "title": "2.5",
          "url": "https://github.com/containerd/containerd/milestone/52",
          "description": "",
          "openIssues": 0,
          "closedIssues": 0
        },
        {
          "repository": "containerd/containerd",
          "kind": "milestone",
          "title": "2.4",
          "url": "https://github.com/containerd/containerd/milestone/51",
          "description": "",
          "dueAt": "2026-08-26T00:00:00Z",
          "progress": 34,
          "openIssues": 37,
          "closedIssues": 19
        }
      ],
      "communityDiscussions": []
    },
    "runs": [
      {
        "collectedAt": "2026-08-13T12:26:38.318Z",
        "since": "2026-08-12T12:26:38.318Z",
        "observedCount": 41,
        "changedCount": 41
      },
      {
        "collectedAt": "2026-08-13T13:48:00.446149Z",
        "since": "2026-08-12T13:48:00.446149Z",
        "observedCount": 40,
        "changedCount": 40
      },
      {
        "collectedAt": "2026-08-13T16:19:22.035158Z",
        "since": "2026-08-12T16:19:22.035158Z",
        "observedCount": 41,
        "changedCount": 5
      },
      {
        "collectedAt": "2026-08-13T17:43:20.785491Z",
        "since": "2026-08-12T17:43:20.785491Z",
        "observedCount": 38,
        "changedCount": 2
      },
      {
        "collectedAt": "2026-08-13T17:47:07.884300Z",
        "since": "2026-08-12T17:47:07.884300Z",
        "observedCount": 38,
        "changedCount": 0
      },
      {
        "collectedAt": "2026-08-13T18:01:55.420671Z",
        "since": "2026-08-12T18:01:55.420671Z",
        "observedCount": 35,
        "changedCount": 0
      }
    ],
    "signals": [
      {
        "id": "github:containerd/containerd:issue:10046",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "Improve image pull progress in ctr",
        "text": "### What is the problem you're trying to solve Show the total size and progress of the total downloaded size when pulling the image ### Describe the solution you'd like Hi When we pull the images, nothing is displayed about the image size, how many layers are there, or how much is downloaded. This can be most user-friendly. However, from my study of the contribution documents, I found that to start working on such a change, I need to get approval from your team first. May I work on this? Any insight? ### Additional context -",
        "url": "https://github.com/containerd/containerd/issues/10046",
        "timestamp": "2026-08-12T12:49:35Z",
        "metrics": {
          "reactions": 0,
          "comments": 4
        },
        "labels": [
          "kind/feature",
          "Stale",
          "status/accepted"
        ],
        "author": "ghorbani-ali",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:containerd/containerd:issue:12131",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "containerd on Windows 11 - cannot override root directory using --root parameter",
        "text": "### Description Hi All, I am trying to use `containerd.exe` on Windows 11. This is a corporate workstation and is locked down. Whenever I run containerd.exe from the command line, I get the following error: ``` containerd: mkdir C:\\ProgramData\\containerd\\root: Access is denied. ``` I cannot create/edit any folders under `c:\\ProgramData\\containerd\\root` since I do not have local Administrator rights. ## Trying the --root parameter I used the `--root` argument to specify a directory as follows: ``` containerd --root C:\\Users\\myusername\\Tools\\containerd\\root ``` But, this too throws the same error as above: ``` containerd: mkdir C:\\ProgramData\\containerd\\state: Access is denied ``` ## Version containerd github.com/containerd/containerd/v2 v2.1.3 c787fb98911740dd3ff2d0e45ce88cdf01410486 Any advice? Thanks, Sau ### Steps to reproduce the issue 1. Download the compressed content for AMD64 platform. 2. Use 7Z on Windows or some equivalent tool to extract the inner folders and files. 3. Copy over all the binaries (containerd.exe , ctr.exe, containerd-stress.exe, containerd-shim-runhcs-v1.exe) to another folder 4. Launch a CMD 5. Change over directory to the folder where you extracted the binaries (containerd.exe) 6. Run the command `containerd.exe --root C:\\Users\\saurabhdasgupta\\Tools\\containerd\\root` You should see the error reported above ### Describe the results you received and expected Containerd throws the following error: ``` containerd: mkdir C:\\ProgramData\\containerd\\state: Access is denied. ``` ### What version of containerd are you using? containerd github.com/containerd/containerd/v2 v2.1.3 c787fb98911740dd3ff2d0e45ce88cdf01410486 ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. _No response_",
        "url": "https://github.com/containerd/containerd/issues/12131",
        "createdAt": "2025-07-21T21:47:39Z",
        "updatedAt": "2026-08-13T00:19:32Z",
        "timestamp": "2026-08-13T00:19:32Z",
        "metrics": {
          "reactions": 0,
          "comments": 4
        },
        "labels": [
          "kind/bug",
          "platform/windows",
          "Stale"
        ],
        "author": "sdg002",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:12280",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "Support container checkpoint/restore for gVisor runtime",
        "text": "### What is the problem you're trying to solve Hey containerd-maintainers team! I am trying to enable container checkpoint/restore support using the gVisor (runsc) runtime. Currently, the CheckpointContainer method is not implemented in both containerd’s CRI service and the containerd-runsc-shim I am working on implementing the missing functionality in containerd-runsc-shim (https://github.com/google/gvisor/issues/11810), but without CRI support in containerd, checkpoint cannot be enabled. Probably the same relates to restore as well. ### Describe the solution you'd like Implement the CheckpointContainer method in the CRI service: https://github.com/containerd/containerd/blob/61ab051136c53fe2b9dc0681c326dc86897a59de/internal/cri/server/container_checkpoint.go#L46 I would appreciate guidance on: - Any known challenges with adding this method. - Whether there are related discussions or design considerations that I should be aware of. - If there are existing efforts or plans for CRI checkpoint/restore support. ### Additional context _No response_",
        "url": "https://github.com/containerd/containerd/issues/12280",
        "createdAt": "2025-09-08T09:50:28Z",
        "updatedAt": "2026-08-13T16:08:48Z",
        "timestamp": "2026-08-13T16:08:48Z",
        "metrics": {
          "reactions": 1,
          "comments": 7
        },
        "labels": [
          "kind/feature",
          "area/cri",
          "Stale"
        ],
        "author": "samorojy",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:12517",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "Support for time namespaces",
        "text": "### What is the problem you're trying to solve Support for time namespaces is added to OCI runtime spec and runc implemented the necessary functionality in https://github.com/opencontainers/runc/issues/2345 but there seems to be no way to utilize this feature in containerd. ### Describe the solution you'd like Similar to network namespaces, a pod can share a time namespace. This would let the processes inside the container change the time without affecting the host or other pods. I think it should be shared in the pod because it's likely that containers in the same pod interact with each other at a low level that they'd not want to have time differences. IMO, this should be a new default since I can't really think of a case where it'd create issues for others but only additional isolation which is preferred. ### Additional context _No response_",
        "url": "https://github.com/containerd/containerd/issues/12517",
        "createdAt": "2025-11-15T10:05:21Z",
        "updatedAt": "2026-08-13T00:19:40Z",
        "timestamp": "2026-08-13T00:19:40Z",
        "metrics": {
          "reactions": 0,
          "comments": 5
        },
        "labels": [
          "kind/feature",
          "Stale",
          "area/runtime"
        ],
        "author": "muvaf",
        "state": "closed",
        "assignees": [
          "R3hankhan123"
        ]
      },
      {
        "id": "github:containerd/containerd:issue:12728",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "容器没有重启功能，是否可以出一个重启功能",
        "text": "### Description 容器没有重启功能，是否可以出一个重启功能，类似于docker restart id这样的功能 ### Steps to reproduce the issue 1. 2. 3. ### Describe the results you received and expected docker restart id ### What version of containerd are you using? 1.7 ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. _No response_",
        "url": "https://github.com/containerd/containerd/issues/12728",
        "createdAt": "2025-12-25T06:35:33Z",
        "updatedAt": "2026-08-13T00:19:43Z",
        "timestamp": "2026-08-13T00:19:43Z",
        "metrics": {
          "reactions": 0,
          "comments": 5
        },
        "labels": [
          "kind/bug",
          "kind/feature",
          "Stale",
          "area/runtime"
        ],
        "author": "957446525",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:12738",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "Continuous memory growth in containerd v2.1.4",
        "text": "### Description We are seeing continuous memory growth on `containerd` version `github.com/containerd/containerd/v2 2.1.4` in our AWS EKS v1.32 cluster. The containerd process memory grows constantly and reaches maximum capacity and causes the node to become non-responsive and also causes OS to kill other things abruptly due to memory pressure causing node to go into `NotReady` status. ### Steps to reproduce the issue We simply keep it running for a few days and memory grows. (Note: We have duplicate metrics. So two same value lines are shown. Ignore one) <img width=\"1831\" height=\"825\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/a25b8918-dae7-41bb-b2db-a3d3374ef8b3\" /> ### Describe the results you received and expected Memory should not grow unbounded like this. ### What version of containerd are you using? containerd github.com/containerd/containerd/v2 2.1.4 75cb2b7193e4e490e9fbdc236c0e811ccaba3376 ### Any other relevant information We also use these nodes for running periodic Argo jobs (short-lived jobs run periodically every 2mins, etc.). I am not sure if this has any relation. ### Show configuration if it is related to CRI plugin. _No response_",
        "url": "https://github.com/containerd/containerd/issues/12738",
        "createdAt": "2026-01-01T08:39:25Z",
        "updatedAt": "2026-08-13T00:19:44Z",
        "timestamp": "2026-08-13T00:19:44Z",
        "metrics": {
          "reactions": 2,
          "comments": 20
        },
        "labels": [
          "kind/bug",
          "area/runtime"
        ],
        "author": "spy16x",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:12752",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "Unpack failure when mixing remote and local snapshotters",
        "text": "### Description We are encountering unpack failures when mixing remote snapshotters (like `stargz`) and local snapshotters (like `overlayfs`) on the same node. If an image is first pulled using a remote snapshotter—which lazy-loads content—and is later used by a container requiring a local snapshotter, the operation fails. `LocalResolve` incorrectly assumes the image is ready for the local snapshotter because the image metadata exists. However, the local snapshotter requires full content blobs that were never downloaded, leading to a \"content not found\" error during unpacking. ### Steps to reproduce the issue 1. Environment: - Kubernetes: 1.35.0 - Containerd: 2.2.0 - Snapshotters: Stargz (0.17.0) and Overlayfs. 2. Run a pod using the stargz runtime handler. The image is pulled, metadata is stored, but content is lazy-loaded. 3. Run a second pod using the same image but with the default overlayfs runtime handler. ### Describe the results you received and expected Received result: - The Overlayfs Pod fails to start with the following error from kubelet: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 1s default-scheduler Successfully assigned default/overlayfs to lc0 Normal Pulled 0s (x2 over 0s) kubelet spec.containers{overlayfs}: Container image \"quay.io/chengyuzhu6/alpine:3.15.3-estargz\" already present on machine and can be accessed by the pod Warning Failed 0s (x2 over 0s) kubelet spec.containers{overlayfs}: Error: failed to create containerd container: error unpacking image: apply layer error for \"quay.io/chengyuzhu6/alpine:3.15.3-estargz\": failed to extract layer sha256:ebb0761a96f14086da76390e335406d5899572f36f1109aadf53f954a471fbec: failed to get reader from content store: content diges ``` Expected result: Containerd should run successfully. It should detect that the local snapshot/content is missing for the requested snapshotter and automatically fetch the missing blobs from the registry, allowing container with local snapshotter(e.g. overlayfs) to start ### What version of containerd are you using? 2.2.0 ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. [plugins.'io.containerd.cri.v1.images'.runtime_platforms.stargz] platform = 'linux' snapshotter = 'stargz' [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.stargz] runtime_type = \"io.containerd.runc.v2\" snapshotter = \"stargz\" [proxy_plugins] [proxy_plugins.stargz] type = \"snapshot\" address = \"/run/containerd-stargz-grpc/containerd-stargz-grpc.sock\" [proxy_plugins.stargz.exports] root = \"/var/lib/containerd-stargz-grpc/\" enable_remote_snapshot_annotations = \"true\" [[plugins.\"io.containerd.transfer.v1.local\".unpack_config]] platform = \"linux\" snapshotter = \"overlayfs\" [[plugins.\"io.containerd.transfer.v1.local\".unpack_config]] platform = \"linux\" snapshotter = \"stargz\"",
        "url": "https://github.com/containerd/containerd/issues/12752",
        "createdAt": "2026-01-07T11:19:28Z",
        "updatedAt": "2026-08-13T00:19:45Z",
        "timestamp": "2026-08-13T00:19:45Z",
        "metrics": {
          "reactions": 3,
          "comments": 3
        },
        "labels": [
          "kind/bug",
          "Stale"
        ],
        "author": "ChengyuZhu6",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:12760",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "Support env variable configuration for logging binary",
        "text": "### What is the problem you're trying to solve When configuring container IO to use a logger binary, containerd only allows me to pass values as command line arguments. ``` // BinaryIO forwards container STDOUT|STDERR directly to a logging binary func BinaryIO(binary string, args map[string]string) Creator ``` ### Describe the solution you'd like I would like the ability to also configure environment variables. E.g., https://github.com/aws/shim-loggers-for-containerd/pull/389 added support for passing the Splunk token as an argument *or* environment variable, and I would like switch to env. ``` TokenKey = \"splunk-token\" TokenEnvKey = \"SPLUNK_TOKEN\" ``` ### Additional context A PR which implements this change: https://github.com/containerd/containerd/pull/12877",
        "url": "https://github.com/containerd/containerd/issues/12760",
        "createdAt": "2026-01-08T22:29:12Z",
        "updatedAt": "2026-08-13T00:19:47Z",
        "timestamp": "2026-08-13T00:19:47Z",
        "metrics": {
          "reactions": 0,
          "comments": 2
        },
        "labels": [
          "kind/enhancement",
          "kind/feature",
          "Stale"
        ],
        "author": "willmyrs",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:12973",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "make benchmark never work",
        "text": "### Description make benchmark only test the current directory, not subdirectories. ### Steps to reproduce the issue ``` make benchmark + benchmark no Go files in /home/nmx/github.com/ningmingxiao/tmp/containerd FAIL . [setup failed] make: *** [Makefile:252：benchmark ``` ### Describe the results you received and expected make benchmark can test subdirectories ### What version of containerd are you using? master ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. _No response_",
        "url": "https://github.com/containerd/containerd/issues/12973",
        "createdAt": "2026-03-04T03:22:51Z",
        "updatedAt": "2026-08-13T00:19:57Z",
        "timestamp": "2026-08-13T00:19:57Z",
        "metrics": {
          "reactions": 0,
          "comments": 2
        },
        "labels": [
          "kind/bug",
          "go"
        ],
        "author": "ningmingxiao",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:13129",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "Cannot run OPA image in user namespace on GKE",
        "text": "## Short description `opa:1.14.1-envoy` image fails to start with `hostUsers: false` (Kubernetes user namespaces) due to missing `/dev/console` `containerd: 2.1.5` ``` $ kubectl version -o yaml clientVersion: buildDate: \"2026-03-18T18:30:07Z\" compiler: gc gitCommit: 6c1cd99aef09161ddb07b8ade6c9564e9b9bfe27 gitTreeState: clean gitVersion: v1.35.3 goVersion: go1.26.1 major: \"1\" minor: \"35\" platform: darwin/arm64 kustomizeVersion: v5.7.1 serverVersion: buildDate: \"2026-02-27T01:05:58Z\" compiler: gc emulationMajor: \"1\" emulationMinor: \"34\" gitCommit: f72eca22c700e6c114f9243d3ca6a1706e45c2a0 gitTreeState: clean gitVersion: v1.34.5-gke.1076000 goVersion: go1.24.13 X:boringcrypto major: \"1\" minCompatibilityMajor: \"1\" minCompatibilityMinor: \"33\" minor: \"34\" platform: linux/amd64 ``` ## Steps To Reproduce - Deploy this pod ``` apiVersion: v1 kind: Pod metadata: name: opa-userns-test spec: hostUsers: false securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true fsGroup: 1000 containers: - name: opa image: openpolicyagent/opa:1.14.1-envoy args: - run - --server - --diagnostic-addr=0.0.0.0:8282 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true ``` - It gets `CreateContainerError` - Set `hostUsers: true` (default) and re-deploy - works ## Expected behavior The above pod spec works in user namespace, i.e. with `hostUsers: false` ## Additional context I built a custom OPA image with the following `Dockerfile`. This image works with `hostUsers: false`. ``` FROM openpolicyagent/opa:1.14.1-envoy AS opa FROM busybox AS setup COPY --from=opa / /opa-root/ RUN mkdir -p /opa-root/dev && touch /opa-root/dev/console FROM openpolicyagent/opa:1.14.1-envoy COPY --from=setup /opa-root/ / ```",
        "url": "https://github.com/containerd/containerd/issues/13129",
        "createdAt": "2026-03-26T14:46:42Z",
        "updatedAt": "2026-08-13T16:08:46Z",
        "timestamp": "2026-08-13T16:08:46Z",
        "metrics": {
          "reactions": 0,
          "comments": 5
        },
        "labels": [
          "kind/support",
          "Stale"
        ],
        "author": "savitha-qs",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:13529",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "CRI: tag+digest sandbox image breaks RunPodSandbox",
        "text": "### Description A CRI sandbox image (`pinned_images.sandbox`, or legacy `sandbox_image`) configured as a reference with both a tag and a digest (name:tag@sha256:…) makes `RunPodSandbox` fail with \"failed to get sandbox image … not found\", even though containerd successfully pulls that exact image during the same call and stores it. No pods can start on the node. Identical config works on 2.2.x and fails on 2.3.x (verified on 2.3.0 and 2.3.1). https://github.com/containerd/containerd/pull/13513 might also fix this issue. ### Steps to reproduce the issue 1. Fetch containerd 2.3.1 + crictl into a temp dir: ```bash mkdir -p /tmp/ctd-repro && cd /tmp/ctd-repro curl -fsSL https://github.com/containerd/containerd/releases/download/v2.3.1/containerd-2.3.1-linux-amd64.tar.gz | tar xz curl -fsSL https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.34.0/crictl-v1.34.0-linux-amd64.tar.gz | tar xz -C bin ``` 2. Write a config with a `tag@digest` sandbox image: ```bash cat > config.toml <<EOF version = 3 root = \"/tmp/ctd-repro/root\" state = \"/tmp/ctd-repro/state\" [grpc] address = \"/tmp/ctd-repro/c.sock\" [plugins.\"io.containerd.cri.v1.images\".pinned_images] sandbox = \"registry.k8s.io/pause:3.10.1@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c\" [plugins.\"io.containerd.cri.v1.runtime\".containerd] default_runtime_name = \"runc\" [plugins.\"io.containerd.cri.v1.runtime\".containerd.runtimes.runc] runtime_type = \"io.containerd.runc.v2\" EOF ``` 3. Start the throwaway containerd (root, with `bin` on PATH for the shim/runc): ```bash sudo env \"PATH=/tmp/ctd-repro/bin:/usr/sbin:/usr/bin:/sbin:/bin\" ./bin/containerd -c config.toml &>containerd.log & sleep 3 ``` 4. Minimal pod-sandbox config (host network, so no CNI is needed): ```bash cat > pod.json <<'EOF' { \"metadata\": {\"name\":\"probe\",\"namespace\":\"default\",\"uid\":\"u-1\",\"attempt\":1}, \"log_directory\":\"/tmp\", \"linux\": {\"security_context\": {\"namespace_options\": {\"network\": 2}}} } EOF ``` 5. Run the sandbox — fails: ```console $ sudo ./bin/crictl -r unix:///tmp/ctd-repro/c.sock -i unix:///tmp/ctd-repro/c.sock runp pod.json ... code = NotFound desc = failed to start sandbox \"...\": failed to get sandbox image \"registry.k8s.io/pause:3.10.1@sha256:278fb9...\": image \"...\": not found ``` 6. …but the image is present — containerd pulled it during step 5 and stored it under its canonical (digest-only) name: ```console $ sudo ./bin/ctr -a /tmp/ctd-repro/c.sock -n k8s.io images ls | grep pause registry.k8s.io/pause@sha256:278fb9... ``` ### Describe the results you received and expected `RunPodSandbox` fails immediately with: ``` code = NotFound desc = failed to start sandbox \"...\": failed to get sandbox image \"registry.k8s.io/pause:3.10.1@sha256:278fb9...\": image \"...\": not found ``` containerd pulled that image during the same call, `ctr -n k8s.io images ls` shows it present as `registry.k8s.io/pause@sha256:278fb9...`, yet the sandbox lookup reports it missing. No pod sandbox is created, so the node can run no pods. `RunPodSandbox` should resolve the configured sandbox image (a valid `name:tag@digest` reference) and start the sandbox, as it does on containerd 2.2.x, and as 2.3.x already does when the same image is configured tag-only (`pause:3.10.1`) or digest-only (`pause@sha256:...`). ### What version of containerd are you using? containerd github.com/containerd/containerd/v2 v2.3.1 64b425cf570b3b8dd1d4cc46da7c1fce65c6651a ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. ```toml version = 3 root = \"/tmp/ctd-repro/root\" state = \"/tmp/ctd-repro/state\" [grpc] address = \"/tmp/ctd-repro/c.sock\" [plugins.\"io.containerd.cri.v1.images\".pinned_images] sandbox = \"registry.k8s.io/pause:3.10.1@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c\" [plugins.\"io.containerd.cri.v1.runtime\".containerd] default_runtime_name = \"runc\" [plugins.\"io.containerd.cri.v1.runtime\".containerd.runtimes.runc] runtime_type = \"io.containerd.runc.v2\" ```",
        "url": "https://github.com/containerd/containerd/issues/13529",
        "createdAt": "2026-06-04T06:31:48Z",
        "updatedAt": "2026-08-13T04:24:06Z",
        "timestamp": "2026-08-13T04:24:06Z",
        "metrics": {
          "reactions": 1,
          "comments": 1
        },
        "labels": [
          "kind/bug",
          "area/cri"
        ],
        "author": "hakman",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:13909",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "CRI image pull is sometimes canceled by image_pull_progress_timeout during unpack",
        "text": "### Description When the CRI plugin pulls an image through the transfer service (`use_local_image_pull = false`), the image pull can be canceled by `image_pull_progress_timeout` even when all blob downloads have finished and the remaining unpack is progressing. The unpack then fails with: ``` cancel pulling image <ref> because of no progress in 5m0s stop pulling image <ref>: active requests=1, bytes read=... failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:...) to overlayfs as \"extract-...\": context canceled ``` The CRI progress reporter's active-request counter leaks because the transfer service can skip the `\"complete\"` progress event due to a race between download-completion detection and the start of extraction. 1. The transfer service `ProgressTracker` polls statuses every 300ms and sends the `\"complete\"` event only while the job is in the `jobAdded`/`jobInProgress` state. https://github.com/containerd/containerd/blob/v2.2.3/core/transfer/local/progress.go#L108-L142 2. However, unpack starts immediately after the layer fetch completes, and the applier's progress callback (`diff.WithProgress(progressTracker.ExtractProgress)`, https://github.com/containerd/containerd/blob/v2.2.3/core/transfer/local/pull.go#L205) moves the job directly to `jobExtracting` via `j.extraction` in `ExtractProgress`. https://github.com/containerd/containerd/blob/v2.2.3/core/transfer/local/progress.go#L242-L247 The tick period in (1) is 300ms while extraction starts a few milliseconds after the fetch completes, so the job usually transitions directly from `jobInProgress` to `jobExtracting` before the next tick can detect the completion. `\"extracting\"`/`\"extracted\"` events are then sent instead of `\"complete\"`. 3. The CRI `transferProgressReporter.handleProgress` only handles `\"waiting\"`, `\"downloading\"`, and `\"complete\"`. `\"extracting\"` and `\"extracted\"` are ignored. https://github.com/containerd/containerd/blob/v2.2.3/internal/cri/server/images/image_pull.go#L923-L972 The status counted on the first `\"downloading\"` event is therefore never decremented, `activeReqs` stays at 1. 4. The watchdog resets its timer only when `activeReqs == 0` or `bytesRead` increases. During unpack neither holds, so after `image_pull_progress_timeout` it cancels the whole pull. https://github.com/containerd/containerd/blob/v2.2.3/internal/cri/server/images/image_pull.go#L1016-L1040 The cancellation lands in the unpacker's `Apply`, which produces the misleading \"failed to extract layer ... context canceled\". Any image whose unpack tail (after the last downloaded byte) takes longer than `image_pull_progress_timeout` can be killed this way. The `\"already exists\"` event is ignored as well. The patch below lets the CRI reporter treat these events as download completion and fixes the issue. ```diff --- a/internal/cri/server/images/image_pull.go +++ b/internal/cri/server/images/image_pull.go @@ -958,7 +958,7 @@ func (reporter *transferProgressReporter) handleProgress(p transfer.Progress) { delete(reporter.statuses, p.Name) } - case \"complete\": + case \"complete\", \"extracting\", \"extracted\", \"already exists\": if node, exists := reporter.statuses[p.Name]; exists { if curProgress := p.Progress - node.Progress; curProgress > 0 { reporter.IncBytesRead(curProgress ``` ### Steps to reproduce the issue 1. Configure the config with a short progress timeout to reproduce. ```toml # /etc/containerd/certs.d/localhost:5000/hosts.toml server = \"http://localhost:5000\" [host.\"http://localhost:5000\"] capabilities = [\"pull\", \"resolve\"] ``` ```toml # /etc/containerd/config.toml version = 3 root = '/var/lib/containerd' [plugins] [plugins.'io.containerd.cri.v1.images'] image_pull_progress_timeout = '5s' [plugins.'io.containerd.cri.v1.images'.registry] config_path = '/etc/containerd/certs.d' ``` ```console $ sudo systemctl restart containerd ``` 2. Push a test image whose unpacking takes over 5s to a local registry ```console $ docker run -d -p 5000:5000 --name registry registry:2 $ docker build -t localhost:5000/pull-timeout-repro:v1 - <<'EOF' FROM busybox RUN dd if=/dev/urandom of=/pad bs=1M count=512 && \\ dd if=/dev/zero of=/bloat bs=1M count=8192 EOF $ docker push localhost:5000/pull-timeout-repro:v1 ``` 3. Pull the image through CRI: ```console $ crictl pull localhost:5000/pull-timeout-repro:v1 ``` 4. Observe the failure and the containerd logs: ```console time=\"2026-08-04T01:08:54.969864113Z\" level=info msg=\"PullImage \\\"localhost:5000/pull-timeout-repro:v1\\\"\" time=\"2026-08-04T01:09:04.970872476Z\" level=error msg=\"cancel pulling image localhost:5000/pull-timeout-repro:v1 because of no progress in 5s\" time=\"2026-08-04T01:09:04.971276186Z\" level=info msg=\"stop pulling image localhost:5000/pull-timeout-repro:v1: active requests=1, bytes read=532676608\" time=\"2026-08-04T01:09:04.975872008Z\" level=error msg=\"failed to cleanup \\\"extract-44496416-qksT sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc\\\"\" error=\"snapshot extract-44496416-qksT sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc does not exist: not found\" time=\"2026-08-04T01:09:04.979959133Z\" level=error msg=\"PullImage \\\"localhost:5000/pull-timeout-repro:v1\\\" failed\" error=\"rpc error: code = Canceled desc = failed to pull and unpack image \\\"localhost:5000/pull-timeout-repro:v1\\\": failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:6351c1a637bce545bf24bf655339b5e123b094fd1ac3036ee79accbfc6879035) to overlayfs as \\\"extract-44496416-qksT sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc\\\": mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3090561630: context canceled\" ``` ### Describe the results you received and expected Received: the pull fails after downloads have completed, while the unpack is progressing. ``` $ crictl pull localhost:5000/pull-timeout-repro:v1 E0804 01:11:11.281711 263 remote_image.go:250] \"PullImage from image service failed\" err=\"rpc error: code = Canceled desc = failed to pull and unpack image \\\"localhost:5000/pull-timeout-repro:v1\\\": failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:6351c1a637bce545bf24bf655339b5e123b094fd1ac3036ee79accbfc6879035) to overlayfs as \\\"extract-346500103-1gP- sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc\\\": mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount4057491: context canceled\" image=\"localhost:5000/pull-timeout-repro:v1\" FATA[0010] pulling image: rpc error: code = Canceled desc = failed to pull and unpack image \"localhost:5000/pull-timeout-repro:v1\": failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:6351c1a637bce545bf24bf655339b5e123b094fd1ac3036ee79accbfc6879035) to overlayfs as \"extract-346500103-1gP- sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc\": mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount4057491: context canceled ``` Expected: the pull succeeds. ### What version of containerd are you using? containerd github.com/containerd/containerd/v2 v2.2.3 77c84241c7cbdd9b4eca2591793e3d4f4317c590 ### Any other relevant information ```console $ crictl --version crictl version v1.36.0 $ uname -r 7.0.0-28-generic ``` The latest release `v2.3.3` is also affected. ### Show configuration if it is related to CRI plugin. ```toml # /etc/containerd/config.toml version = 3 root = '/var/lib/containerd' [plugins] [plugins.'io.containerd.cri.v1.images'] image_pull_progress_timeout = '5s' [plugins.'io.containerd.cri.v1.images'.registry] config_path = '/etc/containerd/certs.d' ```",
        "url": "https://github.com/containerd/containerd/issues/13909",
        "createdAt": "2026-08-06T09:58:46Z",
        "updatedAt": "2026-08-13T00:24:18Z",
        "timestamp": "2026-08-13T00:24:18Z",
        "metrics": {
          "reactions": 1,
          "comments": 1
        },
        "labels": [
          "kind/bug",
          "area/cri"
        ],
        "author": "naoki9911",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:13952",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "containerd crash with program exceeds 10000-thread limit - container-log FIFO open leaks an OS thread per failed CreateContainer",
        "text": "### Description On a busy Kubernetes node, containerd crashed with: ``` runtime: program exceeds 10000-thread limit fatal error: thread exhaustion ``` The process had leaked OS threads steadily over ~17 hours until it hit Go's default maxmcount (10,000), then aborted. A goroutine dump taken at crash time shows the leaked threads are all blocked in a blocking open(2) on a container log FIFO. The CRI container stdout/stderr logger opens the read end of the container's log FIFO with a blocking `open(O_RDONLY)`. That open parks until the container task opens the write end. If the task never starts (e.g. it fails or times out during create/start), the write end never opens, the read open() blocks indefinitely, and, because it's a blocking syscall, it pins a dedicated OS thread that is never released. Under a condition that makes CreateContainer/task-start repeatedly fail, kubelet retries on its sync backoff. Each retry leaks one more thread. With enough churn this climbs linearly to the 10,000-thread limit and containerd aborts. Containerd/fifo does have a release path: a watcher goroutine (`fifo.go:116`, second trace below) unblocks the pending `open()` when its context is cancelled or `Close()` is called, by opening the other end `O_WRONLY|O_NONBLOCK`. The leak is that on this path nothing ever cancels the container-IO context or calls `Close()` - the gRPC client deadline kubelet hits doesn't propagate to it, so the watcher never fires and there is no timeout on the open, leaving the thread pinned for the life of the process. ``` goroutine 163741607 gp=0x4000d81a40 m=8013 mp=0x40b9e00008 [syscall, 492 minutes]: syscall.Syscall6(0x38, 0xffffffffffffff9c, 0x408d7e8618, 0x80000, 0x0, 0x0, 0x0) /usr/local/go/src/syscall/syscall_linux.go:95 syscall.openat(0xffffffffffffff9c, {0x408d7e8600?, 0x204141fc?}, 0x80000, 0x0) /usr/local/go/src/syscall/zsyscall_linux_arm64.go:98 syscall.Open(...) /usr/local/go/src/syscall/syscall_linux.go:284 os.open(...) /usr/local/go/src/os/file_open_unix.go:15 os.openFileNolog({0x408d7e8600, 0x12}, 0x0, 0x0) /usr/local/go/src/os/file_unix.go:278 os.OpenFile({0x408d7e8600, 0x12}, 0x0, 0x0) /usr/local/go/src/os/file.go:392 github.com/containerd/fifo.openFifo.func2() vendor/github.com/containerd/fifo/fifo.go:138 created by github.com/containerd/fifo.openFifo in goroutine 163741605 vendor/github.com/containerd/fifo/fifo.go:131 ``` This is the release path: ``` goroutine 164339856 [select, 466 minutes]: github.com/containerd/fifo.openFifo.func1() vendor/github.com/containerd/fifo/fifo.go:120 created by github.com/containerd/fifo.openFifo vendor/github.com/containerd/fifo/fifo.go:116 ``` The daemon requests `O_RDONLY|O_NONBLOCK`, but containerd/fifo strips `O_NONBLOCK` and performs the real `open() `in a background goroutine, so at the syscall level it's a blocking `open(O_RDONLY)` that parks until the task opens the write end. Threads and memory climbing before the crash: <img width=\"1470\" height=\"272\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/8e8adcb2-c006-485c-8075-3668a15de862\" /> ### Steps to reproduce the issue I cannot actually reproduce this, only happens under extreme load ### Describe the results you received and expected Not to have threads leak ### What version of containerd are you using? 2.2.1 ### Any other relevant information kernel 6.12.80 ### Show configuration if it is related to CRI plugin. _No response_",
        "url": "https://github.com/containerd/containerd/issues/13952",
        "createdAt": "2026-08-12T17:25:59Z",
        "updatedAt": "2026-08-13T09:32:09Z",
        "timestamp": "2026-08-13T09:32:09Z",
        "metrics": {
          "reactions": 0,
          "comments": 7
        },
        "labels": [
          "kind/bug",
          "area/cri"
        ],
        "author": "olyazavr",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:5708",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "containerd-shim process isn't reaped for some killed containers",
        "text": "**Description** We have several EKS clusters which autoscale throughout the day - they handle burst workloads, and in a given day the underlying ASG-s may scale down to 0 nodes, or scale up to tens of nodes. We've noticed that once in a while, we have nodes which have pods stuck in a 'Terminating' status on them for days on-end, until we manually intervene and force-delete them; I've SSH-ed to one of the nodes which experienced this behavior and tried to introspect the behavior, here's what I've gathered, hopefully covering most abstraction layers, but I could not find the root cause unfortunately - I'd love to know what can I do to further debug this. Quick summary (more info below): - Kubelet is trying to kill an already dead container indefinitely - Docker thinks the container is running, although it is not - Containerd shows the task is stopped - The container-shim process wasn't reaped and is still alive - The shim receives a kill request from containerd, it execve's runc to kill the container; it receives \"container not running\" and responds to containerd with \"process already finished: not found\" - The above loops for every kubelet's kill retry The above leads me to conclude that there's something fishy in the distributed orchestration of killing a container, I'd assume that it's somewhere between containerd<->runc, but I'm not entirely sure - and would love to know how can I better pinpoint the exact cause. **Steps to reproduce the issue:** I'm not entirely sure how to reproduce the behavior yet, as it happens sporadically in arbitrary nodes. **Describe the results you received:** Containers are dead, containerd is aware they are stopped, but the shim isn't reaped, docker thinks the container is still running and misleads kubelet to keep it as Terminated until manually intervening (force deleting the pod). **Describe the results you expected:** The shim should go down with the container, docker should be notified the container is stopped so that kubelet will update the pod's status accordingly. **What version of containerd are you using:** ``` $ containerd --version containerd github.com/containerd/containerd 1.4.1 c623d1b36f09f8ef6536a057bd658b3aa8632828 ``` **Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):** Deeper dive per abstraction layer - **Kubernetes / kubelet:** - The pod is stuck in a 'Terminating state' - There are no finalizers on the pod - When I describe the pod, I see something similar to 'Normal Killing 2m49s (x1715 over 2d6h) kubelet Stopping container csi-node' - In journalctl -u kubelet, I see the following near the time the pod started to terminate: ``` Jul 07 03:40:25 ip-10-0-73-87.ec2.internal kubelet[4811]: I0707 03:40:20.474218 4811 kubelet.go:1848] SyncLoop (DELETE, \"api\"): \"csi-driver-aws-csi-driver-node-2qssj_default(6f2f36c4-06f0-406d-9681-b92fa0106441) \" Jul 07 03:40:32 ip-10-0-73-87.ec2.internal kubelet[4811]: I0707 03:40:32.052286 4811 kubelet.go:1870] SyncLoop (PLEG): \"csi-driver-aws-csi-driver-node-2qssj_default(6f2f36c4-06f0-406d-9681-b92fa0106441)\", event: &pleg.PodLifecycleEvent{ID:\"6f2f36c4-06f0-406d-9681-b92fa0106441\", Type:\"ContainerDied\", Data:\"789845b4d5dc620ce62b36ff5a6d2ef380a725226264b0170d33ea645bb837f1\"} Jul 07 03:41:18 ip-10-0-73-87.ec2.internal kubelet[4811]: I0707 03:41:18.375163 4811 kubelet.go:1870] SyncLoop (PLEG): \"csi-driver-aws-csi-driver-node-2qssj_default(6f2f36c4-06f0-406d-9681-b92fa0106441)\", event: &pleg.PodLifecycleEvent{ID:\"6f2f36c4-06f0-406d-9681-b92fa0106441\", Type:\"ContainerDied\", Data:\"650c0009c27bfd04f4578e3a5fe2ce0eea300088acb5e669f1d71c5d187139ff\"} ``` - Then, I see indefinite \"killing container\" messages: ``` Jul 09 10:32:23 ip-10-0-73-87.ec2.internal kubelet[4811]: I0709 10:32:23.591613 4811 kuberuntime_container.go:635] Killing container \"docker://cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\" wi th a 30 second grace period Jul 09 10:34:24 ip-10-0-73-87.ec2.internal kubelet[4811]: I0709 10:34:24.591670 4811 kuberuntime_container.go:635] Killing container \"docker://cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\" wi th a 30 second grace period Jul 09 10:36:09 ip-10-0-73-87.ec2.internal kubelet[4811]: I0709 10:36:09.591654 4811 kuberuntime_container.go:635] Killing container \"docker://cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\" wi th a 30 second grace period ``` - I've uploaded the kubelet stacktrace here - [kubelet-stacktrace.txt](https://github.com/awslabs/amazon-eks-ami/files/6794004/kubelet-stacktrace.txt) **Docker:** - When I docker container ls / docker inspect <container-id> - the container status is 'Running' ``` { \"Id\": \"cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\", \"Created\": \"2021-07-07T00:11:27.14530101Z\", \"Path\": \"/entrypoint.sh\", \"Args\": [ \"--endpoint=unix:///csi/csi.sock\", \"--v=4\", \"--volume-attach-limit=5\" ], \"State\": { \"Status\": \"running\", \"Running\": true, \"Paused\": false, \"Restarting\": false, \"OOMKilled\": false, \"Dead\": false, \"Pid\": 7064, \"ExitCode\": 0, \"Error\": \"\", \"StartedAt\": \"2021-07-07T00:11:36.641265855Z\", \"FinishedAt\": \"0001-01-01T00:00:00Z\" }, ... } ``` - In journalctl, looking near the time the pod started terminating, I see the following: ``` -- Logs begin at Wed 2021-07-07 00:10:20 UTC, end at Fri 2021-07-09 10:46:56 UTC. -- Jul 07 03:30:43 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:30:43.252002398Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:30:44 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:30:44.542537801Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:39:32 ip-10-0-73-87.ec2.internal dockerd[4325]: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11) on returned error: write unix /var/run/docker.sock->@: write: broken pipe\" Jul 07 03:40:35 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:40:35.735692652Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:40:44 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:40:44.165635762Z\" level=error msg=\"stream copy error: reading from a closed fifo\" Jul 07 03:40:47 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:40:47.504174793Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:40:56 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:40:55.687736472Z\" level=info msg=\"Container cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a failed to exit within 30 se conds of signal 15 - using the force\" Jul 07 03:41:07 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:06.802768003Z\" level=info msg=\"Container cd7ed93ae2d1 failed to exit within 10 seconds of kill - trying direct SIGKILL\" Jul 07 03:41:17 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:17.058149021Z\" level=warning msg=\"Published ports are discarded when using host network mode\" Jul 07 03:41:17 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:17.066107174Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:41:48 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:48.397339761Z\" level=info msg=\"Container cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a failed to exit within 30 se conds of signal 15 - using the force\" Jul 07 03:41:58 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:58.415920505Z\" level=info msg=\"Container cd7ed93ae2d1 failed to exit within 10 seconds of kill - trying direct SIGKILL\" Jul 07 03:42:28 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:42:28.634636074Z\" level=info msg=\"Container cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a failed to exit within 30 se conds of signal 15 - using the force\" ``` - The SIGKILL / 'using the force' messages continue indefinitely - I've uploaded the dockerd stacktrace here - [docker-stacktrace.txt](https://github.com/awslabs/amazon-eks-ami/files/6793969/docker-stacktrace.txt) **Containerd:** - In containerd, the task of the container is 'STOPPED' - `cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a 7064 STOPPED` - The task's metrics prints the following: ``` ID TIMESTAMP cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a 2021-07-09 10:52:16.172595382 +0000 UTC METRIC VALUE memory.usage_in_bytes 20029440 memory.limit_in_bytes 9223372036854771712 memory.stat.cache 11411456 cpuacct.usage 69532053147 cpuacct.usage_percpu [14916245050 21098017259 14389165228 19128625610] pids.current 0 pids.limit 0 ``` - The container's info doesn't show something particularly helpful: ``` { \"ID\": \"cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\", \"Labels\": { \"com.docker/engine.bundle.path\": \"/var/run/docker/containerd/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\" }, \"Image\": \"\", \"Runtime\": { \"Name\": \"io.containerd.runtime.v1.linux\", \"Options\": { \"type_url\": \"containerd.linux.runc.RuncOptions\", \"value\": \"CgRydW5jEhwvdmFyL3J1bi9kb2NrZXIvcnVudGltZS1ydW5j\" } }, \"SnapshotKey\": \"\", \"Snapshotter\": \"\", \"CreatedAt\": \"2021-07-07T00:11:36.498772067Z\", \"UpdatedAt\": \"2021-07-07T00:11:36.498772067Z\", \"Extensions\": null, \"Spec\": { \"ociVersion\": \"1.0.1-dev\", ... }, ... } ``` - journalctl -u containerd does not have logs with the given container id - I've uploaded the containerd stack trace here - [containerd-goroutine-stacktrace.txt](https://github.com/awslabs/amazon-eks-ami/files/6793929/containerd-goroutine-stacktrace.txt) **containerd-shim:** - The containerd-shim of the container is still running: ``` root 7023 0.0 0.0 710748 6268 ? Sl Jul07 0:12 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f36 44815387a37a -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc ``` - I tried to peek at the stdout / stderr of the shim in - but either the process hadn't flushed data, or it is simply empty - To find the shim id, I've looked at the process fd-s and correlated the opened sockets w/ ss: ``` [root@ip-10-0-73-87 ~]# ls -al /proc/$pid/fd | awk '/socket/ { print $NF }' | grep -o '[0-9]*' | xargs -I{} sh -c \"ss -apn | grep {}\" u_str LISTEN 0 4096 /run/containerd/s/6e99a634bfa5b915cbeade50e47384f60874a9358e5e96cb59523a46339c138b 25584 * 0 users:((\"containerd-shim\",pid=7023,fd=12),(\"containerd\",pid=3538,fd=80)) u_str ESTAB 0 0 /run/containerd/s/6e99a634bfa5b915cbeade50e47384f60874a9358e5e96cb59523a46339c138b 24369 * 25595 users:((\"containerd-shim\",pid=7023,fd=3)) u_str ESTAB 0 0 * 25595 * 24369 users:((\"containerd\",pid=3538,fd=87)) ``` - Then I looked at the containerd journal entries for the shim id (6e99a6...) and the only thing that came up was: `Jul 07 00:11:36 ip-10-0-73-87.ec2.internal containerd[3538]: time=\"2021-07-07T00:11:36.521930693Z\" level=info msg=\"shim containerd-shim started\" address=\"unix:///run/containerd/s/6e99a634bfa5b915cbeade50e47384f6087 4a9358e5e96cb59523a46339c138b\" debug=false pid=7023` - I've managed to retrieve the shim's go stack trace by strace-ing to a file and sending a kill -USR1 to it, but I don't see anything of particular interest there: <details> <summary>shim stacktrace</summary> <p> ``` write(8</var/lib/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a/shim.stdout.log>, \"time=\"2021-07-09T11:56:32Z\" level=info msg=\"=== BEGIN goroutine stack dump === goroutine 9 [running]: main.dumpStacks(0xc000064150) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:276 +0x74 main.executeShim.func1(0xc00004a1e0, 0xc000064150) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:186 +0x3d created by main.executeShim /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:184 +0x5e9 goroutine 1 [select]: main.handleSignals(0xc000064150, 0xc0000a8720, 0xc00005c090, 0xc00006e000, 0xc0000d9e50, 0x0) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:239 +0x119 main.executeShim(0xc0000d2540, 0x7fe26c883088) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:189 +0x625 main.main() /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:118 +0x20a goroutine 18 [chan receive]: main.main.func1() /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:89 +0x85 created by main.main /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:88 +0x74 goroutine 6 [syscall]: syscall.Syscall6(0xe8, 0xb, 0xc00003b9b8, 0x80, 0xffffffffffffffff, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x4) /usr/lib/golang/src/syscall/asm_linux_amd64.s:41 +0x5 github.com/containerd/containerd/vendor/golang.org/x/sys/unix.EpollWait(0xb, 0xc00003b9b8, 0x80, 0x80, 0xffffffffffffffff, 0xffffffffffffffff, 0x85a8e0, 0xa5cb40) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x72 github.com/containerd/containerd/vendor/github.com/containerd/console.(*Epoller).Wait(0xc00005c2d0, 0xc0000d9a78, 0x8) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/console/console_linux.go:111 +0x7a created by github.com/containerd/containerd/runtime/v1/shim.(*Service).initPlatform /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service_linux.go:113 +0xbb goroutine 5 [chan receive]: github.com/containerd/containerd/runtime/v1/shim.(*Service).processExits(0xc00006e000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service.go:501 +0xd6 created by github.com/containerd/containerd/runtime/v1/shim.NewService /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service.go:92 +0x40d goroutine 24 [runnable]: os/signal.process(0x85ec60, 0xa5cb70) /usr/lib/golang/src/os/signal/signal.go:240 +0x10a os/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x45 created by os/signal.Notify.func1.1 /usr/lib/golang/src/os/signal/signal.go:150 +0x45 goroutine 7 [chan receive, 3375 minutes]: github.com/containerd/containerd/runtime/v1/shim.(*Service).forward(0xc00006e000, 0x85a3e0, 0xc00001a030) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service.go:579 +0x71 created by github.com/containerd/containerd/runtime/v1/shim.NewService /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service.go:96 +0x4bf goroutine 8 [IO wait, 23 minutes]: internal/poll.runtime_pollWait(0x7fe26c882de8, 0x72, 0x0) /usr/lib/golang/src/runtime/netpoll.go:220 +0x55 internal/poll.(*pollDesc).wait(0xc000074098, 0x72, 0x0, 0x0, 0x7fee5f) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000074080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/internal/poll/fd_unix.go:394 +0x1fc net.(*netFD).accept(0xc000074080, 0xc00009e458, 0xc0000121b8, 0x79ccc0) /usr/lib/golang/src/net/fd_unix.go:172 +0x45 net.(*UnixListener).accept(0xc00005c330, 0xc00008fe50, 0xc00008fe58, 0x18) /usr/lib/golang/src/net/unixsock_posix.go:162 +0x32 net.(*UnixListener).Accept(0xc00005c330, 0x812878, 0xc000012190, 0x862f20, 0xc0000a0000) /usr/lib/golang/src/net/unixsock.go:260 +0x65 github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Server).Serve(0xc00005c090, 0x862f20, 0xc0000a0000, 0x861720, 0xc00005c330, 0x0, 0x0) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:87 +0x107 main.serve.func1(0x861720, 0xc00005c330, 0xc00005c090, 0x862f20, 0xc0000a0000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:224 +0x88 created by main.serve /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:222 +0x1fe goroutine 25 [select]: github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run(0xc00009e3c0, 0x862f20, 0xc0000a0000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:431 +0x433 created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Server).Serve /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:127 +0x28d goroutine 10 [IO wait]: internal/poll.runtime_pollWait(0x7fe26c882d08, 0x72, 0x85a8e0) /usr/lib/golang/src/runtime/netpoll.go:220 +0x55 internal/poll.(*pollDesc).wait(0xc0000e2318, 0x72, 0x85a800, 0xa4b248, 0x0) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0000e2300, 0xc000077000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/internal/poll/fd_unix.go:159 +0x1a5 net.(*netFD).Read(0xc0000e2300, 0xc000077000, 0x1000, 0x1000, 0xc000056270, 0xc00018fc28, 0x20) /usr/lib/golang/src/net/fd_posix.go:55 +0x4f net.(*conn).Read(0xc0000a6078, 0xc000077000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/net.go:182 +0x8e bufio.(*Reader).Read(0xc00004a420, 0xc000078060, 0xa, 0xa, 0xc00018fd98, 0x447f74, 0xc00018ff18) /usr/lib/golang/src/bufio/bufio.go:227 +0x222 io.ReadAtLeast(0x859fa0, 0xc00004a420, 0xc000078060, 0xa, 0xa, 0xa, 0xa, 0x2, 0x789f60) /usr/lib/golang/src/io/io.go:314 +0x87 io.ReadFull(...) /usr/lib/golang/src/io/io.go:333 github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc000078060, 0xa, 0xa, 0x859fa0, 0xc00004a420, 0x73, 0xc00002ec00, 0x0, 0x6b0aae) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:53 +0x69 github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc000078040, 0xc00018fe2c, 0x3, 0x2, 0xc00014a400, 0x0, 0x0, 0x0) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:101 +0x6b github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run.func1(0xc00005e120, 0xc00009e3c0, 0xc00005e1e0, 0xc000078040, 0xc00005e180, 0xc00004a480) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:362 +0x1b0 created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:332 +0x2c5 goroutine 17533 [select, 23 minutes]: github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run(0xc00009e1e0, 0x862f20, 0xc0000a0000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:431 +0x433 created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Server).Serve /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:127 +0x28d goroutine 17545 [select, 23 minutes]: github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run(0xc000012190, 0x862f20, 0xc0000a0000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:431 +0x433 created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Server).Serve /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:127 +0x28d === END goroutine stack dump ===\" namespace=moby path=/run/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a pid=7023\" ``` </p> </details> - When strace-ing the shim, I saw that it execve-s runc with the following: ``` /usr/sbin/runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a/log.json --log-format json kill cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a 9 ``` to which runc returns \"container not running\", and in turn the shim reports to containerd - \"process already finished: not found\" **runc:** - There are no hanging runc processes which I could observe, nor any journal logs related; not sure how to introspect this layer after the fact. - `runc --root /var/run/docker/runtime-runc/moby events cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a` shows \"container with id ... is not running\" - `runc list` shows the container as stopped - `runc state` shows the following: ``` { \"ociVersion\": \"1.0.2-dev\", \"id\": \"cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\", \"pid\": 0, \"status\": \"stopped\", \"bundle\": \"/run/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\", \"rootfs\": \"/var/lib/docker/overlay2/91052c9a423d3bee16e65973b097e7d0eb35baa5200e2c8f05f3484850e4c1c1/merged\", \"created\": \"2021-07-07T00:11:36.60742938Z\", \"owner\": \"\" } ``` **os:** - The container's process is gone, meaning it was actually killed **What you expected to happen**: Pods should terminate once their underlying container had died. **How to reproduce it (as minimally and precisely as possible)**: Not actually sure how to reproduce it consistently - it happens when creating and destroying nodes rapidly, I'd assume. <details> <summary>Environment</summary> <p> ``` - Kubernetes version: 1.19 - Docker version output: Client: Version: 19.03.13-ce API version: 1.40 Go version: go1.13.15 Git commit: 4484c46 Built: Mon Oct 12 18:51:20 2020 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 19.03.13-ce API version: 1.40 (minimum version 1.12) Go version: go1.13.15 Git commit: 4484c46 Built: Mon Oct 12 18:51:50 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.1 GitCommit: c623d1b36f09f8ef6536a057bd658b3aa8632828 runc: Version: 1.0.0-rc93 GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec docker-init: Version: 0.19.0 GitCommit: de40ad0 - Docker info output: Client: Debug Mode: false Server: Containers: 14 Running: 9 Paused: 0 Stopped: 5 Images: 8 Server Version: 19.03.13-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: c623d1b36f09f8ef6536a057bd658b3aa8632828 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec init version: de40ad0 (expected: fec3683) Security Options: seccomp Profile: default Kernel Version: 5.4.117-58.216.amzn2.x86_64 Operating System: Amazon Linux 2 OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.46GiB Name: ip-10-0-73-87.ec2.internal ID: BFFR:6SUN:2BSZ:4MB4:K5NO:OBN2:6VHK:Z2YQ:LS3U:KPEW:5TUV:AEBW Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: true ``` </p> </details> <details><summary><code>runc --version</code></summary><br><pre> $ runc --version runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.15.8 libseccomp: 2.4.1 </pre></details> <details><summary><code>uname -a</code></summary><br><pre> $ uname -a Linux ip-10-0-73-87.ec2.internal 5.4.117-58.216.amzn2.x86_64 #1 SMP Tue May 11 20:50:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux </pre></details>",
        "url": "https://github.com/containerd/containerd/issues/5708",
        "createdAt": "2021-07-09T22:51:31Z",
        "updatedAt": "2026-08-12T20:08:24Z",
        "timestamp": "2026-08-12T20:08:24Z",
        "metrics": {
          "reactions": 5,
          "comments": 18
        },
        "labels": [
          "kind/bug"
        ],
        "author": "dany74q",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:issue:8893",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "issue",
        "title": "TaskOOM event lost",
        "text": "### Description https://github.com/kubernetes/kubernetes/issues/119600 When I was troubleshooting the problem that the Reason was not OOMKilled, I found that the container lost the TaskOOM event. ### Steps to reproduce the issue e2e: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-containerd-node-e2e-1-7/1683184526936772608 logfile: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-containerd-node-e2e-1-7/1683184526936772608/artifacts/tmp-node-e2e-92a26bb7-cos-97-16919-353-1-system.log containerd_id: 98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54 containerd_name: oomkill-multi-target-container 1. kernel having trigger oom ``` Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efb6ec_09ab_4eec_8ad6_f953eb1a7466.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efb6ec_09ab_4eec_8ad6_f953eb1a7466.slice/cri-containerd-98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54.scope,task=dd,pid=38287,uid=0 Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 kernel: Memory cgroup out of memory: Killed process 38287 (dd) total-vm:21724kB, anon-rss:14744kB, file-rss:508kB, shmem-rss:0kB, UID:0 pgtables:68kB oom_score_adj:997 Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 kernel: Tasks in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efb6ec_09ab_4eec_8ad6_f953eb1a7466.slice/cri-containerd-98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54.scope are going to be killed due to memory.oom.group set ``` 2. and then containerd exit shim ``` Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 containerd[910]: time=\"2023-07-23T18:56:46.329304474Z\" level=debug msg=\"received exit event container_id:\\\"98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54\\\" id:\\\"98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54\\\" pid:38265 exit_status:137 exited_at:{seconds:1690138606 nanos:325950236}\" Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 containerd[910]: time=\"2023-07-23T18:56:46.389869167Z\" level=info msg=\"shim disconnected\" id=98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54 namespace=k8s.io Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 containerd[910]: time=\"2023-07-23T18:56:46.390142692Z\" level=warning msg=\"cleaning up after shim disconnected\" id=98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54 namespace=k8s.io ``` 3. After that, no TaskOOM event was received until container delete. ### Describe the results you received and expected Expect to receive TaskOOM event. ### What version of containerd are you using? 1.7 ### Any other relevant information ubuntu ### Show configuration if it is related to CRI plugin. no",
        "url": "https://github.com/containerd/containerd/issues/8893",
        "createdAt": "2023-07-30T15:02:09Z",
        "updatedAt": "2026-08-13T16:54:10Z",
        "timestamp": "2026-08-13T16:54:10Z",
        "metrics": {
          "reactions": 0,
          "comments": 17
        },
        "labels": [
          "Stale"
        ],
        "author": "lengrongfu",
        "state": "open",
        "assignees": [
          "lengrongfu",
          "fuweid"
        ]
      },
      {
        "id": "github:containerd/containerd:pull_request:11458",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "ignore update status file when no space",
        "text": "Fix #7247 Failed to start recovre, the root cause is inconsistent data between `cri` and `container in metadata` Ensure consistency, the following manual recovery could be resolve. ``` - stop containerd service, like `systemctl stop containerd` - disable cri plugin in config.toml and start containerd service - find container and task which reversed, and stop task, delete container to release disk, which most like - `ctr -n k8s.io t stop [id]` - `ctr -n k8s.io c rm [id]` - enable cri plugin in config.toml - restart containerd service, `systemctl restart containerd` ``` As described in [here](https://github.com/containerd/containerd/issues/11504), ignoring noSpace error is a more convenient way to handle such issues",
        "url": "https://github.com/containerd/containerd/pull/11458",
        "createdAt": "2025-03-03T06:50:30Z",
        "updatedAt": "2026-08-13T00:19:12Z",
        "timestamp": "2026-08-13T00:19:12Z",
        "metrics": {
          "reactions": 0,
          "comments": 4
        },
        "labels": [
          "area/cri",
          "needs-ok-to-test",
          "Stale",
          "size/L"
        ],
        "author": "yylt",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:11477",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "Draft: ctr support print plugin config",
        "text": "``` bin/ctr plugin inspect runtime { \"cdiSpecDirs\": [ \"/etc/cdi\", \"/var/run/cdi\" ], \"cni\": { \"binDir\": \"/opt/cni/bin\", \"confDir\": \"/etc/cni/net.d\", \"confTemplate\": \"\", \"ipPref\": \"\", \"maxConfNum\": 1, \"setupSerially\": false, \"useInternalLoopback\": false }, \"containerd\": { \"defaultRuntimeName\": \"runc\", \"ignoreBlockIONotEnabledErrors\": false, \"ignoreRdtNotEnabledErrors\": false, \"runtimes\": { \"runc\": { \"ContainerAnnotations\": [], \"PodAnnotations\": [], \"baseRuntimeSpec\": \"\", \"cgroupWritable\": false, \"cniConfDir\": \"\", \"cniMaxConfNum\": 0, \"io_type\": \"\", \"options\": { \"BinaryName\": \"\", \"CriuImagePath\": \"\", \"CriuPath\": \"\", \"CriuWorkPath\": \"\", \"IoGid\": 0, \"IoUid\": 0, \"NoNewKeyring\": false, \"NoPivotRoot\": false, \"Root\": \"\", \"ShimCgroup\": \"\", \"SystemdCgroup\": false }, \"privileged_without_host_devices\": false, \"privileged_without_host_devices_all_devices_allowed\": false, \"runtimePath\": \"\", \"runtimeType\": \"io.containerd.runc.v2\", \"sandboxer\": \"podsandbox\", \"snapshotter\": \"\" } } }, \"containerdEndpoint\": \"/run/containerd/containerd.sock\", \"containerdRootDir\": \"/var/lib/containerd\", \"device_ownership_from_security_context\": false, \"disableApparmor\": false, \"disableHugetlbController\": true, \"disableProcMount\": false, \"drainExecSyncIOTimeout\": \"0s\", \"enableCDI\": false, \"enableSelinux\": true, \"enableUnprivilegedICMP\": false, \"enableUnprivilegedPorts\": false, \"ignoreDeprecationWarnings\": null, \"ignoreImageDefinedVolumes\": false, \"maxContainerLogSize\": 16384, \"netnsMountsUnderStateDir\": false, \"restrictOOMScoreAdj\": false, \"rootDir\": \"/var/lib/containerd/io.containerd.grpc.v1.cri\", \"selinuxCategoryRange\": 1024, \"stateDir\": \"/run/containerd/io.containerd.grpc.v1.cri\", \"tolerateMissingHugetlbController\": true, \"unsetSeccompProfile\": \"\" } ```",
        "url": "https://github.com/containerd/containerd/pull/11477",
        "createdAt": "2025-03-04T11:44:15Z",
        "updatedAt": "2026-08-13T00:19:14Z",
        "timestamp": "2026-08-13T00:19:14Z",
        "metrics": {
          "reactions": 0,
          "comments": 4
        },
        "labels": [
          "kind/feature",
          "area/cri",
          "needs-ok-to-test",
          "Stale",
          "size/M",
          "do-not-merge/work-in-progress"
        ],
        "author": "ningmingxiao",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:12889",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "docker fetcher: strip sensitive headers on descriptor URLs",
        "text": "# docker fetcher: strip sensitive headers for desc.urls fetches context: - containerd supports fetching content from `descriptor.urls` (oci descriptor URLs). - the desc.urls fetch path currently reuses resolver/global headers (e.g., CRI `registry.headers` / hosts headers). - when desc.urls points to a non-registry origin, forwarding well-known sensitive headers (e.g., `Authorization`, `Proxy-Authorization`, `Cookie`) is a cross-origin trust-boundary footgun. change: - keep operator control and existing behavior, but strip a small set of well-known sensitive headers on desc.urls-driven requests: `Authorization`, `Proxy-Authorization`, `Cookie`, `Cookie2`. - this mirrors go standard library redirect behavior where sensitive headers are not forwarded cross-origin. test: - add `TestFetcherDescURLsDoesNotForwardResolverHeaders` to assert sensitive headers are not sent to desc.urls origins, while non-sensitive custom headers still flow. scope: - `core/remotes/docker` only. pins: - repo: https://github.com/containerd/containerd - commit: c6d6c8af753c3b5cb6528231b9063d1372f7d226",
        "url": "https://github.com/containerd/containerd/pull/12889",
        "createdAt": "2026-02-11T20:14:10Z",
        "updatedAt": "2026-08-12T23:29:53Z",
        "timestamp": "2026-08-12T23:29:53Z",
        "metrics": {
          "reactions": 0,
          "comments": 5
        },
        "labels": [
          "kind/enhancement",
          "cherry-pick/1.7.x",
          "size/L",
          "area/distribution",
          "cherry-pick/2.0.x",
          "cherry-pick/2.2.x",
          "cherry-pick/2.3.x"
        ],
        "author": "1seal",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13294",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "vendor: github.com/sirupsen/logrus v1.10.0",
        "text": "### vendor: github.com/sirupsen/logrus v1.10.0 Notable changes: - Add bidirectional `log/slog` integration with a Logrus hook and `slog.Handler`. - Add minimal, composable logging interfaces for individual log levels. - Fix reentrant logging deadlocks and improve concurrency safety around formatters, hooks, and entries. - Fix generic `Log`, `Logf`, `Logln`, and `LogFn` methods unexpectedly panicking at `PanicLevel`. - Allow `Entry.Caller` to be set explicitly and preserved across derived entries. - Improve `TextFormatter` performance and reduce allocations significantly. - Improve common Logger and Entry hot paths and caller-reporting performance. - Update `TextFormatter` handling for `[]byte`, debug/trace colors, and Windows ANSI terminals. - Raise the minimum supported Go version to Go 1.23. - Deprecate `Entry.HasCaller` and `MutexWrap`. release-notes: https://github.com/sirupsen/logrus/releases/tag/v1.10.0 full diff: https://github.com/sirupsen/logrus/compare/v1.9.4...v1.10.0",
        "url": "https://github.com/containerd/containerd/pull/13294",
        "createdAt": "2026-04-27T13:41:27Z",
        "updatedAt": "2026-08-13T15:55:43Z",
        "timestamp": "2026-08-13T15:55:43Z",
        "metrics": {
          "reactions": 0,
          "comments": 1
        },
        "labels": [
          "size/XXL"
        ],
        "author": "thaJeztah",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13359",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "Allow hosts file configuration for proxies per registry",
        "text": "Fixes #8377 The setup: I have a proxy running that has DNS mapped `local.registry` to `127.0.0.1`, because I have a registry (with a self signed certificate) running locally as well with an image tagged `local.registry/localpython` in it. (Setup directions are [here](https://gist.github.com/lauralorenz/085c7368886ea4f8ab8b9e2419db8841).) ``` docker run -d --name local-squid-proxy --net=host --add-host local.registry:127.0.0.1 ubuntu/squid:latest docker run -d --name registry -p 443:443 -v /tmp/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:3 ``` ``` lauralorenz@lauralorenz:bin$ curl -k https://local.registry/v2/localpython/tags/list {\"name\":\"localpython\",\"tags\":[\"latest\"]} ``` In both of the following test scenarios (before the feature and after) I have configured the following hosts.toml: ``` lauralorenz@lauralorenz:containerd$ cat /etc/containerd/certs.d/local.registry/hosts.toml [host.\"https://local.registry\"] capabilities = [\"resolve\",\"pull\"] proxy=\"http://localhost:3128\" skip_verify=true ``` Then I ran against a containerd and ctr built on main (1b71eeeae7d537ec94bcd467089aea136256b83e), and then again on a contained and ctr built from this commit. # Before this PR / on main Does not care about the proxy setting in hosts.toml and simply can't find the host. ``` lauralorenz@lauralorenz:containerd$ sudo /usr/local/google/home/lauralorenz/go/src/containerd/containerd/bin/ctr images pull --hosts-dir=/etc/containerd/certs.d local.registry/localpython:latest ctr: failed to resolve image: failed to do request: Head \"https://local.registry/v2/localpython/manifests/latest\": dial tcp: lookup local.registry on 127.0.0.1:53: no such host ``` # With this PR Uses proxy setting in hosts.toml to get to the forward proxy at `localhost`, which itself forwards the original hostname request `local.registry` to the registry itself. ``` lauralorenz@lauralorenz:containerd$ sudo /usr/local/google/home/lauralorenz/go/src/containerd/containerd/bin/ctr images pull --hosts-dir=/etc/containerd/certs.d local.registry/localpython:latest local.registry/localpython:latest saved └──manifest (a3f5fd4ce3ad) complete |++++++++++++++++++++++++++++++++++++++| ├──config (7c8620a05d35) complete |++++++++++++++++++++++++++++++++++++++| ├──layer (fd8e41ac7277) extracted |++++++++++++++++++++++++++++++++++++++| ├──layer (ea923c2ed79e) extracted |++++++++++++++++++++++++++++++++++++++| ├──layer (5c671a5c7ab3) extracted |++++++++++++++++++++++++++++++++++++++| └──layer (6a0ac1617861) extracted |++++++++++++++++++++++++++++++++++++++| application/vnd.docker.distribution.manifest.v2+json sha256:a3f5fd4ce3adead6336ca2f69f4589b41c7b24317e451e4048116cbd6d761be4 Completed pull from OCI Registry (local.registry/localpython:latest) elapsed: 2.2 s total: 19.4 M (8.9 MiB/s) ``` ``` lauralorenz@lauralorenz:containerd$ sudo /usr/local/google/home/lauralorenz/go/src/containerd/containerd/bin/ctr content list DIGEST SIZE AGE LABELS sha256:5c671a5c7ab3e0fe7526f4ff0a233b54727a1f407035f414a73138d2b0886733 455.7kB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/uncompressed=sha256:611cae45bc2cb7ff6a78488ab602a59918fbc0431a0f12281dacf10cbb7ba94f sha256:6a0ac1617861a677b045b7ff88545213ec31c0ff08763195a70a4a5adda577bb 3.864MB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/uncompressed=sha256:29df493baa13de438d6d2ece3a8333032e0b7b9b9d8cce4ee82194da255f61e1 sha256:7c8620a05d355e188e9c92949198770fd43d150d051f8c3223f506f215a9f12d 5.276kB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/gc.ref.snapshot.overlayfs=sha256:b4f837ae9f4abc0db962ac37b08df79e45a60a17d7e88ffe7aafb537d1e58353 sha256:a3f5fd4ce3adead6336ca2f69f4589b41c7b24317e451e4048116cbd6d761be4 1.157kB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/gc.ref.content.config=sha256:7c8620a05d355e188e9c92949198770fd43d150d051f8c3223f506f215a9f12d,containerd.io/gc.ref.content.l.0=sha256:6a0ac1617861a677b045b7ff88545213ec31c0ff08763195a70a4a5adda577bb,containerd.io/gc.ref.content.l.1=sha256:5c671a5c7ab3e0fe7526f4ff0a233b54727a1f407035f414a73138d2b0886733,containerd.io/gc.ref.content.l.2=sha256:ea923c2ed79ed2d7163aacc54686501cb3339215671be89b50b62814aeffb95c,containerd.io/gc.ref.content.l.3=sha256:fd8e41ac72776720a5ce79d8a0c66e75ae8d3d836cb15cc6ae0682b0b7999a11 sha256:ea923c2ed79ed2d7163aacc54686501cb3339215671be89b50b62814aeffb95c 16.02MB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/uncompressed=sha256:b02b147e872fb22e2550aaaeae713706d551a5321619cf742e89c4144b54c7d4 sha256:fd8e41ac72776720a5ce79d8a0c66e75ae8d3d836cb15cc6ae0682b0b7999a11 248B About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/uncompressed=sha256:6239b5ae5fda090b9f21fcd4bc2b888837f17814c06c0642921cdb35c6a847e5 ```",
        "url": "https://github.com/containerd/containerd/pull/13359",
        "createdAt": "2026-05-07T21:34:54Z",
        "updatedAt": "2026-08-12T17:11:42Z",
        "timestamp": "2026-08-12T17:11:42Z",
        "metrics": {
          "reactions": 0,
          "comments": 8
        },
        "labels": [
          "ok-to-test",
          "size/L"
        ],
        "author": "lauralorenz",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13635",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "oom: avoid per-wakeup allocations in cgroup v2 OOM watcher (#13558)",
        "text": "## What this does Reduces continuous CPU/GC overhead of the per-container cgroup v2 OOM watcher (`internal/oom`), which is the only steadily-active, workload- proportional code path in `containerd-shim-runc-v2` on a static workload. ## Background (#13558) The watcher arms an inotify `IN_MODIFY` watch on `memory.events`. The kernel notifies that file on **every** counter change (`low`/`high`/`max`/`oom`/ `oom_kill`/`oom_group_kill`), not just OOM kills. Containers running near their memory limit churn `high`/`max` constantly (and increasingly as page cache fills toward the limit over the node's uptime), waking the watcher goroutine very frequently. Previously each wakeup allocated a new read buffer **and** a `map[string]uint64`, then opened and fully parsed `memory.events` with `bufio.Scanner`/`strings.Fields` just to read `oom_kill`. Across ~50–60 shims this steady allocation churn is avoidable CPU and GC pressure that accumulates on long-running nodes. ## Change - Allocate the inotify and `memory.events` read buffers once per watcher and reuse them. - Read only the `oom_kill` counter with a small allocation-light parser. - No behavioral change: OOM detection and cgroup-deletion handling are identical; covered by the existing `TestWatcher` plus a new parser unit test.",
        "url": "https://github.com/containerd/containerd/pull/13635",
        "createdAt": "2026-06-20T12:04:35Z",
        "updatedAt": "2026-08-12T20:02:30Z",
        "timestamp": "2026-08-12T20:02:30Z",
        "metrics": {
          "reactions": 0,
          "comments": 5
        },
        "labels": [
          "size/L",
          "cherry-pick/2.3.x"
        ],
        "author": "KeyMoad",
        "state": "open",
        "assignees": [
          "fuweid"
        ]
      },
      {
        "id": "github:containerd/containerd:pull_request:13653",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "shim: send event to a queue to prevent event to be dropped",
        "text": "@dmcgowan @fuweid @mxpv can you take a look. fix https://github.com/containerd/containerd/issues/12678 ``` func (s *service) processExits() { for e := range s.ec { // if consumer use much time more than 1s event will be dropped } ``` because notify will drop new events if consumer use more than 1s ``` func Reap() error { now := time.Now() exits, err := reap(false) for _, e := range exits { done := Default.notify(runc.Exit{ Timestamp: now, Pid: e.Pid, Status: e.Status, }) select { case <-done: case <-time.After(1 * time.Second): } } return err } ``` I create a queue to add this event into it it will make consumer consume event quickly It's easy to reproduce it add some sleep here (simulate high-load scenarios because sometimes wait lifecycleMu lock may quite a while) ``` func (s *service) processExits() { for e := range s.ec { // add some sleep time.Sleep(time.Second * 3) s.lifecycleMu.Lock() ``` reproduce ``` ✦2 ❯ cat container.json { \"metadata\": { \"name\": \"TESTFAILPOINTPROFILE_delayUpdate5\" }, \"image\": { \"image\": \"busybox:latest\" }, \"command\": [ 'sleep','infinity' ], \"log_path\": \"busybox5.log\", \"linux\": { \"resources\": { \"memory\": { \"limit\": 20971 } } } } root in /home/nmx/pods via 🐹 v1.26.3 ✦2 ❯ cat pod.json { \"metadata\": { \"name\": \"busybox-sandbox2\", \"namespace\": \"default\", \"attempt\": 1 }, \"annotations\": { }, \"log_directory\": \"/tmp\" } ``` crictl run --no-pull container.json pod.json 66abdf4d13 go run main.go cat main.go ``` package main import ( \"fmt\" \"os/exec\" \"sync\" \"time\" ) func main() { const ( totalTask = 100 maxConcur = 50 ) var wg sync.WaitGroup ch := make(chan struct{}, maxConcur) go func() { time.Sleep(time.Second*2) cmd := exec.Command(\"sh\", \"-c\", \"kill -9 $(pidof sleep)\") cmd.CombinedOutput() }() for i := 0; i < totalTask; i++ { ch <- struct{}{} wg.Add(1) go func(idx int) { defer func() { <-ch wg.Done() }() cmd := exec.Command(\"sh\", \"-c\", \"crictl exec 66abdf4d13 date\") _, err := cmd.CombinedOutput() if err != nil { fmt.Printf(\"Task %d execution failed: %v\\n\", idx, err) } }(i) } wg.Wait() close(ch) fmt.Println(\"done\") } ``` ``` crictl ps|grep 66a 66abdf4d13b04 busybox:latest About a minute ago Running TESTFAILPOINTPROFILE_delayUpdate5 0 bb4899de19161 unknown unknown crictl exec 66a date FATA[0000] execing command in container 66a: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"35d0a3f6180055dcc2694b39c53d7e4215613850e622222cb78be205f1016fae\": OCI runtime exec failed: exec failed: cannot exec in a stopped container ```` container is still running even main process is exited. @evgfitil",
        "url": "https://github.com/containerd/containerd/pull/13653",
        "createdAt": "2026-06-23T02:48:10Z",
        "updatedAt": "2026-08-13T01:21:19Z",
        "timestamp": "2026-08-13T01:21:19Z",
        "metrics": {
          "reactions": 1,
          "comments": 4
        },
        "labels": [
          "size/L"
        ],
        "author": "ningmingxiao",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13705",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "fix: prevent tar extraction data corruption by limiting reader size",
        "text": "we find sometimes extract a binary file from a blob, binary is broken ls -l fabric-manager_bad fabric-manager_ok -rwxr-xr-x 1 root root 44538840 6月 12 15:44 fabric-manager_bad -rwxr-xr-x 1 root root 44538840 6月 12 15:44 fabric-manager_ok ``` cmp -l fabric-manager_bad fabric-manager_ok 5525505 177 75 5525506 376 272 5525571 377 203 5525572 377 75 5525573 377 167 5525574 377 222 5525575 377 126 5525576 377 2 5525577 377 0 5525578 377 164 5525579 377 13 5525580 377 110 5525581 377 213 5525582 377 20 5525583 377 350 5525584 377 255 5525585 377 107 5525586 377 264 5525588 377 111 5525589 377 211 5525590 377 23 5525591 377 110 5525592 377 215 5525593 377 25 5525594 377 344 5525595 377 314 5525596 377 102 5525597 377 1 5525598 377 110 5525599 377 211 5525600 377 20 5525601 377 110 5525602 377 215 5525603 377 5 5525604 377 231 5525605 377 127 5525606 377 22 5525607 377 1 5525608 377 110 5525609 377 213 5525610 377 134 5525611 377 44 5525612 377 40 5525613 377 110 5525614 377 215 5525615 377 15 5525616 377 51 5525617 377 37 5525618 377 70 5525619 377 1 5525620 377 277 5525621 377 16 5525622 377 0 5525623 377 0 5525624 377 0 5525625 377 350 5525626 377 303 5525627 377 255 5525628 377 254 5525630 377 110 5525631 377 307 5525632 377 100 5525633 377 10 5525634 377 160 5525635 377 0 5525636 377 0 5525637 377 0 5525638 377 203 5525639 377 75 5525640 377 64 5525641 377 222 5525642 377 126 5525643 377 2 5525644 377 0 5525645 377 164 5525646 377 13 5525647 377 110 5525648 377 213 5525649 377 20 5525650 377 350 5525651 377 152 5525652 377 107 5525653 377 264 5525655 377 111 5525656 377 211 5525657 377 23 5525658 377 110 5525659 377 215 5525660 377 25 5525661 377 4 5525662 377 70 5525663 377 100 5525664 377 1 5525665 377 110 5525666 377 211 5525667 377 20 5525668 377 110 5525669 377 215 5525670 377 5 5525671 377 126 5525672 377 127 5525673 377 22 5525674 377 1 5525675 377 110 5525676 377 213 5525677 377 134 5525678 377 44 5525679 377 40 5525680 377 110 5525681 377 215 5525682 377 15 5525683 377 172 5525684 377 140 5525685 377 67 5525686 377 1 5525687 377 277 5525688 377 5 5525689 377 0 5525690 377 0 5525691 377 0 5525692 377 17 5525693 377 37 5525694 377 104 5525695 377 0 5525696 377 0 5525697 377 350 5525698 377 173 5525699 377 255 5525700 377 254 5525702 377 110 5525703 377 307 5525704 377 100 5525705 377 10 5525706 377 124 5525707 377 5 5525708 377 0 5525709 377 0 5525710 377 203 5525711 377 75 5525712 377 354 5525713 377 221 5525714 377 126 5525715 377 2 5525716 377 0 5525717 377 164 5525718 377 13 5525719 377 110 5525720 377 213 5525721 377 20 5525722 377 350 5525723 377 42 5525724 377 107 5525725 377 264 5525727 377 111 5525728 377 211 5525729 377 23 5525730 377 110 5525731 377 215 5525732 377 25 5525733 377 163 5525734 377 305 5525735 377 106 5525736 377 1 5525737 377 110 5525738 377 211 5525739 377 20 5525740 377 110 5525741 377 215 5525742 377 5 5525743 377 16 5525744 377 127 5525745 377 22 5525746 377 1 5525747 377 110 5525748 377 213 5525749 377 134 5525750 377 44 5525751 377 40 5525752 377 110 5525753 377 215 5525754 377 15 5525755 377 77 5525756 377 243 5525757 377 67 5525758 377 1 5525759 377 277 5525760 377 10 5525761 377 0 5525762 377 0 5525763 377 0 5525764 377 350 5525765 377 70 5525766 377 255 5525767 377 254 5525769 377 110 5525770 377 307 5525771 377 100 5525772 377 10 5525773 377 307 5525774 377 4 5525775 377 0 5525776 377 0 5525777 377 203 5525778 377 75 5525779 377 251 5525780 377 221 5525781 377 126 5525782 377 2 5525783 377 0 5525784 377 164 5525785 377 17 5525786 377 110 5525787 377 213 5525788 377 20 5525789 377 17 5525790 377 37 5525791 377 100 5525792 377 0 5525793 377 350 5525794 377 333 5525795 377 106 5525796 377 264 5525798 377 111 5525799 377 211 5525800 377 23 5525801 377 110 5525802 377 215 5525803 377 25 5525804 377 335 5525805 377 214 5525806 377 106 5525807 377 1 5525808 377 110 5525809 377 211 5525810 377 20 5525811 377 110 5525812 377 215 5525813 377 5 5525814 377 307 5525815 377 126 5525816 377 22 5525817 377 1 5525818 377 110 5525819 377 213 5525820 377 134 5525821 377 44 5525822 377 40 5525823 377 110 5525824 377 215 5525825 377 15 5525826 377 260 5525827 377 156 5525828 377 70 5525829 377 1 5525830 377 277 5525831 377 21 5525832 377 0 5525833 377 0 5525834 377 0 5525835 377 350 5525836 377 361 5525837 377 254 5525838 377 254 5525840 377 110 5525841 377 307 5525842 377 100 5525843 377 10 5525844 377 144 5525845 377 5 5525846 377 0 5525847 377 0 5525848 377 203 5525849 377 75 5525850 377 142 5525851 377 221 5525852 377 126 5525853 377 2 5525854 377 0 5525855 377 146 5525856 377 220 5525857 377 165 5525858 377 7 5525859 377 110 5525860 377 213 5525861 377 124 5525862 377 44 5525863 377 40 5525864 377 353 5525865 377 37 5525866 377 110 5525867 377 213 5525868 377 15 5525869 377 300 5525870 377 117 5525871 377 123 5525872 377 2 5525873 377 110 5525874 377 213 5525875 377 20 5525876 377 350 5525877 377 310 5525878 377 106 5525879 377 264 5525881 377 111 5525882 377 211 5525883 377 23 5525884 377 110 5525885 377 213 5525886 377 124 5525887 377 44 5525888 377 40 5525889 377 111 5525890 377 211 5525891 377 123 5525892 377 10 5525893 377 111 5525894 377 211 5525895 377 113 5525896 377 20 5525897 377 110 5525898 377 215 5525899 377 15 5525900 377 40 5525901 377 312 5525902 377 106 5525903 377 1 5525904 377 110 5525905 377 211 5525906 377 10 5525907 377 110 5525908 377 211 5525909 377 25 5525910 377 227 5525911 377 117 5525912 377 123 5525913 377 2 5525914 377 110 5525915 377 203 5525916 377 304 5525917 377 50 5525918 377 135 5525919 377 303 5525920 377 220 5525921 377 350 5525922 377 33 5525923 377 51 5525924 377 264 5525926 377 351 5525927 377 266 5525928 377 374 5525931 377 314 5525932 377 314 5525933 377 314 5525934 377 314 5525935 377 314 5525936 377 314 5525937 377 314 5525938 377 314 5525939 377 314 5525940 377 314 5525941 377 314 5525942 377 314 5525943 377 314 5525944 377 314 5525945 377 314 5525946 377 314 5525947 377 314 5525948 377 314 5525949 377 314 5525950 377 314 5525951 377 314 5525952 377 314 5525953 377 111 5525954 377 73 ```",
        "url": "https://github.com/containerd/containerd/pull/13705",
        "createdAt": "2026-07-02T04:50:56Z",
        "updatedAt": "2026-08-12T21:20:52Z",
        "timestamp": "2026-08-12T21:20:52Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/XS"
        ],
        "author": "ningmingxiao",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13775",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "core/remotes/docker: only fetch descriptor urls for foreign layers",
        "text": "dockerFetcher.Fetch walks desc.URLs before it tries the registry endpoints and issues a GET to each one, checking only that the scheme is http or https. Those urls come straight from the image manifest, so a crafted image can put an arbitrary address on any descriptor (a config, a regular layer, even a child manifest) and make the daemon request it during a pull, for instance an internal service or a cloud metadata endpoint. The urls field is the mechanism for non-distributable (foreign) layers, and distributable content is always available from the registry, which is exactly what the loop already falls through to. Gating on images.IsNonDistributable keeps foreign layers working while dropping urls on every other descriptor, so the check sits next to the request it guards instead of at each caller.",
        "url": "https://github.com/containerd/containerd/pull/13775",
        "createdAt": "2026-07-11T05:25:11Z",
        "updatedAt": "2026-08-12T23:24:59Z",
        "timestamp": "2026-08-12T23:24:59Z",
        "metrics": {
          "reactions": 0,
          "comments": 3
        },
        "labels": [
          "size/M"
        ],
        "author": "aysha-afrah26",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13799",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "metadata: bound snapshotter Remove during garbage collection",
        "text": "Fixes #13798 **Problem** During snapshot GC, `cleanupSnapshotter` calls the snapshotter's `Remove` while holding the metadata store write lock, with no deadline (it even strips the caller's cancellation). If the snapshotter never answers, the lock is held forever. Every snapshot operation on the node queues behind it, including CRI `RunPodSandbox`, so no new container can start. Nothing on the readiness path takes this lock, so the node keeps reporting Ready the whole time. ```mermaid flowchart TD K[kubelet] -->|Ready probe| S[\"CRI Status()\"] K -->|RunPodSandbox| P[snapshot Prepare] S --> OK[\"returns OK - never takes the lock\"] P --> L{{metadata store write lock}} GC[snapshot GC] -->|holds| L GC -->|\"Remove() - no deadline\"| SOCI[soci snapshotter<br/>out of fds, never answers] L -.->|blocked behind GC| P ``` We hit this in production: our soci snapshotter ran out of file descriptors and stopped answering, the GC's `Remove` parked in `waitOnHeader`, and the node could not start a single pod for 4.5 hours while showing Ready. Goroutine dumps are in the issue. The fd leak was a soci bug (fixed upstream in awslabs/soci-snapshotter#2043), but any proxy snapshotter that stalls reproduces this: one stuck RPC should not brick the node. **Fix** Bound each GC-path `Remove` with a `pkg/timeout` key (`io.containerd.timeout.snapshot.remove`, default 5m), the same pattern as shim cleanup and bolt open timeouts. On the first timed-out `Remove` the rest of the pass is abandoned, so a wedged snapshotter costs one timeout per GC cycle instead of one per orphan. Skipped snapshots stay orphaned and are retried next pass. The deferred `Cleanup` call is skipped in that case too, since it is an unbounded RPC to the same snapshotter that just failed to answer. **Not in this PR** `Walk` runs under the same lock and is also unbounded. Left out to keep this change small; can follow up.",
        "url": "https://github.com/containerd/containerd/pull/13799",
        "createdAt": "2026-07-15T16:37:00Z",
        "updatedAt": "2026-08-13T08:15:08Z",
        "timestamp": "2026-08-13T08:15:08Z",
        "metrics": {
          "reactions": 0,
          "comments": 3
        },
        "labels": [
          "size/L",
          "do-not-merge/contains-merge-commits"
        ],
        "author": "NahumLitvin",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13830",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "ctr: dedupe CRI image aliases in images list by default",
        "text": "## Problem Fixes #11000 (related: #8421, #10090, #10303, #10328). CRI writes three image names into the containerd metadata store for each pull (`repo:tag`, `repo@digest`, and a digest-only config/image ID). That is intentional for CRI's own image store, but `ctr -n k8s.io images ls` lists every name, so operators see three rows for one image. ``` $ ctr -n k8s.io i ls REF ... DIGEST docker.io/library/centos:7 ... sha256:be65... docker.io/library/centos@sha256:be65... ... sha256:be65... sha256:eeb6ee3f44bd... ... sha256:be65... ``` ## Design (display-only, minimal UX) **Prefer filtering at list time over changing CRI create behavior.** | Approach | Pros | Cons | | --- | --- | --- | | **A. Display filter in `ctr images list` (this PR)** | No metadata / CRI semantics change; safe for scripts that still know exact names; matches docker-like \"show tags by default\" | Must use `--all` to see aliases; nerdctl not covered | | B. Stop CRI from creating repo@digest / imageID names | True uniqueness in meta.db | High risk: CRI, GC, pinned images, and other tools may depend on those names; needs broader design | | C. Dedupe only by digest (one row total) | Cleanest table | Hides multiple tags that legitimately share a digest | This PR implements **A**, as sketched in the issue. ### Behavior - **Default:** For each target digest, if any name is a **tagged** ref (`name:tag`), only tagged refs are printed. CRI's `repo@digest` and bare digest ID rows are hidden when a tag exists. - **No tagged name:** All refs for that digest are still shown (digest-only images are not dropped). - **`--all` / `-a`:** Previous behavior — print every reference. - Metadata store, delete, pull, and CRI are **unchanged**. Deleting still requires the exact name (or multiple deletes) as today; fixing delete-by-ID is a separate problem. ### Why not filter only on `io.cri-containerd.image=managed`? A pure \"tagged only\" filter (as in the issue snippet) would hide legitimate digests that were never tagged. Grouping by target digest and preferring tags when present avoids that. ## Code - `cmd/ctr/commands/images/images.go` — `--all` flag + `filterImageListForDisplay` - `cmd/ctr/commands/images/list_filter_test.go` — unit tests for CRI triple, multi-tag, digest-only, ordering ## Test ``` go test ./cmd/ctr/commands/images/ -run TestFilterImageListForDisplay ``` ## Notes for maintainers Happy to adjust defaults if preferred: 1. Default stays full list; opt-in `--dedupe` instead of opt-out `--all` 2. Also collapse multi-tag digests to a single row (not done here) 3. Document-only note in `docs/namespaces.md` without code change Open to discussion if approach B (CRI create) is preferred long-term; this PR is intentionally the small, reversible UX fix.",
        "url": "https://github.com/containerd/containerd/pull/13830",
        "createdAt": "2026-07-22T19:18:33Z",
        "updatedAt": "2026-08-12T19:54:56Z",
        "timestamp": "2026-08-12T19:54:56Z",
        "metrics": {
          "reactions": 0,
          "comments": 1
        },
        "labels": [
          "size/L"
        ],
        "author": "locker95",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13910",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "pkg/shim: Report bootstrap API mismatch on startup",
        "text": "- replace / close: https://github.com/containerd/containerd/pull/13764 - address / close: https://github.com/containerd/containerd/issues/13763 A pre-2.3 containerd daemon can start a newly installed shim through the deprecated CLI, environment, and stdin API. The shim currently accepts that request but returns a protobuf bootstrap result, which the old daemon cannot parse. It treats the protobuf bytes as a socket address and fails later with a cryptic error: ``` failed to create TTRPC connection: unsupported protocol: \\b\\x03\\x12Yunix ``` This patch makes the shim reject the deprecated startup request before starting and tell the user that the containerd daemon may need to be restarted. The 2.2 daemon includes the shim's stderr in its start error, so this message reaches the user directly. This explicitly removes input-side support for the deprecated startup API, but it does not break a working standard containerd pairing. Pre-2.3 daemons already cannot consume the response from the new shim. After a package upgrade, a restarted daemon uses the new bootstrap API. Custom callers that send the deprecated request format while supporting the new protobuf response will now also be rejected.",
        "url": "https://github.com/containerd/containerd/pull/13910",
        "createdAt": "2026-08-06T19:31:15Z",
        "updatedAt": "2026-08-12T20:28:43Z",
        "timestamp": "2026-08-12T20:28:43Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/M",
          "cherry-pick/2.3.x"
        ],
        "author": "vvoland",
        "state": "open",
        "assignees": [
          "vvoland"
        ]
      },
      {
        "id": "github:containerd/containerd:pull_request:13915",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "remotes/docker: retry blob fetch on connection reset by peer",
        "text": "## What this does `httpReadSeeker` already resumes a blob fetch when the connection closes cleanly before the full body arrives (`io.ErrUnexpectedEOF`): it reopens the body with a ranged request at the current offset, bounded by `maxRetry` consecutive reopens that make no progress. A mid-body TCP RST, however, surfaces as a connection-reset error, misses that case, and fails the whole fetch — discarding everything already transferred. This PR treats connection resets like unexpected EOFs: retry by reopening at the current offset, under the exact same `maxRetry`/no-progress bound. The reset class is matched per-platform alongside the existing `isConnError` helpers (`ECONNRESET`, plus `WSAECONNRESET` on Windows, where `syscall.ECONNRESET` is a synthetic constant a real reset never matches), unwrapping with `errors.Is`. The unexpected-EOF match also moves from bare equality to `errors.Is`, so a wrapped `io.ErrUnexpectedEOF` now retries as well. ## Why On congested or middlebox-mediated paths, long blob transfers can stall, and far-end idle timers kill stalled flows with a RST after ~30s. In one observed case, a pull repeatedly transferred about 63% of a ~190MB layer at steady throughput, took a mid-body reset, and restarted from byte zero each time, never completing. With reconnect-at-offset covering the reset class, such a pull completes within the existing retry budget. Registries and range-capable blob backends already serve the reopen path today; only the error classification is blind to resets. Symptom report: #11365. Adjacent precedent: google/go-containerregistry#1415, which retries resets during blob reads. ## Caveats and scope - **Deliberate refusal:** retrying on RST could mask a far end intentionally killing the transfer. The existing consecutive-zero-progress bound caps that at `maxRetry` reopens — identical to today's handling of the unexpected-EOF class. - **Deliberately narrow:** `ETIMEDOUT` and generic temporary network errors are *not* covered. Question for maintainers: would you want those classes in a follow-up, or is reset-only the right scope? - **HTTP/2:** stream-level resets (e.g. `RST_STREAM`) surface as different error types and are out of scope here. - **Interaction with #13571:** that PR (activity-based timeout) touches the same area; this change does not restructure the reader. Happy to rebase/adjust if #13571 lands first. ## How it was tested New `httpreadseeker_test.go`, table-driven over both retryable classes (unexpected EOF, and a reset built as `*net.OpError` → `*os.SyscallError` → `ECONNRESET`, as a TCP read surfaces it): - reopen is re-invoked at the **advanced** offset and the transfer completes across mid-body failures when progress occurs; - `maxRetry` consecutive zero-progress failures abort with the original error (exactly `1 + maxRetry` opens); - unit coverage of `isRetryableReadError`, including wrapped forms and negative cases (`io.EOF`, `ECONNREFUSED`). - a Windows-only test (`httpreadseeker_windows_test.go`) covering the `WSAECONNRESET` arm, bare and wrapped in `*net.OpError`/`*os.SyscallError`; Also run locally: `go build`, `go vet`, full `core/remotes/docker/...` tests, `golangci-lint` with the repo config (0 issues), and cross-compiles for `GOOS=windows` (build + vet), `darwin`, and `freebsd`.",
        "url": "https://github.com/containerd/containerd/pull/13915",
        "createdAt": "2026-08-07T08:53:41Z",
        "updatedAt": "2026-08-12T17:15:51Z",
        "timestamp": "2026-08-12T17:15:51Z",
        "metrics": {
          "reactions": 0,
          "comments": 2
        },
        "labels": [
          "size/L"
        ],
        "author": "mabdelmalek-ant",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13924",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "fix(cri): CRI image pull is sometimes canceled by image_pull_progress…",
        "text": "Fixes #13909. ### Description When the CRI plugin pulls an image through the transfer service, the image pull can be canceled by `image_pull_progress_timeout` even when all blob downloads have finished and the remaining unpack is progressing. This happens because the transfer service's polling period is 300ms, and extraction can start within a few milliseconds after download completes. The job transitions directly to `jobExtracting` and sends `\"extracting\"`/`\"extracted\"` events instead of `\"complete\"`. Since the CRI `transferProgressReporter.handleProgress` only handled `\"complete\"`, the active request counter was never decremented, causing the watchdog to eventually time out the pull due to lack of download progress. This patch fixes the issue by allowing the CRI reporter to treat `\"extracting\"`, `\"extracted\"`, and `\"already exists\"` events as completion of the download phase, properly decrementing the active request counter and avoiding the timeout.",
        "url": "https://github.com/containerd/containerd/pull/13924",
        "createdAt": "2026-08-08T23:17:38Z",
        "updatedAt": "2026-08-12T20:28:41Z",
        "timestamp": "2026-08-12T20:28:41Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/M",
          "cherry-pick/2.2.x",
          "cherry-pick/2.3.x"
        ],
        "author": "RohithPariki",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13928",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "tracing: align span attribute keys with OTel semantic conventions",
        "text": "Closes #12191 Rename trace span attribute keys to follow the OpenTelemetry semantic conventions. Keys that have no semconv equivalent (snapshotter, pull ref, unpack) are moved under the `containerd.` namespace instead. | Legacy key | New key | | --- | --- | | `task.container.id` | `container.id` | | `task.runtime.name` | `container.runtime.name` | | `task.pid` | `process.pid` | | `task.process.id` | `process.pid` | | `image.id` | `container.image.id` | | `image.ref` | `containerd.image.ref` | | `snapshotter.name` | `containerd.snapshotter.name` | | `container.snapshotter.name` | `containerd.snapshotter.name` | | `unpack` | `containerd.pull.unpack` | To avoid breaking existing consumers, both the new and legacy keys are emitted for now (same approach as `HTTPStatusCodeAttributes`). The legacy keys are documented as deprecated in `docs/tracing.md` and can be dropped in a future release. `container.image.ref` is kept as is, since an image ref is not equivalent to `container.image.id`, `container.image.name` (see discussion in #12191). Attributes not mentioned in the issue are left untouched.",
        "url": "https://github.com/containerd/containerd/pull/13928",
        "createdAt": "2026-08-09T13:22:35Z",
        "updatedAt": "2026-08-12T16:12:11Z",
        "timestamp": "2026-08-12T16:12:11Z",
        "metrics": {
          "reactions": 0,
          "comments": 5
        },
        "labels": [
          "size/L"
        ],
        "author": "chansuke",
        "state": "open",
        "assignees": [],
        "change": "updated"
      },
      {
        "id": "github:containerd/containerd:pull_request:13941",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "erofs: instrument warm up cache",
        "text": "It'd good go understand: - Rate of cache lookups. - How many cache hits alone / layer bytes served from cache. - Amount of time spent converting layers on cache miss",
        "url": "https://github.com/containerd/containerd/pull/13941",
        "createdAt": "2026-08-11T18:50:53Z",
        "updatedAt": "2026-08-13T15:35:21Z",
        "timestamp": "2026-08-13T15:35:21Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/M"
        ],
        "author": "mxpv",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13942",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "Prepare release notes for v2.3.4",
        "text": "containerd 2.3.4 Welcome to the v2.3.4 release of containerd! The fourth patch release for containerd 2.3 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) * Set default runtimeFeatures.UserNamespacesHostNetwork to true in CRI ([#13914](https://github.com/containerd/containerd/pull/13914)) * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * Support non-UTF-8 binary environment variable values in CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13778](https://github.com/containerd/containerd/pull/13778)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13734](https://github.com/containerd/containerd/pull/13734)) * Normalize sandbox image references in CRI to resolve images without domain prefixes ([#13759](https://github.com/containerd/containerd/pull/13759)) #### Node Resource Interface (NRI) * Emit deprecation warnings for plugins using deprecated NRI interfaces ([#13935](https://github.com/containerd/containerd/pull/13935)) #### Runtime * Enable log scrubbing by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * Fix memory leak in OOM watcher map when stopping container monitoring ([#13870](https://github.com/containerd/containerd/pull/13870)) * Avoid orphaning shims on transient errors when loading process IDs ([#13857](https://github.com/containerd/containerd/pull/13857)) * Fix corruption of binary protobuf shim start responses caused by premature whitespace trimming ([#13803](https://github.com/containerd/containerd/pull/13803)) #### Snapshotters * Fix EROFS snapshotter dropping lower layers stacked above merged filesystem metadata ([#13876](https://github.com/containerd/containerd/pull/13876)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Maksym Pavlenko * Mike Brown * Akihiro Suda * Derek McGowan * Jordan Liggitt * Paweł Gronowski * Amir Alavi * Andrew Halaney * Bing Hongtao * Chris Henzie * Harshal Patel * Krisztian Litkey * Phil Estes * Wei Fu * ningmingxiao ### Changes <details><summary>35 commits</summary> <p> * [`c1f5360ef`](https://github.com/containerd/containerd/commit/c1f5360ef7b9afb748d4119ad7be665ba5144480) Prepare release notes for v2.3.4 * cri: disable restore in CreateContainer by default ([#13913](https://github.com/containerd/containerd/pull/13913)) * [`928c03c68`](https://github.com/containerd/containerd/commit/928c03c6898cad95f795300281fdff0dbc55a32f) cri: disable restore in CreateContainer by default * nri,deprecation: record and emit warnings for NRI deprecations. ([#13935](https://github.com/containerd/containerd/pull/13935)) * [`5966e2edb`](https://github.com/containerd/containerd/commit/5966e2edb15bb939aa2bb73649fbad73a2b04094) nri,deprecation: emit warnings for old NRI plugins. * Set the default of runtimeFeatures.UserNamespacesHostNetwork to true ([#13914](https://github.com/containerd/containerd/pull/13914)) * [`ab52c58f1`](https://github.com/containerd/containerd/commit/ab52c58f16ac50bb6724354bf36cbb4da480c2a2) Set the default of runtimeFeatures.UserNamespacesHostNetwork to true * Use ScrubLogs by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * [`0c255158a`](https://github.com/containerd/containerd/commit/0c255158aa1ecab40aafc42269cfc018a46eb627) ctr: add --scrub-logs flag for Windows * [`1c2b13dc9`](https://github.com/containerd/containerd/commit/1c2b13dc975468197e57321e25fd09163f78a75a) cri/config: use ScrubLogs by default on Windows * snapshots/erofs: keep lowers stacked above a merged fsmeta ([#13876](https://github.com/containerd/containerd/pull/13876)) * [`0c511e068`](https://github.com/containerd/containerd/commit/0c511e068b7d6dd815418185a235b3ba3347c65a) snapshots/erofs: keep lowers stacked above a merged fsmeta * cri: deprecate restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * [`dc98141df`](https://github.com/containerd/containerd/commit/dc98141df70d3d72d46125c6f30de0d2de648086) cri: deprecate restore in CreateContainer * internal/oom: Fix memory leak by removing watcher from map on Stop ([#13870](https://github.com/containerd/containerd/pull/13870)) * [`537d82d54`](https://github.com/containerd/containerd/commit/537d82d545759d4a072d564d4937707af2650163) internal/oom: Fix memory leak by removing watcher from map on Stop * shim_load: Consider shim leaked only if we can't find pids ([#13857](https://github.com/containerd/containerd/pull/13857)) * [`decf97a9c`](https://github.com/containerd/containerd/commit/decf97a9c7c26f92391d2c862d35832e0a291e80) shim_load: Consider shim leaked only if we can't find pids * core/runtime/v2: Drop checkpointctl module dependency ([#13840](https://github.com/containerd/containerd/pull/13840)) * [`796f07dc8`](https://github.com/containerd/containerd/commit/796f07dc853d9ae8a3a1278fe47436755a4bc3b7) core/runtime/v2: Drop checkpointctl module dependency * Handle []byte envvar value for CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * [`751fddddb`](https://github.com/containerd/containerd/commit/751fddddbb630631e12fd63e5dac4a9530188822) Handle []byte envvar value * [`0bce9060e`](https://github.com/containerd/containerd/commit/0bce9060ef3401e690b541dd6d2008c2abc50b0d) update to v0.36.x kubernetes dependencies * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13778](https://github.com/containerd/containerd/pull/13778)) * [`61a8f6f45`](https://github.com/containerd/containerd/commit/61a8f6f45e34660d787988e8a7e7a218f556369c) fix(cri): introspect OCI runtime features for non-runc runtimes * core/runtime/v2: Preserve protobuf shim response bytes ([#13803](https://github.com/containerd/containerd/pull/13803)) * [`1d28017be`](https://github.com/containerd/containerd/commit/1d28017be295005bfa3aee1d47afe86341d391b6) core/runtime/v2: Preserve protobuf shim response bytes * Disable checkpoint restore codepath when CRIU is not installed ([#13734](https://github.com/containerd/containerd/pull/13734)) * [`374091d67`](https://github.com/containerd/containerd/commit/374091d67c1966969b772594ba96a74f4357bc7a) github/workflows: install criu in node-e2e * [`db03e3968`](https://github.com/containerd/containerd/commit/db03e39685448a1d87025a2ca3a1e4e42713102d) cri: add enable_criu configuration option * [`dacd4c7d0`](https://github.com/containerd/containerd/commit/dacd4c7d00f4a5fd71197e67afdb6355e8536e4f) cri: validate CRIU availability and version early * ci: bound Go fuzzing by execution count ([#13785](https://github.com/containerd/containerd/pull/13785)) * [`890a9c86c`](https://github.com/containerd/containerd/commit/890a9c86cd9918e53c1ef1b57c6539a943196cd5) ci: bound Go fuzzing by execution count * cri: auto-add prefix for pause image ([#13759](https://github.com/containerd/containerd/pull/13759)) * [`0b2f1d078`](https://github.com/containerd/containerd/commit/0b2f1d078124b52f4c8030dc686f2d65c1dfc17c) cri: auto-add prefix for pause image </p> </details> ### Dependency Changes * **k8s.io/api** v0.36.0 -> v0.36.3 * **k8s.io/apimachinery** v0.36.0 -> v0.36.3 * **k8s.io/client-go** v0.36.0 -> v0.36.3 * **k8s.io/component-base** v0.36.0 -> v0.36.3 * **k8s.io/cri-api** v0.36.0 -> v0.36.3 * **k8s.io/cri-client** v0.36.0 -> v0.36.3 * **k8s.io/cri-streaming** v0.36.0 -> v0.36.3 * **sigs.k8s.io/structured-merge-diff/v6** v6.3.2 -> v6.3.3 Previous release can be found at [v2.3.3](https://github.com/containerd/containerd/releases/tag/v2.3.3) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
        "url": "https://github.com/containerd/containerd/pull/13942",
        "createdAt": "2026-08-11T23:22:05Z",
        "updatedAt": "2026-08-12T17:47:32Z",
        "timestamp": "2026-08-12T17:47:32Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/S"
        ],
        "author": "samuelkarp",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13943",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "Add log message when a shim is loaded into the shim manager",
        "text": "This change proposes a happy path log for shims loaded into the shim manager for visibility in runtime logs of existing workloads and their version. This is to increase visibility for workloads picked up when the server is bounced or upgraded.",
        "url": "https://github.com/containerd/containerd/pull/13943",
        "createdAt": "2026-08-12T01:38:30Z",
        "updatedAt": "2026-08-12T13:49:16Z",
        "timestamp": "2026-08-12T13:49:16Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/S"
        ],
        "author": "austinvazquez",
        "state": "open",
        "assignees": [],
        "change": "updated"
      },
      {
        "id": "github:containerd/containerd:pull_request:13945",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "[release/2.3] snapshots/erofs: protect snapshot staging from cleanup",
        "text": "Cherry-pick of https://github.com/containerd/containerd/pull/13932",
        "url": "https://github.com/containerd/containerd/pull/13945",
        "createdAt": "2026-08-12T02:40:34Z",
        "updatedAt": "2026-08-12T17:42:38Z",
        "timestamp": "2026-08-12T17:42:38Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/M"
        ],
        "author": "hsiangkao",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13946",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "Prepare release notes for v2.2.7",
        "text": "containerd 2.2.7 Welcome to the v2.2.7 release of containerd! The seventh patch release for containerd 2.2 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * Support non-UTF-8 binary environment variable values in CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13795](https://github.com/containerd/containerd/pull/13795)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13779](https://github.com/containerd/containerd/pull/13779)) #### Runtime * Enable log scrubbing by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * Fix mount manager activation error when activation already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Jordan Liggitt * Akihiro Suda * Amir Alavi * Chris Henzie * Derek McGowan * Maksym Pavlenko * Mike Brown * Phil Estes ### Changes <details><summary>21 commits</summary> <p> * [`43f299db6`](https://github.com/containerd/containerd/commit/43f299db6e087ef066e47c3cf41ee56723983eb3) Prepare release notes for v2.2.7 * cri: disable restore in CreateContainer by default ([#13937](https://github.com/containerd/containerd/pull/13937)) * [`5c3a47036`](https://github.com/containerd/containerd/commit/5c3a47036ec08f8a7517a01a0baf113d6146eca0) cri: disable restore in CreateContainer by default * Use ScrubLogs by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * [`4c93ae6d5`](https://github.com/containerd/containerd/commit/4c93ae6d5e7eaa5957887d469aa610763c2f8b42) ctr: add --scrub-logs flag for Windows * [`a124c7e35`](https://github.com/containerd/containerd/commit/a124c7e354da7252a363f3a3eaafb5a81a6d508e) cri/config: use ScrubLogs by default on Windows * cri: deprecate restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * [`c59d041d8`](https://github.com/containerd/containerd/commit/c59d041d84cb402b991338f2894c0637eb5dc608) cri: deprecate restore in CreateContainer * Handle []byte envvar value for CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * [`6cdc2ddce`](https://github.com/containerd/containerd/commit/6cdc2ddce8aa962135459e02d41ab84772e8e08e) Handle []byte envvar value * [`a60898833`](https://github.com/containerd/containerd/commit/a60898833909a6bb4c38cc1fa47d44434edfe9bf) update to v0.34.x kubernetes dependencies * Fix mount manager activation error when already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) * [`54ecff6ca`](https://github.com/containerd/containerd/commit/54ecff6cabbbbf5b7ffa606cdeeba13131a03968) core/mount: Fix mount manager activation error when already exists * Disable checkpoint restore codepath when CRIU is not installed ([#13795](https://github.com/containerd/containerd/pull/13795)) * [`fd966ba29`](https://github.com/containerd/containerd/commit/fd966ba2904fc92c429fff41b0e4d6cd2384fe7c) github/workflows: install criu in node-e2e * [`7d41b25c9`](https://github.com/containerd/containerd/commit/7d41b25c9884c0f94acdb0f7e78fbab2752fccb6) cri: add enable_criu configuration option * [`48116fa59`](https://github.com/containerd/containerd/commit/48116fa596fd0e926749dcdfd3ef92da00fc6dc8) cri: validate CRIU availability and version early * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13779](https://github.com/containerd/containerd/pull/13779)) * [`6a3d14e8a`](https://github.com/containerd/containerd/commit/6a3d14e8a982e141155f4d39a7b1d34d91fca837) fix(cri): introspect OCI runtime features for non-runc runtimes * ci: bound Go fuzzing by execution count ([#13786](https://github.com/containerd/containerd/pull/13786)) * [`424b714f0`](https://github.com/containerd/containerd/commit/424b714f0c5bc28e840bde698cbbcdc8c4caca0f) ci: bound Go fuzzing by execution count </p> </details> ### Dependency Changes * **go.opentelemetry.io/otel** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/metric** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/trace** v1.38.0 -> v1.41.0 * **google.golang.org/protobuf** v1.36.10 -> f2248ac996af * **k8s.io/api** v0.34.1 -> v0.34.10 * **k8s.io/apimachinery** v0.34.1 -> v0.34.10 * **k8s.io/client-go** v0.34.1 -> v0.34.10 * **k8s.io/cri-api** v0.34.1 -> v0.34.10 Previous release can be found at [v2.2.6](https://github.com/containerd/containerd/releases/tag/v2.2.6) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
        "url": "https://github.com/containerd/containerd/pull/13946",
        "createdAt": "2026-08-12T06:28:52Z",
        "updatedAt": "2026-08-12T17:51:11Z",
        "timestamp": "2026-08-12T17:51:11Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/S"
        ],
        "author": "samuelkarp",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13948",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "[pkg/shim] implement Windows support for the shim server",
        "text": "## What Implements the Windows-specific methods in `pkg/shim` that were previously `ErrNotImplemented` stubs, so a shim built on this package can actually run and serve on Windows (not just Unix). ## Why On Windows the package exposed the shim entry points but left every platform hook unimplemented, so the serve path failed immediately (no ttrpc server, no listener, no logging). This fills those gaps using the Windows named-pipe model. An existing [Windows shim](https://github.com/microsoft/hcsshim/tree/b6b988ac86ef715646ab1b04481ebff88af03cce/cmd/containerd-shim-lcow-v2) uses the same package ([currently forked and modified in project](https://github.com/microsoft/hcsshim/tree/b6b988ac86ef715646ab1b04481ebff88af03cce/internal/shim)). Post this change, the `pkg/shim` can be used by any Windows shim end to end. ## Changes - **ttrpc serving over named pipes** — the shim now creates and serves its API on a Windows named pipe instead of a Unix socket. - **Log streaming over a reconnecting named pipe** — shim logs are served on a pipe that containerd connects to, and transparently re-accepts the reader if it reconnects (e.g. after a containerd restart). Unlike a Unix FIFO, a connection-oriented pipe requires this reconnection handling explicitly. - **Process & signal lifecycle** — signal setup, graceful shutdown on interrupt/terminate, the reap loop, sub-reaper, and stack-dump hooks are wired up with Windows-appropriate (mostly no-op where the OS handles it) behavior. ## Testing Adds unit tests covering the pipe listener, the reconnecting log writer (pre-connection drop, write-through, reconnection, concurrency, close), signal and lifecycle handling, and pipe-readiness. All four platforms (windows/linux/darwin/freebsd) build; the Windows suite passes. ## Notes Windows-only files; no behavior change on Linux/Unix.",
        "url": "https://github.com/containerd/containerd/pull/13948",
        "createdAt": "2026-08-12T10:55:37Z",
        "updatedAt": "2026-08-12T18:02:47Z",
        "timestamp": "2026-08-12T18:02:47Z",
        "metrics": {
          "reactions": 0,
          "comments": 1
        },
        "labels": [
          "size/XL"
        ],
        "author": "rawahars",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13949",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "Ensure credentials are forwarded to configured mirrors",
        "text": "This PR fixes an issue with image credentials being forwarded to unrelated mirrors",
        "url": "https://github.com/containerd/containerd/pull/13949",
        "createdAt": "2026-08-12T16:30:53Z",
        "updatedAt": "2026-08-12T20:21:09Z",
        "timestamp": "2026-08-12T20:21:09Z",
        "metrics": {
          "reactions": 0,
          "comments": 2
        },
        "labels": [
          "size/L",
          "do-not-merge/work-in-progress"
        ],
        "author": "MikeZappa87",
        "state": "open",
        "assignees": [
          "MikeZappa87"
        ]
      },
      {
        "id": "github:containerd/containerd:pull_request:13950",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "[release/2.2] snapshots/erofs: protect snapshot staging from cleanup",
        "text": "Cherry-pick of https://github.com/containerd/containerd/pull/13932",
        "url": "https://github.com/containerd/containerd/pull/13950",
        "createdAt": "2026-08-12T16:31:43Z",
        "updatedAt": "2026-08-13T13:50:20Z",
        "timestamp": "2026-08-13T13:50:20Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/M"
        ],
        "author": "hsiangkao",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13951",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "treat missing runtime state as container already dead during kill",
        "text": "Pods get stuck in terminating forever when the DeleteContainer call times out, but does partially delete the state. Example: ``` time=\"2026-08-04T16:21:31.095039308Z\" level=info msg=\"StopPodSandbox for \\\"f37ee9eca302510909867fae89544fae2b480852fb1ccca45e46833aca5ab229\\\"\" time=\"2026-08-04T16:21:45.129960940Z\" level=error msg=\"failed to delete task\" error=\"context deadline exceeded\" id=f37ee9eca302510909867fae89544fae2b480852fb1ccca45e46833aca5ab229 ``` or ``` time=\"2026-08-04T16:50:41.170399937Z\" level=info msg=\"StopContainer for \\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" with timeout 240 (s)\" time=\"2026-08-04T16:50:59.314985313Z\" level=error msg=\"failed to handle container TaskExit event container_id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" pid:206839 exit_status:143 exited_at:{seconds:1785862249 nanos:295565617}\" error=\"failed to stop container: failed to delete task: context deadline exceeded\" ``` both end up stuck with: ``` Aug 04 19:57:05 ip-172-18-72-147 containerd[28731]: time=\"2026-08-04T19:57:05.033712712Z\" level=info msg=\"TaskExit event container_id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" pid:206839 exit_status:143 exited_at:{seconds:1785862249 nanos:295565617}\" Aug 04 19:57:05 ip-172-18-72-147 containerd[28731]: time=\"2026-08-04T19:57:05.851121769Z\" level=error msg=\"Failed to handle backOff event container_id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" pid:206839 exit_status:143 exited_at:{seconds:1785862249 nanos:295565617} for 8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\" error=\"failed to handle container TaskExit event: failed to stop container:.....: error opening file `/run/containerd/runc/k8s.io/8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a/status`: No such file or directory\\n\" ``` The only way to recover is to force delete the pods. This seems similar to https://github.com/containerd/containerd/issues/7496 which was solved by https://github.com/containerd/containerd/pull/8954, but that only includes the case where we get `errdefs.ErrNotFound` and not `No such file or directory` This adds that `No such file or directory` case alongside the `errdefs.ErrNotFound` catch",
        "url": "https://github.com/containerd/containerd/pull/13951",
        "createdAt": "2026-08-12T16:59:27Z",
        "updatedAt": "2026-08-12T17:12:13Z",
        "timestamp": "2026-08-12T17:12:13Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/M"
        ],
        "author": "olyazavr",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13953",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "Forward credentials to cr mirrors",
        "url": "https://github.com/containerd/containerd/pull/13953",
        "createdAt": "2026-08-12T17:34:21Z",
        "updatedAt": "2026-08-12T18:48:31Z",
        "timestamp": "2026-08-12T18:48:31Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/L",
          "do-not-merge/work-in-progress"
        ],
        "author": "MikeZappa87",
        "state": "closed",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13954",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "fix(runtime): apply load timeout to load shim",
        "url": "https://github.com/containerd/containerd/pull/13954",
        "createdAt": "2026-08-12T20:47:51Z",
        "updatedAt": "2026-08-13T00:45:57Z",
        "timestamp": "2026-08-13T00:45:57Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/M",
          "do-not-merge/work-in-progress"
        ],
        "author": "austinvazquez",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13955",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "ci: add lima image list for fedora images",
        "text": "This change adds fallback image entries for Fedora's canonical dl.fedoraproject.org.",
        "url": "https://github.com/containerd/containerd/pull/13955",
        "createdAt": "2026-08-13T01:08:22Z",
        "updatedAt": "2026-08-13T17:05:15Z",
        "timestamp": "2026-08-13T17:05:15Z",
        "metrics": {
          "reactions": 0,
          "comments": 1
        },
        "labels": [
          "size/S"
        ],
        "author": "austinvazquez",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:pull_request:13956",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "pull_request",
        "title": "cri: mv io create from create stage to start stage",
        "text": "because k8s will reserve several dead containers https://kubernetes.io/docs/concepts/architecture/garbage-collection/#container-image-garbage-collection if start container failed container I/O should be closed as soon as possible.",
        "url": "https://github.com/containerd/containerd/pull/13956",
        "createdAt": "2026-08-13T03:02:34Z",
        "updatedAt": "2026-08-13T03:06:06Z",
        "timestamp": "2026-08-13T03:06:06Z",
        "metrics": {
          "reactions": 0,
          "comments": 0
        },
        "labels": [
          "size/M"
        ],
        "author": "ningmingxiao",
        "state": "open",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:release:369432727",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "release",
        "title": "containerd 2.3.4",
        "text": "Welcome to the v2.3.4 release of containerd! The fourth patch release for containerd 2.3 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) * Set default runtimeFeatures.UserNamespacesHostNetwork to true in CRI ([#13914](https://github.com/containerd/containerd/pull/13914)) * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * Support non-UTF-8 binary environment variable values in CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13778](https://github.com/containerd/containerd/pull/13778)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13734](https://github.com/containerd/containerd/pull/13734)) * Normalize sandbox image references in CRI to resolve images without domain prefixes ([#13759](https://github.com/containerd/containerd/pull/13759)) #### Node Resource Interface (NRI) * Emit deprecation warnings for plugins using deprecated NRI interfaces ([#13935](https://github.com/containerd/containerd/pull/13935)) #### Runtime * Enable log scrubbing by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * Fix memory leak in OOM watcher map when stopping container monitoring ([#13870](https://github.com/containerd/containerd/pull/13870)) * Avoid orphaning shims on transient errors when loading process IDs ([#13857](https://github.com/containerd/containerd/pull/13857)) * Fix corruption of binary protobuf shim start responses caused by premature whitespace trimming ([#13803](https://github.com/containerd/containerd/pull/13803)) #### Snapshotters * Fix EROFS snapshotter dropping lower layers stacked above merged filesystem metadata ([#13876](https://github.com/containerd/containerd/pull/13876)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Maksym Pavlenko * Mike Brown * Akihiro Suda * Derek McGowan * Jordan Liggitt * Paweł Gronowski * Amir Alavi * Andrew Halaney * Bing Hongtao * Chris Henzie * Harshal Patel * Krisztian Litkey * Phil Estes * Wei Fu * ningmingxiao ### Changes <details><summary>35 commits</summary> <p> * [`c1f5360ef`](https://github.com/containerd/containerd/commit/c1f5360ef7b9afb748d4119ad7be665ba5144480) Prepare release notes for v2.3.4 * cri: disable restore in CreateContainer by default ([#13913](https://github.com/containerd/containerd/pull/13913)) * [`928c03c68`](https://github.com/containerd/containerd/commit/928c03c6898cad95f795300281fdff0dbc55a32f) cri: disable restore in CreateContainer by default * nri,deprecation: record and emit warnings for NRI deprecations. ([#13935](https://github.com/containerd/containerd/pull/13935)) * [`5966e2edb`](https://github.com/containerd/containerd/commit/5966e2edb15bb939aa2bb73649fbad73a2b04094) nri,deprecation: emit warnings for old NRI plugins. * Set the default of runtimeFeatures.UserNamespacesHostNetwork to true ([#13914](https://github.com/containerd/containerd/pull/13914)) * [`ab52c58f1`](https://github.com/containerd/containerd/commit/ab52c58f16ac50bb6724354bf36cbb4da480c2a2) Set the default of runtimeFeatures.UserNamespacesHostNetwork to true * Use ScrubLogs by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * [`0c255158a`](https://github.com/containerd/containerd/commit/0c255158aa1ecab40aafc42269cfc018a46eb627) ctr: add --scrub-logs flag for Windows * [`1c2b13dc9`](https://github.com/containerd/containerd/commit/1c2b13dc975468197e57321e25fd09163f78a75a) cri/config: use ScrubLogs by default on Windows * snapshots/erofs: keep lowers stacked above a merged fsmeta ([#13876](https://github.com/containerd/containerd/pull/13876)) * [`0c511e068`](https://github.com/containerd/containerd/commit/0c511e068b7d6dd815418185a235b3ba3347c65a) snapshots/erofs: keep lowers stacked above a merged fsmeta * cri: deprecate restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * [`dc98141df`](https://github.com/containerd/containerd/commit/dc98141df70d3d72d46125c6f30de0d2de648086) cri: deprecate restore in CreateContainer * internal/oom: Fix memory leak by removing watcher from map on Stop ([#13870](https://github.com/containerd/containerd/pull/13870)) * [`537d82d54`](https://github.com/containerd/containerd/commit/537d82d545759d4a072d564d4937707af2650163) internal/oom: Fix memory leak by removing watcher from map on Stop * shim_load: Consider shim leaked only if we can't find pids ([#13857](https://github.com/containerd/containerd/pull/13857)) * [`decf97a9c`](https://github.com/containerd/containerd/commit/decf97a9c7c26f92391d2c862d35832e0a291e80) shim_load: Consider shim leaked only if we can't find pids * core/runtime/v2: Drop checkpointctl module dependency ([#13840](https://github.com/containerd/containerd/pull/13840)) * [`796f07dc8`](https://github.com/containerd/containerd/commit/796f07dc853d9ae8a3a1278fe47436755a4bc3b7) core/runtime/v2: Drop checkpointctl module dependency * Handle []byte envvar value for CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * [`751fddddb`](https://github.com/containerd/containerd/commit/751fddddbb630631e12fd63e5dac4a9530188822) Handle []byte envvar value * [`0bce9060e`](https://github.com/containerd/containerd/commit/0bce9060ef3401e690b541dd6d2008c2abc50b0d) update to v0.36.x kubernetes dependencies * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13778](https://github.com/containerd/containerd/pull/13778)) * [`61a8f6f45`](https://github.com/containerd/containerd/commit/61a8f6f45e34660d787988e8a7e7a218f556369c) fix(cri): introspect OCI runtime features for non-runc runtimes * core/runtime/v2: Preserve protobuf shim response bytes ([#13803](https://github.com/containerd/containerd/pull/13803)) * [`1d28017be`](https://github.com/containerd/containerd/commit/1d28017be295005bfa3aee1d47afe86341d391b6) core/runtime/v2: Preserve protobuf shim response bytes * Disable checkpoint restore codepath when CRIU is not installed ([#13734](https://github.com/containerd/containerd/pull/13734)) * [`374091d67`](https://github.com/containerd/containerd/commit/374091d67c1966969b772594ba96a74f4357bc7a) github/workflows: install criu in node-e2e * [`db03e3968`](https://github.com/containerd/containerd/commit/db03e39685448a1d87025a2ca3a1e4e42713102d) cri: add enable_criu configuration option * [`dacd4c7d0`](https://github.com/containerd/containerd/commit/dacd4c7d00f4a5fd71197e67afdb6355e8536e4f) cri: validate CRIU availability and version early * ci: bound Go fuzzing by execution count ([#13785](https://github.com/containerd/containerd/pull/13785)) * [`890a9c86c`](https://github.com/containerd/containerd/commit/890a9c86cd9918e53c1ef1b57c6539a943196cd5) ci: bound Go fuzzing by execution count * cri: auto-add prefix for pause image ([#13759](https://github.com/containerd/containerd/pull/13759)) * [`0b2f1d078`](https://github.com/containerd/containerd/commit/0b2f1d078124b52f4c8030dc686f2d65c1dfc17c) cri: auto-add prefix for pause image </p> </details> ### Dependency Changes * **k8s.io/api** v0.36.0 -> v0.36.3 * **k8s.io/apimachinery** v0.36.0 -> v0.36.3 * **k8s.io/client-go** v0.36.0 -> v0.36.3 * **k8s.io/component-base** v0.36.0 -> v0.36.3 * **k8s.io/cri-api** v0.36.0 -> v0.36.3 * **k8s.io/cri-client** v0.36.0 -> v0.36.3 * **k8s.io/cri-streaming** v0.36.0 -> v0.36.3 * **sigs.k8s.io/structured-merge-diff/v6** v6.3.2 -> v6.3.3 Previous release can be found at [v2.3.3](https://github.com/containerd/containerd/releases/tag/v2.3.3) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
        "url": "https://github.com/containerd/containerd/releases/tag/v2.3.4",
        "createdAt": "2026-08-12T17:59:47Z",
        "updatedAt": "2026-08-12T17:59:47Z",
        "timestamp": "2026-08-12T17:59:47Z",
        "labels": [],
        "author": "github-actions[bot]",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:release:369434467",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "release",
        "title": "containerd 2.2.7",
        "text": "Welcome to the v2.2.7 release of containerd! The seventh patch release for containerd 2.2 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * Support non-UTF-8 binary environment variable values in CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13795](https://github.com/containerd/containerd/pull/13795)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13779](https://github.com/containerd/containerd/pull/13779)) #### Runtime * Enable log scrubbing by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * Fix mount manager activation error when activation already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Jordan Liggitt * Akihiro Suda * Amir Alavi * Chris Henzie * Derek McGowan * Maksym Pavlenko * Mike Brown * Phil Estes ### Changes <details><summary>21 commits</summary> <p> * [`43f299db6`](https://github.com/containerd/containerd/commit/43f299db6e087ef066e47c3cf41ee56723983eb3) Prepare release notes for v2.2.7 * cri: disable restore in CreateContainer by default ([#13937](https://github.com/containerd/containerd/pull/13937)) * [`5c3a47036`](https://github.com/containerd/containerd/commit/5c3a47036ec08f8a7517a01a0baf113d6146eca0) cri: disable restore in CreateContainer by default * Use ScrubLogs by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * [`4c93ae6d5`](https://github.com/containerd/containerd/commit/4c93ae6d5e7eaa5957887d469aa610763c2f8b42) ctr: add --scrub-logs flag for Windows * [`a124c7e35`](https://github.com/containerd/containerd/commit/a124c7e354da7252a363f3a3eaafb5a81a6d508e) cri/config: use ScrubLogs by default on Windows * cri: deprecate restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * [`c59d041d8`](https://github.com/containerd/containerd/commit/c59d041d84cb402b991338f2894c0637eb5dc608) cri: deprecate restore in CreateContainer * Handle []byte envvar value for CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * [`6cdc2ddce`](https://github.com/containerd/containerd/commit/6cdc2ddce8aa962135459e02d41ab84772e8e08e) Handle []byte envvar value * [`a60898833`](https://github.com/containerd/containerd/commit/a60898833909a6bb4c38cc1fa47d44434edfe9bf) update to v0.34.x kubernetes dependencies * Fix mount manager activation error when already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) * [`54ecff6ca`](https://github.com/containerd/containerd/commit/54ecff6cabbbbf5b7ffa606cdeeba13131a03968) core/mount: Fix mount manager activation error when already exists * Disable checkpoint restore codepath when CRIU is not installed ([#13795](https://github.com/containerd/containerd/pull/13795)) * [`fd966ba29`](https://github.com/containerd/containerd/commit/fd966ba2904fc92c429fff41b0e4d6cd2384fe7c) github/workflows: install criu in node-e2e * [`7d41b25c9`](https://github.com/containerd/containerd/commit/7d41b25c9884c0f94acdb0f7e78fbab2752fccb6) cri: add enable_criu configuration option * [`48116fa59`](https://github.com/containerd/containerd/commit/48116fa596fd0e926749dcdfd3ef92da00fc6dc8) cri: validate CRIU availability and version early * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13779](https://github.com/containerd/containerd/pull/13779)) * [`6a3d14e8a`](https://github.com/containerd/containerd/commit/6a3d14e8a982e141155f4d39a7b1d34d91fca837) fix(cri): introspect OCI runtime features for non-runc runtimes * ci: bound Go fuzzing by execution count ([#13786](https://github.com/containerd/containerd/pull/13786)) * [`424b714f0`](https://github.com/containerd/containerd/commit/424b714f0c5bc28e840bde698cbbcdc8c4caca0f) ci: bound Go fuzzing by execution count </p> </details> ### Dependency Changes * **go.opentelemetry.io/otel** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/metric** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/trace** v1.38.0 -> v1.41.0 * **google.golang.org/protobuf** v1.36.10 -> f2248ac996af * **k8s.io/api** v0.34.1 -> v0.34.10 * **k8s.io/apimachinery** v0.34.1 -> v0.34.10 * **k8s.io/client-go** v0.34.1 -> v0.34.10 * **k8s.io/cri-api** v0.34.1 -> v0.34.10 Previous release can be found at [v2.2.6](https://github.com/containerd/containerd/releases/tag/v2.2.6) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
        "url": "https://github.com/containerd/containerd/releases/tag/v2.2.7",
        "createdAt": "2026-08-12T18:02:25Z",
        "updatedAt": "2026-08-12T18:02:25Z",
        "timestamp": "2026-08-12T18:02:25Z",
        "labels": [],
        "author": "github-actions[bot]",
        "assignees": []
      },
      {
        "id": "github:containerd/containerd:release:https://github.com/containerd/containerd/releases/tag/v2.2.7",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "release",
        "title": "containerd 2.2.7",
        "text": "Welcome to the v2.2.7 release of containerd! The seventh patch release for containerd 2.2 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * Support non-UTF-8 binary environment variable values in CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13795](https://github.com/containerd/containerd/pull/13795)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13779](https://github.com/containerd/containerd/pull/13779)) #### Runtime * Enable log scrubbing by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * Fix mount manager activation error when activation already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Jordan Liggitt * Akihiro Suda * Amir Alavi * Chris Henzie * Derek McGowan * Maksym Pavlenko * Mike Brown * Phil Estes ### Changes <details><summary>21 commits</summary> <p> * [`43f299db6`](https://github.com/containerd/containerd/commit/43f299db6e087ef066e47c3cf41ee56723983eb3) Prepare release notes for v2.2.7 * cri: disable restore in CreateContainer by default ([#13937](https://github.com/containerd/containerd/pull/13937)) * [`5c3a47036`](https://github.com/containerd/containerd/commit/5c3a47036ec08f8a7517a01a0baf113d6146eca0) cri: disable restore in CreateContainer by default * Use ScrubLogs by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * [`4c93ae6d5`](https://github.com/containerd/containerd/commit/4c93ae6d5e7eaa5957887d469aa610763c2f8b42) ctr: add --scrub-logs flag for Windows * [`a124c7e35`](https://github.com/containerd/containerd/commit/a124c7e354da7252a363f3a3eaafb5a81a6d508e) cri/config: use ScrubLogs by default on Windows * cri: deprecate restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * [`c59d041d8`](https://github.com/containerd/containerd/commit/c59d041d84cb402b991338f2894c0637eb5dc608) cri: deprecate restore in CreateContainer * Handle []byte envvar value for CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * [`6cdc2ddce`](https://github.com/containerd/containerd/commit/6cdc2ddce8aa962135459e02d41ab84772e8e08e) Handle []byte envvar value * [`a60898833`](https://github.com/containerd/containerd/commit/a60898833909a6bb4c38cc1fa47d44434edfe9bf) update to v0.34.x kubernetes dependencies * Fix mount manager activation error when already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) * [`54ecff6ca`](https://github.com/containerd/containerd/commit/54ecff6cabbbbf5b7ffa606cdeeba13131a03968) core/mount: Fix mount manager activation error when already exists * Disable checkpoint restore codepath when CRIU is not installed ([#13795](https://github.com/containerd/containerd/pull/13795)) * [`fd966ba29`](https://github.com/containerd/containerd/commit/fd966ba2904fc92c429fff41b0e4d6cd2384fe7c) github/workflows: install criu in node-e2e * [`7d41b25c9`](https://github.com/containerd/containerd/commit/7d41b25c9884c0f94acdb0f7e78fbab2752fccb6) cri: add enable_criu configuration option * [`48116fa59`](https://github.com/containerd/containerd/commit/48116fa596fd0e926749dcdfd3ef92da00fc6dc8) cri: validate CRIU availability and version early * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13779](https://github.com/containerd/containerd/pull/13779)) * [`6a3d14e8a`](https://github.com/containerd/containerd/commit/6a3d14e8a982e141155f4d39a7b1d34d91fca837) fix(cri): introspect OCI runtime features for non-runc runtimes * ci: bound Go fuzzing by execution count ([#13786](https://github.com/containerd/containerd/pull/13786)) * [`424b714f0`](https://github.com/containerd/containerd/commit/424b714f0c5bc28e840bde698cbbcdc8c4caca0f) ci: bound Go fuzzing by execution count </p> </details> ### Dependency Changes * **go.opentelemetry.io/otel** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/metric** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/trace** v1.38.0 -> v1.41.0 * **google.golang.org/protobuf** v1.36.10 -> f2248ac996af * **k8s.io/api** v0.34.1 -> v0.34.10 * **k8s.io/apimachinery** v0.34.1 -> v0.34.10 * **k8s.io/client-go** v0.34.1 -> v0.34.10 * **k8s.io/cri-api** v0.34.1 -> v0.34.10 Previous release can be found at [v2.2.6](https://github.com/containerd/containerd/releases/tag/v2.2.6) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
        "url": "https://github.com/containerd/containerd/releases/tag/v2.2.7",
        "timestamp": "2026-08-12T18:02:25Z",
        "labels": [],
        "author": "github-actions[bot]",
        "assignees": [],
        "change": "new"
      },
      {
        "id": "github:containerd/containerd:release:https://github.com/containerd/containerd/releases/tag/v2.3.4",
        "source": "github",
        "group": "platform-infrastructure",
        "project": "containerd/containerd",
        "kind": "release",
        "title": "containerd 2.3.4",
        "text": "Welcome to the v2.3.4 release of containerd! The fourth patch release for containerd 2.3 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) * Set default runtimeFeatures.UserNamespacesHostNetwork to true in CRI ([#13914](https://github.com/containerd/containerd/pull/13914)) * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * Support non-UTF-8 binary environment variable values in CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13778](https://github.com/containerd/containerd/pull/13778)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13734](https://github.com/containerd/containerd/pull/13734)) * Normalize sandbox image references in CRI to resolve images without domain prefixes ([#13759](https://github.com/containerd/containerd/pull/13759)) #### Node Resource Interface (NRI) * Emit deprecation warnings for plugins using deprecated NRI interfaces ([#13935](https://github.com/containerd/containerd/pull/13935)) #### Runtime * Enable log scrubbing by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * Fix memory leak in OOM watcher map when stopping container monitoring ([#13870](https://github.com/containerd/containerd/pull/13870)) * Avoid orphaning shims on transient errors when loading process IDs ([#13857](https://github.com/containerd/containerd/pull/13857)) * Fix corruption of binary protobuf shim start responses caused by premature whitespace trimming ([#13803](https://github.com/containerd/containerd/pull/13803)) #### Snapshotters * Fix EROFS snapshotter dropping lower layers stacked above merged filesystem metadata ([#13876](https://github.com/containerd/containerd/pull/13876)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Maksym Pavlenko * Mike Brown * Akihiro Suda * Derek McGowan * Jordan Liggitt * Paweł Gronowski * Amir Alavi * Andrew Halaney * Bing Hongtao * Chris Henzie * Harshal Patel * Krisztian Litkey * Phil Estes * Wei Fu * ningmingxiao ### Changes <details><summary>35 commits</summary> <p> * [`c1f5360ef`](https://github.com/containerd/containerd/commit/c1f5360ef7b9afb748d4119ad7be665ba5144480) Prepare release notes for v2.3.4 * cri: disable restore in CreateContainer by default ([#13913](https://github.com/containerd/containerd/pull/13913)) * [`928c03c68`](https://github.com/containerd/containerd/commit/928c03c6898cad95f795300281fdff0dbc55a32f) cri: disable restore in CreateContainer by default * nri,deprecation: record and emit warnings for NRI deprecations. ([#13935](https://github.com/containerd/containerd/pull/13935)) * [`5966e2edb`](https://github.com/containerd/containerd/commit/5966e2edb15bb939aa2bb73649fbad73a2b04094) nri,deprecation: emit warnings for old NRI plugins. * Set the default of runtimeFeatures.UserNamespacesHostNetwork to true ([#13914](https://github.com/containerd/containerd/pull/13914)) * [`ab52c58f1`](https://github.com/containerd/containerd/commit/ab52c58f16ac50bb6724354bf36cbb4da480c2a2) Set the default of runtimeFeatures.UserNamespacesHostNetwork to true * Use ScrubLogs by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * [`0c255158a`](https://github.com/containerd/containerd/commit/0c255158aa1ecab40aafc42269cfc018a46eb627) ctr: add --scrub-logs flag for Windows * [`1c2b13dc9`](https://github.com/containerd/containerd/commit/1c2b13dc975468197e57321e25fd09163f78a75a) cri/config: use ScrubLogs by default on Windows * snapshots/erofs: keep lowers stacked above a merged fsmeta ([#13876](https://github.com/containerd/containerd/pull/13876)) * [`0c511e068`](https://github.com/containerd/containerd/commit/0c511e068b7d6dd815418185a235b3ba3347c65a) snapshots/erofs: keep lowers stacked above a merged fsmeta * cri: deprecate restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * [`dc98141df`](https://github.com/containerd/containerd/commit/dc98141df70d3d72d46125c6f30de0d2de648086) cri: deprecate restore in CreateContainer * internal/oom: Fix memory leak by removing watcher from map on Stop ([#13870](https://github.com/containerd/containerd/pull/13870)) * [`537d82d54`](https://github.com/containerd/containerd/commit/537d82d545759d4a072d564d4937707af2650163) internal/oom: Fix memory leak by removing watcher from map on Stop * shim_load: Consider shim leaked only if we can't find pids ([#13857](https://github.com/containerd/containerd/pull/13857)) * [`decf97a9c`](https://github.com/containerd/containerd/commit/decf97a9c7c26f92391d2c862d35832e0a291e80) shim_load: Consider shim leaked only if we can't find pids * core/runtime/v2: Drop checkpointctl module dependency ([#13840](https://github.com/containerd/containerd/pull/13840)) * [`796f07dc8`](https://github.com/containerd/containerd/commit/796f07dc853d9ae8a3a1278fe47436755a4bc3b7) core/runtime/v2: Drop checkpointctl module dependency * Handle []byte envvar value for CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * [`751fddddb`](https://github.com/containerd/containerd/commit/751fddddbb630631e12fd63e5dac4a9530188822) Handle []byte envvar value * [`0bce9060e`](https://github.com/containerd/containerd/commit/0bce9060ef3401e690b541dd6d2008c2abc50b0d) update to v0.36.x kubernetes dependencies * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13778](https://github.com/containerd/containerd/pull/13778)) * [`61a8f6f45`](https://github.com/containerd/containerd/commit/61a8f6f45e34660d787988e8a7e7a218f556369c) fix(cri): introspect OCI runtime features for non-runc runtimes * core/runtime/v2: Preserve protobuf shim response bytes ([#13803](https://github.com/containerd/containerd/pull/13803)) * [`1d28017be`](https://github.com/containerd/containerd/commit/1d28017be295005bfa3aee1d47afe86341d391b6) core/runtime/v2: Preserve protobuf shim response bytes * Disable checkpoint restore codepath when CRIU is not installed ([#13734](https://github.com/containerd/containerd/pull/13734)) * [`374091d67`](https://github.com/containerd/containerd/commit/374091d67c1966969b772594ba96a74f4357bc7a) github/workflows: install criu in node-e2e * [`db03e3968`](https://github.com/containerd/containerd/commit/db03e39685448a1d87025a2ca3a1e4e42713102d) cri: add enable_criu configuration option * [`dacd4c7d0`](https://github.com/containerd/containerd/commit/dacd4c7d00f4a5fd71197e67afdb6355e8536e4f) cri: validate CRIU availability and version early * ci: bound Go fuzzing by execution count ([#13785](https://github.com/containerd/containerd/pull/13785)) * [`890a9c86c`](https://github.com/containerd/containerd/commit/890a9c86cd9918e53c1ef1b57c6539a943196cd5) ci: bound Go fuzzing by execution count * cri: auto-add prefix for pause image ([#13759](https://github.com/containerd/containerd/pull/13759)) * [`0b2f1d078`](https://github.com/containerd/containerd/commit/0b2f1d078124b52f4c8030dc686f2d65c1dfc17c) cri: auto-add prefix for pause image </p> </details> ### Dependency Changes * **k8s.io/api** v0.36.0 -> v0.36.3 * **k8s.io/apimachinery** v0.36.0 -> v0.36.3 * **k8s.io/client-go** v0.36.0 -> v0.36.3 * **k8s.io/component-base** v0.36.0 -> v0.36.3 * **k8s.io/cri-api** v0.36.0 -> v0.36.3 * **k8s.io/cri-client** v0.36.0 -> v0.36.3 * **k8s.io/cri-streaming** v0.36.0 -> v0.36.3 * **sigs.k8s.io/structured-merge-diff/v6** v6.3.2 -> v6.3.3 Previous release can be found at [v2.3.3](https://github.com/containerd/containerd/releases/tag/v2.3.3) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
        "url": "https://github.com/containerd/containerd/releases/tag/v2.3.4",
        "timestamp": "2026-08-12T17:59:47Z",
        "labels": [],
        "author": "github-actions[bot]",
        "assignees": [],
        "change": "new"
      }
    ],
    "events": [
      {
        "id": "event:25ac6253a3153eef89b4",
        "signalId": "github:containerd/containerd:pull_request:13950",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13950",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "[release/2.2] snapshots/erofs: protect snapshot staging from cleanup",
          "text": "Cherry-pick of https://github.com/containerd/containerd/pull/13932",
          "url": "https://github.com/containerd/containerd/pull/13950",
          "createdAt": "2026-08-12T16:31:43Z",
          "updatedAt": "2026-08-13T11:52:05Z",
          "timestamp": "2026-08-13T11:52:05Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M"
          ],
          "author": "hsiangkao",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:efac7f4f5a1215561b95",
        "signalId": "github:containerd/containerd:pull_request:13294",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13294",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "vendor: github.com/sirupsen/logrus v1.10.0",
          "text": "Not tagged yet; - https://github.com/sirupsen/logrus/pull/1568 full diff: https://github.com/sirupsen/logrus/compare/v1.9.4...84cedf9d8b02e719f44be1b8a384be724ad49097",
          "url": "https://github.com/containerd/containerd/pull/13294",
          "createdAt": "2026-04-27T13:41:27Z",
          "updatedAt": "2026-08-13T10:57:00Z",
          "timestamp": "2026-08-13T10:57:00Z",
          "metrics": {
            "reactions": 0,
            "comments": 1
          },
          "labels": [
            "size/XXL",
            "do-not-merge/work-in-progress"
          ],
          "author": "thaJeztah",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:a273f631a13f9a4d61be",
        "signalId": "github:containerd/containerd:issue:13952",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:13952",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "containerd crash with program exceeds 10000-thread limit - container-log FIFO open leaks an OS thread per failed CreateContainer",
          "text": "### Description On a busy Kubernetes node, containerd crashed with: ``` runtime: program exceeds 10000-thread limit fatal error: thread exhaustion ``` The process had leaked OS threads steadily over ~17 hours until it hit Go's default maxmcount (10,000), then aborted. A goroutine dump taken at crash time shows the leaked threads are all blocked in a blocking open(2) on a container log FIFO. The CRI container stdout/stderr logger opens the read end of the container's log FIFO with a blocking `open(O_RDONLY)`. That open parks until the container task opens the write end. If the task never starts (e.g. it fails or times out during create/start), the write end never opens, the read open() blocks indefinitely, and, because it's a blocking syscall, it pins a dedicated OS thread that is never released. Under a condition that makes CreateContainer/task-start repeatedly fail, kubelet retries on its sync backoff. Each retry leaks one more thread. With enough churn this climbs linearly to the 10,000-thread limit and containerd aborts. Containerd/fifo does have a release path: a watcher goroutine (`fifo.go:116`, second trace below) unblocks the pending `open()` when its context is cancelled or `Close()` is called, by opening the other end `O_WRONLY|O_NONBLOCK`. The leak is that on this path nothing ever cancels the container-IO context or calls `Close()` - the gRPC client deadline kubelet hits doesn't propagate to it, so the watcher never fires and there is no timeout on the open, leaving the thread pinned for the life of the process. ``` goroutine 163741607 gp=0x4000d81a40 m=8013 mp=0x40b9e00008 [syscall, 492 minutes]: syscall.Syscall6(0x38, 0xffffffffffffff9c, 0x408d7e8618, 0x80000, 0x0, 0x0, 0x0) /usr/local/go/src/syscall/syscall_linux.go:95 syscall.openat(0xffffffffffffff9c, {0x408d7e8600?, 0x204141fc?}, 0x80000, 0x0) /usr/local/go/src/syscall/zsyscall_linux_arm64.go:98 syscall.Open(...) /usr/local/go/src/syscall/syscall_linux.go:284 os.open(...) /usr/local/go/src/os/file_open_unix.go:15 os.openFileNolog({0x408d7e8600, 0x12}, 0x0, 0x0) /usr/local/go/src/os/file_unix.go:278 os.OpenFile({0x408d7e8600, 0x12}, 0x0, 0x0) /usr/local/go/src/os/file.go:392 github.com/containerd/fifo.openFifo.func2() vendor/github.com/containerd/fifo/fifo.go:138 created by github.com/containerd/fifo.openFifo in goroutine 163741605 vendor/github.com/containerd/fifo/fifo.go:131 ``` This is the release path: ``` goroutine 164339856 [select, 466 minutes]: github.com/containerd/fifo.openFifo.func1() vendor/github.com/containerd/fifo/fifo.go:120 created by github.com/containerd/fifo.openFifo vendor/github.com/containerd/fifo/fifo.go:116 ``` The daemon requests `O_RDONLY|O_NONBLOCK`, but containerd/fifo strips `O_NONBLOCK` and performs the real `open() `in a background goroutine, so at the syscall level it's a blocking `open(O_RDONLY)` that parks until the task opens the write end. Threads and memory climbing before the crash: <img width=\"1470\" height=\"272\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/8e8adcb2-c006-485c-8075-3668a15de862\" /> ### Steps to reproduce the issue I cannot actually reproduce this, only happens under extreme load ### Describe the results you received and expected Not to have threads leak ### What version of containerd are you using? 2.2.1 ### Any other relevant information kernel 6.12.80 ### Show configuration if it is related to CRI plugin. _No response_",
          "url": "https://github.com/containerd/containerd/issues/13952",
          "createdAt": "2026-08-12T17:25:59Z",
          "updatedAt": "2026-08-13T09:32:09Z",
          "timestamp": "2026-08-13T09:32:09Z",
          "metrics": {
            "reactions": 0,
            "comments": 7
          },
          "labels": [
            "kind/bug",
            "area/cri"
          ],
          "author": "olyazavr",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:e799af0dc0a4be24e0fd",
        "signalId": "github:containerd/containerd:pull_request:13799",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13799",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "metadata: bound snapshotter Remove during garbage collection",
          "text": "Fixes #13798 **Problem** During snapshot GC, `cleanupSnapshotter` calls the snapshotter's `Remove` while holding the metadata store write lock, with no deadline (it even strips the caller's cancellation). If the snapshotter never answers, the lock is held forever. Every snapshot operation on the node queues behind it, including CRI `RunPodSandbox`, so no new container can start. Nothing on the readiness path takes this lock, so the node keeps reporting Ready the whole time. ```mermaid flowchart TD K[kubelet] -->|Ready probe| S[\"CRI Status()\"] K -->|RunPodSandbox| P[snapshot Prepare] S --> OK[\"returns OK - never takes the lock\"] P --> L{{metadata store write lock}} GC[snapshot GC] -->|holds| L GC -->|\"Remove() - no deadline\"| SOCI[soci snapshotter<br/>out of fds, never answers] L -.->|blocked behind GC| P ``` We hit this in production: our soci snapshotter ran out of file descriptors and stopped answering, the GC's `Remove` parked in `waitOnHeader`, and the node could not start a single pod for 4.5 hours while showing Ready. Goroutine dumps are in the issue. The fd leak was a soci bug (fixed upstream in awslabs/soci-snapshotter#2043), but any proxy snapshotter that stalls reproduces this: one stuck RPC should not brick the node. **Fix** Bound each GC-path `Remove` with a `pkg/timeout` key (`io.containerd.timeout.snapshot.remove`, default 5m), the same pattern as shim cleanup and bolt open timeouts. On the first timed-out `Remove` the rest of the pass is abandoned, so a wedged snapshotter costs one timeout per GC cycle instead of one per orphan. Skipped snapshots stay orphaned and are retried next pass. The deferred `Cleanup` call is skipped in that case too, since it is an unbounded RPC to the same snapshotter that just failed to answer. **Not in this PR** `Walk` runs under the same lock and is also unbounded. Left out to keep this change small; can follow up.",
          "url": "https://github.com/containerd/containerd/pull/13799",
          "createdAt": "2026-07-15T16:37:00Z",
          "updatedAt": "2026-08-13T08:15:08Z",
          "timestamp": "2026-08-13T08:15:08Z",
          "metrics": {
            "reactions": 0,
            "comments": 3
          },
          "labels": [
            "size/L",
            "do-not-merge/contains-merge-commits"
          ],
          "author": "NahumLitvin",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:bfb8e8eedf601aee2b52",
        "signalId": "github:containerd/containerd:issue:13529",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:13529",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "CRI: tag+digest sandbox image breaks RunPodSandbox",
          "text": "### Description A CRI sandbox image (`pinned_images.sandbox`, or legacy `sandbox_image`) configured as a reference with both a tag and a digest (name:tag@sha256:…) makes `RunPodSandbox` fail with \"failed to get sandbox image … not found\", even though containerd successfully pulls that exact image during the same call and stores it. No pods can start on the node. Identical config works on 2.2.x and fails on 2.3.x (verified on 2.3.0 and 2.3.1). https://github.com/containerd/containerd/pull/13513 might also fix this issue. ### Steps to reproduce the issue 1. Fetch containerd 2.3.1 + crictl into a temp dir: ```bash mkdir -p /tmp/ctd-repro && cd /tmp/ctd-repro curl -fsSL https://github.com/containerd/containerd/releases/download/v2.3.1/containerd-2.3.1-linux-amd64.tar.gz | tar xz curl -fsSL https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.34.0/crictl-v1.34.0-linux-amd64.tar.gz | tar xz -C bin ``` 2. Write a config with a `tag@digest` sandbox image: ```bash cat > config.toml <<EOF version = 3 root = \"/tmp/ctd-repro/root\" state = \"/tmp/ctd-repro/state\" [grpc] address = \"/tmp/ctd-repro/c.sock\" [plugins.\"io.containerd.cri.v1.images\".pinned_images] sandbox = \"registry.k8s.io/pause:3.10.1@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c\" [plugins.\"io.containerd.cri.v1.runtime\".containerd] default_runtime_name = \"runc\" [plugins.\"io.containerd.cri.v1.runtime\".containerd.runtimes.runc] runtime_type = \"io.containerd.runc.v2\" EOF ``` 3. Start the throwaway containerd (root, with `bin` on PATH for the shim/runc): ```bash sudo env \"PATH=/tmp/ctd-repro/bin:/usr/sbin:/usr/bin:/sbin:/bin\" ./bin/containerd -c config.toml &>containerd.log & sleep 3 ``` 4. Minimal pod-sandbox config (host network, so no CNI is needed): ```bash cat > pod.json <<'EOF' { \"metadata\": {\"name\":\"probe\",\"namespace\":\"default\",\"uid\":\"u-1\",\"attempt\":1}, \"log_directory\":\"/tmp\", \"linux\": {\"security_context\": {\"namespace_options\": {\"network\": 2}}} } EOF ``` 5. Run the sandbox — fails: ```console $ sudo ./bin/crictl -r unix:///tmp/ctd-repro/c.sock -i unix:///tmp/ctd-repro/c.sock runp pod.json ... code = NotFound desc = failed to start sandbox \"...\": failed to get sandbox image \"registry.k8s.io/pause:3.10.1@sha256:278fb9...\": image \"...\": not found ``` 6. …but the image is present — containerd pulled it during step 5 and stored it under its canonical (digest-only) name: ```console $ sudo ./bin/ctr -a /tmp/ctd-repro/c.sock -n k8s.io images ls | grep pause registry.k8s.io/pause@sha256:278fb9... ``` ### Describe the results you received and expected `RunPodSandbox` fails immediately with: ``` code = NotFound desc = failed to start sandbox \"...\": failed to get sandbox image \"registry.k8s.io/pause:3.10.1@sha256:278fb9...\": image \"...\": not found ``` containerd pulled that image during the same call, `ctr -n k8s.io images ls` shows it present as `registry.k8s.io/pause@sha256:278fb9...`, yet the sandbox lookup reports it missing. No pod sandbox is created, so the node can run no pods. `RunPodSandbox` should resolve the configured sandbox image (a valid `name:tag@digest` reference) and start the sandbox, as it does on containerd 2.2.x, and as 2.3.x already does when the same image is configured tag-only (`pause:3.10.1`) or digest-only (`pause@sha256:...`). ### What version of containerd are you using? containerd github.com/containerd/containerd/v2 v2.3.1 64b425cf570b3b8dd1d4cc46da7c1fce65c6651a ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. ```toml version = 3 root = \"/tmp/ctd-repro/root\" state = \"/tmp/ctd-repro/state\" [grpc] address = \"/tmp/ctd-repro/c.sock\" [plugins.\"io.containerd.cri.v1.images\".pinned_images] sandbox = \"registry.k8s.io/pause:3.10.1@sha256:278fb9dbcca9518083ad1e11276933a2e96f23de604a3a08cc3c80002767d24c\" [plugins.\"io.containerd.cri.v1.runtime\".containerd] default_runtime_name = \"runc\" [plugins.\"io.containerd.cri.v1.runtime\".containerd.runtimes.runc] runtime_type = \"io.containerd.runc.v2\" ```",
          "url": "https://github.com/containerd/containerd/issues/13529",
          "createdAt": "2026-06-04T06:31:48Z",
          "updatedAt": "2026-08-13T04:24:06Z",
          "timestamp": "2026-08-13T04:24:06Z",
          "metrics": {
            "reactions": 1,
            "comments": 1
          },
          "labels": [
            "kind/bug",
            "area/cri"
          ],
          "author": "hakman",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:4428b331758635f6b625",
        "signalId": "github:containerd/containerd:pull_request:13956",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13956",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "cri: mv io create from create stage to start stage",
          "text": "because k8s will reserve several dead containers https://kubernetes.io/docs/concepts/architecture/garbage-collection/#container-image-garbage-collection if start container failed container I/O should be closed as soon as possible.",
          "url": "https://github.com/containerd/containerd/pull/13956",
          "createdAt": "2026-08-13T03:02:34Z",
          "updatedAt": "2026-08-13T03:06:06Z",
          "timestamp": "2026-08-13T03:06:06Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M"
          ],
          "author": "ningmingxiao",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:ec269d75f57f70bc72e6",
        "signalId": "github:containerd/containerd:pull_request:13955",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13955",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "ci: add lima image list for fedora images",
          "text": "This change adds fallback image entries for Fedora's canonical dl.fedoraproject.org.",
          "url": "https://github.com/containerd/containerd/pull/13955",
          "createdAt": "2026-08-13T01:08:22Z",
          "updatedAt": "2026-08-13T01:52:47Z",
          "timestamp": "2026-08-13T01:52:47Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/S"
          ],
          "author": "austinvazquez",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:c323d381270229e87942",
        "signalId": "github:containerd/containerd:pull_request:13653",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13653",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "shim: send event to a queue to prevent event to be dropped",
          "text": "@dmcgowan @fuweid @mxpv can you take a look. fix https://github.com/containerd/containerd/issues/12678 ``` func (s *service) processExits() { for e := range s.ec { // if consumer use much time more than 1s event will be dropped } ``` because notify will drop new events if consumer use more than 1s ``` func Reap() error { now := time.Now() exits, err := reap(false) for _, e := range exits { done := Default.notify(runc.Exit{ Timestamp: now, Pid: e.Pid, Status: e.Status, }) select { case <-done: case <-time.After(1 * time.Second): } } return err } ``` I create a queue to add this event into it it will make consumer consume event quickly It's easy to reproduce it add some sleep here (simulate high-load scenarios because sometimes wait lifecycleMu lock may quite a while) ``` func (s *service) processExits() { for e := range s.ec { // add some sleep time.Sleep(time.Second * 3) s.lifecycleMu.Lock() ``` reproduce ``` ✦2 ❯ cat container.json { \"metadata\": { \"name\": \"TESTFAILPOINTPROFILE_delayUpdate5\" }, \"image\": { \"image\": \"busybox:latest\" }, \"command\": [ 'sleep','infinity' ], \"log_path\": \"busybox5.log\", \"linux\": { \"resources\": { \"memory\": { \"limit\": 20971 } } } } root in /home/nmx/pods via 🐹 v1.26.3 ✦2 ❯ cat pod.json { \"metadata\": { \"name\": \"busybox-sandbox2\", \"namespace\": \"default\", \"attempt\": 1 }, \"annotations\": { }, \"log_directory\": \"/tmp\" } ``` crictl run --no-pull container.json pod.json 66abdf4d13 go run main.go cat main.go ``` package main import ( \"fmt\" \"os/exec\" \"sync\" \"time\" ) func main() { const ( totalTask = 100 maxConcur = 50 ) var wg sync.WaitGroup ch := make(chan struct{}, maxConcur) go func() { time.Sleep(time.Second*2) cmd := exec.Command(\"sh\", \"-c\", \"kill -9 $(pidof sleep)\") cmd.CombinedOutput() }() for i := 0; i < totalTask; i++ { ch <- struct{}{} wg.Add(1) go func(idx int) { defer func() { <-ch wg.Done() }() cmd := exec.Command(\"sh\", \"-c\", \"crictl exec 66abdf4d13 date\") _, err := cmd.CombinedOutput() if err != nil { fmt.Printf(\"Task %d execution failed: %v\\n\", idx, err) } }(i) } wg.Wait() close(ch) fmt.Println(\"done\") } ``` ``` crictl ps|grep 66a 66abdf4d13b04 busybox:latest About a minute ago Running TESTFAILPOINTPROFILE_delayUpdate5 0 bb4899de19161 unknown unknown crictl exec 66a date FATA[0000] execing command in container 66a: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"35d0a3f6180055dcc2694b39c53d7e4215613850e622222cb78be205f1016fae\": OCI runtime exec failed: exec failed: cannot exec in a stopped container ```` container is still running even main process is exited. @evgfitil",
          "url": "https://github.com/containerd/containerd/pull/13653",
          "createdAt": "2026-06-23T02:48:10Z",
          "updatedAt": "2026-08-13T01:21:19Z",
          "timestamp": "2026-08-13T01:21:19Z",
          "metrics": {
            "reactions": 1,
            "comments": 4
          },
          "labels": [
            "size/L"
          ],
          "author": "ningmingxiao",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:e32f8160716b900f8a31",
        "signalId": "github:containerd/containerd:pull_request:13954",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13954",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "fix(runtime): apply load timeout to load shim",
          "url": "https://github.com/containerd/containerd/pull/13954",
          "createdAt": "2026-08-12T20:47:51Z",
          "updatedAt": "2026-08-13T00:45:57Z",
          "timestamp": "2026-08-13T00:45:57Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M",
            "do-not-merge/work-in-progress"
          ],
          "author": "austinvazquez",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:a4f4320481df84a0ef32",
        "signalId": "github:containerd/containerd:issue:13909",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:13909",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "CRI image pull is sometimes canceled by image_pull_progress_timeout during unpack",
          "text": "### Description When the CRI plugin pulls an image through the transfer service (`use_local_image_pull = false`), the image pull can be canceled by `image_pull_progress_timeout` even when all blob downloads have finished and the remaining unpack is progressing. The unpack then fails with: ``` cancel pulling image <ref> because of no progress in 5m0s stop pulling image <ref>: active requests=1, bytes read=... failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:...) to overlayfs as \"extract-...\": context canceled ``` The CRI progress reporter's active-request counter leaks because the transfer service can skip the `\"complete\"` progress event due to a race between download-completion detection and the start of extraction. 1. The transfer service `ProgressTracker` polls statuses every 300ms and sends the `\"complete\"` event only while the job is in the `jobAdded`/`jobInProgress` state. https://github.com/containerd/containerd/blob/v2.2.3/core/transfer/local/progress.go#L108-L142 2. However, unpack starts immediately after the layer fetch completes, and the applier's progress callback (`diff.WithProgress(progressTracker.ExtractProgress)`, https://github.com/containerd/containerd/blob/v2.2.3/core/transfer/local/pull.go#L205) moves the job directly to `jobExtracting` via `j.extraction` in `ExtractProgress`. https://github.com/containerd/containerd/blob/v2.2.3/core/transfer/local/progress.go#L242-L247 The tick period in (1) is 300ms while extraction starts a few milliseconds after the fetch completes, so the job usually transitions directly from `jobInProgress` to `jobExtracting` before the next tick can detect the completion. `\"extracting\"`/`\"extracted\"` events are then sent instead of `\"complete\"`. 3. The CRI `transferProgressReporter.handleProgress` only handles `\"waiting\"`, `\"downloading\"`, and `\"complete\"`. `\"extracting\"` and `\"extracted\"` are ignored. https://github.com/containerd/containerd/blob/v2.2.3/internal/cri/server/images/image_pull.go#L923-L972 The status counted on the first `\"downloading\"` event is therefore never decremented, `activeReqs` stays at 1. 4. The watchdog resets its timer only when `activeReqs == 0` or `bytesRead` increases. During unpack neither holds, so after `image_pull_progress_timeout` it cancels the whole pull. https://github.com/containerd/containerd/blob/v2.2.3/internal/cri/server/images/image_pull.go#L1016-L1040 The cancellation lands in the unpacker's `Apply`, which produces the misleading \"failed to extract layer ... context canceled\". Any image whose unpack tail (after the last downloaded byte) takes longer than `image_pull_progress_timeout` can be killed this way. The `\"already exists\"` event is ignored as well. The patch below lets the CRI reporter treat these events as download completion and fixes the issue. ```diff --- a/internal/cri/server/images/image_pull.go +++ b/internal/cri/server/images/image_pull.go @@ -958,7 +958,7 @@ func (reporter *transferProgressReporter) handleProgress(p transfer.Progress) { delete(reporter.statuses, p.Name) } - case \"complete\": + case \"complete\", \"extracting\", \"extracted\", \"already exists\": if node, exists := reporter.statuses[p.Name]; exists { if curProgress := p.Progress - node.Progress; curProgress > 0 { reporter.IncBytesRead(curProgress ``` ### Steps to reproduce the issue 1. Configure the config with a short progress timeout to reproduce. ```toml # /etc/containerd/certs.d/localhost:5000/hosts.toml server = \"http://localhost:5000\" [host.\"http://localhost:5000\"] capabilities = [\"pull\", \"resolve\"] ``` ```toml # /etc/containerd/config.toml version = 3 root = '/var/lib/containerd' [plugins] [plugins.'io.containerd.cri.v1.images'] image_pull_progress_timeout = '5s' [plugins.'io.containerd.cri.v1.images'.registry] config_path = '/etc/containerd/certs.d' ``` ```console $ sudo systemctl restart containerd ``` 2. Push a test image whose unpacking takes over 5s to a local registry ```console $ docker run -d -p 5000:5000 --name registry registry:2 $ docker build -t localhost:5000/pull-timeout-repro:v1 - <<'EOF' FROM busybox RUN dd if=/dev/urandom of=/pad bs=1M count=512 && \\ dd if=/dev/zero of=/bloat bs=1M count=8192 EOF $ docker push localhost:5000/pull-timeout-repro:v1 ``` 3. Pull the image through CRI: ```console $ crictl pull localhost:5000/pull-timeout-repro:v1 ``` 4. Observe the failure and the containerd logs: ```console time=\"2026-08-04T01:08:54.969864113Z\" level=info msg=\"PullImage \\\"localhost:5000/pull-timeout-repro:v1\\\"\" time=\"2026-08-04T01:09:04.970872476Z\" level=error msg=\"cancel pulling image localhost:5000/pull-timeout-repro:v1 because of no progress in 5s\" time=\"2026-08-04T01:09:04.971276186Z\" level=info msg=\"stop pulling image localhost:5000/pull-timeout-repro:v1: active requests=1, bytes read=532676608\" time=\"2026-08-04T01:09:04.975872008Z\" level=error msg=\"failed to cleanup \\\"extract-44496416-qksT sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc\\\"\" error=\"snapshot extract-44496416-qksT sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc does not exist: not found\" time=\"2026-08-04T01:09:04.979959133Z\" level=error msg=\"PullImage \\\"localhost:5000/pull-timeout-repro:v1\\\" failed\" error=\"rpc error: code = Canceled desc = failed to pull and unpack image \\\"localhost:5000/pull-timeout-repro:v1\\\": failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:6351c1a637bce545bf24bf655339b5e123b094fd1ac3036ee79accbfc6879035) to overlayfs as \\\"extract-44496416-qksT sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc\\\": mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3090561630: context canceled\" ``` ### Describe the results you received and expected Received: the pull fails after downloads have completed, while the unpack is progressing. ``` $ crictl pull localhost:5000/pull-timeout-repro:v1 E0804 01:11:11.281711 263 remote_image.go:250] \"PullImage from image service failed\" err=\"rpc error: code = Canceled desc = failed to pull and unpack image \\\"localhost:5000/pull-timeout-repro:v1\\\": failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:6351c1a637bce545bf24bf655339b5e123b094fd1ac3036ee79accbfc6879035) to overlayfs as \\\"extract-346500103-1gP- sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc\\\": mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount4057491: context canceled\" image=\"localhost:5000/pull-timeout-repro:v1\" FATA[0010] pulling image: rpc error: code = Canceled desc = failed to pull and unpack image \"localhost:5000/pull-timeout-repro:v1\": failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:6351c1a637bce545bf24bf655339b5e123b094fd1ac3036ee79accbfc6879035) to overlayfs as \"extract-346500103-1gP- sha256:ea398c04f2bdff858f6901539d699e2220ffb255f6d734b1c09b7251168569dc\": mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount4057491: context canceled ``` Expected: the pull succeeds. ### What version of containerd are you using? containerd github.com/containerd/containerd/v2 v2.2.3 77c84241c7cbdd9b4eca2591793e3d4f4317c590 ### Any other relevant information ```console $ crictl --version crictl version v1.36.0 $ uname -r 7.0.0-28-generic ``` The latest release `v2.3.3` is also affected. ### Show configuration if it is related to CRI plugin. ```toml # /etc/containerd/config.toml version = 3 root = '/var/lib/containerd' [plugins] [plugins.'io.containerd.cri.v1.images'] image_pull_progress_timeout = '5s' [plugins.'io.containerd.cri.v1.images'.registry] config_path = '/etc/containerd/certs.d' ```",
          "url": "https://github.com/containerd/containerd/issues/13909",
          "createdAt": "2026-08-06T09:58:46Z",
          "updatedAt": "2026-08-13T00:24:18Z",
          "timestamp": "2026-08-13T00:24:18Z",
          "metrics": {
            "reactions": 1,
            "comments": 1
          },
          "labels": [
            "kind/bug",
            "area/cri"
          ],
          "author": "naoki9911",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:3347c68877454625ca5b",
        "signalId": "github:containerd/containerd:issue:12973",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:12973",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "make benchmark never work",
          "text": "### Description make benchmark only test the current directory, not subdirectories. ### Steps to reproduce the issue ``` make benchmark + benchmark no Go files in /home/nmx/github.com/ningmingxiao/tmp/containerd FAIL . [setup failed] make: *** [Makefile:252：benchmark ``` ### Describe the results you received and expected make benchmark can test subdirectories ### What version of containerd are you using? master ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. _No response_",
          "url": "https://github.com/containerd/containerd/issues/12973",
          "createdAt": "2026-03-04T03:22:51Z",
          "updatedAt": "2026-08-13T00:19:57Z",
          "timestamp": "2026-08-13T00:19:57Z",
          "metrics": {
            "reactions": 0,
            "comments": 2
          },
          "labels": [
            "kind/bug",
            "go"
          ],
          "author": "ningmingxiao",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:22aef41226fe96b9623c",
        "signalId": "github:containerd/containerd:issue:12760",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:12760",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "Support env variable configuration for logging binary",
          "text": "### What is the problem you're trying to solve When configuring container IO to use a logger binary, containerd only allows me to pass values as command line arguments. ``` // BinaryIO forwards container STDOUT|STDERR directly to a logging binary func BinaryIO(binary string, args map[string]string) Creator ``` ### Describe the solution you'd like I would like the ability to also configure environment variables. E.g., https://github.com/aws/shim-loggers-for-containerd/pull/389 added support for passing the Splunk token as an argument *or* environment variable, and I would like switch to env. ``` TokenKey = \"splunk-token\" TokenEnvKey = \"SPLUNK_TOKEN\" ``` ### Additional context A PR which implements this change: https://github.com/containerd/containerd/pull/12877",
          "url": "https://github.com/containerd/containerd/issues/12760",
          "createdAt": "2026-01-08T22:29:12Z",
          "updatedAt": "2026-08-13T00:19:47Z",
          "timestamp": "2026-08-13T00:19:47Z",
          "metrics": {
            "reactions": 0,
            "comments": 2
          },
          "labels": [
            "kind/enhancement",
            "kind/feature",
            "Stale"
          ],
          "author": "willmyrs",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:92756954b9686f7fd03d",
        "signalId": "github:containerd/containerd:issue:12752",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:12752",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "Unpack failure when mixing remote and local snapshotters",
          "text": "### Description We are encountering unpack failures when mixing remote snapshotters (like `stargz`) and local snapshotters (like `overlayfs`) on the same node. If an image is first pulled using a remote snapshotter—which lazy-loads content—and is later used by a container requiring a local snapshotter, the operation fails. `LocalResolve` incorrectly assumes the image is ready for the local snapshotter because the image metadata exists. However, the local snapshotter requires full content blobs that were never downloaded, leading to a \"content not found\" error during unpacking. ### Steps to reproduce the issue 1. Environment: - Kubernetes: 1.35.0 - Containerd: 2.2.0 - Snapshotters: Stargz (0.17.0) and Overlayfs. 2. Run a pod using the stargz runtime handler. The image is pulled, metadata is stored, but content is lazy-loaded. 3. Run a second pod using the same image but with the default overlayfs runtime handler. ### Describe the results you received and expected Received result: - The Overlayfs Pod fails to start with the following error from kubelet: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 1s default-scheduler Successfully assigned default/overlayfs to lc0 Normal Pulled 0s (x2 over 0s) kubelet spec.containers{overlayfs}: Container image \"quay.io/chengyuzhu6/alpine:3.15.3-estargz\" already present on machine and can be accessed by the pod Warning Failed 0s (x2 over 0s) kubelet spec.containers{overlayfs}: Error: failed to create containerd container: error unpacking image: apply layer error for \"quay.io/chengyuzhu6/alpine:3.15.3-estargz\": failed to extract layer sha256:ebb0761a96f14086da76390e335406d5899572f36f1109aadf53f954a471fbec: failed to get reader from content store: content diges ``` Expected result: Containerd should run successfully. It should detect that the local snapshot/content is missing for the requested snapshotter and automatically fetch the missing blobs from the registry, allowing container with local snapshotter(e.g. overlayfs) to start ### What version of containerd are you using? 2.2.0 ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. [plugins.'io.containerd.cri.v1.images'.runtime_platforms.stargz] platform = 'linux' snapshotter = 'stargz' [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.stargz] runtime_type = \"io.containerd.runc.v2\" snapshotter = \"stargz\" [proxy_plugins] [proxy_plugins.stargz] type = \"snapshot\" address = \"/run/containerd-stargz-grpc/containerd-stargz-grpc.sock\" [proxy_plugins.stargz.exports] root = \"/var/lib/containerd-stargz-grpc/\" enable_remote_snapshot_annotations = \"true\" [[plugins.\"io.containerd.transfer.v1.local\".unpack_config]] platform = \"linux\" snapshotter = \"overlayfs\" [[plugins.\"io.containerd.transfer.v1.local\".unpack_config]] platform = \"linux\" snapshotter = \"stargz\"",
          "url": "https://github.com/containerd/containerd/issues/12752",
          "createdAt": "2026-01-07T11:19:28Z",
          "updatedAt": "2026-08-13T00:19:45Z",
          "timestamp": "2026-08-13T00:19:45Z",
          "metrics": {
            "reactions": 3,
            "comments": 3
          },
          "labels": [
            "kind/bug",
            "Stale"
          ],
          "author": "ChengyuZhu6",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:75bea2e953ff7e87966d",
        "signalId": "github:containerd/containerd:issue:12738",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:12738",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "Continuous memory growth in containerd v2.1.4",
          "text": "### Description We are seeing continuous memory growth on `containerd` version `github.com/containerd/containerd/v2 2.1.4` in our AWS EKS v1.32 cluster. The containerd process memory grows constantly and reaches maximum capacity and causes the node to become non-responsive and also causes OS to kill other things abruptly due to memory pressure causing node to go into `NotReady` status. ### Steps to reproduce the issue We simply keep it running for a few days and memory grows. (Note: We have duplicate metrics. So two same value lines are shown. Ignore one) <img width=\"1831\" height=\"825\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/a25b8918-dae7-41bb-b2db-a3d3374ef8b3\" /> ### Describe the results you received and expected Memory should not grow unbounded like this. ### What version of containerd are you using? containerd github.com/containerd/containerd/v2 2.1.4 75cb2b7193e4e490e9fbdc236c0e811ccaba3376 ### Any other relevant information We also use these nodes for running periodic Argo jobs (short-lived jobs run periodically every 2mins, etc.). I am not sure if this has any relation. ### Show configuration if it is related to CRI plugin. _No response_",
          "url": "https://github.com/containerd/containerd/issues/12738",
          "createdAt": "2026-01-01T08:39:25Z",
          "updatedAt": "2026-08-13T00:19:44Z",
          "timestamp": "2026-08-13T00:19:44Z",
          "metrics": {
            "reactions": 2,
            "comments": 20
          },
          "labels": [
            "kind/bug",
            "area/runtime"
          ],
          "author": "spy16x",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:8609a39c0cbd6faff520",
        "signalId": "github:containerd/containerd:issue:12728",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:12728",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "容器没有重启功能，是否可以出一个重启功能",
          "text": "### Description 容器没有重启功能，是否可以出一个重启功能，类似于docker restart id这样的功能 ### Steps to reproduce the issue 1. 2. 3. ### Describe the results you received and expected docker restart id ### What version of containerd are you using? 1.7 ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. _No response_",
          "url": "https://github.com/containerd/containerd/issues/12728",
          "createdAt": "2025-12-25T06:35:33Z",
          "updatedAt": "2026-08-13T00:19:43Z",
          "timestamp": "2026-08-13T00:19:43Z",
          "metrics": {
            "reactions": 0,
            "comments": 5
          },
          "labels": [
            "kind/bug",
            "kind/feature",
            "Stale",
            "area/runtime"
          ],
          "author": "957446525",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:c784c35dddb47310b957",
        "signalId": "github:containerd/containerd:issue:12517",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:12517",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "Support for time namespaces",
          "text": "### What is the problem you're trying to solve Support for time namespaces is added to OCI runtime spec and runc implemented the necessary functionality in https://github.com/opencontainers/runc/issues/2345 but there seems to be no way to utilize this feature in containerd. ### Describe the solution you'd like Similar to network namespaces, a pod can share a time namespace. This would let the processes inside the container change the time without affecting the host or other pods. I think it should be shared in the pod because it's likely that containers in the same pod interact with each other at a low level that they'd not want to have time differences. IMO, this should be a new default since I can't really think of a case where it'd create issues for others but only additional isolation which is preferred. ### Additional context _No response_",
          "url": "https://github.com/containerd/containerd/issues/12517",
          "createdAt": "2025-11-15T10:05:21Z",
          "updatedAt": "2026-08-13T00:19:40Z",
          "timestamp": "2026-08-13T00:19:40Z",
          "metrics": {
            "reactions": 0,
            "comments": 5
          },
          "labels": [
            "kind/feature",
            "Stale",
            "area/runtime"
          ],
          "author": "muvaf",
          "state": "closed",
          "assignees": [
            "R3hankhan123"
          ],
          "change": "updated"
        }
      },
      {
        "id": "event:f377c659f96092e9304c",
        "signalId": "github:containerd/containerd:issue:12131",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:12131",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "containerd on Windows 11 - cannot override root directory using --root parameter",
          "text": "### Description Hi All, I am trying to use `containerd.exe` on Windows 11. This is a corporate workstation and is locked down. Whenever I run containerd.exe from the command line, I get the following error: ``` containerd: mkdir C:\\ProgramData\\containerd\\root: Access is denied. ``` I cannot create/edit any folders under `c:\\ProgramData\\containerd\\root` since I do not have local Administrator rights. ## Trying the --root parameter I used the `--root` argument to specify a directory as follows: ``` containerd --root C:\\Users\\myusername\\Tools\\containerd\\root ``` But, this too throws the same error as above: ``` containerd: mkdir C:\\ProgramData\\containerd\\state: Access is denied ``` ## Version containerd github.com/containerd/containerd/v2 v2.1.3 c787fb98911740dd3ff2d0e45ce88cdf01410486 Any advice? Thanks, Sau ### Steps to reproduce the issue 1. Download the compressed content for AMD64 platform. 2. Use 7Z on Windows or some equivalent tool to extract the inner folders and files. 3. Copy over all the binaries (containerd.exe , ctr.exe, containerd-stress.exe, containerd-shim-runhcs-v1.exe) to another folder 4. Launch a CMD 5. Change over directory to the folder where you extracted the binaries (containerd.exe) 6. Run the command `containerd.exe --root C:\\Users\\saurabhdasgupta\\Tools\\containerd\\root` You should see the error reported above ### Describe the results you received and expected Containerd throws the following error: ``` containerd: mkdir C:\\ProgramData\\containerd\\state: Access is denied. ``` ### What version of containerd are you using? containerd github.com/containerd/containerd/v2 v2.1.3 c787fb98911740dd3ff2d0e45ce88cdf01410486 ### Any other relevant information _No response_ ### Show configuration if it is related to CRI plugin. _No response_",
          "url": "https://github.com/containerd/containerd/issues/12131",
          "createdAt": "2025-07-21T21:47:39Z",
          "updatedAt": "2026-08-13T00:19:32Z",
          "timestamp": "2026-08-13T00:19:32Z",
          "metrics": {
            "reactions": 0,
            "comments": 4
          },
          "labels": [
            "kind/bug",
            "platform/windows",
            "Stale"
          ],
          "author": "sdg002",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:6d028503e99b27a18e8d",
        "signalId": "github:containerd/containerd:pull_request:11477",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:11477",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "Draft: ctr support print plugin config",
          "text": "``` bin/ctr plugin inspect runtime { \"cdiSpecDirs\": [ \"/etc/cdi\", \"/var/run/cdi\" ], \"cni\": { \"binDir\": \"/opt/cni/bin\", \"confDir\": \"/etc/cni/net.d\", \"confTemplate\": \"\", \"ipPref\": \"\", \"maxConfNum\": 1, \"setupSerially\": false, \"useInternalLoopback\": false }, \"containerd\": { \"defaultRuntimeName\": \"runc\", \"ignoreBlockIONotEnabledErrors\": false, \"ignoreRdtNotEnabledErrors\": false, \"runtimes\": { \"runc\": { \"ContainerAnnotations\": [], \"PodAnnotations\": [], \"baseRuntimeSpec\": \"\", \"cgroupWritable\": false, \"cniConfDir\": \"\", \"cniMaxConfNum\": 0, \"io_type\": \"\", \"options\": { \"BinaryName\": \"\", \"CriuImagePath\": \"\", \"CriuPath\": \"\", \"CriuWorkPath\": \"\", \"IoGid\": 0, \"IoUid\": 0, \"NoNewKeyring\": false, \"NoPivotRoot\": false, \"Root\": \"\", \"ShimCgroup\": \"\", \"SystemdCgroup\": false }, \"privileged_without_host_devices\": false, \"privileged_without_host_devices_all_devices_allowed\": false, \"runtimePath\": \"\", \"runtimeType\": \"io.containerd.runc.v2\", \"sandboxer\": \"podsandbox\", \"snapshotter\": \"\" } } }, \"containerdEndpoint\": \"/run/containerd/containerd.sock\", \"containerdRootDir\": \"/var/lib/containerd\", \"device_ownership_from_security_context\": false, \"disableApparmor\": false, \"disableHugetlbController\": true, \"disableProcMount\": false, \"drainExecSyncIOTimeout\": \"0s\", \"enableCDI\": false, \"enableSelinux\": true, \"enableUnprivilegedICMP\": false, \"enableUnprivilegedPorts\": false, \"ignoreDeprecationWarnings\": null, \"ignoreImageDefinedVolumes\": false, \"maxContainerLogSize\": 16384, \"netnsMountsUnderStateDir\": false, \"restrictOOMScoreAdj\": false, \"rootDir\": \"/var/lib/containerd/io.containerd.grpc.v1.cri\", \"selinuxCategoryRange\": 1024, \"stateDir\": \"/run/containerd/io.containerd.grpc.v1.cri\", \"tolerateMissingHugetlbController\": true, \"unsetSeccompProfile\": \"\" } ```",
          "url": "https://github.com/containerd/containerd/pull/11477",
          "createdAt": "2025-03-04T11:44:15Z",
          "updatedAt": "2026-08-13T00:19:14Z",
          "timestamp": "2026-08-13T00:19:14Z",
          "metrics": {
            "reactions": 0,
            "comments": 4
          },
          "labels": [
            "kind/feature",
            "area/cri",
            "needs-ok-to-test",
            "Stale",
            "size/M",
            "do-not-merge/work-in-progress"
          ],
          "author": "ningmingxiao",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:cbc37323a867d8359a3e",
        "signalId": "github:containerd/containerd:pull_request:11458",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:11458",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "ignore update status file when no space",
          "text": "Fix #7247 Failed to start recovre, the root cause is inconsistent data between `cri` and `container in metadata` Ensure consistency, the following manual recovery could be resolve. ``` - stop containerd service, like `systemctl stop containerd` - disable cri plugin in config.toml and start containerd service - find container and task which reversed, and stop task, delete container to release disk, which most like - `ctr -n k8s.io t stop [id]` - `ctr -n k8s.io c rm [id]` - enable cri plugin in config.toml - restart containerd service, `systemctl restart containerd` ``` As described in [here](https://github.com/containerd/containerd/issues/11504), ignoring noSpace error is a more convenient way to handle such issues",
          "url": "https://github.com/containerd/containerd/pull/11458",
          "createdAt": "2025-03-03T06:50:30Z",
          "updatedAt": "2026-08-13T00:19:12Z",
          "timestamp": "2026-08-13T00:19:12Z",
          "metrics": {
            "reactions": 0,
            "comments": 4
          },
          "labels": [
            "area/cri",
            "needs-ok-to-test",
            "Stale",
            "size/L"
          ],
          "author": "yylt",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:c6ef4053044f5266e20f",
        "signalId": "github:containerd/containerd:pull_request:12889",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:12889",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "docker fetcher: strip sensitive headers on descriptor URLs",
          "text": "# docker fetcher: strip sensitive headers for desc.urls fetches context: - containerd supports fetching content from `descriptor.urls` (oci descriptor URLs). - the desc.urls fetch path currently reuses resolver/global headers (e.g., CRI `registry.headers` / hosts headers). - when desc.urls points to a non-registry origin, forwarding well-known sensitive headers (e.g., `Authorization`, `Proxy-Authorization`, `Cookie`) is a cross-origin trust-boundary footgun. change: - keep operator control and existing behavior, but strip a small set of well-known sensitive headers on desc.urls-driven requests: `Authorization`, `Proxy-Authorization`, `Cookie`, `Cookie2`. - this mirrors go standard library redirect behavior where sensitive headers are not forwarded cross-origin. test: - add `TestFetcherDescURLsDoesNotForwardResolverHeaders` to assert sensitive headers are not sent to desc.urls origins, while non-sensitive custom headers still flow. scope: - `core/remotes/docker` only. pins: - repo: https://github.com/containerd/containerd - commit: c6d6c8af753c3b5cb6528231b9063d1372f7d226",
          "url": "https://github.com/containerd/containerd/pull/12889",
          "createdAt": "2026-02-11T20:14:10Z",
          "updatedAt": "2026-08-12T23:29:53Z",
          "timestamp": "2026-08-12T23:29:53Z",
          "metrics": {
            "reactions": 0,
            "comments": 5
          },
          "labels": [
            "kind/enhancement",
            "cherry-pick/1.7.x",
            "size/L",
            "area/distribution",
            "cherry-pick/2.0.x",
            "cherry-pick/2.2.x",
            "cherry-pick/2.3.x"
          ],
          "author": "1seal",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:9ff0960e577657f5663a",
        "signalId": "github:containerd/containerd:pull_request:13775",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13775",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "core/remotes/docker: only fetch descriptor urls for foreign layers",
          "text": "dockerFetcher.Fetch walks desc.URLs before it tries the registry endpoints and issues a GET to each one, checking only that the scheme is http or https. Those urls come straight from the image manifest, so a crafted image can put an arbitrary address on any descriptor (a config, a regular layer, even a child manifest) and make the daemon request it during a pull, for instance an internal service or a cloud metadata endpoint. The urls field is the mechanism for non-distributable (foreign) layers, and distributable content is always available from the registry, which is exactly what the loop already falls through to. Gating on images.IsNonDistributable keeps foreign layers working while dropping urls on every other descriptor, so the check sits next to the request it guards instead of at each caller.",
          "url": "https://github.com/containerd/containerd/pull/13775",
          "createdAt": "2026-07-11T05:25:11Z",
          "updatedAt": "2026-08-12T23:24:59Z",
          "timestamp": "2026-08-12T23:24:59Z",
          "metrics": {
            "reactions": 0,
            "comments": 3
          },
          "labels": [
            "size/M"
          ],
          "author": "aysha-afrah26",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:e4045f8d1482a42be1bc",
        "signalId": "github:containerd/containerd:pull_request:13705",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13705",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "fix: prevent tar extraction data corruption by limiting reader size",
          "text": "we find sometimes extract a binary file from a blob, binary is broken ls -l fabric-manager_bad fabric-manager_ok -rwxr-xr-x 1 root root 44538840 6月 12 15:44 fabric-manager_bad -rwxr-xr-x 1 root root 44538840 6月 12 15:44 fabric-manager_ok ``` cmp -l fabric-manager_bad fabric-manager_ok 5525505 177 75 5525506 376 272 5525571 377 203 5525572 377 75 5525573 377 167 5525574 377 222 5525575 377 126 5525576 377 2 5525577 377 0 5525578 377 164 5525579 377 13 5525580 377 110 5525581 377 213 5525582 377 20 5525583 377 350 5525584 377 255 5525585 377 107 5525586 377 264 5525588 377 111 5525589 377 211 5525590 377 23 5525591 377 110 5525592 377 215 5525593 377 25 5525594 377 344 5525595 377 314 5525596 377 102 5525597 377 1 5525598 377 110 5525599 377 211 5525600 377 20 5525601 377 110 5525602 377 215 5525603 377 5 5525604 377 231 5525605 377 127 5525606 377 22 5525607 377 1 5525608 377 110 5525609 377 213 5525610 377 134 5525611 377 44 5525612 377 40 5525613 377 110 5525614 377 215 5525615 377 15 5525616 377 51 5525617 377 37 5525618 377 70 5525619 377 1 5525620 377 277 5525621 377 16 5525622 377 0 5525623 377 0 5525624 377 0 5525625 377 350 5525626 377 303 5525627 377 255 5525628 377 254 5525630 377 110 5525631 377 307 5525632 377 100 5525633 377 10 5525634 377 160 5525635 377 0 5525636 377 0 5525637 377 0 5525638 377 203 5525639 377 75 5525640 377 64 5525641 377 222 5525642 377 126 5525643 377 2 5525644 377 0 5525645 377 164 5525646 377 13 5525647 377 110 5525648 377 213 5525649 377 20 5525650 377 350 5525651 377 152 5525652 377 107 5525653 377 264 5525655 377 111 5525656 377 211 5525657 377 23 5525658 377 110 5525659 377 215 5525660 377 25 5525661 377 4 5525662 377 70 5525663 377 100 5525664 377 1 5525665 377 110 5525666 377 211 5525667 377 20 5525668 377 110 5525669 377 215 5525670 377 5 5525671 377 126 5525672 377 127 5525673 377 22 5525674 377 1 5525675 377 110 5525676 377 213 5525677 377 134 5525678 377 44 5525679 377 40 5525680 377 110 5525681 377 215 5525682 377 15 5525683 377 172 5525684 377 140 5525685 377 67 5525686 377 1 5525687 377 277 5525688 377 5 5525689 377 0 5525690 377 0 5525691 377 0 5525692 377 17 5525693 377 37 5525694 377 104 5525695 377 0 5525696 377 0 5525697 377 350 5525698 377 173 5525699 377 255 5525700 377 254 5525702 377 110 5525703 377 307 5525704 377 100 5525705 377 10 5525706 377 124 5525707 377 5 5525708 377 0 5525709 377 0 5525710 377 203 5525711 377 75 5525712 377 354 5525713 377 221 5525714 377 126 5525715 377 2 5525716 377 0 5525717 377 164 5525718 377 13 5525719 377 110 5525720 377 213 5525721 377 20 5525722 377 350 5525723 377 42 5525724 377 107 5525725 377 264 5525727 377 111 5525728 377 211 5525729 377 23 5525730 377 110 5525731 377 215 5525732 377 25 5525733 377 163 5525734 377 305 5525735 377 106 5525736 377 1 5525737 377 110 5525738 377 211 5525739 377 20 5525740 377 110 5525741 377 215 5525742 377 5 5525743 377 16 5525744 377 127 5525745 377 22 5525746 377 1 5525747 377 110 5525748 377 213 5525749 377 134 5525750 377 44 5525751 377 40 5525752 377 110 5525753 377 215 5525754 377 15 5525755 377 77 5525756 377 243 5525757 377 67 5525758 377 1 5525759 377 277 5525760 377 10 5525761 377 0 5525762 377 0 5525763 377 0 5525764 377 350 5525765 377 70 5525766 377 255 5525767 377 254 5525769 377 110 5525770 377 307 5525771 377 100 5525772 377 10 5525773 377 307 5525774 377 4 5525775 377 0 5525776 377 0 5525777 377 203 5525778 377 75 5525779 377 251 5525780 377 221 5525781 377 126 5525782 377 2 5525783 377 0 5525784 377 164 5525785 377 17 5525786 377 110 5525787 377 213 5525788 377 20 5525789 377 17 5525790 377 37 5525791 377 100 5525792 377 0 5525793 377 350 5525794 377 333 5525795 377 106 5525796 377 264 5525798 377 111 5525799 377 211 5525800 377 23 5525801 377 110 5525802 377 215 5525803 377 25 5525804 377 335 5525805 377 214 5525806 377 106 5525807 377 1 5525808 377 110 5525809 377 211 5525810 377 20 5525811 377 110 5525812 377 215 5525813 377 5 5525814 377 307 5525815 377 126 5525816 377 22 5525817 377 1 5525818 377 110 5525819 377 213 5525820 377 134 5525821 377 44 5525822 377 40 5525823 377 110 5525824 377 215 5525825 377 15 5525826 377 260 5525827 377 156 5525828 377 70 5525829 377 1 5525830 377 277 5525831 377 21 5525832 377 0 5525833 377 0 5525834 377 0 5525835 377 350 5525836 377 361 5525837 377 254 5525838 377 254 5525840 377 110 5525841 377 307 5525842 377 100 5525843 377 10 5525844 377 144 5525845 377 5 5525846 377 0 5525847 377 0 5525848 377 203 5525849 377 75 5525850 377 142 5525851 377 221 5525852 377 126 5525853 377 2 5525854 377 0 5525855 377 146 5525856 377 220 5525857 377 165 5525858 377 7 5525859 377 110 5525860 377 213 5525861 377 124 5525862 377 44 5525863 377 40 5525864 377 353 5525865 377 37 5525866 377 110 5525867 377 213 5525868 377 15 5525869 377 300 5525870 377 117 5525871 377 123 5525872 377 2 5525873 377 110 5525874 377 213 5525875 377 20 5525876 377 350 5525877 377 310 5525878 377 106 5525879 377 264 5525881 377 111 5525882 377 211 5525883 377 23 5525884 377 110 5525885 377 213 5525886 377 124 5525887 377 44 5525888 377 40 5525889 377 111 5525890 377 211 5525891 377 123 5525892 377 10 5525893 377 111 5525894 377 211 5525895 377 113 5525896 377 20 5525897 377 110 5525898 377 215 5525899 377 15 5525900 377 40 5525901 377 312 5525902 377 106 5525903 377 1 5525904 377 110 5525905 377 211 5525906 377 10 5525907 377 110 5525908 377 211 5525909 377 25 5525910 377 227 5525911 377 117 5525912 377 123 5525913 377 2 5525914 377 110 5525915 377 203 5525916 377 304 5525917 377 50 5525918 377 135 5525919 377 303 5525920 377 220 5525921 377 350 5525922 377 33 5525923 377 51 5525924 377 264 5525926 377 351 5525927 377 266 5525928 377 374 5525931 377 314 5525932 377 314 5525933 377 314 5525934 377 314 5525935 377 314 5525936 377 314 5525937 377 314 5525938 377 314 5525939 377 314 5525940 377 314 5525941 377 314 5525942 377 314 5525943 377 314 5525944 377 314 5525945 377 314 5525946 377 314 5525947 377 314 5525948 377 314 5525949 377 314 5525950 377 314 5525951 377 314 5525952 377 314 5525953 377 111 5525954 377 73 ```",
          "url": "https://github.com/containerd/containerd/pull/13705",
          "createdAt": "2026-07-02T04:50:56Z",
          "updatedAt": "2026-08-12T21:20:52Z",
          "timestamp": "2026-08-12T21:20:52Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/XS"
          ],
          "author": "ningmingxiao",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:da35486791bbbb85d558",
        "signalId": "github:containerd/containerd:pull_request:13910",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13910",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "pkg/shim: Report bootstrap API mismatch on startup",
          "text": "- replace / close: https://github.com/containerd/containerd/pull/13764 - address / close: https://github.com/containerd/containerd/issues/13763 A pre-2.3 containerd daemon can start a newly installed shim through the deprecated CLI, environment, and stdin API. The shim currently accepts that request but returns a protobuf bootstrap result, which the old daemon cannot parse. It treats the protobuf bytes as a socket address and fails later with a cryptic error: ``` failed to create TTRPC connection: unsupported protocol: \\b\\x03\\x12Yunix ``` This patch makes the shim reject the deprecated startup request before starting and tell the user that the containerd daemon may need to be restarted. The 2.2 daemon includes the shim's stderr in its start error, so this message reaches the user directly. This explicitly removes input-side support for the deprecated startup API, but it does not break a working standard containerd pairing. Pre-2.3 daemons already cannot consume the response from the new shim. After a package upgrade, a restarted daemon uses the new bootstrap API. Custom callers that send the deprecated request format while supporting the new protobuf response will now also be rejected.",
          "url": "https://github.com/containerd/containerd/pull/13910",
          "createdAt": "2026-08-06T19:31:15Z",
          "updatedAt": "2026-08-12T20:28:43Z",
          "timestamp": "2026-08-12T20:28:43Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M",
            "cherry-pick/2.3.x"
          ],
          "author": "vvoland",
          "state": "open",
          "assignees": [
            "vvoland"
          ],
          "change": "updated"
        }
      },
      {
        "id": "event:91bc6db0a1ac446add81",
        "signalId": "github:containerd/containerd:pull_request:13924",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13924",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "fix(cri): CRI image pull is sometimes canceled by image_pull_progress…",
          "text": "Fixes #13909. ### Description When the CRI plugin pulls an image through the transfer service, the image pull can be canceled by `image_pull_progress_timeout` even when all blob downloads have finished and the remaining unpack is progressing. This happens because the transfer service's polling period is 300ms, and extraction can start within a few milliseconds after download completes. The job transitions directly to `jobExtracting` and sends `\"extracting\"`/`\"extracted\"` events instead of `\"complete\"`. Since the CRI `transferProgressReporter.handleProgress` only handled `\"complete\"`, the active request counter was never decremented, causing the watchdog to eventually time out the pull due to lack of download progress. This patch fixes the issue by allowing the CRI reporter to treat `\"extracting\"`, `\"extracted\"`, and `\"already exists\"` events as completion of the download phase, properly decrementing the active request counter and avoiding the timeout.",
          "url": "https://github.com/containerd/containerd/pull/13924",
          "createdAt": "2026-08-08T23:17:38Z",
          "updatedAt": "2026-08-12T20:28:41Z",
          "timestamp": "2026-08-12T20:28:41Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M",
            "cherry-pick/2.2.x",
            "cherry-pick/2.3.x"
          ],
          "author": "RohithPariki",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:5ce11054e80787563652",
        "signalId": "github:containerd/containerd:pull_request:13949",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13949",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "Ensure credentials are forwarded to configured mirrors",
          "text": "This PR fixes an issue with image credentials being forwarded to unrelated mirrors",
          "url": "https://github.com/containerd/containerd/pull/13949",
          "createdAt": "2026-08-12T16:30:53Z",
          "updatedAt": "2026-08-12T20:21:09Z",
          "timestamp": "2026-08-12T20:21:09Z",
          "metrics": {
            "reactions": 0,
            "comments": 2
          },
          "labels": [
            "size/L",
            "do-not-merge/work-in-progress"
          ],
          "author": "MikeZappa87",
          "state": "open",
          "assignees": [
            "MikeZappa87"
          ],
          "change": "updated"
        }
      },
      {
        "id": "event:ae4f817b98c72d13d2af",
        "signalId": "github:containerd/containerd:issue:5708",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:5708",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "containerd-shim process isn't reaped for some killed containers",
          "text": "**Description** We have several EKS clusters which autoscale throughout the day - they handle burst workloads, and in a given day the underlying ASG-s may scale down to 0 nodes, or scale up to tens of nodes. We've noticed that once in a while, we have nodes which have pods stuck in a 'Terminating' status on them for days on-end, until we manually intervene and force-delete them; I've SSH-ed to one of the nodes which experienced this behavior and tried to introspect the behavior, here's what I've gathered, hopefully covering most abstraction layers, but I could not find the root cause unfortunately - I'd love to know what can I do to further debug this. Quick summary (more info below): - Kubelet is trying to kill an already dead container indefinitely - Docker thinks the container is running, although it is not - Containerd shows the task is stopped - The container-shim process wasn't reaped and is still alive - The shim receives a kill request from containerd, it execve's runc to kill the container; it receives \"container not running\" and responds to containerd with \"process already finished: not found\" - The above loops for every kubelet's kill retry The above leads me to conclude that there's something fishy in the distributed orchestration of killing a container, I'd assume that it's somewhere between containerd<->runc, but I'm not entirely sure - and would love to know how can I better pinpoint the exact cause. **Steps to reproduce the issue:** I'm not entirely sure how to reproduce the behavior yet, as it happens sporadically in arbitrary nodes. **Describe the results you received:** Containers are dead, containerd is aware they are stopped, but the shim isn't reaped, docker thinks the container is still running and misleads kubelet to keep it as Terminated until manually intervening (force deleting the pod). **Describe the results you expected:** The shim should go down with the container, docker should be notified the container is stopped so that kubelet will update the pod's status accordingly. **What version of containerd are you using:** ``` $ containerd --version containerd github.com/containerd/containerd 1.4.1 c623d1b36f09f8ef6536a057bd658b3aa8632828 ``` **Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):** Deeper dive per abstraction layer - **Kubernetes / kubelet:** - The pod is stuck in a 'Terminating state' - There are no finalizers on the pod - When I describe the pod, I see something similar to 'Normal Killing 2m49s (x1715 over 2d6h) kubelet Stopping container csi-node' - In journalctl -u kubelet, I see the following near the time the pod started to terminate: ``` Jul 07 03:40:25 ip-10-0-73-87.ec2.internal kubelet[4811]: I0707 03:40:20.474218 4811 kubelet.go:1848] SyncLoop (DELETE, \"api\"): \"csi-driver-aws-csi-driver-node-2qssj_default(6f2f36c4-06f0-406d-9681-b92fa0106441) \" Jul 07 03:40:32 ip-10-0-73-87.ec2.internal kubelet[4811]: I0707 03:40:32.052286 4811 kubelet.go:1870] SyncLoop (PLEG): \"csi-driver-aws-csi-driver-node-2qssj_default(6f2f36c4-06f0-406d-9681-b92fa0106441)\", event: &pleg.PodLifecycleEvent{ID:\"6f2f36c4-06f0-406d-9681-b92fa0106441\", Type:\"ContainerDied\", Data:\"789845b4d5dc620ce62b36ff5a6d2ef380a725226264b0170d33ea645bb837f1\"} Jul 07 03:41:18 ip-10-0-73-87.ec2.internal kubelet[4811]: I0707 03:41:18.375163 4811 kubelet.go:1870] SyncLoop (PLEG): \"csi-driver-aws-csi-driver-node-2qssj_default(6f2f36c4-06f0-406d-9681-b92fa0106441)\", event: &pleg.PodLifecycleEvent{ID:\"6f2f36c4-06f0-406d-9681-b92fa0106441\", Type:\"ContainerDied\", Data:\"650c0009c27bfd04f4578e3a5fe2ce0eea300088acb5e669f1d71c5d187139ff\"} ``` - Then, I see indefinite \"killing container\" messages: ``` Jul 09 10:32:23 ip-10-0-73-87.ec2.internal kubelet[4811]: I0709 10:32:23.591613 4811 kuberuntime_container.go:635] Killing container \"docker://cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\" wi th a 30 second grace period Jul 09 10:34:24 ip-10-0-73-87.ec2.internal kubelet[4811]: I0709 10:34:24.591670 4811 kuberuntime_container.go:635] Killing container \"docker://cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\" wi th a 30 second grace period Jul 09 10:36:09 ip-10-0-73-87.ec2.internal kubelet[4811]: I0709 10:36:09.591654 4811 kuberuntime_container.go:635] Killing container \"docker://cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\" wi th a 30 second grace period ``` - I've uploaded the kubelet stacktrace here - [kubelet-stacktrace.txt](https://github.com/awslabs/amazon-eks-ami/files/6794004/kubelet-stacktrace.txt) **Docker:** - When I docker container ls / docker inspect <container-id> - the container status is 'Running' ``` { \"Id\": \"cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\", \"Created\": \"2021-07-07T00:11:27.14530101Z\", \"Path\": \"/entrypoint.sh\", \"Args\": [ \"--endpoint=unix:///csi/csi.sock\", \"--v=4\", \"--volume-attach-limit=5\" ], \"State\": { \"Status\": \"running\", \"Running\": true, \"Paused\": false, \"Restarting\": false, \"OOMKilled\": false, \"Dead\": false, \"Pid\": 7064, \"ExitCode\": 0, \"Error\": \"\", \"StartedAt\": \"2021-07-07T00:11:36.641265855Z\", \"FinishedAt\": \"0001-01-01T00:00:00Z\" }, ... } ``` - In journalctl, looking near the time the pod started terminating, I see the following: ``` -- Logs begin at Wed 2021-07-07 00:10:20 UTC, end at Fri 2021-07-09 10:46:56 UTC. -- Jul 07 03:30:43 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:30:43.252002398Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:30:44 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:30:44.542537801Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:39:32 ip-10-0-73-87.ec2.internal dockerd[4325]: http: superfluous response.WriteHeader call from github.com/docker/docker/api/server/httputils.WriteJSON (httputils_write_json.go:11) on returned error: write unix /var/run/docker.sock->@: write: broken pipe\" Jul 07 03:40:35 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:40:35.735692652Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:40:44 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:40:44.165635762Z\" level=error msg=\"stream copy error: reading from a closed fifo\" Jul 07 03:40:47 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:40:47.504174793Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:40:56 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:40:55.687736472Z\" level=info msg=\"Container cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a failed to exit within 30 se conds of signal 15 - using the force\" Jul 07 03:41:07 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:06.802768003Z\" level=info msg=\"Container cd7ed93ae2d1 failed to exit within 10 seconds of kill - trying direct SIGKILL\" Jul 07 03:41:17 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:17.058149021Z\" level=warning msg=\"Published ports are discarded when using host network mode\" Jul 07 03:41:17 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:17.066107174Z\" level=info msg=\"ignoring event\" module=libcontainerd namespace=moby topic=/tasks/delete type=\"*events.TaskDelete\" Jul 07 03:41:48 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:48.397339761Z\" level=info msg=\"Container cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a failed to exit within 30 se conds of signal 15 - using the force\" Jul 07 03:41:58 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:41:58.415920505Z\" level=info msg=\"Container cd7ed93ae2d1 failed to exit within 10 seconds of kill - trying direct SIGKILL\" Jul 07 03:42:28 ip-10-0-73-87.ec2.internal dockerd[4325]: time=\"2021-07-07T03:42:28.634636074Z\" level=info msg=\"Container cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a failed to exit within 30 se conds of signal 15 - using the force\" ``` - The SIGKILL / 'using the force' messages continue indefinitely - I've uploaded the dockerd stacktrace here - [docker-stacktrace.txt](https://github.com/awslabs/amazon-eks-ami/files/6793969/docker-stacktrace.txt) **Containerd:** - In containerd, the task of the container is 'STOPPED' - `cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a 7064 STOPPED` - The task's metrics prints the following: ``` ID TIMESTAMP cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a 2021-07-09 10:52:16.172595382 +0000 UTC METRIC VALUE memory.usage_in_bytes 20029440 memory.limit_in_bytes 9223372036854771712 memory.stat.cache 11411456 cpuacct.usage 69532053147 cpuacct.usage_percpu [14916245050 21098017259 14389165228 19128625610] pids.current 0 pids.limit 0 ``` - The container's info doesn't show something particularly helpful: ``` { \"ID\": \"cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\", \"Labels\": { \"com.docker/engine.bundle.path\": \"/var/run/docker/containerd/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\" }, \"Image\": \"\", \"Runtime\": { \"Name\": \"io.containerd.runtime.v1.linux\", \"Options\": { \"type_url\": \"containerd.linux.runc.RuncOptions\", \"value\": \"CgRydW5jEhwvdmFyL3J1bi9kb2NrZXIvcnVudGltZS1ydW5j\" } }, \"SnapshotKey\": \"\", \"Snapshotter\": \"\", \"CreatedAt\": \"2021-07-07T00:11:36.498772067Z\", \"UpdatedAt\": \"2021-07-07T00:11:36.498772067Z\", \"Extensions\": null, \"Spec\": { \"ociVersion\": \"1.0.1-dev\", ... }, ... } ``` - journalctl -u containerd does not have logs with the given container id - I've uploaded the containerd stack trace here - [containerd-goroutine-stacktrace.txt](https://github.com/awslabs/amazon-eks-ami/files/6793929/containerd-goroutine-stacktrace.txt) **containerd-shim:** - The containerd-shim of the container is still running: ``` root 7023 0.0 0.0 710748 6268 ? Sl Jul07 0:12 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f36 44815387a37a -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc ``` - I tried to peek at the stdout / stderr of the shim in - but either the process hadn't flushed data, or it is simply empty - To find the shim id, I've looked at the process fd-s and correlated the opened sockets w/ ss: ``` [root@ip-10-0-73-87 ~]# ls -al /proc/$pid/fd | awk '/socket/ { print $NF }' | grep -o '[0-9]*' | xargs -I{} sh -c \"ss -apn | grep {}\" u_str LISTEN 0 4096 /run/containerd/s/6e99a634bfa5b915cbeade50e47384f60874a9358e5e96cb59523a46339c138b 25584 * 0 users:((\"containerd-shim\",pid=7023,fd=12),(\"containerd\",pid=3538,fd=80)) u_str ESTAB 0 0 /run/containerd/s/6e99a634bfa5b915cbeade50e47384f60874a9358e5e96cb59523a46339c138b 24369 * 25595 users:((\"containerd-shim\",pid=7023,fd=3)) u_str ESTAB 0 0 * 25595 * 24369 users:((\"containerd\",pid=3538,fd=87)) ``` - Then I looked at the containerd journal entries for the shim id (6e99a6...) and the only thing that came up was: `Jul 07 00:11:36 ip-10-0-73-87.ec2.internal containerd[3538]: time=\"2021-07-07T00:11:36.521930693Z\" level=info msg=\"shim containerd-shim started\" address=\"unix:///run/containerd/s/6e99a634bfa5b915cbeade50e47384f6087 4a9358e5e96cb59523a46339c138b\" debug=false pid=7023` - I've managed to retrieve the shim's go stack trace by strace-ing to a file and sending a kill -USR1 to it, but I don't see anything of particular interest there: <details> <summary>shim stacktrace</summary> <p> ``` write(8</var/lib/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a/shim.stdout.log>, \"time=\"2021-07-09T11:56:32Z\" level=info msg=\"=== BEGIN goroutine stack dump === goroutine 9 [running]: main.dumpStacks(0xc000064150) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:276 +0x74 main.executeShim.func1(0xc00004a1e0, 0xc000064150) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:186 +0x3d created by main.executeShim /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:184 +0x5e9 goroutine 1 [select]: main.handleSignals(0xc000064150, 0xc0000a8720, 0xc00005c090, 0xc00006e000, 0xc0000d9e50, 0x0) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:239 +0x119 main.executeShim(0xc0000d2540, 0x7fe26c883088) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:189 +0x625 main.main() /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:118 +0x20a goroutine 18 [chan receive]: main.main.func1() /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:89 +0x85 created by main.main /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:88 +0x74 goroutine 6 [syscall]: syscall.Syscall6(0xe8, 0xb, 0xc00003b9b8, 0x80, 0xffffffffffffffff, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x4) /usr/lib/golang/src/syscall/asm_linux_amd64.s:41 +0x5 github.com/containerd/containerd/vendor/golang.org/x/sys/unix.EpollWait(0xb, 0xc00003b9b8, 0x80, 0x80, 0xffffffffffffffff, 0xffffffffffffffff, 0x85a8e0, 0xa5cb40) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x72 github.com/containerd/containerd/vendor/github.com/containerd/console.(*Epoller).Wait(0xc00005c2d0, 0xc0000d9a78, 0x8) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/console/console_linux.go:111 +0x7a created by github.com/containerd/containerd/runtime/v1/shim.(*Service).initPlatform /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service_linux.go:113 +0xbb goroutine 5 [chan receive]: github.com/containerd/containerd/runtime/v1/shim.(*Service).processExits(0xc00006e000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service.go:501 +0xd6 created by github.com/containerd/containerd/runtime/v1/shim.NewService /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service.go:92 +0x40d goroutine 24 [runnable]: os/signal.process(0x85ec60, 0xa5cb70) /usr/lib/golang/src/os/signal/signal.go:240 +0x10a os/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x45 created by os/signal.Notify.func1.1 /usr/lib/golang/src/os/signal/signal.go:150 +0x45 goroutine 7 [chan receive, 3375 minutes]: github.com/containerd/containerd/runtime/v1/shim.(*Service).forward(0xc00006e000, 0x85a3e0, 0xc00001a030) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service.go:579 +0x71 created by github.com/containerd/containerd/runtime/v1/shim.NewService /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/runtime/v1/shim/service.go:96 +0x4bf goroutine 8 [IO wait, 23 minutes]: internal/poll.runtime_pollWait(0x7fe26c882de8, 0x72, 0x0) /usr/lib/golang/src/runtime/netpoll.go:220 +0x55 internal/poll.(*pollDesc).wait(0xc000074098, 0x72, 0x0, 0x0, 0x7fee5f) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000074080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/lib/golang/src/internal/poll/fd_unix.go:394 +0x1fc net.(*netFD).accept(0xc000074080, 0xc00009e458, 0xc0000121b8, 0x79ccc0) /usr/lib/golang/src/net/fd_unix.go:172 +0x45 net.(*UnixListener).accept(0xc00005c330, 0xc00008fe50, 0xc00008fe58, 0x18) /usr/lib/golang/src/net/unixsock_posix.go:162 +0x32 net.(*UnixListener).Accept(0xc00005c330, 0x812878, 0xc000012190, 0x862f20, 0xc0000a0000) /usr/lib/golang/src/net/unixsock.go:260 +0x65 github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Server).Serve(0xc00005c090, 0x862f20, 0xc0000a0000, 0x861720, 0xc00005c330, 0x0, 0x0) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:87 +0x107 main.serve.func1(0x861720, 0xc00005c330, 0xc00005c090, 0x862f20, 0xc0000a0000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:224 +0x88 created by main.serve /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/cmd/containerd-shim/main_unix.go:222 +0x1fe goroutine 25 [select]: github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run(0xc00009e3c0, 0x862f20, 0xc0000a0000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:431 +0x433 created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Server).Serve /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:127 +0x28d goroutine 10 [IO wait]: internal/poll.runtime_pollWait(0x7fe26c882d08, 0x72, 0x85a8e0) /usr/lib/golang/src/runtime/netpoll.go:220 +0x55 internal/poll.(*pollDesc).wait(0xc0000e2318, 0x72, 0x85a800, 0xa4b248, 0x0) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0000e2300, 0xc000077000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/internal/poll/fd_unix.go:159 +0x1a5 net.(*netFD).Read(0xc0000e2300, 0xc000077000, 0x1000, 0x1000, 0xc000056270, 0xc00018fc28, 0x20) /usr/lib/golang/src/net/fd_posix.go:55 +0x4f net.(*conn).Read(0xc0000a6078, 0xc000077000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/net.go:182 +0x8e bufio.(*Reader).Read(0xc00004a420, 0xc000078060, 0xa, 0xa, 0xc00018fd98, 0x447f74, 0xc00018ff18) /usr/lib/golang/src/bufio/bufio.go:227 +0x222 io.ReadAtLeast(0x859fa0, 0xc00004a420, 0xc000078060, 0xa, 0xa, 0xa, 0xa, 0x2, 0x789f60) /usr/lib/golang/src/io/io.go:314 +0x87 io.ReadFull(...) /usr/lib/golang/src/io/io.go:333 github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc000078060, 0xa, 0xa, 0x859fa0, 0xc00004a420, 0x73, 0xc00002ec00, 0x0, 0x6b0aae) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:53 +0x69 github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc000078040, 0xc00018fe2c, 0x3, 0x2, 0xc00014a400, 0x0, 0x0, 0x0) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:101 +0x6b github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run.func1(0xc00005e120, 0xc00009e3c0, 0xc00005e1e0, 0xc000078040, 0xc00005e180, 0xc00004a480) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:362 +0x1b0 created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:332 +0x2c5 goroutine 17533 [select, 23 minutes]: github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run(0xc00009e1e0, 0x862f20, 0xc0000a0000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:431 +0x433 created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Server).Serve /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:127 +0x28d goroutine 17545 [select, 23 minutes]: github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*serverConn).run(0xc000012190, 0x862f20, 0xc0000a0000) /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:431 +0x433 created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Server).Serve /builddir/build/BUILD/containerd-1.4.1-2.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/server.go:127 +0x28d === END goroutine stack dump ===\" namespace=moby path=/run/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a pid=7023\" ``` </p> </details> - When strace-ing the shim, I saw that it execve-s runc with the following: ``` /usr/sbin/runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a/log.json --log-format json kill cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a 9 ``` to which runc returns \"container not running\", and in turn the shim reports to containerd - \"process already finished: not found\" **runc:** - There are no hanging runc processes which I could observe, nor any journal logs related; not sure how to introspect this layer after the fact. - `runc --root /var/run/docker/runtime-runc/moby events cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a` shows \"container with id ... is not running\" - `runc list` shows the container as stopped - `runc state` shows the following: ``` { \"ociVersion\": \"1.0.2-dev\", \"id\": \"cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\", \"pid\": 0, \"status\": \"stopped\", \"bundle\": \"/run/containerd/io.containerd.runtime.v1.linux/moby/cd7ed93ae2d106564609055e17b24679860bc6cfbfdb5c845f3644815387a37a\", \"rootfs\": \"/var/lib/docker/overlay2/91052c9a423d3bee16e65973b097e7d0eb35baa5200e2c8f05f3484850e4c1c1/merged\", \"created\": \"2021-07-07T00:11:36.60742938Z\", \"owner\": \"\" } ``` **os:** - The container's process is gone, meaning it was actually killed **What you expected to happen**: Pods should terminate once their underlying container had died. **How to reproduce it (as minimally and precisely as possible)**: Not actually sure how to reproduce it consistently - it happens when creating and destroying nodes rapidly, I'd assume. <details> <summary>Environment</summary> <p> ``` - Kubernetes version: 1.19 - Docker version output: Client: Version: 19.03.13-ce API version: 1.40 Go version: go1.13.15 Git commit: 4484c46 Built: Mon Oct 12 18:51:20 2020 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 19.03.13-ce API version: 1.40 (minimum version 1.12) Go version: go1.13.15 Git commit: 4484c46 Built: Mon Oct 12 18:51:50 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.1 GitCommit: c623d1b36f09f8ef6536a057bd658b3aa8632828 runc: Version: 1.0.0-rc93 GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec docker-init: Version: 0.19.0 GitCommit: de40ad0 - Docker info output: Client: Debug Mode: false Server: Containers: 14 Running: 9 Paused: 0 Stopped: 5 Images: 8 Server Version: 19.03.13-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: c623d1b36f09f8ef6536a057bd658b3aa8632828 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec init version: de40ad0 (expected: fec3683) Security Options: seccomp Profile: default Kernel Version: 5.4.117-58.216.amzn2.x86_64 Operating System: Amazon Linux 2 OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.46GiB Name: ip-10-0-73-87.ec2.internal ID: BFFR:6SUN:2BSZ:4MB4:K5NO:OBN2:6VHK:Z2YQ:LS3U:KPEW:5TUV:AEBW Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: true ``` </p> </details> <details><summary><code>runc --version</code></summary><br><pre> $ runc --version runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.15.8 libseccomp: 2.4.1 </pre></details> <details><summary><code>uname -a</code></summary><br><pre> $ uname -a Linux ip-10-0-73-87.ec2.internal 5.4.117-58.216.amzn2.x86_64 #1 SMP Tue May 11 20:50:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux </pre></details>",
          "url": "https://github.com/containerd/containerd/issues/5708",
          "createdAt": "2021-07-09T22:51:31Z",
          "updatedAt": "2026-08-12T20:08:24Z",
          "timestamp": "2026-08-12T20:08:24Z",
          "metrics": {
            "reactions": 5,
            "comments": 18
          },
          "labels": [
            "kind/bug"
          ],
          "author": "dany74q",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:7387645976e1a40d450b",
        "signalId": "github:containerd/containerd:pull_request:13635",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13635",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "oom: avoid per-wakeup allocations in cgroup v2 OOM watcher (#13558)",
          "text": "## What this does Reduces continuous CPU/GC overhead of the per-container cgroup v2 OOM watcher (`internal/oom`), which is the only steadily-active, workload- proportional code path in `containerd-shim-runc-v2` on a static workload. ## Background (#13558) The watcher arms an inotify `IN_MODIFY` watch on `memory.events`. The kernel notifies that file on **every** counter change (`low`/`high`/`max`/`oom`/ `oom_kill`/`oom_group_kill`), not just OOM kills. Containers running near their memory limit churn `high`/`max` constantly (and increasingly as page cache fills toward the limit over the node's uptime), waking the watcher goroutine very frequently. Previously each wakeup allocated a new read buffer **and** a `map[string]uint64`, then opened and fully parsed `memory.events` with `bufio.Scanner`/`strings.Fields` just to read `oom_kill`. Across ~50–60 shims this steady allocation churn is avoidable CPU and GC pressure that accumulates on long-running nodes. ## Change - Allocate the inotify and `memory.events` read buffers once per watcher and reuse them. - Read only the `oom_kill` counter with a small allocation-light parser. - No behavioral change: OOM detection and cgroup-deletion handling are identical; covered by the existing `TestWatcher` plus a new parser unit test.",
          "url": "https://github.com/containerd/containerd/pull/13635",
          "createdAt": "2026-06-20T12:04:35Z",
          "updatedAt": "2026-08-12T20:02:30Z",
          "timestamp": "2026-08-12T20:02:30Z",
          "metrics": {
            "reactions": 0,
            "comments": 5
          },
          "labels": [
            "size/L",
            "cherry-pick/2.3.x"
          ],
          "author": "KeyMoad",
          "state": "open",
          "assignees": [
            "fuweid"
          ],
          "change": "updated"
        }
      },
      {
        "id": "event:9633ead05126034da931",
        "signalId": "github:containerd/containerd:pull_request:13830",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13830",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "ctr: dedupe CRI image aliases in images list by default",
          "text": "## Problem Fixes #11000 (related: #8421, #10090, #10303, #10328). CRI writes three image names into the containerd metadata store for each pull (`repo:tag`, `repo@digest`, and a digest-only config/image ID). That is intentional for CRI's own image store, but `ctr -n k8s.io images ls` lists every name, so operators see three rows for one image. ``` $ ctr -n k8s.io i ls REF ... DIGEST docker.io/library/centos:7 ... sha256:be65... docker.io/library/centos@sha256:be65... ... sha256:be65... sha256:eeb6ee3f44bd... ... sha256:be65... ``` ## Design (display-only, minimal UX) **Prefer filtering at list time over changing CRI create behavior.** | Approach | Pros | Cons | | --- | --- | --- | | **A. Display filter in `ctr images list` (this PR)** | No metadata / CRI semantics change; safe for scripts that still know exact names; matches docker-like \"show tags by default\" | Must use `--all` to see aliases; nerdctl not covered | | B. Stop CRI from creating repo@digest / imageID names | True uniqueness in meta.db | High risk: CRI, GC, pinned images, and other tools may depend on those names; needs broader design | | C. Dedupe only by digest (one row total) | Cleanest table | Hides multiple tags that legitimately share a digest | This PR implements **A**, as sketched in the issue. ### Behavior - **Default:** For each target digest, if any name is a **tagged** ref (`name:tag`), only tagged refs are printed. CRI's `repo@digest` and bare digest ID rows are hidden when a tag exists. - **No tagged name:** All refs for that digest are still shown (digest-only images are not dropped). - **`--all` / `-a`:** Previous behavior — print every reference. - Metadata store, delete, pull, and CRI are **unchanged**. Deleting still requires the exact name (or multiple deletes) as today; fixing delete-by-ID is a separate problem. ### Why not filter only on `io.cri-containerd.image=managed`? A pure \"tagged only\" filter (as in the issue snippet) would hide legitimate digests that were never tagged. Grouping by target digest and preferring tags when present avoids that. ## Code - `cmd/ctr/commands/images/images.go` — `--all` flag + `filterImageListForDisplay` - `cmd/ctr/commands/images/list_filter_test.go` — unit tests for CRI triple, multi-tag, digest-only, ordering ## Test ``` go test ./cmd/ctr/commands/images/ -run TestFilterImageListForDisplay ``` ## Notes for maintainers Happy to adjust defaults if preferred: 1. Default stays full list; opt-in `--dedupe` instead of opt-out `--all` 2. Also collapse multi-tag digests to a single row (not done here) 3. Document-only note in `docs/namespaces.md` without code change Open to discussion if approach B (CRI create) is preferred long-term; this PR is intentionally the small, reversible UX fix.",
          "url": "https://github.com/containerd/containerd/pull/13830",
          "createdAt": "2026-07-22T19:18:33Z",
          "updatedAt": "2026-08-12T19:54:56Z",
          "timestamp": "2026-08-12T19:54:56Z",
          "metrics": {
            "reactions": 0,
            "comments": 1
          },
          "labels": [
            "size/L"
          ],
          "author": "locker95",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:324f510d494156038a7c",
        "signalId": "github:containerd/containerd:pull_request:13953",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13953",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "Forward credentials to cr mirrors",
          "url": "https://github.com/containerd/containerd/pull/13953",
          "createdAt": "2026-08-12T17:34:21Z",
          "updatedAt": "2026-08-12T18:48:31Z",
          "timestamp": "2026-08-12T18:48:31Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/L",
            "do-not-merge/work-in-progress"
          ],
          "author": "MikeZappa87",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:e9baa565552a56bcc7d1",
        "signalId": "github:containerd/containerd:pull_request:13948",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13948",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "[pkg/shim] implement Windows support for the shim server",
          "text": "## What Implements the Windows-specific methods in `pkg/shim` that were previously `ErrNotImplemented` stubs, so a shim built on this package can actually run and serve on Windows (not just Unix). ## Why On Windows the package exposed the shim entry points but left every platform hook unimplemented, so the serve path failed immediately (no ttrpc server, no listener, no logging). This fills those gaps using the Windows named-pipe model. An existing [Windows shim](https://github.com/microsoft/hcsshim/tree/b6b988ac86ef715646ab1b04481ebff88af03cce/cmd/containerd-shim-lcow-v2) uses the same package ([currently forked and modified in project](https://github.com/microsoft/hcsshim/tree/b6b988ac86ef715646ab1b04481ebff88af03cce/internal/shim)). Post this change, the `pkg/shim` can be used by any Windows shim end to end. ## Changes - **ttrpc serving over named pipes** — the shim now creates and serves its API on a Windows named pipe instead of a Unix socket. - **Log streaming over a reconnecting named pipe** — shim logs are served on a pipe that containerd connects to, and transparently re-accepts the reader if it reconnects (e.g. after a containerd restart). Unlike a Unix FIFO, a connection-oriented pipe requires this reconnection handling explicitly. - **Process & signal lifecycle** — signal setup, graceful shutdown on interrupt/terminate, the reap loop, sub-reaper, and stack-dump hooks are wired up with Windows-appropriate (mostly no-op where the OS handles it) behavior. ## Testing Adds unit tests covering the pipe listener, the reconnecting log writer (pre-connection drop, write-through, reconnection, concurrency, close), signal and lifecycle handling, and pipe-readiness. All four platforms (windows/linux/darwin/freebsd) build; the Windows suite passes. ## Notes Windows-only files; no behavior change on Linux/Unix.",
          "url": "https://github.com/containerd/containerd/pull/13948",
          "createdAt": "2026-08-12T10:55:37Z",
          "updatedAt": "2026-08-12T18:02:47Z",
          "timestamp": "2026-08-12T18:02:47Z",
          "metrics": {
            "reactions": 0,
            "comments": 1
          },
          "labels": [
            "size/XL"
          ],
          "author": "rawahars",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:5bacf297e091f5d1c617",
        "signalId": "github:containerd/containerd:pull_request:13946",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13946",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "Prepare release notes for v2.2.7",
          "text": "containerd 2.2.7 Welcome to the v2.2.7 release of containerd! The seventh patch release for containerd 2.2 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * Support non-UTF-8 binary environment variable values in CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13795](https://github.com/containerd/containerd/pull/13795)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13779](https://github.com/containerd/containerd/pull/13779)) #### Runtime * Enable log scrubbing by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * Fix mount manager activation error when activation already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Jordan Liggitt * Akihiro Suda * Amir Alavi * Chris Henzie * Derek McGowan * Maksym Pavlenko * Mike Brown * Phil Estes ### Changes <details><summary>21 commits</summary> <p> * [`43f299db6`](https://github.com/containerd/containerd/commit/43f299db6e087ef066e47c3cf41ee56723983eb3) Prepare release notes for v2.2.7 * cri: disable restore in CreateContainer by default ([#13937](https://github.com/containerd/containerd/pull/13937)) * [`5c3a47036`](https://github.com/containerd/containerd/commit/5c3a47036ec08f8a7517a01a0baf113d6146eca0) cri: disable restore in CreateContainer by default * Use ScrubLogs by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * [`4c93ae6d5`](https://github.com/containerd/containerd/commit/4c93ae6d5e7eaa5957887d469aa610763c2f8b42) ctr: add --scrub-logs flag for Windows * [`a124c7e35`](https://github.com/containerd/containerd/commit/a124c7e354da7252a363f3a3eaafb5a81a6d508e) cri/config: use ScrubLogs by default on Windows * cri: deprecate restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * [`c59d041d8`](https://github.com/containerd/containerd/commit/c59d041d84cb402b991338f2894c0637eb5dc608) cri: deprecate restore in CreateContainer * Handle []byte envvar value for CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * [`6cdc2ddce`](https://github.com/containerd/containerd/commit/6cdc2ddce8aa962135459e02d41ab84772e8e08e) Handle []byte envvar value * [`a60898833`](https://github.com/containerd/containerd/commit/a60898833909a6bb4c38cc1fa47d44434edfe9bf) update to v0.34.x kubernetes dependencies * Fix mount manager activation error when already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) * [`54ecff6ca`](https://github.com/containerd/containerd/commit/54ecff6cabbbbf5b7ffa606cdeeba13131a03968) core/mount: Fix mount manager activation error when already exists * Disable checkpoint restore codepath when CRIU is not installed ([#13795](https://github.com/containerd/containerd/pull/13795)) * [`fd966ba29`](https://github.com/containerd/containerd/commit/fd966ba2904fc92c429fff41b0e4d6cd2384fe7c) github/workflows: install criu in node-e2e * [`7d41b25c9`](https://github.com/containerd/containerd/commit/7d41b25c9884c0f94acdb0f7e78fbab2752fccb6) cri: add enable_criu configuration option * [`48116fa59`](https://github.com/containerd/containerd/commit/48116fa596fd0e926749dcdfd3ef92da00fc6dc8) cri: validate CRIU availability and version early * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13779](https://github.com/containerd/containerd/pull/13779)) * [`6a3d14e8a`](https://github.com/containerd/containerd/commit/6a3d14e8a982e141155f4d39a7b1d34d91fca837) fix(cri): introspect OCI runtime features for non-runc runtimes * ci: bound Go fuzzing by execution count ([#13786](https://github.com/containerd/containerd/pull/13786)) * [`424b714f0`](https://github.com/containerd/containerd/commit/424b714f0c5bc28e840bde698cbbcdc8c4caca0f) ci: bound Go fuzzing by execution count </p> </details> ### Dependency Changes * **go.opentelemetry.io/otel** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/metric** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/trace** v1.38.0 -> v1.41.0 * **google.golang.org/protobuf** v1.36.10 -> f2248ac996af * **k8s.io/api** v0.34.1 -> v0.34.10 * **k8s.io/apimachinery** v0.34.1 -> v0.34.10 * **k8s.io/client-go** v0.34.1 -> v0.34.10 * **k8s.io/cri-api** v0.34.1 -> v0.34.10 Previous release can be found at [v2.2.6](https://github.com/containerd/containerd/releases/tag/v2.2.6) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
          "url": "https://github.com/containerd/containerd/pull/13946",
          "createdAt": "2026-08-12T06:28:52Z",
          "updatedAt": "2026-08-12T17:51:11Z",
          "timestamp": "2026-08-12T17:51:11Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/S"
          ],
          "author": "samuelkarp",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:b903234fecfa2005c6eb",
        "signalId": "github:containerd/containerd:pull_request:13942",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13942",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "Prepare release notes for v2.3.4",
          "text": "containerd 2.3.4 Welcome to the v2.3.4 release of containerd! The fourth patch release for containerd 2.3 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) * Set default runtimeFeatures.UserNamespacesHostNetwork to true in CRI ([#13914](https://github.com/containerd/containerd/pull/13914)) * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * Support non-UTF-8 binary environment variable values in CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13778](https://github.com/containerd/containerd/pull/13778)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13734](https://github.com/containerd/containerd/pull/13734)) * Normalize sandbox image references in CRI to resolve images without domain prefixes ([#13759](https://github.com/containerd/containerd/pull/13759)) #### Node Resource Interface (NRI) * Emit deprecation warnings for plugins using deprecated NRI interfaces ([#13935](https://github.com/containerd/containerd/pull/13935)) #### Runtime * Enable log scrubbing by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * Fix memory leak in OOM watcher map when stopping container monitoring ([#13870](https://github.com/containerd/containerd/pull/13870)) * Avoid orphaning shims on transient errors when loading process IDs ([#13857](https://github.com/containerd/containerd/pull/13857)) * Fix corruption of binary protobuf shim start responses caused by premature whitespace trimming ([#13803](https://github.com/containerd/containerd/pull/13803)) #### Snapshotters * Fix EROFS snapshotter dropping lower layers stacked above merged filesystem metadata ([#13876](https://github.com/containerd/containerd/pull/13876)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Maksym Pavlenko * Mike Brown * Akihiro Suda * Derek McGowan * Jordan Liggitt * Paweł Gronowski * Amir Alavi * Andrew Halaney * Bing Hongtao * Chris Henzie * Harshal Patel * Krisztian Litkey * Phil Estes * Wei Fu * ningmingxiao ### Changes <details><summary>35 commits</summary> <p> * [`c1f5360ef`](https://github.com/containerd/containerd/commit/c1f5360ef7b9afb748d4119ad7be665ba5144480) Prepare release notes for v2.3.4 * cri: disable restore in CreateContainer by default ([#13913](https://github.com/containerd/containerd/pull/13913)) * [`928c03c68`](https://github.com/containerd/containerd/commit/928c03c6898cad95f795300281fdff0dbc55a32f) cri: disable restore in CreateContainer by default * nri,deprecation: record and emit warnings for NRI deprecations. ([#13935](https://github.com/containerd/containerd/pull/13935)) * [`5966e2edb`](https://github.com/containerd/containerd/commit/5966e2edb15bb939aa2bb73649fbad73a2b04094) nri,deprecation: emit warnings for old NRI plugins. * Set the default of runtimeFeatures.UserNamespacesHostNetwork to true ([#13914](https://github.com/containerd/containerd/pull/13914)) * [`ab52c58f1`](https://github.com/containerd/containerd/commit/ab52c58f16ac50bb6724354bf36cbb4da480c2a2) Set the default of runtimeFeatures.UserNamespacesHostNetwork to true * Use ScrubLogs by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * [`0c255158a`](https://github.com/containerd/containerd/commit/0c255158aa1ecab40aafc42269cfc018a46eb627) ctr: add --scrub-logs flag for Windows * [`1c2b13dc9`](https://github.com/containerd/containerd/commit/1c2b13dc975468197e57321e25fd09163f78a75a) cri/config: use ScrubLogs by default on Windows * snapshots/erofs: keep lowers stacked above a merged fsmeta ([#13876](https://github.com/containerd/containerd/pull/13876)) * [`0c511e068`](https://github.com/containerd/containerd/commit/0c511e068b7d6dd815418185a235b3ba3347c65a) snapshots/erofs: keep lowers stacked above a merged fsmeta * cri: deprecate restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * [`dc98141df`](https://github.com/containerd/containerd/commit/dc98141df70d3d72d46125c6f30de0d2de648086) cri: deprecate restore in CreateContainer * internal/oom: Fix memory leak by removing watcher from map on Stop ([#13870](https://github.com/containerd/containerd/pull/13870)) * [`537d82d54`](https://github.com/containerd/containerd/commit/537d82d545759d4a072d564d4937707af2650163) internal/oom: Fix memory leak by removing watcher from map on Stop * shim_load: Consider shim leaked only if we can't find pids ([#13857](https://github.com/containerd/containerd/pull/13857)) * [`decf97a9c`](https://github.com/containerd/containerd/commit/decf97a9c7c26f92391d2c862d35832e0a291e80) shim_load: Consider shim leaked only if we can't find pids * core/runtime/v2: Drop checkpointctl module dependency ([#13840](https://github.com/containerd/containerd/pull/13840)) * [`796f07dc8`](https://github.com/containerd/containerd/commit/796f07dc853d9ae8a3a1278fe47436755a4bc3b7) core/runtime/v2: Drop checkpointctl module dependency * Handle []byte envvar value for CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * [`751fddddb`](https://github.com/containerd/containerd/commit/751fddddbb630631e12fd63e5dac4a9530188822) Handle []byte envvar value * [`0bce9060e`](https://github.com/containerd/containerd/commit/0bce9060ef3401e690b541dd6d2008c2abc50b0d) update to v0.36.x kubernetes dependencies * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13778](https://github.com/containerd/containerd/pull/13778)) * [`61a8f6f45`](https://github.com/containerd/containerd/commit/61a8f6f45e34660d787988e8a7e7a218f556369c) fix(cri): introspect OCI runtime features for non-runc runtimes * core/runtime/v2: Preserve protobuf shim response bytes ([#13803](https://github.com/containerd/containerd/pull/13803)) * [`1d28017be`](https://github.com/containerd/containerd/commit/1d28017be295005bfa3aee1d47afe86341d391b6) core/runtime/v2: Preserve protobuf shim response bytes * Disable checkpoint restore codepath when CRIU is not installed ([#13734](https://github.com/containerd/containerd/pull/13734)) * [`374091d67`](https://github.com/containerd/containerd/commit/374091d67c1966969b772594ba96a74f4357bc7a) github/workflows: install criu in node-e2e * [`db03e3968`](https://github.com/containerd/containerd/commit/db03e39685448a1d87025a2ca3a1e4e42713102d) cri: add enable_criu configuration option * [`dacd4c7d0`](https://github.com/containerd/containerd/commit/dacd4c7d00f4a5fd71197e67afdb6355e8536e4f) cri: validate CRIU availability and version early * ci: bound Go fuzzing by execution count ([#13785](https://github.com/containerd/containerd/pull/13785)) * [`890a9c86c`](https://github.com/containerd/containerd/commit/890a9c86cd9918e53c1ef1b57c6539a943196cd5) ci: bound Go fuzzing by execution count * cri: auto-add prefix for pause image ([#13759](https://github.com/containerd/containerd/pull/13759)) * [`0b2f1d078`](https://github.com/containerd/containerd/commit/0b2f1d078124b52f4c8030dc686f2d65c1dfc17c) cri: auto-add prefix for pause image </p> </details> ### Dependency Changes * **k8s.io/api** v0.36.0 -> v0.36.3 * **k8s.io/apimachinery** v0.36.0 -> v0.36.3 * **k8s.io/client-go** v0.36.0 -> v0.36.3 * **k8s.io/component-base** v0.36.0 -> v0.36.3 * **k8s.io/cri-api** v0.36.0 -> v0.36.3 * **k8s.io/cri-client** v0.36.0 -> v0.36.3 * **k8s.io/cri-streaming** v0.36.0 -> v0.36.3 * **sigs.k8s.io/structured-merge-diff/v6** v6.3.2 -> v6.3.3 Previous release can be found at [v2.3.3](https://github.com/containerd/containerd/releases/tag/v2.3.3) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
          "url": "https://github.com/containerd/containerd/pull/13942",
          "createdAt": "2026-08-11T23:22:05Z",
          "updatedAt": "2026-08-12T17:47:32Z",
          "timestamp": "2026-08-12T17:47:32Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/S"
          ],
          "author": "samuelkarp",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:8588e8500ab9e35d9ad3",
        "signalId": "github:containerd/containerd:pull_request:13945",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13945",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "[release/2.3] snapshots/erofs: protect snapshot staging from cleanup",
          "text": "Cherry-pick of https://github.com/containerd/containerd/pull/13932",
          "url": "https://github.com/containerd/containerd/pull/13945",
          "createdAt": "2026-08-12T02:40:34Z",
          "updatedAt": "2026-08-12T17:42:38Z",
          "timestamp": "2026-08-12T17:42:38Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M"
          ],
          "author": "hsiangkao",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:fa0d6c8e06223caeb914",
        "signalId": "github:containerd/containerd:pull_request:13915",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13915",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "remotes/docker: retry blob fetch on connection reset by peer",
          "text": "## What this does `httpReadSeeker` already resumes a blob fetch when the connection closes cleanly before the full body arrives (`io.ErrUnexpectedEOF`): it reopens the body with a ranged request at the current offset, bounded by `maxRetry` consecutive reopens that make no progress. A mid-body TCP RST, however, surfaces as a connection-reset error, misses that case, and fails the whole fetch — discarding everything already transferred. This PR treats connection resets like unexpected EOFs: retry by reopening at the current offset, under the exact same `maxRetry`/no-progress bound. The reset class is matched per-platform alongside the existing `isConnError` helpers (`ECONNRESET`, plus `WSAECONNRESET` on Windows, where `syscall.ECONNRESET` is a synthetic constant a real reset never matches), unwrapping with `errors.Is`. The unexpected-EOF match also moves from bare equality to `errors.Is`, so a wrapped `io.ErrUnexpectedEOF` now retries as well. ## Why On congested or middlebox-mediated paths, long blob transfers can stall, and far-end idle timers kill stalled flows with a RST after ~30s. In one observed case, a pull repeatedly transferred about 63% of a ~190MB layer at steady throughput, took a mid-body reset, and restarted from byte zero each time, never completing. With reconnect-at-offset covering the reset class, such a pull completes within the existing retry budget. Registries and range-capable blob backends already serve the reopen path today; only the error classification is blind to resets. Symptom report: #11365. Adjacent precedent: google/go-containerregistry#1415, which retries resets during blob reads. ## Caveats and scope - **Deliberate refusal:** retrying on RST could mask a far end intentionally killing the transfer. The existing consecutive-zero-progress bound caps that at `maxRetry` reopens — identical to today's handling of the unexpected-EOF class. - **Deliberately narrow:** `ETIMEDOUT` and generic temporary network errors are *not* covered. Question for maintainers: would you want those classes in a follow-up, or is reset-only the right scope? - **HTTP/2:** stream-level resets (e.g. `RST_STREAM`) surface as different error types and are out of scope here. - **Interaction with #13571:** that PR (activity-based timeout) touches the same area; this change does not restructure the reader. Happy to rebase/adjust if #13571 lands first. ## How it was tested New `httpreadseeker_test.go`, table-driven over both retryable classes (unexpected EOF, and a reset built as `*net.OpError` → `*os.SyscallError` → `ECONNRESET`, as a TCP read surfaces it): - reopen is re-invoked at the **advanced** offset and the transfer completes across mid-body failures when progress occurs; - `maxRetry` consecutive zero-progress failures abort with the original error (exactly `1 + maxRetry` opens); - unit coverage of `isRetryableReadError`, including wrapped forms and negative cases (`io.EOF`, `ECONNREFUSED`). - a Windows-only test (`httpreadseeker_windows_test.go`) covering the `WSAECONNRESET` arm, bare and wrapped in `*net.OpError`/`*os.SyscallError`; Also run locally: `go build`, `go vet`, full `core/remotes/docker/...` tests, `golangci-lint` with the repo config (0 issues), and cross-compiles for `GOOS=windows` (build + vet), `darwin`, and `freebsd`.",
          "url": "https://github.com/containerd/containerd/pull/13915",
          "createdAt": "2026-08-07T08:53:41Z",
          "updatedAt": "2026-08-12T17:15:51Z",
          "timestamp": "2026-08-12T17:15:51Z",
          "metrics": {
            "reactions": 0,
            "comments": 2
          },
          "labels": [
            "size/L"
          ],
          "author": "mabdelmalek-ant",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:cea122a9e9790a241965",
        "signalId": "github:containerd/containerd:pull_request:13951",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13951",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "treat missing runtime state as container already dead during kill",
          "text": "Pods get stuck in terminating forever when the DeleteContainer call times out, but does partially delete the state. Example: ``` time=\"2026-08-04T16:21:31.095039308Z\" level=info msg=\"StopPodSandbox for \\\"f37ee9eca302510909867fae89544fae2b480852fb1ccca45e46833aca5ab229\\\"\" time=\"2026-08-04T16:21:45.129960940Z\" level=error msg=\"failed to delete task\" error=\"context deadline exceeded\" id=f37ee9eca302510909867fae89544fae2b480852fb1ccca45e46833aca5ab229 ``` or ``` time=\"2026-08-04T16:50:41.170399937Z\" level=info msg=\"StopContainer for \\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" with timeout 240 (s)\" time=\"2026-08-04T16:50:59.314985313Z\" level=error msg=\"failed to handle container TaskExit event container_id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" pid:206839 exit_status:143 exited_at:{seconds:1785862249 nanos:295565617}\" error=\"failed to stop container: failed to delete task: context deadline exceeded\" ``` both end up stuck with: ``` Aug 04 19:57:05 ip-172-18-72-147 containerd[28731]: time=\"2026-08-04T19:57:05.033712712Z\" level=info msg=\"TaskExit event container_id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" pid:206839 exit_status:143 exited_at:{seconds:1785862249 nanos:295565617}\" Aug 04 19:57:05 ip-172-18-72-147 containerd[28731]: time=\"2026-08-04T19:57:05.851121769Z\" level=error msg=\"Failed to handle backOff event container_id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" id:\\\"8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\\\" pid:206839 exit_status:143 exited_at:{seconds:1785862249 nanos:295565617} for 8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a\" error=\"failed to handle container TaskExit event: failed to stop container:.....: error opening file `/run/containerd/runc/k8s.io/8882359507b80e801be6b7dd5b21db3c65bd72fc028436a56ceffa1dfeec380a/status`: No such file or directory\\n\" ``` The only way to recover is to force delete the pods. This seems similar to https://github.com/containerd/containerd/issues/7496 which was solved by https://github.com/containerd/containerd/pull/8954, but that only includes the case where we get `errdefs.ErrNotFound` and not `No such file or directory` This adds that `No such file or directory` case alongside the `errdefs.ErrNotFound` catch",
          "url": "https://github.com/containerd/containerd/pull/13951",
          "createdAt": "2026-08-12T16:59:27Z",
          "updatedAt": "2026-08-12T17:12:13Z",
          "timestamp": "2026-08-12T17:12:13Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M"
          ],
          "author": "olyazavr",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:d46ea4ab57e353fdc4b1",
        "signalId": "github:containerd/containerd:pull_request:13359",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13359",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "Allow hosts file configuration for proxies per registry",
          "text": "Fixes #8377 The setup: I have a proxy running that has DNS mapped `local.registry` to `127.0.0.1`, because I have a registry (with a self signed certificate) running locally as well with an image tagged `local.registry/localpython` in it. (Setup directions are [here](https://gist.github.com/lauralorenz/085c7368886ea4f8ab8b9e2419db8841).) ``` docker run -d --name local-squid-proxy --net=host --add-host local.registry:127.0.0.1 ubuntu/squid:latest docker run -d --name registry -p 443:443 -v /tmp/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:3 ``` ``` lauralorenz@lauralorenz:bin$ curl -k https://local.registry/v2/localpython/tags/list {\"name\":\"localpython\",\"tags\":[\"latest\"]} ``` In both of the following test scenarios (before the feature and after) I have configured the following hosts.toml: ``` lauralorenz@lauralorenz:containerd$ cat /etc/containerd/certs.d/local.registry/hosts.toml [host.\"https://local.registry\"] capabilities = [\"resolve\",\"pull\"] proxy=\"http://localhost:3128\" skip_verify=true ``` Then I ran against a containerd and ctr built on main (1b71eeeae7d537ec94bcd467089aea136256b83e), and then again on a contained and ctr built from this commit. # Before this PR / on main Does not care about the proxy setting in hosts.toml and simply can't find the host. ``` lauralorenz@lauralorenz:containerd$ sudo /usr/local/google/home/lauralorenz/go/src/containerd/containerd/bin/ctr images pull --hosts-dir=/etc/containerd/certs.d local.registry/localpython:latest ctr: failed to resolve image: failed to do request: Head \"https://local.registry/v2/localpython/manifests/latest\": dial tcp: lookup local.registry on 127.0.0.1:53: no such host ``` # With this PR Uses proxy setting in hosts.toml to get to the forward proxy at `localhost`, which itself forwards the original hostname request `local.registry` to the registry itself. ``` lauralorenz@lauralorenz:containerd$ sudo /usr/local/google/home/lauralorenz/go/src/containerd/containerd/bin/ctr images pull --hosts-dir=/etc/containerd/certs.d local.registry/localpython:latest local.registry/localpython:latest saved └──manifest (a3f5fd4ce3ad) complete |++++++++++++++++++++++++++++++++++++++| ├──config (7c8620a05d35) complete |++++++++++++++++++++++++++++++++++++++| ├──layer (fd8e41ac7277) extracted |++++++++++++++++++++++++++++++++++++++| ├──layer (ea923c2ed79e) extracted |++++++++++++++++++++++++++++++++++++++| ├──layer (5c671a5c7ab3) extracted |++++++++++++++++++++++++++++++++++++++| └──layer (6a0ac1617861) extracted |++++++++++++++++++++++++++++++++++++++| application/vnd.docker.distribution.manifest.v2+json sha256:a3f5fd4ce3adead6336ca2f69f4589b41c7b24317e451e4048116cbd6d761be4 Completed pull from OCI Registry (local.registry/localpython:latest) elapsed: 2.2 s total: 19.4 M (8.9 MiB/s) ``` ``` lauralorenz@lauralorenz:containerd$ sudo /usr/local/google/home/lauralorenz/go/src/containerd/containerd/bin/ctr content list DIGEST SIZE AGE LABELS sha256:5c671a5c7ab3e0fe7526f4ff0a233b54727a1f407035f414a73138d2b0886733 455.7kB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/uncompressed=sha256:611cae45bc2cb7ff6a78488ab602a59918fbc0431a0f12281dacf10cbb7ba94f sha256:6a0ac1617861a677b045b7ff88545213ec31c0ff08763195a70a4a5adda577bb 3.864MB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/uncompressed=sha256:29df493baa13de438d6d2ece3a8333032e0b7b9b9d8cce4ee82194da255f61e1 sha256:7c8620a05d355e188e9c92949198770fd43d150d051f8c3223f506f215a9f12d 5.276kB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/gc.ref.snapshot.overlayfs=sha256:b4f837ae9f4abc0db962ac37b08df79e45a60a17d7e88ffe7aafb537d1e58353 sha256:a3f5fd4ce3adead6336ca2f69f4589b41c7b24317e451e4048116cbd6d761be4 1.157kB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/gc.ref.content.config=sha256:7c8620a05d355e188e9c92949198770fd43d150d051f8c3223f506f215a9f12d,containerd.io/gc.ref.content.l.0=sha256:6a0ac1617861a677b045b7ff88545213ec31c0ff08763195a70a4a5adda577bb,containerd.io/gc.ref.content.l.1=sha256:5c671a5c7ab3e0fe7526f4ff0a233b54727a1f407035f414a73138d2b0886733,containerd.io/gc.ref.content.l.2=sha256:ea923c2ed79ed2d7163aacc54686501cb3339215671be89b50b62814aeffb95c,containerd.io/gc.ref.content.l.3=sha256:fd8e41ac72776720a5ce79d8a0c66e75ae8d3d836cb15cc6ae0682b0b7999a11 sha256:ea923c2ed79ed2d7163aacc54686501cb3339215671be89b50b62814aeffb95c 16.02MB About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/uncompressed=sha256:b02b147e872fb22e2550aaaeae713706d551a5321619cf742e89c4144b54c7d4 sha256:fd8e41ac72776720a5ce79d8a0c66e75ae8d3d836cb15cc6ae0682b0b7999a11 248B About a minute containerd.io/distribution.source.local.registry=localpython,containerd.io/uncompressed=sha256:6239b5ae5fda090b9f21fcd4bc2b888837f17814c06c0642921cdb35c6a847e5 ```",
          "url": "https://github.com/containerd/containerd/pull/13359",
          "createdAt": "2026-05-07T21:34:54Z",
          "updatedAt": "2026-08-12T17:11:42Z",
          "timestamp": "2026-08-12T17:11:42Z",
          "metrics": {
            "reactions": 0,
            "comments": 8
          },
          "labels": [
            "ok-to-test",
            "size/L"
          ],
          "author": "lauralorenz",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:98937a36f589ccc57290",
        "signalId": "github:containerd/containerd:pull_request:13928",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13928",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "tracing: align span attribute keys with OTel semantic conventions",
          "text": "Closes #12191 Rename trace span attribute keys to follow the OpenTelemetry semantic conventions. Keys that have no semconv equivalent (snapshotter, pull ref, unpack) are moved under the `containerd.` namespace instead. | Legacy key | New key | | --- | --- | | `task.container.id` | `container.id` | | `task.runtime.name` | `container.runtime.name` | | `task.pid` | `process.pid` | | `task.process.id` | `process.pid` | | `image.id` | `container.image.id` | | `image.ref` | `containerd.image.ref` | | `snapshotter.name` | `containerd.snapshotter.name` | | `container.snapshotter.name` | `containerd.snapshotter.name` | | `unpack` | `containerd.pull.unpack` | To avoid breaking existing consumers, both the new and legacy keys are emitted for now (same approach as `HTTPStatusCodeAttributes`). The legacy keys are documented as deprecated in `docs/tracing.md` and can be dropped in a future release. `container.image.ref` is kept as is, since an image ref is not equivalent to `container.image.id`, `container.image.name` (see discussion in #12191). Attributes not mentioned in the issue are left untouched.",
          "url": "https://github.com/containerd/containerd/pull/13928",
          "createdAt": "2026-08-09T13:22:35Z",
          "updatedAt": "2026-08-12T16:12:11Z",
          "timestamp": "2026-08-12T16:12:11Z",
          "metrics": {
            "reactions": 0,
            "comments": 5
          },
          "labels": [
            "size/L"
          ],
          "author": "chansuke",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:35d4021e0cc6a3881898",
        "signalId": "github:containerd/containerd:pull_request:13943",
        "event": "changed",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13943",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "Add log message when a shim is loaded into the shim manager",
          "text": "This change proposes a happy path log for shims loaded into the shim manager for visibility in runtime logs of existing workloads and their version. This is to increase visibility for workloads picked up when the server is bounced or upgraded.",
          "url": "https://github.com/containerd/containerd/pull/13943",
          "createdAt": "2026-08-12T01:38:30Z",
          "updatedAt": "2026-08-12T13:49:16Z",
          "timestamp": "2026-08-12T13:49:16Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/S"
          ],
          "author": "austinvazquez",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:ca7e0b23b95d32702b9e",
        "signalId": "github:containerd/containerd:release:369432727",
        "event": "discovered",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:release:369432727",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "release",
          "title": "containerd 2.3.4",
          "text": "Welcome to the v2.3.4 release of containerd! The fourth patch release for containerd 2.3 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) * Set default runtimeFeatures.UserNamespacesHostNetwork to true in CRI ([#13914](https://github.com/containerd/containerd/pull/13914)) * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * Support non-UTF-8 binary environment variable values in CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13778](https://github.com/containerd/containerd/pull/13778)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13734](https://github.com/containerd/containerd/pull/13734)) * Normalize sandbox image references in CRI to resolve images without domain prefixes ([#13759](https://github.com/containerd/containerd/pull/13759)) #### Node Resource Interface (NRI) * Emit deprecation warnings for plugins using deprecated NRI interfaces ([#13935](https://github.com/containerd/containerd/pull/13935)) #### Runtime * Enable log scrubbing by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * Fix memory leak in OOM watcher map when stopping container monitoring ([#13870](https://github.com/containerd/containerd/pull/13870)) * Avoid orphaning shims on transient errors when loading process IDs ([#13857](https://github.com/containerd/containerd/pull/13857)) * Fix corruption of binary protobuf shim start responses caused by premature whitespace trimming ([#13803](https://github.com/containerd/containerd/pull/13803)) #### Snapshotters * Fix EROFS snapshotter dropping lower layers stacked above merged filesystem metadata ([#13876](https://github.com/containerd/containerd/pull/13876)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13913](https://github.com/containerd/containerd/pull/13913)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Maksym Pavlenko * Mike Brown * Akihiro Suda * Derek McGowan * Jordan Liggitt * Paweł Gronowski * Amir Alavi * Andrew Halaney * Bing Hongtao * Chris Henzie * Harshal Patel * Krisztian Litkey * Phil Estes * Wei Fu * ningmingxiao ### Changes <details><summary>35 commits</summary> <p> * [`c1f5360ef`](https://github.com/containerd/containerd/commit/c1f5360ef7b9afb748d4119ad7be665ba5144480) Prepare release notes for v2.3.4 * cri: disable restore in CreateContainer by default ([#13913](https://github.com/containerd/containerd/pull/13913)) * [`928c03c68`](https://github.com/containerd/containerd/commit/928c03c6898cad95f795300281fdff0dbc55a32f) cri: disable restore in CreateContainer by default * nri,deprecation: record and emit warnings for NRI deprecations. ([#13935](https://github.com/containerd/containerd/pull/13935)) * [`5966e2edb`](https://github.com/containerd/containerd/commit/5966e2edb15bb939aa2bb73649fbad73a2b04094) nri,deprecation: emit warnings for old NRI plugins. * Set the default of runtimeFeatures.UserNamespacesHostNetwork to true ([#13914](https://github.com/containerd/containerd/pull/13914)) * [`ab52c58f1`](https://github.com/containerd/containerd/commit/ab52c58f16ac50bb6724354bf36cbb4da480c2a2) Set the default of runtimeFeatures.UserNamespacesHostNetwork to true * Use ScrubLogs by default on Windows ([#13904](https://github.com/containerd/containerd/pull/13904)) * [`0c255158a`](https://github.com/containerd/containerd/commit/0c255158aa1ecab40aafc42269cfc018a46eb627) ctr: add --scrub-logs flag for Windows * [`1c2b13dc9`](https://github.com/containerd/containerd/commit/1c2b13dc975468197e57321e25fd09163f78a75a) cri/config: use ScrubLogs by default on Windows * snapshots/erofs: keep lowers stacked above a merged fsmeta ([#13876](https://github.com/containerd/containerd/pull/13876)) * [`0c511e068`](https://github.com/containerd/containerd/commit/0c511e068b7d6dd815418185a235b3ba3347c65a) snapshots/erofs: keep lowers stacked above a merged fsmeta * cri: deprecate restore in CreateContainer ([#13868](https://github.com/containerd/containerd/pull/13868)) * [`dc98141df`](https://github.com/containerd/containerd/commit/dc98141df70d3d72d46125c6f30de0d2de648086) cri: deprecate restore in CreateContainer * internal/oom: Fix memory leak by removing watcher from map on Stop ([#13870](https://github.com/containerd/containerd/pull/13870)) * [`537d82d54`](https://github.com/containerd/containerd/commit/537d82d545759d4a072d564d4937707af2650163) internal/oom: Fix memory leak by removing watcher from map on Stop * shim_load: Consider shim leaked only if we can't find pids ([#13857](https://github.com/containerd/containerd/pull/13857)) * [`decf97a9c`](https://github.com/containerd/containerd/commit/decf97a9c7c26f92391d2c862d35832e0a291e80) shim_load: Consider shim leaked only if we can't find pids * core/runtime/v2: Drop checkpointctl module dependency ([#13840](https://github.com/containerd/containerd/pull/13840)) * [`796f07dc8`](https://github.com/containerd/containerd/commit/796f07dc853d9ae8a3a1278fe47436755a4bc3b7) core/runtime/v2: Drop checkpointctl module dependency * Handle []byte envvar value for CRI ([#13454](https://github.com/containerd/containerd/pull/13454)) * [`751fddddb`](https://github.com/containerd/containerd/commit/751fddddbb630631e12fd63e5dac4a9530188822) Handle []byte envvar value * [`0bce9060e`](https://github.com/containerd/containerd/commit/0bce9060ef3401e690b541dd6d2008c2abc50b0d) update to v0.36.x kubernetes dependencies * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13778](https://github.com/containerd/containerd/pull/13778)) * [`61a8f6f45`](https://github.com/containerd/containerd/commit/61a8f6f45e34660d787988e8a7e7a218f556369c) fix(cri): introspect OCI runtime features for non-runc runtimes * core/runtime/v2: Preserve protobuf shim response bytes ([#13803](https://github.com/containerd/containerd/pull/13803)) * [`1d28017be`](https://github.com/containerd/containerd/commit/1d28017be295005bfa3aee1d47afe86341d391b6) core/runtime/v2: Preserve protobuf shim response bytes * Disable checkpoint restore codepath when CRIU is not installed ([#13734](https://github.com/containerd/containerd/pull/13734)) * [`374091d67`](https://github.com/containerd/containerd/commit/374091d67c1966969b772594ba96a74f4357bc7a) github/workflows: install criu in node-e2e * [`db03e3968`](https://github.com/containerd/containerd/commit/db03e39685448a1d87025a2ca3a1e4e42713102d) cri: add enable_criu configuration option * [`dacd4c7d0`](https://github.com/containerd/containerd/commit/dacd4c7d00f4a5fd71197e67afdb6355e8536e4f) cri: validate CRIU availability and version early * ci: bound Go fuzzing by execution count ([#13785](https://github.com/containerd/containerd/pull/13785)) * [`890a9c86c`](https://github.com/containerd/containerd/commit/890a9c86cd9918e53c1ef1b57c6539a943196cd5) ci: bound Go fuzzing by execution count * cri: auto-add prefix for pause image ([#13759](https://github.com/containerd/containerd/pull/13759)) * [`0b2f1d078`](https://github.com/containerd/containerd/commit/0b2f1d078124b52f4c8030dc686f2d65c1dfc17c) cri: auto-add prefix for pause image </p> </details> ### Dependency Changes * **k8s.io/api** v0.36.0 -> v0.36.3 * **k8s.io/apimachinery** v0.36.0 -> v0.36.3 * **k8s.io/client-go** v0.36.0 -> v0.36.3 * **k8s.io/component-base** v0.36.0 -> v0.36.3 * **k8s.io/cri-api** v0.36.0 -> v0.36.3 * **k8s.io/cri-client** v0.36.0 -> v0.36.3 * **k8s.io/cri-streaming** v0.36.0 -> v0.36.3 * **sigs.k8s.io/structured-merge-diff/v6** v6.3.2 -> v6.3.3 Previous release can be found at [v2.3.3](https://github.com/containerd/containerd/releases/tag/v2.3.3) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
          "url": "https://github.com/containerd/containerd/releases/tag/v2.3.4",
          "createdAt": "2026-08-12T17:59:47Z",
          "updatedAt": "2026-08-12T17:59:47Z",
          "timestamp": "2026-08-12T17:59:47Z",
          "labels": [],
          "author": "github-actions[bot]",
          "assignees": [],
          "change": "new"
        }
      },
      {
        "id": "event:732307b718e3d3f91958",
        "signalId": "github:containerd/containerd:release:369434467",
        "event": "discovered",
        "observedAt": "2026-08-13T13:48:00.446149Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:release:369434467",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "release",
          "title": "containerd 2.2.7",
          "text": "Welcome to the v2.2.7 release of containerd! The seventh patch release for containerd 2.2 contains various fixes and updates. ### Highlights #### Container Runtime Interface (CRI) * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * Support non-UTF-8 binary environment variable values in CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * Disable checkpoint restore codepaths when CRIU is not installed and add enable_criu configuration option ([#13795](https://github.com/containerd/containerd/pull/13795)) * Enable OCI runtime feature introspection for non-runc runtimes in CRI ([#13779](https://github.com/containerd/containerd/pull/13779)) #### Runtime * Enable log scrubbing by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * Fix mount manager activation error when activation already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) #### Breaking * Disable checkpoint restore in CreateContainer by default, requiring the enable_experimental_restore_via_create configuration option to enable ([#13937](https://github.com/containerd/containerd/pull/13937)) #### Deprecations * Deprecate checkpoint restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) Please try out the release binaries and report any issues at https://github.com/containerd/containerd/issues. ### Contributors * Samuel Karp * Jordan Liggitt * Akihiro Suda * Amir Alavi * Chris Henzie * Derek McGowan * Maksym Pavlenko * Mike Brown * Phil Estes ### Changes <details><summary>21 commits</summary> <p> * [`43f299db6`](https://github.com/containerd/containerd/commit/43f299db6e087ef066e47c3cf41ee56723983eb3) Prepare release notes for v2.2.7 * cri: disable restore in CreateContainer by default ([#13937](https://github.com/containerd/containerd/pull/13937)) * [`5c3a47036`](https://github.com/containerd/containerd/commit/5c3a47036ec08f8a7517a01a0baf113d6146eca0) cri: disable restore in CreateContainer by default * Use ScrubLogs by default on Windows ([#13882](https://github.com/containerd/containerd/pull/13882)) * [`4c93ae6d5`](https://github.com/containerd/containerd/commit/4c93ae6d5e7eaa5957887d469aa610763c2f8b42) ctr: add --scrub-logs flag for Windows * [`a124c7e35`](https://github.com/containerd/containerd/commit/a124c7e354da7252a363f3a3eaafb5a81a6d508e) cri/config: use ScrubLogs by default on Windows * cri: deprecate restore in CreateContainer ([#13872](https://github.com/containerd/containerd/pull/13872)) * [`c59d041d8`](https://github.com/containerd/containerd/commit/c59d041d84cb402b991338f2894c0637eb5dc608) cri: deprecate restore in CreateContainer * Handle []byte envvar value for CRI ([#13455](https://github.com/containerd/containerd/pull/13455)) * [`6cdc2ddce`](https://github.com/containerd/containerd/commit/6cdc2ddce8aa962135459e02d41ab84772e8e08e) Handle []byte envvar value * [`a60898833`](https://github.com/containerd/containerd/commit/a60898833909a6bb4c38cc1fa47d44434edfe9bf) update to v0.34.x kubernetes dependencies * Fix mount manager activation error when already exists ([#13823](https://github.com/containerd/containerd/pull/13823)) * [`54ecff6ca`](https://github.com/containerd/containerd/commit/54ecff6cabbbbf5b7ffa606cdeeba13131a03968) core/mount: Fix mount manager activation error when already exists * Disable checkpoint restore codepath when CRIU is not installed ([#13795](https://github.com/containerd/containerd/pull/13795)) * [`fd966ba29`](https://github.com/containerd/containerd/commit/fd966ba2904fc92c429fff41b0e4d6cd2384fe7c) github/workflows: install criu in node-e2e * [`7d41b25c9`](https://github.com/containerd/containerd/commit/7d41b25c9884c0f94acdb0f7e78fbab2752fccb6) cri: add enable_criu configuration option * [`48116fa59`](https://github.com/containerd/containerd/commit/48116fa596fd0e926749dcdfd3ef92da00fc6dc8) cri: validate CRIU availability and version early * fix(cri): introspect OCI runtime features for non-runc runtimes ([#13779](https://github.com/containerd/containerd/pull/13779)) * [`6a3d14e8a`](https://github.com/containerd/containerd/commit/6a3d14e8a982e141155f4d39a7b1d34d91fca837) fix(cri): introspect OCI runtime features for non-runc runtimes * ci: bound Go fuzzing by execution count ([#13786](https://github.com/containerd/containerd/pull/13786)) * [`424b714f0`](https://github.com/containerd/containerd/commit/424b714f0c5bc28e840bde698cbbcdc8c4caca0f) ci: bound Go fuzzing by execution count </p> </details> ### Dependency Changes * **go.opentelemetry.io/otel** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/metric** v1.38.0 -> v1.41.0 * **go.opentelemetry.io/otel/trace** v1.38.0 -> v1.41.0 * **google.golang.org/protobuf** v1.36.10 -> f2248ac996af * **k8s.io/api** v0.34.1 -> v0.34.10 * **k8s.io/apimachinery** v0.34.1 -> v0.34.10 * **k8s.io/client-go** v0.34.1 -> v0.34.10 * **k8s.io/cri-api** v0.34.1 -> v0.34.10 Previous release can be found at [v2.2.6](https://github.com/containerd/containerd/releases/tag/v2.2.6) ### Which file should I download? * `containerd-<VERSION>-<OS>-<ARCH>.tar.gz`: ✅Recommended. Dynamically linked with glibc 2.35 (Ubuntu 22.04). * `containerd-static-<VERSION>-<OS>-<ARCH>.tar.gz`: Statically linked. Expected to be used on Linux distributions that do not use glibc >= 2.35. Not position-independent. In addition to containerd, typically you will have to install [runc](https://github.com/opencontainers/runc/releases) and [CNI plugins](https://github.com/containernetworking/plugins/releases) from their official sites too. See also the [Getting Started](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) documentation.",
          "url": "https://github.com/containerd/containerd/releases/tag/v2.2.7",
          "createdAt": "2026-08-12T18:02:25Z",
          "updatedAt": "2026-08-12T18:02:25Z",
          "timestamp": "2026-08-12T18:02:25Z",
          "labels": [],
          "author": "github-actions[bot]",
          "assignees": [],
          "change": "new"
        }
      },
      {
        "id": "event:406e2ffc6975a8cc24cc",
        "signalId": "github:containerd/containerd:issue:12280",
        "event": "discovered",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:12280",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "Support container checkpoint/restore for gVisor runtime",
          "text": "### What is the problem you're trying to solve Hey containerd-maintainers team! I am trying to enable container checkpoint/restore support using the gVisor (runsc) runtime. Currently, the CheckpointContainer method is not implemented in both containerd’s CRI service and the containerd-runsc-shim I am working on implementing the missing functionality in containerd-runsc-shim (https://github.com/google/gvisor/issues/11810), but without CRI support in containerd, checkpoint cannot be enabled. Probably the same relates to restore as well. ### Describe the solution you'd like Implement the CheckpointContainer method in the CRI service: https://github.com/containerd/containerd/blob/61ab051136c53fe2b9dc0681c326dc86897a59de/internal/cri/server/container_checkpoint.go#L46 I would appreciate guidance on: - Any known challenges with adding this method. - Whether there are related discussions or design considerations that I should be aware of. - If there are existing efforts or plans for CRI checkpoint/restore support. ### Additional context _No response_",
          "url": "https://github.com/containerd/containerd/issues/12280",
          "createdAt": "2025-09-08T09:50:28Z",
          "updatedAt": "2026-08-13T16:08:48Z",
          "timestamp": "2026-08-13T16:08:48Z",
          "metrics": {
            "reactions": 1,
            "comments": 7
          },
          "labels": [
            "kind/feature",
            "area/cri",
            "Stale"
          ],
          "author": "samorojy",
          "state": "open",
          "assignees": [],
          "change": "new"
        }
      },
      {
        "id": "event:e8adf0c668065777ad07",
        "signalId": "github:containerd/containerd:issue:13129",
        "event": "discovered",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:13129",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "Cannot run OPA image in user namespace on GKE",
          "text": "## Short description `opa:1.14.1-envoy` image fails to start with `hostUsers: false` (Kubernetes user namespaces) due to missing `/dev/console` `containerd: 2.1.5` ``` $ kubectl version -o yaml clientVersion: buildDate: \"2026-03-18T18:30:07Z\" compiler: gc gitCommit: 6c1cd99aef09161ddb07b8ade6c9564e9b9bfe27 gitTreeState: clean gitVersion: v1.35.3 goVersion: go1.26.1 major: \"1\" minor: \"35\" platform: darwin/arm64 kustomizeVersion: v5.7.1 serverVersion: buildDate: \"2026-02-27T01:05:58Z\" compiler: gc emulationMajor: \"1\" emulationMinor: \"34\" gitCommit: f72eca22c700e6c114f9243d3ca6a1706e45c2a0 gitTreeState: clean gitVersion: v1.34.5-gke.1076000 goVersion: go1.24.13 X:boringcrypto major: \"1\" minCompatibilityMajor: \"1\" minCompatibilityMinor: \"33\" minor: \"34\" platform: linux/amd64 ``` ## Steps To Reproduce - Deploy this pod ``` apiVersion: v1 kind: Pod metadata: name: opa-userns-test spec: hostUsers: false securityContext: runAsUser: 1000 runAsGroup: 1000 runAsNonRoot: true fsGroup: 1000 containers: - name: opa image: openpolicyagent/opa:1.14.1-envoy args: - run - --server - --diagnostic-addr=0.0.0.0:8282 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true ``` - It gets `CreateContainerError` - Set `hostUsers: true` (default) and re-deploy - works ## Expected behavior The above pod spec works in user namespace, i.e. with `hostUsers: false` ## Additional context I built a custom OPA image with the following `Dockerfile`. This image works with `hostUsers: false`. ``` FROM openpolicyagent/opa:1.14.1-envoy AS opa FROM busybox AS setup COPY --from=opa / /opa-root/ RUN mkdir -p /opa-root/dev && touch /opa-root/dev/console FROM openpolicyagent/opa:1.14.1-envoy COPY --from=setup /opa-root/ / ```",
          "url": "https://github.com/containerd/containerd/issues/13129",
          "createdAt": "2026-03-26T14:46:42Z",
          "updatedAt": "2026-08-13T16:08:46Z",
          "timestamp": "2026-08-13T16:08:46Z",
          "metrics": {
            "reactions": 0,
            "comments": 5
          },
          "labels": [
            "kind/support",
            "Stale"
          ],
          "author": "savitha-qs",
          "state": "open",
          "assignees": [],
          "change": "new"
        }
      },
      {
        "id": "event:57810638a610bd56bd3d",
        "signalId": "github:containerd/containerd:pull_request:13294",
        "event": "changed",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [
          "text",
          "updatedAt",
          "labels"
        ],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13294",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "vendor: github.com/sirupsen/logrus v1.10.0",
          "text": "### vendor: github.com/sirupsen/logrus v1.10.0 Notable changes: - Add bidirectional `log/slog` integration with a Logrus hook and `slog.Handler`. - Add minimal, composable logging interfaces for individual log levels. - Fix reentrant logging deadlocks and improve concurrency safety around formatters, hooks, and entries. - Fix generic `Log`, `Logf`, `Logln`, and `LogFn` methods unexpectedly panicking at `PanicLevel`. - Allow `Entry.Caller` to be set explicitly and preserved across derived entries. - Improve `TextFormatter` performance and reduce allocations significantly. - Improve common Logger and Entry hot paths and caller-reporting performance. - Update `TextFormatter` handling for `[]byte`, debug/trace colors, and Windows ANSI terminals. - Raise the minimum supported Go version to Go 1.23. - Deprecate `Entry.HasCaller` and `MutexWrap`. release-notes: https://github.com/sirupsen/logrus/releases/tag/v1.10.0 full diff: https://github.com/sirupsen/logrus/compare/v1.9.4...v1.10.0",
          "url": "https://github.com/containerd/containerd/pull/13294",
          "createdAt": "2026-04-27T13:41:27Z",
          "updatedAt": "2026-08-13T15:55:43Z",
          "timestamp": "2026-08-13T15:55:43Z",
          "metrics": {
            "reactions": 0,
            "comments": 1
          },
          "labels": [
            "size/XXL"
          ],
          "author": "thaJeztah",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:ab0d40177310cd057cea",
        "signalId": "github:containerd/containerd:pull_request:13941",
        "event": "discovered",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13941",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "erofs: instrument warm up cache",
          "text": "It'd good go understand: - Rate of cache lookups. - How many cache hits alone / layer bytes served from cache. - Amount of time spent converting layers on cache miss",
          "url": "https://github.com/containerd/containerd/pull/13941",
          "createdAt": "2026-08-11T18:50:53Z",
          "updatedAt": "2026-08-13T15:35:21Z",
          "timestamp": "2026-08-13T15:35:21Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M"
          ],
          "author": "mxpv",
          "state": "open",
          "assignees": [],
          "change": "new"
        }
      },
      {
        "id": "event:0553bd625602df7cb4e0",
        "signalId": "github:containerd/containerd:pull_request:13950",
        "event": "changed",
        "observedAt": "2026-08-13T16:19:22.035158Z",
        "changedFields": [
          "updatedAt",
          "state"
        ],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13950",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "[release/2.2] snapshots/erofs: protect snapshot staging from cleanup",
          "text": "Cherry-pick of https://github.com/containerd/containerd/pull/13932",
          "url": "https://github.com/containerd/containerd/pull/13950",
          "createdAt": "2026-08-12T16:31:43Z",
          "updatedAt": "2026-08-13T13:50:20Z",
          "timestamp": "2026-08-13T13:50:20Z",
          "metrics": {
            "reactions": 0,
            "comments": 0
          },
          "labels": [
            "size/M"
          ],
          "author": "hsiangkao",
          "state": "closed",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:e0c6cf3adc088db3ad15",
        "signalId": "github:containerd/containerd:pull_request:13955",
        "event": "changed",
        "observedAt": "2026-08-13T17:43:20.785491Z",
        "changedFields": [
          "updatedAt",
          "metrics"
        ],
        "signal": {
          "id": "github:containerd/containerd:pull_request:13955",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "pull_request",
          "title": "ci: add lima image list for fedora images",
          "text": "This change adds fallback image entries for Fedora's canonical dl.fedoraproject.org.",
          "url": "https://github.com/containerd/containerd/pull/13955",
          "createdAt": "2026-08-13T01:08:22Z",
          "updatedAt": "2026-08-13T17:05:15Z",
          "timestamp": "2026-08-13T17:05:15Z",
          "metrics": {
            "reactions": 0,
            "comments": 1
          },
          "labels": [
            "size/S"
          ],
          "author": "austinvazquez",
          "state": "open",
          "assignees": [],
          "change": "updated"
        }
      },
      {
        "id": "event:268065bf0ce8f87d4b96",
        "signalId": "github:containerd/containerd:issue:8893",
        "event": "discovered",
        "observedAt": "2026-08-13T17:43:20.785491Z",
        "changedFields": [],
        "signal": {
          "id": "github:containerd/containerd:issue:8893",
          "source": "github",
          "group": "platform-infrastructure",
          "project": "containerd/containerd",
          "kind": "issue",
          "title": "TaskOOM event lost",
          "text": "### Description https://github.com/kubernetes/kubernetes/issues/119600 When I was troubleshooting the problem that the Reason was not OOMKilled, I found that the container lost the TaskOOM event. ### Steps to reproduce the issue e2e: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-containerd-node-e2e-1-7/1683184526936772608 logfile: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-containerd-node-e2e-1-7/1683184526936772608/artifacts/tmp-node-e2e-92a26bb7-cos-97-16919-353-1-system.log containerd_id: 98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54 containerd_name: oomkill-multi-target-container 1. kernel having trigger oom ``` Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efb6ec_09ab_4eec_8ad6_f953eb1a7466.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efb6ec_09ab_4eec_8ad6_f953eb1a7466.slice/cri-containerd-98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54.scope,task=dd,pid=38287,uid=0 Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 kernel: Memory cgroup out of memory: Killed process 38287 (dd) total-vm:21724kB, anon-rss:14744kB, file-rss:508kB, shmem-rss:0kB, UID:0 pgtables:68kB oom_score_adj:997 Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 kernel: Tasks in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efb6ec_09ab_4eec_8ad6_f953eb1a7466.slice/cri-containerd-98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54.scope are going to be killed due to memory.oom.group set ``` 2. and then containerd exit shim ``` Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 containerd[910]: time=\"2023-07-23T18:56:46.329304474Z\" level=debug msg=\"received exit event container_id:\\\"98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54\\\" id:\\\"98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54\\\" pid:38265 exit_status:137 exited_at:{seconds:1690138606 nanos:325950236}\" Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 containerd[910]: time=\"2023-07-23T18:56:46.389869167Z\" level=info msg=\"shim disconnected\" id=98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54 namespace=k8s.io Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 containerd[910]: time=\"2023-07-23T18:56:46.390142692Z\" level=warning msg=\"cleaning up after shim disconnected\" id=98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54 namespace=k8s.io ``` 3. After that, no TaskOOM event was received until container delete. ### Describe the results you received and expected Expect to receive TaskOOM event. ### What version of containerd are you using? 1.7 ### Any other relevant information ubuntu ### Show configuration if it is related to CRI plugin. no",
          "url": "https://github.com/containerd/containerd/issues/8893",
          "createdAt": "2023-07-30T15:02:09Z",
          "updatedAt": "2026-08-13T16:54:10Z",
          "timestamp": "2026-08-13T16:54:10Z",
          "metrics": {
            "reactions": 0,
            "comments": 17
          },
          "labels": [
            "Stale"
          ],
          "author": "lengrongfu",
          "state": "open",
          "assignees": [
            "lengrongfu",
            "fuweid"
          ],
          "change": "new"
        }
      }
    ]
  }
}