One cgroup budget for a Tekton taskrun's step and its dind sidecar

A Tekton TaskRun that builds container images runs as one pod holding one or more step containers, alongside a privileged Docker-in-Docker sidecar. A step issues docker build and docker run against the sidecar’s daemon socket. The dockerd process inside the sidecar is what actually does the work: pulling layers, running RUN instructions, spawning child containers. During a build, the sidecar is the busiest thing in the pod.

The two limits have to be sized independently, so the pod requests the sum of two peaks that never coincide: 4 GiB for the step, 4 GiB for the sidecar, 8 GiB reserved against node capacity for a workload that never exceeds 4. One shared ceiling would fit the actual usage.

The step and the sidecar sit in sibling cgroups under the same pod slice. Their paths are identical except for a container hash, so a one-line substitution in the sidecar’s entrypoint is enough to point dockerd at the step’s cgroup instead of its own. Everything the build creates is then charged to the step’s limit, and the kernel holds that limit all the way to an OOM kill. The same nesting that makes it work is what erases the build container from the metrics every memory dashboard is built on.

The problem

The unit of budget you want here is the pod: give this whole TaskRun 4 GiB, let the step and the build children draw from it, and stop paying for a peak that never happens. Kubernetes has that concept. PodLevelResources, KEP-2837, sets spec.resources on the pod itself. It went beta and default-on in 1.34, and is still beta in 1.36, which graduates InPlacePodLevelResourcesVerticalScaling to beta on by default. Two further gates around pod-level resources, PodLevelResourcesFixDefaulting and PodLevelResourcesFixKubeletQOSClass, land in 1.37.

But Tekton can’t use it. A TaskRun’s podTemplate is a fixed allowlist of pod-spec fields: env, nodeSelector, tolerations, affinity, securityContext, volumes, runtimeClassName, automountServiceAccountToken, dnsPolicy, dnsConfig, enableServiceLinks, priorityClassName, schedulerName, imagePullSecrets, hostNetwork, hostUsers, hostAliases, and topologySpreadConstraints. There’s no resources. The other lever, computeResources, rewrites the resources block of individual containers, which is per-container by construction and says nothing about the pod as a whole.

So a platform team that wants one number per TaskRun pod is stuck between a Kubernetes feature that’s still beta and a Tekton API surface that doesn’t expose it.

The options

Four candidates get the step and the build sharing one ceiling.

PodLevelResources. Set the budget on the pod and let the kernel divide it. Correct, but unavailable in Tekton’s podTemplate and still beta.

PID migration via cgroup.procs. Write the sidecar’s own dockerd PID into the step container’s cgroup.procs, moving the daemon itself under the step’s budget. This charges dockerd’s own footprint, including image pulls and layer extraction, to the step, which is a different and much blunter thing than charging the build’s children.

--cgroup-parent borrowing. Start dockerd with --cgroup-parent pointed at the step container’s scope, so every container dockerd creates is nested inside the step’s cgroup while dockerd itself stays where the kubelet put it.

Writing memory.max into the pod slice. Have something with node access compute a budget and write it directly into kubelet-kubepods-*-pod<uid>.slice/memory.max, bypassing the API entirely.

I tested --cgroup-parent borrowing. It’s the only one of the four that both works inside a normal pod today and targets the build’s children specifically rather than the daemon that spawns them.

The chosen method

The sidecar’s entrypoint runs before dockerd and does four things: read its own cgroup path, identify its own pod, ask the API server for the step container’s containerd ID, and substitute that ID into the path it read.

The critical property is that dockerd’s own PID never moves. The daemon stays in the sidecar’s cgroup, where the kubelet put it and where the kubelet’s own accounting expects it. Only the containers it goes on to create land under the step.

Step one reads the sidecar’s own cgroup from procfs, which under cgroup v2 unified hierarchy is a single 0:: line:

1
$ awk -F: '/^0::/ {print $3}' /proc/self/cgroup
2
/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-podcd57a787_d809_4a55_b6f8_ab38de73aeb8.slice/cri-containerd-4354439d2f1668c37baaabaa5a33b5ce71a1b24c182e4ebe0d16455889c14617.scope

Steps two and three read the pod name from /etc/hostname and the namespace from /var/run/secrets/kubernetes.io/serviceaccount/namespace, then poll the API server for the step container’s ID, filtering out the sidecar’s own entry:

1
kubectl get pod "$podname" -n "$namespace" -o json | \
2
  jq -r 'first(.status.containerStatuses[] | select(.name != "sidecar-dind" and .started == true) | .containerID)' | \
3
  awk -F'://' '{print $2}'

Step four is the whole trick. The sidecar’s cgroup path and the step’s cgroup path are siblings under the same pod slice, differing only in the container hash, so one substitution converts one into the other:

1
NEW_CG_PATH=$(echo "$CGROUP_PATH" | sed -E "s/cri-containerd-[a-f0-9]+\.scope/cri-containerd-$CONTAINERD_HASH.scope/")

Then dockerd starts with that path as its parent:

1
exec /usr/local/bin/dockerd-entrypoint.sh \
2
  --cgroup-parent="$NEW_CG_PATH" \
3
  "$@"

The sequence, with the step’s own commands arriving after dockerd is already reparented:

sequenceDiagram
    participant Init as sidecar entrypoint
    participant API as kube-apiserver
    participant D as dockerd (sidecar cgroup)
    participant Step as step container
    participant CG as step .scope cgroup

    Init->>Init: read /proc/self/cgroup
    Init->>API: get pod, read step containerID
    API-->>Init: cri-containerd-<step hash>
    Init->>Init: sed sidecar hash -> step hash
    Init->>D: exec dockerd --cgroup-parent=<step scope>
    Step->>D: docker run / docker build
    D->>CG: create child cgroup under step scope
    Note over CG: child's memory charges here

One detail makes the substitution viable, and it’s worth checking before copying any of this. The two sides of the pod run different cgroup drivers:

1
$ kubectl exec <pod> -c sidecar-dind -- docker info --format '{{.CgroupDriver}} / v{{.CgroupVersion}}'
2
cgroupfs / v2
3
$ kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq -r .kubeletconfig.cgroupDriver
4
systemd

The kubelet builds the systemd-style .slice/.scope paths the sidecar reads out of procfs, while dockerd inside the sidecar runs cgroupfs and treats --cgroup-parent as a literal directory to create beneath. That mismatch is what lets a path lifted from the kubelet’s hierarchy be handed to dockerd unchanged. A dind daemon configured with the systemd driver would expect a slice name and derive its own scope instead, and the borrowed path wouldn’t nest the same way.

Two prerequisites beyond that. The sidecar’s service account needs get on pods in its own namespace, because the container ID is only available through the API. And the sidecar needs privileged: true, both for dockerd’s normal reasons and because without a host cgroup namespace the container reads only the namespace-relative 0::/ and has no host path to rewrite.

What the kernel does with it

The nesting is literal. After the step tells dockerd to run a container that writes a 200 MB file, that container’s cgroup appears as a subdirectory of the step’s scope on the node:

1
$ ls .../cri-containerd-84c3e193...abe1.scope/
2
4c4095d78188d1ba957135bd72bf672eec1de8a40ed8e011e3b30fb1a811d247
3
cgroup.controllers
4
cgroup.subtree_control
5
...
6
memory.current
7
memory.max
8
...

The kernel charges it to the step. Reading the step scope’s memory.current directly on the node, with a 200 MB child running:

1
$ cat .../cri-containerd-84c3e193...abe1.scope/memory.current
2
213598208

That’s roughly 203.7 MiB against a step container whose own resident footprint before the child started was about 7 MB. The charge landed exactly where the mechanism aimed it.

That happens because of one file. A cgroup only accounts for a resource if its parent enabled that controller in cgroup.subtree_control, and the step’s scope lists memory as available without enabling it:

1
$ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.controllers
2
cpuset cpu io memory hugetlb pids rdma misc
3
$ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.subtree_control
4
cpuset cpu pids
5
$ cat .../4c4095d78188...247/cgroup.controllers
6
cpuset cpu pids
7
$ cat .../4c4095d78188...247/memory.current
8
cat: ...: No such file or directory

So the child gets no memory.* files at all, which makes it no boundary for memory accounting: the pages its processes instantiate are charged to the nearest enclosing cgroup that is one, and that’s the step’s own .scope. The mechanism works precisely because the child is not a memcg.

There’s a second reason, and it’s the stronger one. Reading the child’s cgroup.procs returns Operation not supported, which the kernel documents for threaded cgroups, and cgroup.type confirms it. Probing a separate run for those values: the child reads threaded, the step scope reads domain threaded, and a step scope with no dind child beneath it yet reads a plain domain. Threaded cgroups carry only the threaded controllers, cpu, cpuset, perf_event, and pids, and memory isn’t among them. The two facts are the same one seen from either end: delegating only threaded controllers is what let the child become threaded at all, and once threaded it can never carry a memory.current. Its processes show up in cgroup.threads instead.

That’s the whole finding, and everything an operator would reach for follows from it. There is no per-child memory object, so nothing reading cgroup memory files can report one. cadvisor’s line for the child reads zero on working set, usage, and RSS alike:

1
container_memory_working_set_bytes{container="",id=".../cri-containerd-84c3e193...abe1.scope/4c4095d78188...247",...} 0
2
container_memory_usage_bytes{container="",id=".../4c4095d78188...247",...} 0
3
container_memory_rss{container="",id=".../4c4095d78188...247",...} 0

docker stats inside the sidecar, looking at the same container through the daemon that created it, agrees:

1
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
2
4c4095d78188   memhog    0.00%     0B / 0B             0.00%     1.32kB / 126B   0B / 0B     1

Neither is wrong. There’s no memory.current at the child to read, so both correctly report the absence as zero.

The step’s own line is the one that misleads, because it does move, just nowhere near enough. cadvisor’s container_memory_working_set_bytes for step-step went from about 7.2 MB before the child to about 13.9 MB with it running: roughly 6.4 MiB of movement against a kernel charge of roughly 204 MiB. Those are sampled gauges, so the individual readings drift a few hundred kB between runs, but the ratio is stable at around thirty. The metric captures about a thirtieth of the charge.

One footnote on where the 200 MB went. The step scope’s memory.stat shows anon 6541312 and file 200101888, all of it inactive_file. A container writing a blob to disk charges the step as reclaimable page cache, not anonymous memory, which is why that particular workload never triggers a kill.

What happens when it OOMs

Forcing a kill needs anonymous memory. With the step capped at 64Mi via stepSpecs.computeResources and a child allocating 500 MiB of anon, the kernel does exactly what the budget says.

The kill is a real memcg kill, not a node-pressure eviction, and the cgroup it names is the step’s:

1
oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/docker/b132849958c0.../kubelet.slice/.../kubelet-kubepods-burstable-podee4ed01f_c5b6_401d_84e6_cd45e7a94890.slice/cri-containerd-d5a62de9d463d23595850010701c41f8cd07c14626727281560162300d0f7d98.scope,task_memcg=...,task=entrypoint,pid=79431,uid=0
2
memory: usage 65536kB, limit 65536kB, failcnt 59

The oom_memcg path carries this run’s pod UID (ee4ed01f_c5b6_401d_84e6_cd45e7a94890) and this run’s step container hash, both matched against what kubectl reported for the same pod. usage 65536kB, limit 65536kB is the 64Mi cap hit exactly, with 59 prior failed charge attempts.

The kernel then killed five processes, not one, because memory.oom.group is set on the step scope:

The python3 at oom_score_adj: 0 is the child inside dind, the process that actually invoked the killer. Everything at 996 is Tekton’s own step machinery: two entrypoint processes, the generated step script, and its sleep. The kernel log states the rule plainly, that tasks in the step scope “are going to be killed due to memory.oom.group set.” The step is billed for the child’s overrun and executed for it.

Note that the killer only got 58084 kB of anon resident before hitting the wall, against a total-vm reservation of 523528 kB. The 500 MiB is what it asked for, not what it held.

Tekton usually reports the failure, but not as an OOM:

1
$ kubectl get taskrun cgroup-budget-run-oom -o jsonpath='{.status.conditions}'
2
[{"lastTransitionTime":"2026-08-18T08:32:12Z","message":"\"step-step\" exited with code 137: Error","reason":"StepFailed","status":"False","type":"Succeeded"}]

This is the nuance worth getting right, because the obvious reading is wrong. Tekton does have OOM-specific reasons: TaskRunReasonStepOOM (“StepOOM”), TaskRunReasonSidecarOOM, and TaskRunReasonInitContainerOOM all exist in the v1 TaskRun types. Tekton’s getFailureInfo() calls isOOMKilled(), which tests s.State.Terminated.Reason == "OOMKilled" and nothing else. It never looks at the exit code. The step’s terminated state here was exitCode: 137 with reason: "Error", so the check correctly declined to fire.

The interesting part is one layer down, because Tekton isn’t the component that decides. containerd writes that string, and it reaches the opposite conclusion from the opposite evidence: on a 137 exit it checks whether the cgroup’s memory.events counter shows an oom_kill, and only then sets the reason. So containerd gates on the exit code Tekton ignores, and Tekton gates on the string containerd may or may not get around to writing.

It usually doesn’t. Polling the step scope’s own memory.events in a busy loop through the kill catches the counter arriving and the cgroup disappearing almost simultaneously:

1
09:17:12.938 | oom_kill 5 | oom_group_kill 1 | current=774144
2
09:17:12.940 | directory gone

Two milliseconds. At a 20 ms sampling interval the directory was already gone. The kernel’s accounting is correct and hierarchical throughout: the pod slice’s memory.events reads oom_kill 5 while its memory.events.local stays at 0, which is exactly what a kill charged to a descendant should look like. The counter is there to be read. Under the systemd cgroup driver, the scope unit is garbage-collected as soon as its last process exits, and containerd is racing that GC to read a file that is about to stop existing. containerd’s own source comments name this race.

Repeat the same OOM six times and the reason comes back Error five times and OOMKilled once, with exitCode: 137 every time. So StepOOM isn’t dead code and this isn’t a Tekton bug: it’s a race one layer below Tekton that Tekton faithfully reports the losing side of. An operator sees the same generic failure most of the time and the correct one occasionally, which is worse than either being consistent.

There’s a trap in verifying any of this. The kernel ring buffer inside a kind node belongs to the host VM, not the cluster, and it survives cluster deletion. Create a fresh cluster and dmesg on its node already holds OOM lines from clusters that no longer exist: 20 of them here, including a python3 kill reading anon-rss:57964kB, close enough to the live number 58084kB to pass a careless eyeball check. Grepping dmesg | grep -i oom and reading whatever comes back will happily “confirm” a several-hour-old event from a deleted cluster. Three distinct oom_memcg values sat in that buffer and only one belonged to the pod under test. Extract oom_memcg and match it against the current pod UID and step container hash, or the number you quote is fiction.

The remaining tests:

Why not to run this

The enforcement is real, and the operability isn’t, and those are separable properties.

Nothing downstream of the kernel sees the charge at the granularity it happened. Dashboards built on container_memory_working_set_bytes stay roughly flat while the kernel accumulates two hundred megabytes against the step, and kubectl top consumes the same cadvisor-derived series, though this cluster had no metrics-server installed so that path wasn’t exercised directly. Anything keyed on that per-container series would inherit the same gap: an autoscaler or a sizing recommender reading step memory sees a number roughly thirty times too small. Neither was deployed here, only the cadvisor series underneath them. Recovering the truth means reading memory.current on the node, which means node access and a scrape path that doesn’t exist by default.

Tekton’s status surface can’t distinguish a step that overran its own budget from a step that was killed for a child’s. Both usually arrive as StepFailed with exited with code 137: Error, and memory.oom.group means the same set of Tekton processes dies either way, so nothing in the process-exit shape tells them apart. The occasional run that does win the race and report StepOOM is no better for this purpose: it correctly says the step was OOM-killed, which is still the wrong container to go looking at. An on-call engineer reading only the TaskRun has no signal pointing at the build container, and an intermittent reason string is harder to build an alert on than a consistently wrong one.

The pod also still reserves two numbers, not one. dockerd’s own PID stays in the sidecar’s cgroup under the sidecar’s own limit, so the 8 GiB of the opening never collapses all the way to 4. What changes is their size: the sidecar’s cadvisor line read 26.9 MB idle and 42.8 MB while a 200 MB child ran, because the child’s memory is charged elsewhere. The second reservation covers a daemon’s own footprint instead of a peak build, and can be sized in tens of megabytes rather than gigabytes.

You can’t drop privileged: true later as a hardening pass. Without it the sidecar reads 0::/ instead of a real cgroup path, the substitution has nothing to match, and the daemon would start with --cgroup-parent=/. dockerd never gets that far, dying on a mount failure first, but nothing in the script detects or reports the broken path either way.

Both failure modes leave a dead sidecar inside a green TaskRun. RBAC denial and the unprivileged case both end with the sidecar at exit code 1 and the TaskRun at Succeeded / All Steps have completed executing. The step ran, passed, and was never subject to the budget the pipeline thought it had. There is no annotation, condition, or event distinguishing an enforced run from an unenforced one.

There’s a related Docker issue, and it’s worth being precise about how it differs. moby/moby#45378, DinD cgroupv2 problem inside K8s, filed 2023-04-21 and still open, reports containers inside DinD exceeding the pod memory limit without being OOM-killed, an enforcement failure not seen on cgroup v1. That’s the opposite half of what shows up here, where enforcement works and attribution is missing. Both point at nested cgroup accounting under DinD being fragile, in different directions.

Everything above ran on a single-node kind cluster, kind v0.32.0, node image kindest/node:v1.36.1, Kubernetes v1.36.1 on Debian 13 with containerd 2.3.1, kernel 6.8.0-117-generic on aarch64, cgroup v2 unified hierarchy, and Tekton Pipelines v1.15.x. The sidecar is docker:28-dind (Docker 28.5.2, Alpine 3.22) with apk add bash kubectl jq on top, since that base image ships no bash and the entrypoint needs it. The step is alpine:3.20. Load was generated by kubectl exec into the sidecar and running docker run against its own dockerd, on a disposable local cluster and nothing resembling production. That deliberately tests the plain container path: docker build reaches the same daemon, but BuildKit places its workers on its own terms, so treat the placement result here as proven for containers dockerd creates directly and unverified for every build topology.

What to use instead

PodLevelResources is the answer as soon as your cluster is on 1.34 or later, where it’s beta and enabled by default. It puts the limit on the pod, and it keeps the kubelet’s accounting and the kernel’s accounting pointing at the same object. It’s still beta in 1.36, with two fix gates landing in 1.37, so read the release notes before you build a platform commitment on it. What it leaves unsolved is delivery: spec.resources still has to reach the pod, and the TaskRun’s podTemplate allowlist won’t carry it. Use mutating admission webhook to patch spec.resources on the pod and you have the shared budget.

Writing memory.max into the pod slice from a node-level agent is the conservative fallback. It needs node access and hardcodes the kubelet’s slice naming, which varies with the cgroup driver and cgroupRoot, but it works on any version and needs no API changes. It also charges the whole pod, sidecar included, which for a dind pod is usually what you wanted anyway.

None of this is really about Tekton or Docker. cgroup.subtree_control decides which level owns a resource, and Kubernetes doesn’t delegate memory below the container cgroup. That makes enforcement and observability separable, which is the part worth remembering: the kernel will happily hold you to a budget nobody can see you spending.