Bhawani Singh

Pod Startup Forensics: The Problem

Bhawani Singh — Sat, 18 Jul 2026 00:00:00 GMT

A pod takes 40 seconds to start instead of 4. It doesn't look like an incident. Nothing pages. kubectl get pods eventually shows Running. The deploy finishes. The only trace is a vague sense that something felt slow, and by the time anyone goes looking, the pod that was actually slow is usually gone.

Kubernetes has a real metric for this. kubelet exposes kubelet_pod_start_sli_duration_seconds, a live histogram, and SIG-Scalability maintains an SLO that the 99th percentile of pod startup latency, per cluster-day, stays under 5 seconds, built on exactly that signal. So the instinct is to assume the problem is already solved.

It isn't. That metric tells you the cluster-wide p99 crept up this week. It can't tell you which pod, which phase, or why.

Autoscaling only works if the scale-up is fast

The entire value of a Horizontal Pod Autoscaler or Cluster Autoscaler reacting to load is that new capacity shows up before the old capacity falls over. Imagine a scale-up event that takes 45 seconds to go from "decision made" to "pod serving traffic," against a load spike that doubles request volume in under a minute. The autoscaler technically did its job. It decided to add capacity. The outcome is the same as not having an autoscaler at all: the existing pods absorb the overflow, latency degrades, and the new pod shows up in a system that's already stopped hurting on its own, or has already tripped a different alarm.

Pod startup latency isn't one number. It's schedule delay, image pull, PVC attach and mount as two separate operations, init containers running in sequence, sidecars starting alongside or before the main container, and the main container itself becoming ready. Every one of those phases is a place autoscaling responsiveness can quietly evaporate, and none of them show up as a single knob to tune. Picture a cluster that scales in a few seconds on a warm node and takes a minute or more on a node that has to pull a fresh image: that cluster doesn't have one autoscaling latency. It has a distribution, and the tail of that distribution is exactly the case autoscaling exists to handle: sudden load, no warm capacity, has to happen fast.

CI/CD pipelines spend most of their wall clock waiting on pods, not running work

Tekton, and any CI system built the same way, one pod per unit of work, pays the pod startup cost on every Task in a pipeline. A PipelineRun with five sequential Tasks doesn't pay pod startup once. It pays it five times, because each Task runs as its own pod with its own schedule, pull, init, and ready sequence. The individual Steps inside a Task run as containers sequenced within that one pod, so they share the pod's startup cost rather than each paying it separately.

This matters more than it looks like it should, because the actual work inside a Step is frequently fast. In this project's own testing, a Tekton TaskRun pod's step containers finished in under two seconds, fast enough that five separate one-shot lookups run right after the pod was deployed never once caught a step container still running; they'd already gone terminated in the time it took to issue the command. That's one data point, not a universal one, but it illustrates a real dynamic: if the startup overhead wrapped around that Task's pod, schedule plus image pull plus init container plus container create, itself runs several seconds, the overhead isn't a rounding error next to the work happening inside it. A pipeline that feels slow is often a pipeline where the build and test commands run fine and the pods wrapping each Task are the bottleneck, and nothing in a typical CI dashboard separates those two costs from each other.

The on-call cost: Guessing with kubectl describe pod

When a pod is slow to start in production, not failing, not crash-looping, just slow, the standard move is kubectl describe pod followed by reading the Events section top to bottom. That gives a chronological list: Scheduled, Pulling, Pulled, Created, Started, maybe a FailedMount or a repeated Unhealthy if something is actually wrong. What it doesn't give is any indication of why a phase took as long as it did.

So the on-call engineer does arithmetic by hand: subtracting firstTimestamp values between events to guess how long image pull took, squinting at whether the gap between Pulled and Started looks unusually large. If the answer isn't obvious from the event list, there's nowhere else to look. If the slow phase is an init container blocking on a connect() call to a service that's still coming up, the Event stream shows nothing at all during that block. No event fires while a process is just waiting on a syscall to return. The engineer either already knows this failure mode from having seen it before, or starts guessing: the CNI, DNS, the CSI driver, the registry. That guessing has a real cost: time spent per incident, and because nothing captures the answer, the same ambiguity can get re-diagnosed from scratch the next time it happens.

A fair fraction of the pods worth interrogating are already gone by the time anyone looks. Kubernetes garbage-collects completed pods, CI systems delete TaskRun pods after a retention window, and a pod that failed to schedule and got evicted leaves no long-lived object to describe. kubectl describe pod only works on a pod that still exists. Kubernetes' own Event objects carry a bounded retention window independent of the pod's own lifecycle, so the moment a pod is deleted, its events go with it on their own separate clock. "Why was that slow" becomes permanently unanswerable through the tools most people reach for first, and for a Tekton Task pod whose steps finish in a couple of seconds, that clock starts running almost immediately.

Why kubectl get events isn't a phase breakdown

kubectl get events, or describe pod's embedded events section, gives the raw material a phase breakdown would be built from, not the breakdown itself. It's a flat, chronological list of whatever the kubelet and the various controllers happened to report, in the order they happened to report it. There's no concept of a phase in that stream. No line says "image pull took 10.4 seconds" or "this was the slowest part of startup." Pulling shows up at one timestamp and Pulled at another, and it's on the reader to subtract them, notice which containers were init containers versus sidecars versus the main container (which, depending on Kubernetes version, can report state in genuinely different shapes: a native sidecar with restartPolicy: Always shows up as running in the same status array where an ordinary init container shows terminated), and reconstruct the ordering by hand. There's no root cause in that stream by construction, either. An Event is a report that something happened, not an explanation of what the process was doing while it was blocked. If a container spent eight seconds inside a connect() call, no Event says so, because Events cover what Kubernetes' own controllers observed, and no controller is watching syscalls.

Why kubelet's own latency metric doesn't answer this either

kubelet_pod_start_sli_duration_seconds is a genuinely real, useful signal for a specific question, which makes it tempting to assume it solves this one too. The gap is structural, not a matter of granularity.

That histogram answers "is pod startup latency across this cluster within SLO, in aggregate, over time." It's a single histogram per node, rolled up again across every node in the cluster to produce the SIG-Scalability SLO number: a fast-starting sidecar-free stateless pod and a slow multi-init-container stateful pod both feed the same buckets. No per-pod breakout, no phase attribution within a single pod's startup, no timestamp to point at and say "this second is where the time went." A cluster-wide p99 can trend upward on a dashboard for a week straight with no way to go from that trend to a single pod name worth investigating.

kubectl get events gives raw, unstructured chronology for one pod at a time, with no phase model and no root cause. kubelet's own metric gives an aggregate trend across the whole cluster, with no per-pod resolution at all. Between a single pod's raw chronology and the cluster's aggregate trend sits the actual question: which pod, which second, why. Neither tool covers that middle ground, and nothing else currently does either.

Pod Startup Forensics: The Tooling Gap

Bhawani Singh — Wed, 22 Jul 2026 00:00:00 GMT

kubectl get events gives raw chronology with no phase model. kubelet_pod_start_sli_duration_seconds gives a cluster-wide aggregate with no per-pod resolution. Nothing sits between them. Kubernetes observability isn't short on eBPF-based tooling, though, and eBPF is exactly the mechanism you'd reach for to explain why a phase blocked. So it's worth checking, tool by tool, what each one actually does today, not what its name suggests it might do.

Tetragon: Real syscall visibility, no concept of a phase

Tetragon is Cilium's eBPF-based runtime security and observability tool. It hooks kernel functions directly and streams events for process execution, system call activity, and file/network I/O. Every event carries a process_exec block identifying the process that generated it, a node_name and time, and, because Tetragon is Kubernetes-aware, the pod, namespace, container, and labels the process belongs to. That's real, fine-grained, per-process attribution, and it's exactly the kind of signal you'd want during a slow init container: which syscall, which process, which pod, down to the timestamp.

What Tetragon doesn't have is any notion of a phase to attach that signal to. Each event type, process_exec, process_exit, process_kprobe, process_uprobe, process_tracepoint, is a standalone record. Correlation across events happens through shared process IDs and parent/child exec_id relationships, not through any built-in lifecycle or phase concept, and nothing in the documentation describes or gestures at one. Tetragon will tell you a process inside a given pod called connect() and give you the pod identity and a timestamp. It won't tell you that call happened during the init-container phase rather than the main-container phase, because it has no documented model of "phase" at all. That's a Kubernetes pod-lifecycle concept, and Tetragon operates one layer below it, at the process and syscall level. Getting from a Tetragon event stream to "this pod's startup was slow because of X" requires bringing in a phase model from somewhere else, Tetragon has none, and hand-correlating a syscall trace against it. Right primitive, missing the other half.

Pixie: Deep runtime tracing, built for services that are already running

Pixie (px.dev) is a CNCF sandbox project, also eBPF-based, that automatically instruments HTTP, gRPC, DNS, and SQL traffic without code changes, builds service maps from the traffic it observes, and produces CPU flame graphs and resource profiles broken down by pod, node, and namespace. It stores everything it collects locally in-cluster, in memory, which keeps it fast and avoids exporting data externally. That also means its data is short-lived by design, not a long-term store.

Pixie's entire value proposition is built around instant visibility into services that are already handling traffic: request/response bodies, query latencies, who's calling whom. None of that maps onto pod startup. A pod that's still in its init-container phase isn't serving HTTP, issuing SQL queries, or generating the kind of application-layer traffic Pixie's protocol parsers are built to detect automatically, so there's nothing for its tracers to attach to yet. Its resource-profiling and flame-graph features apply to a pod's CPU behavior once the main container is up and running, not to the sequence of schedule, image pull, PVC attach and mount, and init-container execution that happens before that point. Pixie is a strong answer to "why is this running service slow right now" and has no documented model of the pod-startup path, because that was never the problem it set out to solve.

Parca and Pyroscope: Continuous profilers, sampling code that's already executing

Parca and Grafana Pyroscope are both continuous profilers, and both work on the same basic principle: sample stack traces from running processes at a fixed rate and aggregate them into flame graphs over time. Parca's agent, also eBPF-based, samples user-space and kernel-space stack traces 19 times per second and discovers targets across Kubernetes and systemd automatically, with no code changes or restarts required. Pyroscope does the same continuous aggregation across a longer language list, Go, Java, Python, Ruby, Node.js, .NET, Rust, plus eBPF-based system-level profiling, and correlates its profiles with a project's existing metrics, logs, and traces.

Both tools answer "where's this process spending its CPU time right now, and how has that changed over the last hour." Sampling stack traces at a fixed frequency only produces a meaningful flame graph once a process has run long enough to accumulate samples across representative code paths. That's a statistical technique, and pod startup is a one-shot sequence of distinct phases (schedule, pull, attach, mount, init, ready), not a steady-state workload to sample against. Neither tool's documentation mentions process startup, container initialization, or anything resembling a "pod startup phase." That tracks: a continuous profiler answers questions about ongoing CPU cost. It has no documented mechanism for asking why a container took four seconds to go from Created to Started, because that question isn't about steady-state CPU time. It's about a specific, short, one-time sequence of blocking operations.

The generic platforms: Broad coverage, no startup-specific model

The remaining category is the generic Kubernetes observability platform: the kind that bundles logs, metrics, traces, and often eBPF-based profiling into one product and sells breadth. Metoro is representative. It combines eBPF-based kernel telemetry, APM-style dashboards and service maps, log collection, and continuous CPU/memory profiling into what it describes as seven integrated signals (logs, metrics, traces, profiling, Kubernetes events, resources, deployment context), installed with a single Helm chart. That's a real, wide net, and it explicitly counts Kubernetes events as one of its signal types. It's still a production-monitoring and incident-detection platform at its core, built around dashboards, alerting, and automated root-cause analysis for services under load, not a dedicated model of the schedule-to-ready sequence a pod goes through before it starts serving anything.

OpenObserve sits a layer further out: a unified logs/metrics/traces/RUM store, OpenTelemetry-native, built and marketed primarily on cost and query performance rather than any Kubernetes-specific diagnostic feature, and that focus makes it a genuinely strong place to centralize telemetry cheaply at scale. It's infrastructure you'd point other tools' telemetry at, not a pod-lifecycle tool by itself.

The shape repeats across this category: broad signal coverage, real engineering behind each individual signal, and no dedicated concept of a pod-startup phase anywhere in the product. They're built to answer "what's wrong with my running system," which is a different question, asked at a different point in a pod's life, than "which of these six startup phases was slow, and why."

What's actually missing

Line these up and the shape of the gap gets sharper, not vaguer. Tetragon has the syscall-level ground truth but no phase model to hang it on. Pixie has deep runtime tracing but nothing to trace before a pod's main container is serving traffic. Parca and Pyroscope have continuous sampling profilers built for steady-state code, a different shape of problem entirely from a one-shot startup sequence. The generic platforms have breadth without a startup-specific model anywhere in that breadth. Every one of them is a legitimately strong tool at what it was built for, and every one of them was built for a question other than which pod, which second, why. That's not four small gaps between four adjacent categories of tool. It's one gap, looked at from four angles.

Pod Startup Forensics: The Idea

Bhawani Singh — Sun, 26 Jul 2026 00:00:00 GMT

Tetragon has the syscall trace. It doesn't have a phase. The Kubernetes API has the phase. It doesn't have the syscall trace. Every tool in the last part's survey sits on one side of that line, and no amount of squinting at any single one of them closes the gap, because the gap isn't a missing feature. It's a missing join.

Two sources, two different kinds of authority

The Kubernetes API knows, with certainty, when the scheduler binds a pod to a node, when the kubelet started pulling an image and when it finished, when a container's status flipped from waiting to running to terminated. That certainty comes from a specific place: every one of those transitions is reported by a controller that owns the transition. The scheduler reports scheduling. The kubelet reports image pull and container state. Nothing downstream has to guess, because the component doing the work is the same component filing the event.

That authority has a sharp edge. An Event is a report that something happened, not an account of what the process was doing while it happened. The gap between Pulled and the next event can be eight seconds of an init container blocked inside connect(), and the Event stream says nothing about it, because no controller is watching syscalls. It can't say more than it knows, and what it knows stops at the boundary between phases, not at what happens inside one.

eBPF sits on the opposite side of that boundary. A tracepoint on sys_enter_connect and sys_exit_connect doesn't know what a "phase" is, doesn't know what a Tekton TaskRun is, doesn't know that the process it's watching belongs to a pod at all until something tells it. What it knows, with just as much certainty as the Kubernetes API knows its own transitions, is the syscall: which process called connect(), when it entered the kernel, when it returned, what the return code was. That's exactly the evidence missing from the Event stream. It's also evidence with no label. A syscall trace on its own is a wall of pid, comm, and timestamps with nothing in it that says "startup" or "init container" or "this belongs to the pod that's currently 8 seconds into a phase someone should be looking at."

Neither source can be patched to cover the other's blind spot, because the blind spot isn't a bug in either one. The Kubernetes API doesn't lack syscall visibility by oversight; it lacks it because no controller in the chain from scheduler to kubelet operates at that level, and adding syscall tracing to a controller would be a different, much stranger design than the one Kubernetes has. eBPF doesn't lack a phase model by oversight either; a tracepoint fires on a kernel event, not a Kubernetes concept, and there's no phase field in a sys_enter_connect argument list to read one from. The right design isn't "make the API smarter about syscalls" or "make eBPF smarter about Kubernetes." It's to keep each source doing what it's actually good at, and build the missing piece neither one provides on its own: a join keyed on pod identity, with the API supplying the authoritative phase labels and eBPF supplying the syscall-level evidence for what happened during a phase flagged as slow.

That's a design decision with real content, not a truism. It means the API is never asked to explain why a phase was slow, and eBPF is never asked to say which phase anything happened in. Each source answers only the question it's actually positioned to answer, and the tool's job is entirely in the join: watch for a phase that's taking too long, and when one is flagged, attach the syscall evidence from that pod's cgroup during that exact window. The output isn't "container start took N seconds." It's "container start took N seconds, and the process spent nearly all of it blocked inside connect() to an address that never answered." One source says which seconds mattered. The other says what filled them.

The pod that's already gone by the time you'd query it

There's a second design constraint that doesn't come from the tooling gap directly. It comes from a fact Part 1 already established: a Tekton TaskRun pod can run its steps and terminate in under two seconds. Any design that answers "which pod, which phase, why" by querying live cluster state is structurally blind to exactly the pods most worth investigating after the fact, because by the time a human goes looking, the pod that mattered is gone and so is everything about it.

This rules out an entire category of otherwise-reasonable design: a tool that watches the API and the kernel live, holds the correlated result in memory or in a dashboard, and answers questions against whatever's currently running. That shape works for the pod that's slow right now, in front of you, while you're watching. It fails the pod that was slow ten minutes ago and no longer exists, which is the more common case for anything that resembles CI. Kubernetes' own Event objects carry a retention window independent of the pod's lifecycle, so even the raw material for a manual reconstruction disappears on its own clock, faster than most people think to look.

So durability isn't a feature to add once the phase-and-syscall correlation is working. It has to be there from the first design decision, because the alternative isn't a smaller version of the same tool. It's a tool that answers a different, less useful question: not "what happened to that pod," but "what's happening to pods in general right now." Those look similar from a distance and aren't the same product. The design has to capture and persist the correlated result as it happens, continuously, for every pod, not on demand for whichever pod someone happens to ask about while it's still alive. That single constraint decides most of the shape below: an on-demand query design and an always-watching, always-persisting design are different architectures, not different configurations of the same one.

The shape the idea takes

Put the two constraints together and the design isn't a single new tool bolted where the gap is. It's four pieces, each answering to one of the constraints above, wired together by pod identity: a watcher on the Kubernetes API, a kernel-level tracer, a correlation step, and durable storage.

A continuous watcher on the Kubernetes API. Not a poller that checks in on a fixed interval, because a fixed interval can miss a phase that starts and ends inside the gap between checks, exactly the failure mode that makes a fast Tekton step invisible to anything slower than the step itself. A watch reacts to each transition as the API reports it: pod scheduled, image pull started and finished, each init and sidecar and main container's state change, in order, as they happen. This is the source of authoritative phase boundaries and labels, and it runs whether or not anything downstream is currently interesting, because the pod that turns out to matter isn't known in advance.

A kernel-level tracer that runs continuously, and a phase timeline that decides what to keep looking at. The tracer itself has to watch every syscall on every pod, all the time, because there's no way to go back and capture a syscall after it already happened; nothing about the design can afford to be selective at the point of capture. What's selective is the read side. The watcher's own phase timeline is what decides which stretch of that stream is worth pulling: when a phase crosses a threshold that marks it as unusually slow, that's the signal for the correlation step to pull the syscall evidence already captured for that pod's processes during that window, instead of surfacing all of it undifferentiated. The syscall-level tracer's job isn't to explain everything a container does. It's to have already captured the specific stretch of time the phase-level view goes on to flag as worth explaining.

A correlation step keyed on pod identity, not on time alone. A timestamp overlap between an API event and a kernel event isn't enough on a real node, because a real node runs more than one pod's processes concurrently, and two pods can be inside their own slow phases at the same moment for entirely unrelated reasons. Attributing a syscall to the wrong pod because they happened to overlap in time would produce a confident, wrong answer, which is worse than an honest "unknown." The join has to resolve which pod a given process actually belongs to, and hold that resolution stable across a phase boundary that spans multiple wake-ups of the correlator, not just check it once and assume it still holds.

Durable storage that survives the pod's own deletion. The output of the watch-and-correlate pipeline can't live only in memory or only behind a live query, for the reason the pod-lifecycle constraint above already argued: the moment matters, but by the time anyone asks about it, the pod producing that moment is frequently gone. Every phase transition and every correlated syscall gets written down as it's observed, keyed on the pod's identity rather than on the pod object still existing, so a later lookup against a pod that kubectl has never heard of still has somewhere to go.

None of these four pieces is individually new. Watching the Kubernetes API for state transitions is what every controller in the system already does. Kernel-level tracing on syscalls is the mechanism Tetragon, Pixie, and Parca all already build on. Persisting observed state so it outlives the object it describes isn't a novel idea in distributed systems. What's missing from every existing tool surveyed against this problem isn't any one of these four pieces. It's the combination: authoritative phase labels from one source, syscall-level evidence from another, joined on identity rather than assumed from timing, and captured continuously rather than queried on demand. That combination is the actual shape of an answer to "which pod, which second, why," for a pod that might not exist by the time anyone asks the question.

Disk I/O QoS for Kubernetes with cgroup v2 io.weight

Bhawani Singh — Sun, 09 Aug 2026 00:00:00 GMT

Kubernetes can request and limit CPU, memory, and ephemeral-storage capacity. It has no native resource model for disk I/O bandwidth, IOPS, or latency QoS.

CPU has requests and limits. Memory has requests, limits, and OOM eviction. ResourceQuota can account for requests.ephemeral-storage/limits.ephemeral-storage, but that's disk capacity, not disk performance. A backup job writing two gigabytes to the same device your database reads from can stay well inside its storage quota while still saturating the shared device, and neither the scheduler nor ResourceQuota sees that contention at all. Kubernetes' own resource-management docs describe capacity accounting for storage; nothing performance-related. cgroup v2 has had mechanisms in this area for years; Kubernetes itself has not exposed them as native Pod resource semantics.

The mechanism I went in planning to use was io.latency: a cgroup v2 interface that takes a target completion latency for one cgroup and, once that cgroup misses its target, throttles peer cgroups whose own latency target is higher (via queue-depth throttling and, for some I/O, artificial delay). That's a mechanism for protecting a workload toward a configured latency target, not a hard SLA, but it maps almost directly onto "protect the database from the backup job."

It wasn't there. Not disabled, not zero-valued: find /sys/fs/cgroup -name "io.latency" across every cgroup on the node returned nothing at all. The node runs Debian 13, kernel 6.8.0-117-generic, inside Docker Desktop's Linux VM on macOS, and that VM's kernel doesn't build in CONFIG_BLK_CGROUP_IOLATENCY support. (io.latency's config option is separate from iocost's CONFIG_BLK_CGROUP_IOCOST, a different facility, not tested here either.)

So this post measures the fallback. I built a daemon around io.weight, ran two fio jobs against the same block device with and without it running, and recovered 52% of the protected workload's lost throughput. That number is real, and it's also the weaker half of the story, because io.weight is a ratio rather than a reservation and the benchmark shows exactly what that costs.

What io.weight actually is

Every non-root cgroup v2 group with the io controller enabled exposes io.weight as a plain text file, default 100, range 1 to 10000. Writing default 500 into it sets a proportional share of I/O service time against every device the cgroup touches; under contention, I/O service is distributed roughly by weight ratio between the competing cgroups, and the resulting bandwidth/IOPS split depends on request size, workload, and device. The two workloads below differ by 256x in request size specifically to surface that.

That answers "who gets more when everyone's competing." It doesn't answer "can this workload maintain a target latency under contention," which is the problem io.latency is designed to address. The substitution isn't cosmetic: io.weight shifts a ratio and doesn't reserve a floor, so everything measured below is a proportional-share mechanism standing in for a target-latency one. If you're building this for real, check io.latency on your actual node image before you design around it. It isn't universal, even now.

The cgroup path is not where the docs suggest

My starting assumption was the commonly cited /sys/fs/cgroup/kubepods.slice/.... On this cluster no such directory exists at that level. With the systemd cgroup driver and kubelet's cgroupRoot set to /kubelet, the live hierarchy is:

/sys/fs/cgroup/kubelet.slice/
  kubelet.service
  kubelet-kubepods.slice/
    io.max, io.weight, io.stat, io.prio.class, cgroup.subtree_control
    kubelet-kubepods-besteffort.slice/
      kubelet-kubepods-besteffort-pod<uid_with_underscores>.slice/
    kubelet-kubepods-burstable.slice/
      kubelet-kubepods-burstable-pod<uid_with_underscores>.slice/   <- write target

Guaranteed pods sit directly under kubelet-kubepods.slice/, while Burstable and BestEffort pods get an extra QoS-class slice layer in between. UID dashes become underscores in the leaf name.

The important part isn't the exact path. It's that the path is derived from cgroupRoot, QoS class, and pod UID, which is why it's a flag in the daemon rather than a hardcoded constant: this depends on the cgroup driver and kubelet's configuration, and a cgroupfs-driver cluster wouldn't follow this systemd slice naming at all.

func PodSlicePath(cgroupRoot, qosClass, uid string) (string, error) {
	if uid == "" {
		return "", fmt.Errorf("cgroup: empty pod UID")
	}
	base, err := kubepodsBase(cgroupRoot)
	if err != nil {
		return "", err
	}

	normalizedUID := strings.ReplaceAll(uid, "-", "_")
	lowerQOS := strings.ToLower(qosClass)

	switch lowerQOS {
	case "guaranteed":
		leaf := fmt.Sprintf("kubelet-kubepods-pod%s.slice", normalizedUID)
		return base + "/" + leaf, nil
	case "burstable", "besteffort":
		qosSlice := fmt.Sprintf("kubelet-kubepods-%s.slice", lowerQOS)
		leaf := fmt.Sprintf("kubelet-kubepods-%s-pod%s.slice", lowerQOS, normalizedUID)
		return base + "/" + qosSlice + "/" + leaf, nil
	default:
		return "", fmt.Errorf("cgroup: unknown QOS class %q", qosClass)
	}
}

Per-device weight returns EIO

The first write form I tried was per-device, one ratio per block device, which is the form the interface documentation describes:

$ echo "253:16 200" > .../low-priority.../io.weight
sh: 7: echo: echo: I/O error
exit code: 1
$ cat .../low-priority.../io.weight
default 10

The shell reports I/O error and the file retains its previous value, so the write was rejected outright rather than silently ignored. The cgroup-wide form, on the same file, in the same cgroup, seconds later:

$ echo 150 > .../kubelet-kubepods-burstable-podX.../io.weight
$ cat    .../kubelet-kubepods-burstable-podX.../io.weight
default 150

The scheduler is the wrong place to look. In cgroup v2 the file named io.weight is registered by the iocost controller (CONFIG_BLK_CGROUP_IOCOST); BFQ exposes its own separate file, io.bfq.weight. So vdb running mq-deadline rather than BFQ (/sys/block/vdb/queue/scheduler shows none [mq-deadline]) doesn't by itself explain the rejection. The likelier candidate is -EOPNOTSUPP, which the per-device path returns when the policy isn't enabled for that device: blk-iocost.c has no EIO return at all, and sh: echo: I/O error is the shell's strerror output rather than a captured errno. I didn't capture the raw errno, so I can't map it to a specific return path.

What I can say is what reproduced: the per-device write fails and the cgroup-wide write succeeds, twice, in separate sessions. The daemon writes the cgroup-wide form exclusively, because it's the only form I was able to apply successfully on this kernel and device.

The daemon

It polls two pods, reads an io-qos.demo/tier annotation (high/low, no PriorityClass wiring), maps tier to weight, and writes it.

The write-then-verify pattern below exists directly because of the EIO above: a write can fail depending on form, so the daemon reads the value back rather than trusting that the write call succeeded.

func (d *Daemon) reconcileOne(ctx context.Context, t Target) error {
	pod, err := d.Client.CoreV1().Pods(t.Namespace).Get(ctx, t.Name, metav1.GetOptions{})
	if err != nil {
		return fmt.Errorf("get pod: %w", err)
	}

	if pod.Status.Phase != corev1.PodRunning {
		log.Printf("qosd: %s/%s: phase=%s, waiting", t.Namespace, t.Name, pod.Status.Phase)
		return nil
	}

	tier, ok := pod.Annotations[TierAnnotation]
	if !ok {
		return fmt.Errorf("missing annotation %s", TierAnnotation)
	}
	weight, ok := TierWeight[tier]
	if !ok {
		return fmt.Errorf("unknown tier %q", tier)
	}

	if applied, ok := d.applied[string(pod.UID)]; ok && applied == weight {
		return nil
	}

	slicePath, err := cgroup.PodSlicePath(d.CgroupRoot, string(pod.Status.QOSClass), string(pod.UID))
	if err != nil {
		return fmt.Errorf("resolve cgroup path: %w", err)
	}

	written, readback, err := cgroup.WriteWeight(slicePath, weight)
	if err != nil {
		return fmt.Errorf("write weight: %w", err)
	}
	...
}

func WriteWeight(slicePath string, weight int) (written string, readback string, err error) {
	line, err := WeightLine(weight)
	if err != nil {
		return "", "", err
	}

	weightFile := slicePath + "/io.weight"
	if err := os.WriteFile(weightFile, []byte(line), 0644); err != nil {
		return line, "", fmt.Errorf("cgroup: write %s: %w", weightFile, err)
	}

	data, err := os.ReadFile(weightFile)
	if err != nil {
		return line, "", fmt.Errorf("cgroup: read back %s: %w", weightFile, err)
	}

	return line, string(data), nil
}

go test ./... covers path construction for all three QoS classes, weight formatting at the range boundaries, and write/read-back against a temp dir. It ran as a plain process inside nginx-lab-control-plane, cross-compiled for linux/arm64 and docker cp'd in, pointed at /etc/kubernetes/admin.conf. No DaemonSet, no image build.

The test

Two Burstable pods, low-priority-writer and high-priority-reader, running fio inside Alpine containers against a hostPath on the same device (/dev/vdb1). Low priority: --rw=write --bs=1M --size=2G --numjobs=4, heavy sequential. High priority: --rw=randwrite --bs=4k --numjobs=1 --size=256M, database-shaped.

fio has to run inside the pods rather than on the node, and that's not a stylistic choice. I/O issued from the node's own shell doesn't run inside any pod's cgroup, so the weights under test wouldn't apply to it at all. That meant a separate Alpine image with fio built into it, loaded into the kind cluster.

All three stages use --ioengine=psync rather than a queued engine. Same engine across all three runs, so it moves the absolute numbers, not the comparison.

kubectl exec high-priority-reader -- fio --name=highprio --directory=/data \
  --rw=randwrite --bs=4k --ioengine=psync --numjobs=1 --size=256M \
  --time_based --runtime=60s --output-format=json --group_reporting

Stage	IOPS	Bandwidth (KB/s)	clat mean (µs)	clat p99 (µs)	% of solo ceiling
1. Solo baseline	163,061	652,244	4.5	9.0	100%
2. Contended, no daemon	71,255	285,019	11.4	7.6	43.7%
3. Protected, daemon running	118,970	475,879	6.5	6.9	73.0%

Before the daemon, both pods sat at the kernel default:

$ cat .../kubelet-kubepods-burstable-pod938cca59_..._.slice/io.weight
default 100
$ cat .../kubelet-kubepods-burstable-pode24d2cb2_..._.slice/io.weight
default 100

Daemon log, after reconciling both:

2026/08/09 04:15:39 qosd: default/low-priority-writer uid=938cca59-498a-41ef-86d0-80c39c1d8a83 tier=low qos=Burstable path=/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod938cca59_498a_41ef_86d0_80c39c1d8a83.slice/io.weight wrote="default 10" readback="default 10"
2026/08/09 04:15:39 qosd: default/high-priority-reader uid=e24d2cb2-c787-4862-8a73-7f3e4d93ba92 tier=high qos=Burstable path=/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pode24d2cb2_c787_4862_8a73_7f3e4d93ba92.slice/io.weight wrote="default 500" readback="default 500"

sequenceDiagram
    participant Low as low-priority-writer<br/>(2G sequential write)
    participant Dev as /dev/vdb1
    participant High as high-priority-reader<br/>(4k randwrite)

    Note over Low,High: weight 100/100
    Low->>Dev: heavy sequential I/O
    High->>Dev: competes equally
    Note over High: 71,255 IOPS

    Note over Low,High: weight 10/500
    Low->>Dev: throttled share
    High->>Dev: proportionally favored
    Note over High: 118,970 IOPS

What the numbers say, and what they don't

Distance from the solo ceiling went from 56.3 points contended to 27.0 points protected, which is 52% of the contention-induced gap closed. That's real. It's also half.

The remaining 27 points are the cost of the substitution. A 50:1 weight ratio changes who wins the contest for bandwidth without guaranteeing anyone a floor, so under sustained sequential writes from the low-priority pod the high-priority pod still pays real tax even with 50 times the competing weight. That's what a proportional-share mechanism does when the workload underneath it doesn't stop, and it's the specific failure mode io.latency addresses by targeting a number instead of a ratio.

Latency moved further, relatively, than IOPS: 4.5µs solo, 11.4µs contended, 6.5µs protected. The contended row's own p99 (7.6µs) sits below its mean, which looks like an error until you check the tail: p50 for that run is 1.18µs and max is 740ms, an extremely heavy-tailed distribution where a handful of massive outliers (queued behind the low-priority pod's sequential writes) drag the mean far above the 99th-percentile bucket. Both numbers are real, straight from the same fio run; the mean is dominated by rare, huge stalls that the percentile itself doesn't capture.

The p99 column has a second oddity the table makes plain: the solo run has the worst tail of the three, 9.0µs against 7.6µs contended and 6.9µs protected. p99 gets better as contention is added. With one run per stage and a heavy tail in all three, the 99th percentile here isn't measuring what the mean is measuring, and I wouldn't argue anything from that column. The mean is the number that tracks the weight change.

The low-priority pod didn't lose anything under weight 10. It improved on every axis: 614 to 820 write IOPS, 629 to 840 MB/s, mean latency 6.5ms down to 4.8ms. Both metrics move together because its block size is fixed at 1M, which makes IOPS and bandwidth the same measurement in different units. Aggregate throughput across both pods went from 914 to 1,316 MB/s, up 44%. A proportional-share mechanism redistributes a roughly fixed amount of service; it doesn't hand both competitors a third more of everything, and that the supposedly-starved pod came out ahead is a reason to treat the recovery figure as observational rather than as arbitration doing its job.

All three runs are ext4 buffered writeback on a virtio block device inside Docker Desktop's VM, and 163k IOPS for 4K random writes is writeback cache absorption rather than physical device throughput. None of the absolute numbers transfer to real storage. The workload, device, filesystem, and fio parameters were held constant between stages 2 and 3; the daemon changed the cgroup weights. The runs were sequential rather than randomized or repeated, so this establishes an observed 52% recovery in this experiment, not an isolated, controlled measurement of the weight change as the sole variable: page cache and writeback state carry over between sequential runs on the same device. That's still the number worth trusting more than the six-figure IOPS counts around it, which are cache-absorption artifacts first and device behavior second.

io.max as a second lever

The same controller exposes io.max, keyed by $MAJ:$MIN with rbps/wbps/riops/wiops fields, which caps a cgroup outright instead of ranking it against others:

$ echo "253:16 wiops=200" > .../low-priority.../io.max
$ cat .../low-priority.../io.max
253:16 rbps=max wbps=max riops=max wiops=200

It applied cleanly, which makes it the only per-device write form on this kernel that worked at all. A production version could combine both, using io.max as a hard ceiling on device I/O for known offenders and io.weight to arbitrate everything else, with the cap sized against the workload's own latency sensitivity, since the kernel documents that I/O is delayed once a cgroup hits its io.max limit, and a tight cap on a latency-sensitive workload trades one failure mode for another. A cap and a ratio solve different problems, and neither one alone gives you both.

Scope

Single kind node, no cross-node coordination tested. Neither io.latency nor iocost was benchmarked, since both are absent on this kernel. No network-attached storage; whether any of this reaches an NFS or iSCSI volume is a separate question this doesn't touch. Tier assignment is annotation-driven, reading a fixed annotation off two named pods rather than watching scheduling.k8s.io/v1 cluster-wide.

This is an observational experiment, not a controlled benchmark: one node, one virtual block device, one workload pair, one sequential execution order, one measurement per stage. It demonstrates that changing io.weight coincided with a substantial improvement for this workload on this kernel and device. It doesn't quantify the general performance benefit of io.weight, and it doesn't rule out sequential-run cache/writeback drift as a partial contributor to the specific numbers above.

What it would take to go further

The path resolution logic already treats cgroupRoot as a flag rather than a constant, which was the right call given how much it varies. Making tier assignment PriorityClass-driven instead of annotation-driven is a smaller change than it looks, since the daemon already reads pod.Status.QOSClass off the live object, so adding a PriorityClass watch is additive rather than a redesign.

The bigger open question is io.latency. Any real deployment needs a capability check for CONFIG_BLK_CGROUP_IOLATENCY before committing to it, because some nodes will have it built in and some won't. A daemon that assumes universal availability fails the same way this build's first attempt at per-device weights did: quietly, and only on specific kernels. (iocost, CONFIG_BLK_CGROUP_IOCOST, is a separate mechanism and shouldn't be treated as interchangeable with io.latency.)

Which is the actual lesson here, and it's about interface names rather than about my benchmark. io.weight gives you arbitration, io.max gives you a ceiling, and io.latency gives you a target. Those are three different control semantics wearing one label, and calling all of them "I/O QoS" hides the distinction until a benchmark forces it back out. A production controller shouldn't start by picking a policy. It should start by checking which of the three the node underneath it can actually make.

One cgroup budget for a Tekton taskrun's step and its dind sidecar

Bhawani Singh — Tue, 18 Aug 2026 00:00:00 GMT

A Tekton TaskRun that builds container images runs as one pod holding one or more step containers, alongside a privileged Docker-in-Docker sidecar. A step issues docker build and docker run against the sidecar's daemon socket. The dockerd process inside the sidecar is what actually does the work: pulling layers, running RUN instructions, spawning child containers. During a build, the sidecar is the busiest thing in the pod.

The two limits have to be sized independently, so the pod requests the sum of two peaks that never coincide: 4 GiB for the step, 4 GiB for the sidecar, 8 GiB reserved against node capacity for a workload that never exceeds 4. One shared ceiling would fit the actual usage.

The step and the sidecar sit in sibling cgroups under the same pod slice. Their paths are identical except for a container hash, so a one-line substitution in the sidecar's entrypoint is enough to point dockerd at the step's cgroup instead of its own. Everything the build creates is then charged to the step's limit, and the kernel holds that limit all the way to an OOM kill. The same nesting that makes it work is what erases the build container from the metrics every memory dashboard is built on.

The problem

The unit of budget you want here is the pod: give this whole TaskRun 4 GiB, let the step and the build children draw from it, and stop paying for a peak that never happens. Kubernetes has that concept. PodLevelResources, KEP-2837, sets spec.resources on the pod itself. It went beta and default-on in 1.34, and is still beta in 1.36, which graduates InPlacePodLevelResourcesVerticalScaling to beta on by default. Two further gates around pod-level resources, PodLevelResourcesFixDefaulting and PodLevelResourcesFixKubeletQOSClass, land in 1.37.

But Tekton can't use it. A TaskRun's podTemplate is a fixed allowlist of pod-spec fields: env, nodeSelector, tolerations, affinity, securityContext, volumes, runtimeClassName, automountServiceAccountToken, dnsPolicy, dnsConfig, enableServiceLinks, priorityClassName, schedulerName, imagePullSecrets, hostNetwork, hostUsers, hostAliases, and topologySpreadConstraints. There's no resources. The other lever, computeResources, rewrites the resources block of individual containers, which is per-container by construction and says nothing about the pod as a whole.

So a platform team that wants one number per TaskRun pod is stuck between a Kubernetes feature that's still beta and a Tekton API surface that doesn't expose it.

The options

Four candidates get the step and the build sharing one ceiling.

PodLevelResources. Set the budget on the pod and let the kernel divide it. Correct, but unavailable in Tekton's podTemplate and still beta.

PID migration via cgroup.procs. Write the sidecar's own dockerd PID into the step container's cgroup.procs, moving the daemon itself under the step's budget. This charges dockerd's own footprint, including image pulls and layer extraction, to the step, which is a different and much blunter thing than charging the build's children.

--cgroup-parent borrowing. Start dockerd with --cgroup-parent pointed at the step container's scope, so every container dockerd creates is nested inside the step's cgroup while dockerd itself stays where the kubelet put it.

Writing memory.max into the pod slice. Have something with node access compute a budget and write it directly into kubelet-kubepods-*-pod<uid>.slice/memory.max, bypassing the API entirely.

I tested --cgroup-parent borrowing. It's the only one of the four that both works inside a normal pod today and targets the build's children specifically rather than the daemon that spawns them.

The chosen method

The sidecar's entrypoint runs before dockerd and does four things: read its own cgroup path, identify its own pod, ask the API server for the step container's containerd ID, and substitute that ID into the path it read.

The critical property is that dockerd's own PID never moves. The daemon stays in the sidecar's cgroup, where the kubelet put it and where the kubelet's own accounting expects it. Only the containers it goes on to create land under the step.

Step one reads the sidecar's own cgroup from procfs, which under cgroup v2 unified hierarchy is a single 0:: line:

$ awk -F: '/^0::/ {print $3}' /proc/self/cgroup
/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-podcd57a787_d809_4a55_b6f8_ab38de73aeb8.slice/cri-containerd-4354439d2f1668c37baaabaa5a33b5ce71a1b24c182e4ebe0d16455889c14617.scope

Steps two and three read the pod name from /etc/hostname and the namespace from /var/run/secrets/kubernetes.io/serviceaccount/namespace, then poll the API server for the step container's ID, filtering out the sidecar's own entry:

kubectl get pod "$podname" -n "$namespace" -o json | \
  jq -r 'first(.status.containerStatuses[] | select(.name != "sidecar-dind" and .started == true) | .containerID)' | \
  awk -F'://' '{print $2}'

Step four is the whole trick. The sidecar's cgroup path and the step's cgroup path are siblings under the same pod slice, differing only in the container hash, so one substitution converts one into the other:

NEW_CG_PATH=$(echo "$CGROUP_PATH" | sed -E "s/cri-containerd-[a-f0-9]+\.scope/cri-containerd-$CONTAINERD_HASH.scope/")

Then dockerd starts with that path as its parent:

exec /usr/local/bin/dockerd-entrypoint.sh \
  --cgroup-parent="$NEW_CG_PATH" \
  "$@"

The sequence, with the step's own commands arriving after dockerd is already reparented:

sequenceDiagram
    participant Init as sidecar entrypoint
    participant API as kube-apiserver
    participant D as dockerd (sidecar cgroup)
    participant Step as step container
    participant CG as step .scope cgroup

    Init->>Init: read /proc/self/cgroup
    Init->>API: get pod, read step containerID
    API-->>Init: cri-containerd-<step hash>
    Init->>Init: sed sidecar hash -> step hash
    Init->>D: exec dockerd --cgroup-parent=<step scope>
    Step->>D: docker run / docker build
    D->>CG: create child cgroup under step scope
    Note over CG: child's memory charges here

One detail makes the substitution viable, and it's worth checking before copying any of this. The two sides of the pod run different cgroup drivers:

$ kubectl exec <pod> -c sidecar-dind -- docker info --format '{{.CgroupDriver}} / v{{.CgroupVersion}}'
cgroupfs / v2
$ kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq -r .kubeletconfig.cgroupDriver
systemd

The kubelet builds the systemd-style .slice/.scope paths the sidecar reads out of procfs, while dockerd inside the sidecar runs cgroupfs and treats --cgroup-parent as a literal directory to create beneath. That mismatch is what lets a path lifted from the kubelet's hierarchy be handed to dockerd unchanged. A dind daemon configured with the systemd driver would expect a slice name and derive its own scope instead, and the borrowed path wouldn't nest the same way.

Two prerequisites beyond that. The sidecar's service account needs get on pods in its own namespace, because the container ID is only available through the API. And the sidecar needs privileged: true, both for dockerd's normal reasons and because without a host cgroup namespace the container reads only the namespace-relative 0::/ and has no host path to rewrite.

What the kernel does with it

The nesting is literal. After the step tells dockerd to run a container that writes a 200 MB file, that container's cgroup appears as a subdirectory of the step's scope on the node:

$ ls .../cri-containerd-84c3e193...abe1.scope/
4c4095d78188d1ba957135bd72bf672eec1de8a40ed8e011e3b30fb1a811d247
cgroup.controllers
cgroup.subtree_control
...
memory.current
memory.max
...

The kernel charges it to the step. Reading the step scope's memory.current directly on the node, with a 200 MB child running:

$ cat .../cri-containerd-84c3e193...abe1.scope/memory.current
213598208

That's roughly 203.7 MiB against a step container whose own resident footprint before the child started was about 7 MB. The charge landed exactly where the mechanism aimed it.

That happens because of one file. A cgroup only accounts for a resource if its parent enabled that controller in cgroup.subtree_control, and the step's scope lists memory as available without enabling it:

$ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc
$ cat .../cri-containerd-84c3e193...abe1.scope/cgroup.subtree_control
cpuset cpu pids
$ cat .../4c4095d78188...247/cgroup.controllers
cpuset cpu pids
$ cat .../4c4095d78188...247/memory.current
cat: ...: No such file or directory

So the child gets no memory.* files at all, which makes it no boundary for memory accounting: the pages its processes instantiate are charged to the nearest enclosing cgroup that is one, and that's the step's own .scope. The mechanism works precisely because the child is not a memcg.

There's a second reason, and it's the stronger one. Reading the child's cgroup.procs returns Operation not supported, which the kernel documents for threaded cgroups, and cgroup.type confirms it. Probing a separate run for those values: the child reads threaded, the step scope reads domain threaded, and a step scope with no dind child beneath it yet reads a plain domain. Threaded cgroups carry only the threaded controllers, cpu, cpuset, perf_event, and pids, and memory isn't among them. The two facts are the same one seen from either end: delegating only threaded controllers is what let the child become threaded at all, and once threaded it can never carry a memory.current. Its processes show up in cgroup.threads instead.

That's the whole finding, and everything an operator would reach for follows from it. There is no per-child memory object, so nothing reading cgroup memory files can report one. cadvisor's line for the child reads zero on working set, usage, and RSS alike:

container_memory_working_set_bytes{container="",id=".../cri-containerd-84c3e193...abe1.scope/4c4095d78188...247",...} 0
container_memory_usage_bytes{container="",id=".../4c4095d78188...247",...} 0
container_memory_rss{container="",id=".../4c4095d78188...247",...} 0

docker stats inside the sidecar, looking at the same container through the daemon that created it, agrees:

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
4c4095d78188   memhog    0.00%     0B / 0B             0.00%     1.32kB / 126B   0B / 0B     1

Neither is wrong. There's no memory.current at the child to read, so both correctly report the absence as zero.

The step's own line is the one that misleads, because it does move, just nowhere near enough. cadvisor's container_memory_working_set_bytes for step-step went from about 7.2 MB before the child to about 13.9 MB with it running: roughly 6.4 MiB of movement against a kernel charge of roughly 204 MiB. Those are sampled gauges, so the individual readings drift a few hundred kB between runs, but the ratio is stable at around thirty. The metric captures about a thirtieth of the charge.

One footnote on where the 200 MB went. The step scope's memory.stat shows anon 6541312 and file 200101888, all of it inactive_file. A container writing a blob to disk charges the step as reclaimable page cache, not anonymous memory, which is why that particular workload never triggers a kill.

What happens when it OOMs

Forcing a kill needs anonymous memory. With the step capped at 64Mi via stepSpecs.computeResources and a child allocating 500 MiB of anon, the kernel does exactly what the budget says.

The kill is a real memcg kill, not a node-pressure eviction, and the cgroup it names is the step's:

oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/docker/b132849958c0.../kubelet.slice/.../kubelet-kubepods-burstable-podee4ed01f_c5b6_401d_84e6_cd45e7a94890.slice/cri-containerd-d5a62de9d463d23595850010701c41f8cd07c14626727281560162300d0f7d98.scope,task_memcg=...,task=entrypoint,pid=79431,uid=0
memory: usage 65536kB, limit 65536kB, failcnt 59

The oom_memcg path carries this run's pod UID (ee4ed01f_c5b6_401d_84e6_cd45e7a94890) and this run's step container hash, both matched against what kubectl reported for the same pod. usage 65536kB, limit 65536kB is the 64Mi cap hit exactly, with 59 prior failed charge attempts.

The kernel then killed five processes, not one, because memory.oom.group is set on the step scope:

process	pid	total-vm	anon-rss	oom_score_adj
entrypoint	79431	1301884 kB	5760 kB	996
script-0-fp877	80150	1720 kB	0 kB	996
sleep	80152	1704 kB	0 kB	996
entrypoint	79452	1301884 kB	5760 kB	996
python3	80829	523528 kB	58084 kB	0

The python3 at oom_score_adj: 0 is the child inside dind, the process that actually invoked the killer. Everything at 996 is Tekton's own step machinery: two entrypoint processes, the generated step script, and its sleep. The kernel log states the rule plainly, that tasks in the step scope "are going to be killed due to memory.oom.group set." The step is billed for the child's overrun and executed for it.

Note that the killer only got 58084 kB of anon resident before hitting the wall, against a total-vm reservation of 523528 kB. The 500 MiB is what it asked for, not what it held.

Tekton usually reports the failure, but not as an OOM:

$ kubectl get taskrun cgroup-budget-run-oom -o jsonpath='{.status.conditions}'
[{"lastTransitionTime":"2026-08-18T08:32:12Z","message":"\"step-step\" exited with code 137: Error","reason":"StepFailed","status":"False","type":"Succeeded"}]

This is the nuance worth getting right, because the obvious reading is wrong. Tekton does have OOM-specific reasons: TaskRunReasonStepOOM ("StepOOM"), TaskRunReasonSidecarOOM, and TaskRunReasonInitContainerOOM all exist in the v1 TaskRun types. Tekton's getFailureInfo() calls isOOMKilled(), which tests s.State.Terminated.Reason == "OOMKilled" and nothing else. It never looks at the exit code. The step's terminated state here was exitCode: 137 with reason: "Error", so the check correctly declined to fire.

The interesting part is one layer down, because Tekton isn't the component that decides. containerd writes that string, and it reaches the opposite conclusion from the opposite evidence: on a 137 exit it checks whether the cgroup's memory.events counter shows an oom_kill, and only then sets the reason. So containerd gates on the exit code Tekton ignores, and Tekton gates on the string containerd may or may not get around to writing.

It usually doesn't. Polling the step scope's own memory.events in a busy loop through the kill catches the counter arriving and the cgroup disappearing almost simultaneously:

09:17:12.938 | oom_kill 5 | oom_group_kill 1 | current=774144
09:17:12.940 | directory gone

Two milliseconds. At a 20 ms sampling interval the directory was already gone. The kernel's accounting is correct and hierarchical throughout: the pod slice's memory.events reads oom_kill 5 while its memory.events.local stays at 0, which is exactly what a kill charged to a descendant should look like. The counter is there to be read. Under the systemd cgroup driver, the scope unit is garbage-collected as soon as its last process exits, and containerd is racing that GC to read a file that is about to stop existing. containerd's own source comments name this race.

Repeat the same OOM six times and the reason comes back Error five times and OOMKilled once, with exitCode: 137 every time. So StepOOM isn't dead code and this isn't a Tekton bug: it's a race one layer below Tekton that Tekton faithfully reports the losing side of. An operator sees the same generic failure most of the time and the correct one occasionally, which is worse than either being consistent.

There's a trap in verifying any of this. The kernel ring buffer inside a kind node belongs to the host VM, not the cluster, and it survives cluster deletion. Create a fresh cluster and dmesg on its node already holds OOM lines from clusters that no longer exist: 20 of them here, including a python3 kill reading anon-rss:57964kB, close enough to the live number 58084kB to pass a careless eyeball check. Grepping dmesg | grep -i oom and reading whatever comes back will happily "confirm" a several-hour-old event from a deleted cluster. Three distinct oom_memcg values sat in that buffer and only one belonged to the pod under test. Extract oom_memcg and match it against the current pod UID and step container hash, or the number you quote is fiction.

The remaining tests:

Test	Result
Path substitution	The rewritten path's pod UID and 64-char container hash match what `kubectl` reports and what appears on the node's cgroupfs. Not a byte-for-byte string match: `crictl` returns runc's colon form (`...slice:cri-containerd:<hash>`) while the script builds the systemd-unit form (`.../cri-containerd-<hash>.scope`).
RBAC denied	The sidecar's first `kubectl get pod` returns `Forbidden` and the script dies immediately. `set -eo pipefail` on line 2 plus a piped assignment means the non-zero exit propagates out of the pipeline and terminates the script on the first iteration, before the "retrying" message. Retry count is 0 despite `max_retries=50`. The sidecar exits 1 after about a second. The `TaskRun` reports `Succeeded`.
Unprivileged sidecar	`/proc/self/cgroup` reads a bare `0::/`, the namespaced view, so the `sed` finds no `cri-containerd-<hash>.scope` to replace and silently no-ops. The script logs `New cgroup path: /` and proceeds. That silent no-op was never observed to matter, because dockerd dies first on `mount: permission denied (are you root?)` and `Could not mount /sys/kernel/security.`. Sidecar exits 1, `TaskRun` reports `Succeeded`.
Lifecycle inversion	Doesn't happen. dockerd's own PID never leaves the sidecar's cgroup, so Tekton's ordinary nop image-swap stops the sidecar exactly as it would any other, a few seconds after the step exits.

Why not to run this

The enforcement is real, and the operability isn't, and those are separable properties.

Nothing downstream of the kernel sees the charge at the granularity it happened. Dashboards built on container_memory_working_set_bytes stay roughly flat while the kernel accumulates two hundred megabytes against the step, and kubectl top consumes the same cadvisor-derived series, though this cluster had no metrics-server installed so that path wasn't exercised directly. Anything keyed on that per-container series would inherit the same gap: an autoscaler or a sizing recommender reading step memory sees a number roughly thirty times too small. Neither was deployed here, only the cadvisor series underneath them. Recovering the truth means reading memory.current on the node, which means node access and a scrape path that doesn't exist by default.

Tekton's status surface can't distinguish a step that overran its own budget from a step that was killed for a child's. Both usually arrive as StepFailed with exited with code 137: Error, and memory.oom.group means the same set of Tekton processes dies either way, so nothing in the process-exit shape tells them apart. The occasional run that does win the race and report StepOOM is no better for this purpose: it correctly says the step was OOM-killed, which is still the wrong container to go looking at. An on-call engineer reading only the TaskRun has no signal pointing at the build container, and an intermittent reason string is harder to build an alert on than a consistently wrong one.

The pod also still reserves two numbers, not one. dockerd's own PID stays in the sidecar's cgroup under the sidecar's own limit, so the 8 GiB of the opening never collapses all the way to 4. What changes is their size: the sidecar's cadvisor line read 26.9 MB idle and 42.8 MB while a 200 MB child ran, because the child's memory is charged elsewhere. The second reservation covers a daemon's own footprint instead of a peak build, and can be sized in tens of megabytes rather than gigabytes.

You can't drop privileged: true later as a hardening pass. Without it the sidecar reads 0::/ instead of a real cgroup path, the substitution has nothing to match, and the daemon would start with --cgroup-parent=/. dockerd never gets that far, dying on a mount failure first, but nothing in the script detects or reports the broken path either way.

Both failure modes leave a dead sidecar inside a green TaskRun. RBAC denial and the unprivileged case both end with the sidecar at exit code 1 and the TaskRun at Succeeded / All Steps have completed executing. The step ran, passed, and was never subject to the budget the pipeline thought it had. There is no annotation, condition, or event distinguishing an enforced run from an unenforced one.

There's a related Docker issue, and it's worth being precise about how it differs. moby/moby#45378, DinD cgroupv2 problem inside K8s, filed 2023-04-21 and still open, reports containers inside DinD exceeding the pod memory limit without being OOM-killed, an enforcement failure not seen on cgroup v1. That's the opposite half of what shows up here, where enforcement works and attribution is missing. Both point at nested cgroup accounting under DinD being fragile, in different directions.

Everything above ran on a single-node kind cluster, kind v0.32.0, node image kindest/node:v1.36.1, Kubernetes v1.36.1 on Debian 13 with containerd 2.3.1, kernel 6.8.0-117-generic on aarch64, cgroup v2 unified hierarchy, and Tekton Pipelines v1.15.x. The sidecar is docker:28-dind (Docker 28.5.2, Alpine 3.22) with apk add bash kubectl jq on top, since that base image ships no bash and the entrypoint needs it. The step is alpine:3.20. Load was generated by kubectl exec into the sidecar and running docker run against its own dockerd, on a disposable local cluster and nothing resembling production. That deliberately tests the plain container path: docker build reaches the same daemon, but BuildKit places its workers on its own terms, so treat the placement result here as proven for containers dockerd creates directly and unverified for every build topology.

What to use instead

PodLevelResources is the answer as soon as your cluster is on 1.34 or later, where it's beta and enabled by default. It puts the limit on the pod, and it keeps the kubelet's accounting and the kernel's accounting pointing at the same object. It's still beta in 1.36, with two fix gates landing in 1.37, so read the release notes before you build a platform commitment on it. What it leaves unsolved is delivery: spec.resources still has to reach the pod, and the TaskRun's podTemplate allowlist won't carry it. Use mutating admission webhook to patch spec.resources on the pod and you have the shared budget.

Writing memory.max into the pod slice from a node-level agent is the conservative fallback. It needs node access and hardcodes the kubelet's slice naming, which varies with the cgroup driver and cgroupRoot, but it works on any version and needs no API changes. It also charges the whole pod, sidecar included, which for a dind pod is usually what you wanted anyway.

None of this is really about Tekton or Docker. cgroup.subtree_control decides which level owns a resource, and Kubernetes doesn't delegate memory below the container cgroup. That makes enforcement and observability separable, which is the part worth remembering: the kernel will happily hold you to a budget nobody can see you spending.