Pod Startup Forensics: Closure and What's Next

Part 6 fixed the last correctness bug this series set out to find: a cgroup-lookup race that left most socket syscalls unattributed to a pod. This part closes the series itself: what six parts of building actually proved, and what remains an open question rather than a finding.

What got built

The problem, back in Part 1, was that a slow pod startup produces almost no explanation of where the time went. Kubernetes Events are coarse, and their retention is controlled by the cluster rather than by a durable forensic store; kubelet’s own telemetry doesn’t say why a probe kept failing. Part 2 checked whether an existing tool already closed that gap and found that eBPF observability platforms, continuous profilers, and generic APM tools each solve a related but different problem, none of them putting pod lifecycle records and kernel-level syscall evidence on one shared timeline.

The design that followed, in Part 3, was to capture both kinds of evidence and join them by pod identity and timestamp rather than treat them as separate systems a person has to correlate by hand. Part 4 split that design into three cooperating programs: a CLI that queries stored evidence and renders a timeline, an exporter that watches the Kubernetes API and writes lifecycle records, and an eBPF tracer that hooks socket syscalls in the kernel and streams what it observes.

Two problems remained once those pieces existed. The first was durability: a pod that has already been deleted is exactly the pod someone is most likely to ask about, and by then kubectl describe returns nothing. Part 5 added persistent storage and a --timeline command that reconstructs a deleted pod’s full startup history from Postgres or SQLite alone, verified by deleting a real three-pod pipeline’s pods and rebuilding their timeline afterward with no gaps.

The second was correctness in the merge itself. Building the correlation path that joins eBPF syscalls to the HTTP and TCP probe attempts visible through kubelet’s own connect() calls surfaced two real bugs rather than confirming the design worked on the first try: a connect() return code that was being read as a success signal when it was actually a non-blocking call still in progress, and a cgroup-lookup race that left the large majority of syscalls unattributed to any pod because a lookup miss was treated as final instead of retried. Part 6 fixed both: in the fixed run, every captured bind() and listen() event for a resolvable workload-pod cgroup received an identity, 195 bind() events and 17 listen() events, up from a cluster-wide pre-fix rate where most such events landed as unattributed. A concurrent two-pod test then verified the fix doesn’t cross-attribute one pod’s syscalls to another racing alongside it.

That’s what six parts actually established: a working phase-attribution pipeline that matches Kubernetes’ own lifecycle ground truth, a syscall-to-pod correlation path verified isolated under concurrent load, and a durable storage layer that survives pod deletion, each backed by a captured before-and-after run rather than a description of the intended behavior.

What was never claimed

None of that’s the same as knowing how the system behaves at production scale. Every test in this series ran against a single-node kind cluster with a handful of pods at a time. Two scaling questions matter enough to name directly instead of leaving implicit, and neither one has been answered by anything this series actually did.

Postgres under sustained write load

The tracer records each correlated syscall event to a local buffer on the node and forwards it to the exporter, which attempts to persist it to Postgres or SQLite. That local buffer exists precisely because the forwarding path isn’t guaranteed to keep up: on a busy node during this series’ earlier storage testing, unrelated background syscall traffic drove that local backlog past 60,000 unsent rows, and replaying all of it in one allocation on reconnect contributed to a real OOM kill, a separate incident from the memory ceiling discussed below. The fix bounded the replay to 2,000-row chunks rather than one unbounded read. The code’s own comment on that fix names the exporter’s Postgres pool as a suspected contributor to why the backlog built up, but nothing in this series independently measured Postgres itself while that happened: no pool metrics, no connection counts, no insert-latency capture. The confirmed finding is the local buffer’s own unbounded-replay bug, now fixed; whether Postgres write throughput was actually a bottleneck that day, as opposed to just downstream of a noisy node, was never isolated and measured on its own.

That distinction matters for what’s still genuinely untested: sustained insert throughput, connection pool behavior, and query latency under deliberate, sustained concurrent write pressure, the kind a cluster running hundreds of pods starting at once would produce. Every test in this series that touched Postgres checked correctness, not load: did the right rows land, did a --timeline reconstruction after deletion match what actually happened, did a regression run produce output identical in shape to the original. None of them measured the database under sustained write volume. A platform team adopting this design for that kind of cluster would be the first to find out whether the write path holds up, because nothing here ever generated that load on purpose or measured it directly. Postgres write-volume behavior under sustained high-frequency ingestion remains untested as its own question, separate from the local-buffer bug this series already found and fixed.

The tracer’s syscall-rate ceiling

The eBPF tracer, unlike the Postgres question, does have a real finding behind it: the tracer restarted mid-test after hitting its configured memory limit during Part 6’s two-pod isolation test, and that restart is worth the detail here as a real, measured limit rather than a footnote.

That test needed two pods generating deliberately different, high-volume syscall patterns at the same time, to prove the cgroup-retry fix wouldn’t cross-attribute one pod’s events to the other. The first attempt ran both workloads as fast as the shell would allow, with no pacing. The tracer was OOMKilled, confirmed on the node with dmesg, showing the tracer process killed for exceeding a 384Mi memory limit set on its container, with anonymous resident memory around 391MB at the moment of the kill (the quoted dmesg line survives in this project’s running build notes; no separate raw capture file was saved for it, unlike the numeric captures elsewhere in this series). The test had destroyed its own instrument before it could produce a result.

No allocation profiler ran during the OOM, so what actually consumed the memory was never measured directly. What’s known is that the run exceeded its budget. The kernel ring buffer that delivers raw events is fixed in size, and the outbound gRPC stream is bounded too, so a higher event rate fills those and causes stalling or dropped events rather than growing their own allocation without limit. The more plausible candidates are the parts of the pipeline that aren’t capacity-bounded: the unbounded in-process handoff that moves each event from the ring-buffer reader to the correlator, and the correlator’s own in-memory state, the map of syscalls entered but not yet matched to their exit, and the retry buffer holding events whose cgroup lookup hasn’t resolved yet, both of which can grow with however many events are in flight at once rather than draining at a fixed rate. The correlator also holds a separate in-memory index mapping every pod and container cgroup on the node to its identity, refreshed every few seconds, so pod and container count are candidate contributors too, not just event rate, and this test didn’t isolate one from the other.

The two unthrottled loops in this test were deliberately artificial rather than production-shaped; this series never measured how their syscall rate compares with a real pod startup’s. The response was to pace the test workloads rather than raise the memory limit again to make the symptom go away. With pacing, the same two-pod test completed correctly, with no cross-attribution between the two pods, which is the finding Part 6 actually reported; the tracer still restarted once mid-test after hitting the same limit, which is why that capture verifies final attribution rather than an uninterrupted trace.

The honest framing of that result is narrower than “the tracer has a memory problem,” and narrower still than a general operating threshold. For this specific artificial workload against this specific container’s 384Mi budget, the tracer hit a real ceiling, not evidence that its memory usage is unbounded or unpredictable in ordinary operation. What it doesn’t establish is where that ceiling sits for a production-shaped workload, or against a different memory budget, because the workload that hit it was constructed deliberately to be unrealistic, not sampled from a real deployment. Where the safe operating envelope actually is, and whether 384Mi is a reasonable default cgroup limit for a real cluster’s pod-startup traffic rather than just this test’s own two artificial loops, is future validation work, not something this series measured.

Other open questions, named plainly

A few smaller items surfaced during the build and were deliberately left alone, because fixing them was out of scope for the specific problem each part was solving.

Static and mirror control-plane pods still don’t resolve. kube-apiserver, kube-controller-manager, kube-scheduler, and etcd, when run as static pods, get a cgroup slice path derived from a kubelet-computed hash rather than the UID the Kubernetes API reports for the same pod. The correctness fix in Part 6 retries a cgroup lookup that initially misses; it doesn’t help here, because the mismatch isn’t a timing problem retrying would fix. It’s two different identifiers for the same pod that never converge no matter how long the tracer waits. The race Part 6 closed and this identifier mismatch are different bugs entirely.

Reacting to pod-create events directly, instead of relying on a periodic refresh, was considered and set aside. The cgroup-to-pod index that the correlator consults refreshes on a timer rather than in response to the Kubernetes API announcing a new pod. The exporter already watches the Kubernetes API for lifecycle records, but the cgroup index is the tracer’s own separate structure with no watch of its own. Wiring a second, tracer-side watch stream specifically for that index was a real alternative, and was set aside because it adds its own reconnect and backoff handling for a latency win that isn’t actually guaranteed: the cgroup directory a new pod’s syscalls need to resolve against can still lag behind the watch event announcing the pod, so the added complexity might not even close the gap it’s meant to close. In this series’ own single-node testing, the timer-based refresh interval narrowed that lag enough that the retry fix (see Part 6) resolved every captured pod-scoped bind(), listen(), and accept4() event; a watch-based design stays a real option that wasn’t pursued rather than a known necessity that was skipped.

The default noise-filtering threshold has only been checked against one workload’s timing. Part 6’s filtering pass chose 10 milliseconds as the cutoff below which routine bookkeeping syscalls collapse into summary lines. That number wasn’t arbitrary: the one test scenario it was checked against showed 643 calls landing under the threshold and exactly two calls landing above it, at 10.16ms and 14.28ms, so 10ms cleanly separated the two groups for that capture. Whether that same threshold cleanly separates noise from signal for a container doing heavier synchronous I/O during startup, where ordinary reads might legitimately take several milliseconds without being noise, is untested. The threshold is exposed as a command-line flag specifically so a workload that needs a different value doesn’t require a code change, which is a reasonable hedge against the untested case, not a claim that the default generalizes.

Exec probes don’t fit the tracer’s existing correlation signal. Every probe correlation this series built keys off kubelet dialing the pod over a socket, so its connect() shows up as a kernel event the tracer can already observe and match to a probe attempt. An exec probe runs a command inside the container instead, so that particular kubelet-side connect() signal doesn’t exist for it. That’s not the same as saying no host-side signal could ever work; it means the existing correlation mechanism doesn’t cover this case, and building one that does, most likely tracing the probe process’s own execution inside the container, is separate work nothing in this series attempted.

The tracer has only been exercised on one kernel. Every capture ran on Docker Desktop’s Linux VM kernel through kind, not on a production worker-node kernel. The hooks use tracepoint formats validated on that kernel, but tracepoints aren’t a stable Linux ABI; attachment points, fields, and offsets may differ elsewhere. Compatibility with production nodes such as EKS or Amazon Linux 2023 remains untested.

What this series was and wasn’t

What six parts actually produced is a working mechanism for joining Kubernetes lifecycle records to kernel-level syscall evidence, proven correct at the scale it was tested at, because each fix was caught by testing against a real cluster instead of trusted on paper. The questions that only show up at a different scale, or on infrastructure this series never touched, are named here rather than left implicit: sustained database write volume, the tracer’s own memory ceiling, exec-probe correlation, and eBPF portability to a production node kernel.

The next work is validation: load-test sustained database writes and tracer memory under realistic multi-pod, multi-node startup traffic; verify the tracer on production kernels; and add a correlation path for exec probes. Those tests should report observed behavior, not assumed design intent.

I’ll continue evolving the tool and, when it’s ready for production use, I plan to release it under an open-source license so others can evaluate it, adapt it to their clusters, and contribute improvements.