Pod Startup Forensics: The Idea

Tetragon has the syscall trace. It doesn’t have a phase. The Kubernetes API has the phase. It doesn’t have the syscall trace. Every tool in the last part’s survey sits on one side of that line, and no amount of squinting at any single one of them closes the gap, because the gap isn’t a missing feature. It’s a missing join.

Two sources, two different kinds of authority

The Kubernetes API knows, with certainty, when the scheduler binds a pod to a node, when the kubelet started pulling an image and when it finished, when a container’s status flipped from waiting to running to terminated. That certainty comes from a specific place: every one of those transitions is reported by a controller that owns the transition. The scheduler reports scheduling. The kubelet reports image pull and container state. Nothing downstream has to guess, because the component doing the work is the same component filing the event.

That authority has a sharp edge. An Event is a report that something happened, not an account of what the process was doing while it happened. The gap between Pulled and the next event can be eight seconds of an init container blocked inside connect(), and the Event stream says nothing about it, because no controller is watching syscalls. It can’t say more than it knows, and what it knows stops at the boundary between phases, not at what happens inside one.

eBPF sits on the opposite side of that boundary. A tracepoint on sys_enter_connect and sys_exit_connect doesn’t know what a “phase” is, doesn’t know what a Tekton TaskRun is, doesn’t know that the process it’s watching belongs to a pod at all until something tells it. What it knows, with just as much certainty as the Kubernetes API knows its own transitions, is the syscall: which process called connect(), when it entered the kernel, when it returned, what the return code was. That’s exactly the evidence missing from the Event stream. It’s also evidence with no label. A syscall trace on its own is a wall of pid, comm, and timestamps with nothing in it that says “startup” or “init container” or “this belongs to the pod that’s currently 8 seconds into a phase someone should be looking at.”

Neither source can be patched to cover the other’s blind spot, because the blind spot isn’t a bug in either one. The Kubernetes API doesn’t lack syscall visibility by oversight; it lacks it because no controller in the chain from scheduler to kubelet operates at that level, and adding syscall tracing to a controller would be a different, much stranger design than the one Kubernetes has. eBPF doesn’t lack a phase model by oversight either; a tracepoint fires on a kernel event, not a Kubernetes concept, and there’s no phase field in a sys_enter_connect argument list to read one from. The right design isn’t “make the API smarter about syscalls” or “make eBPF smarter about Kubernetes.” It’s to keep each source doing what it’s actually good at, and build the missing piece neither one provides on its own: a join keyed on pod identity, with the API supplying the authoritative phase labels and eBPF supplying the syscall-level evidence for what happened during a phase flagged as slow.

That’s a design decision with real content, not a truism. It means the API is never asked to explain why a phase was slow, and eBPF is never asked to say which phase anything happened in. Each source answers only the question it’s actually positioned to answer, and the tool’s job is entirely in the join: watch for a phase that’s taking too long, and when one is flagged, attach the syscall evidence from that pod’s cgroup during that exact window. The output isn’t “container start took N seconds.” It’s “container start took N seconds, and the process spent nearly all of it blocked inside connect() to an address that never answered.” One source says which seconds mattered. The other says what filled them.

The pod that’s already gone by the time you’d query it

There’s a second design constraint that doesn’t come from the tooling gap directly. It comes from a fact Part 1 already established: a Tekton TaskRun pod can run its steps and terminate in under two seconds. Any design that answers “which pod, which phase, why” by querying live cluster state is structurally blind to exactly the pods most worth investigating after the fact, because by the time a human goes looking, the pod that mattered is gone and so is everything about it.

This rules out an entire category of otherwise-reasonable design: a tool that watches the API and the kernel live, holds the correlated result in memory or in a dashboard, and answers questions against whatever’s currently running. That shape works for the pod that’s slow right now, in front of you, while you’re watching. It fails the pod that was slow ten minutes ago and no longer exists, which is the more common case for anything that resembles CI. Kubernetes’ own Event objects carry a retention window independent of the pod’s lifecycle, so even the raw material for a manual reconstruction disappears on its own clock, faster than most people think to look.

So durability isn’t a feature to add once the phase-and-syscall correlation is working. It has to be there from the first design decision, because the alternative isn’t a smaller version of the same tool. It’s a tool that answers a different, less useful question: not “what happened to that pod,” but “what’s happening to pods in general right now.” Those look similar from a distance and aren’t the same product. The design has to capture and persist the correlated result as it happens, continuously, for every pod, not on demand for whichever pod someone happens to ask about while it’s still alive. That single constraint decides most of the shape below: an on-demand query design and an always-watching, always-persisting design are different architectures, not different configurations of the same one.

The shape the idea takes

Put the two constraints together and the design isn’t a single new tool bolted where the gap is. It’s four pieces, each answering to one of the constraints above, wired together by pod identity: a watcher on the Kubernetes API, a kernel-level tracer, a correlation step, and durable storage.

A continuous watcher on the Kubernetes API. Not a poller that checks in on a fixed interval, because a fixed interval can miss a phase that starts and ends inside the gap between checks, exactly the failure mode that makes a fast Tekton step invisible to anything slower than the step itself. A watch reacts to each transition as the API reports it: pod scheduled, image pull started and finished, each init and sidecar and main container’s state change, in order, as they happen. This is the source of authoritative phase boundaries and labels, and it runs whether or not anything downstream is currently interesting, because the pod that turns out to matter isn’t known in advance.

A kernel-level tracer that runs continuously, and a phase timeline that decides what to keep looking at. The tracer itself has to watch every syscall on every pod, all the time, because there’s no way to go back and capture a syscall after it already happened; nothing about the design can afford to be selective at the point of capture. What’s selective is the read side. The watcher’s own phase timeline is what decides which stretch of that stream is worth pulling: when a phase crosses a threshold that marks it as unusually slow, that’s the signal for the correlation step to pull the syscall evidence already captured for that pod’s processes during that window, instead of surfacing all of it undifferentiated. The syscall-level tracer’s job isn’t to explain everything a container does. It’s to have already captured the specific stretch of time the phase-level view goes on to flag as worth explaining.

A correlation step keyed on pod identity, not on time alone. A timestamp overlap between an API event and a kernel event isn’t enough on a real node, because a real node runs more than one pod’s processes concurrently, and two pods can be inside their own slow phases at the same moment for entirely unrelated reasons. Attributing a syscall to the wrong pod because they happened to overlap in time would produce a confident, wrong answer, which is worse than an honest “unknown.” The join has to resolve which pod a given process actually belongs to, and hold that resolution stable across a phase boundary that spans multiple wake-ups of the correlator, not just check it once and assume it still holds.

Durable storage that survives the pod’s own deletion. The output of the watch-and-correlate pipeline can’t live only in memory or only behind a live query, for the reason the pod-lifecycle constraint above already argued: the moment matters, but by the time anyone asks about it, the pod producing that moment is frequently gone. Every phase transition and every correlated syscall gets written down as it’s observed, keyed on the pod’s identity rather than on the pod object still existing, so a later lookup against a pod that kubectl has never heard of still has somewhere to go.

None of these four pieces is individually new. Watching the Kubernetes API for state transitions is what every controller in the system already does. Kernel-level tracing on syscalls is the mechanism Tetragon, Pixie, and Parca all already build on. Persisting observed state so it outlives the object it describes isn’t a novel idea in distributed systems. What’s missing from every existing tool surveyed against this problem isn’t any one of these four pieces. It’s the combination: authoritative phase labels from one source, syscall-level evidence from another, joined on identity rather than assumed from timing, and captured continuously rather than queried on demand. That combination is the actual shape of an answer to “which pod, which second, why,” for a pod that might not exist by the time anyone asks the question.

References

1. Kubernetes Event v1 API Reference
2. Kubernetes Pod Lifecycle
3. eBPF Introduction