Pod Startup Forensics: The Architecture

Part 3 ended on a design: two sources of truth joined on pod identity, written before the pod disappears. This part shows what that design became when run against a real cluster: a command run by hand, a background watcher that never stops, and a tracer that talks directly to the kernel. Every claim below comes from a command actually run and checked against the tool’s behavior.

One lookup path for a pod, and for anything that owns a pod

The core CLI path is Kubernetes-native: give it a Pod name and namespace, and it builds that Pod’s timeline directly. The timeline builder does not know or care whether the Pod came from a Deployment, Job, CI system, or a hand-written manifest.

The resolver can be extended for resources that own Pods without changing that core path. Tekton PipelineRun is one example, chosen here because it produces short-lived workload Pods. The extension finds the TaskRun Pods carrying the PipelineRun label and passes them to the same timeline builder used for a direct Pod lookup.

Tekton changes only how the CLI finds the Pod. Once it has one, the timeline path is identical. The two captures below show the direct Kubernetes path and the extension path feeding that same timeline model.

1
$ ./target/release/profiler-cli lookup --kind v1.pods --name baseline-demo --namespace default
2
Pod startup timeline: default/baseline-demo
3
  image_pull                          6.000s
4
  init_container (init-setup)         1.000s
5
  sidecar (sidecar-logger)            1.000s
6
  main_container (main)               0.000s
7
  TOTAL                               8.000s

That’s the direct path: a Pod with an init container, a native Kubernetes sidecar, and a main container. The CLI renders each interval to whole seconds; this capture reports a six-second image-pull interval. The extension path renders the same kind of phase timeline for a Pod found through a PipelineRun:

1
Pod startup timeline: default/demo-pipeline-run-run-demo-task-pod
2
────────────────────────────────────────────────────────────
3
  init_container (prepare)            0.000s
4
  main_container (step-hello)         1.000s
5
────────────────────────────────────────────────────────────
6
  TOTAL                               2.000s
7
  (pass --metrics-url http://<tracer-pod-ip-or-port-forward>:9091/metrics for eBPF root cause)

The resolver changes discovery, not timeline construction. Once it finds one or more Pods, the same Kubernetes-native timeline builder handles them all.

For this capture, Kubernetes supplied only whole-second status and Event timestamps, so the CLI’s one-second display is the appropriate precision.

A container’s status field has more shapes than “running” or “terminated”

Getting that exact match required noticing something kubectl describe pod doesn’t explicitly call out: a regular init container and a native Kubernetes sidecar report their status differently, even though Kubernetes lists them together. After a regular init container completes, its current status is terminated. A native sidecar—an initContainers entry with restartPolicy: Always, stable in Kubernetes 1.33—is normally running while it stays up alongside the main containers, although it can later restart or terminate. A tool that only looks for terminated silently drops every sidecar from the timeline: no crash, no error, just a container that never shows up in the output. The fix is checking its restartPolicy before deciding which status to trust.

A separate problem arises in calculating phase durations. Container-status timestamps are absolute wall-clock times. For a still-running sidecar or main container, the implementation represents the gap from the latest earlier phase end to its startedAt; otherwise it keeps a terminated container’s own startedAt-to-finishedAt duration. This prevents overlapping status times from being rendered as duplicated startup time.

A tracer that watches syscalls, not a wrapper around a log line

Looking up a Pod’s history and building its phase timeline can answer “which phase was slow.” Neither can answer the syscall-level “why”: Kubernetes can report a container state, reason, or message, but it does not record what a process was doing while blocked. That’s the other half of Part 3’s design: a real tracer that watches the kernel directly, not a script that reads log output and guesses.

It’s built with Aya, a Rust framework for writing eBPF programs, and runs on every node as its own background process with the permissions needed to watch the kernel. Fifteen tracepoints observe sched_process_exec plus enter and exit events for seven selected syscalls. Each syscall class was chosen to test a specific hypothesis about why a container might be stuck: opening a file, making a network connection, mounting a filesystem, reading data, or serving traffic through a socket. bind, listen, and accept4 are server-side socket operations; an outbound client commonly calls connect() instead. Paired enter and exit events let the tracer measure the duration of each selected syscall.

Once built, the finished program is copied onto a cluster node and loaded there, a point where many projects like this only work on paper. The captured run showed that the eBPF object attached successfully and emitted duration metrics for all seven traced syscalls. Here is what the tracer reported across several kinds of activity at once:

1
$ curl -s http://127.0.0.1:19191/metrics | grep pod_startup_blocking_syscall_duration_seconds_count
2
...,pod="readiness-slow-listen-demo",syscall="bind"} 1
3
...,pod="readiness-slow-listen-demo",syscall="connect"} 2
4
...,pod="readiness-slow-listen-demo",syscall="listen"} 1
5
...,pod="readiness-slow-listen-demo",syscall="mount"} 77
6
...,pod="readiness-slow-listen-demo",syscall="openat"} 280
7
...,pod="readiness-slow-listen-demo",syscall="read"} 144

Those counts prove the tracer is capturing syscall durations. They do not, by themselves, show which calls belong to the startup interval under investigation. That correlation boundary is the next problem.

Three programs, three jobs

The CLI, exporter, and tracer are separate binaries with different jobs. The CLI provides a live query and a historical query. The exporter watches and stores lifecycle evidence, then serves the historical query. The tracer captures selected syscall events and sends them to the metrics and storage paths.

For a live query, the CLI resolves a Pod from the Kubernetes API, builds its timeline, prints it, and exits. For a historical query, it instead asks the exporter for the persisted timeline of a Pod; this works after the Pods are deleted. The exporter runs continuously. It watches Pod create, update, and delete events; on a Pod update, it derives the timeline and queries retained Pod and referenced-PVC Events for persistence. The tracer runs on each node, pairs the enter and exit of its selected syscalls, and makes those events available to the metrics and storage paths.

The separation follows from their lifetimes. The captured short-lived Pod completed in about two seconds, leaving little time for an ad-hoc live lookup; once its API record is gone, only the exporter-backed historical path can reconstruct it. The exporter and tracer must keep collecting without an operator waiting at a terminal. Combining them with the one-off CLI would couple an interactive command to two always-on services with different availability and storage needs.

References

1. Aya (Rust eBPF library)
2. Aya documentation (docs.rs)
3. eBPF Introduction
4. Tekton PipelineRuns
5. Kubernetes Event v1 API
6. Kubernetes sidecar containers
7. bpf-linker