Pod Startup Forensics: The Tooling Gap
Series · Pod Startup Forensics
Post 2 of 3
Series · Pod Startup Forensics
Post 2 of 3
- Pod Startup Forensics: The Problem
- Pod Startup Forensics: The Tooling Gap
- Pod Startup Forensics: The Idea
Table of Contents
kubectl get events gives raw chronology with no phase model. kubelet_pod_start_sli_duration_seconds gives a cluster-wide aggregate with no per-pod resolution. Nothing sits between them. Kubernetes observability isn’t short on eBPF-based tooling, though, and eBPF is exactly the mechanism you’d reach for to explain why a phase blocked. So it’s worth checking, tool by tool, what each one actually does today, not what its name suggests it might do.
Tetragon: Real syscall visibility, no concept of a phase
Tetragon is Cilium’s eBPF-based runtime security and observability tool. It hooks kernel functions directly and streams events for process execution, system call activity, and file/network I/O. Every event carries a process_exec block identifying the process that generated it, a node_name and time, and, because Tetragon is Kubernetes-aware, the pod, namespace, container, and labels the process belongs to. That’s real, fine-grained, per-process attribution, and it’s exactly the kind of signal you’d want during a slow init container: which syscall, which process, which pod, down to the timestamp.
What Tetragon doesn’t have is any notion of a phase to attach that signal to. Each event type, process_exec, process_exit, process_kprobe, process_uprobe, process_tracepoint, is a standalone record. Correlation across events happens through shared process IDs and parent/child exec_id relationships, not through any built-in lifecycle or phase concept, and nothing in the documentation describes or gestures at one. Tetragon will tell you a process inside a given pod called connect() and give you the pod identity and a timestamp. It won’t tell you that call happened during the init-container phase rather than the main-container phase, because it has no documented model of “phase” at all. That’s a Kubernetes pod-lifecycle concept, and Tetragon operates one layer below it, at the process and syscall level. Getting from a Tetragon event stream to “this pod’s startup was slow because of X” requires bringing in a phase model from somewhere else, Tetragon has none, and hand-correlating a syscall trace against it. Right primitive, missing the other half.
Pixie: Deep runtime tracing, built for services that are already running
Pixie (px.dev) is a CNCF sandbox project, also eBPF-based, that automatically instruments HTTP, gRPC, DNS, and SQL traffic without code changes, builds service maps from the traffic it observes, and produces CPU flame graphs and resource profiles broken down by pod, node, and namespace. It stores everything it collects locally in-cluster, in memory, which keeps it fast and avoids exporting data externally. That also means its data is short-lived by design, not a long-term store.
Pixie’s entire value proposition is built around instant visibility into services that are already handling traffic: request/response bodies, query latencies, who’s calling whom. None of that maps onto pod startup. A pod that’s still in its init-container phase isn’t serving HTTP, issuing SQL queries, or generating the kind of application-layer traffic Pixie’s protocol parsers are built to detect automatically, so there’s nothing for its tracers to attach to yet. Its resource-profiling and flame-graph features apply to a pod’s CPU behavior once the main container is up and running, not to the sequence of schedule, image pull, PVC attach and mount, and init-container execution that happens before that point. Pixie is a strong answer to “why is this running service slow right now” and has no documented model of the pod-startup path, because that was never the problem it set out to solve.
Parca and Pyroscope: Continuous profilers, sampling code that’s already executing
Parca and Grafana Pyroscope are both continuous profilers, and both work on the same basic principle: sample stack traces from running processes at a fixed rate and aggregate them into flame graphs over time. Parca’s agent, also eBPF-based, samples user-space and kernel-space stack traces 19 times per second and discovers targets across Kubernetes and systemd automatically, with no code changes or restarts required. Pyroscope does the same continuous aggregation across a longer language list, Go, Java, Python, Ruby, Node.js, .NET, Rust, plus eBPF-based system-level profiling, and correlates its profiles with a project’s existing metrics, logs, and traces.
Both tools answer “where’s this process spending its CPU time right now, and how has that changed over the last hour.” Sampling stack traces at a fixed frequency only produces a meaningful flame graph once a process has run long enough to accumulate samples across representative code paths. That’s a statistical technique, and pod startup is a one-shot sequence of distinct phases (schedule, pull, attach, mount, init, ready), not a steady-state workload to sample against. Neither tool’s documentation mentions process startup, container initialization, or anything resembling a “pod startup phase.” That tracks: a continuous profiler answers questions about ongoing CPU cost. It has no documented mechanism for asking why a container took four seconds to go from Created to Started, because that question isn’t about steady-state CPU time. It’s about a specific, short, one-time sequence of blocking operations.
The generic platforms: Broad coverage, no startup-specific model
The remaining category is the generic Kubernetes observability platform: the kind that bundles logs, metrics, traces, and often eBPF-based profiling into one product and sells breadth. Metoro is representative. It combines eBPF-based kernel telemetry, APM-style dashboards and service maps, log collection, and continuous CPU/memory profiling into what it describes as seven integrated signals (logs, metrics, traces, profiling, Kubernetes events, resources, deployment context), installed with a single Helm chart. That’s a real, wide net, and it explicitly counts Kubernetes events as one of its signal types. It’s still a production-monitoring and incident-detection platform at its core, built around dashboards, alerting, and automated root-cause analysis for services under load, not a dedicated model of the schedule-to-ready sequence a pod goes through before it starts serving anything.
OpenObserve sits a layer further out: a unified logs/metrics/traces/RUM store, OpenTelemetry-native, built and marketed primarily on cost and query performance rather than any Kubernetes-specific diagnostic feature, and that focus makes it a genuinely strong place to centralize telemetry cheaply at scale. It’s infrastructure you’d point other tools’ telemetry at, not a pod-lifecycle tool by itself.
The shape repeats across this category: broad signal coverage, real engineering behind each individual signal, and no dedicated concept of a pod-startup phase anywhere in the product. They’re built to answer “what’s wrong with my running system,” which is a different question, asked at a different point in a pod’s life, than “which of these six startup phases was slow, and why.”
What’s actually missing
Line these up and the shape of the gap gets sharper, not vaguer. Tetragon has the syscall-level ground truth but no phase model to hang it on. Pixie has deep runtime tracing but nothing to trace before a pod’s main container is serving traffic. Parca and Pyroscope have continuous sampling profilers built for steady-state code, a different shape of problem entirely from a one-shot startup sequence. The generic platforms have breadth without a startup-specific model anywhere in that breadth. Every one of them is a legitimately strong tool at what it was built for, and every one of them was built for a question other than which pod, which second, why. That’s not four small gaps between four adjacent categories of tool. It’s one gap, looked at from four angles.
