Pod Startup Forensics: Surviving Deletion

A pod that started slowly is often gone by the time anyone asks why. kubectl describe returns NotFound; its Events may linger briefly, but on a retention schedule unrelated to the pod itself. The evidence needed to explain the startup has become a race against deletion and garbage collection.

To answer after deletion, this exporter has to capture the evidence while the pod exists and reconstruct it from durable storage later. This part shows the storage design behind that reconstruction and the --timeline command that reads it back.

Capture it before deletion

The exporter writes the observations it needs to reconstruct a startup to a database. Postgres is the deployment choice for shared, durable history; SQLite offers the same --timeline behavior in one local file for development or a single-node demo. With either storage option, the owner, child, pod, phase, and event records form one hierarchy, so a deleted pod can still be found under its original PipelineRun, Job, or CronJob.

What actually gets stored

For the deletion reconstruction in this part, two stored record types carry the core answer: phase records for duration and an ordered log of Kubernetes Events for explanation. Phase records retain timings that Events can’t provide, especially container start and finish times. Events retain the verbatim reasons, messages, and retry history that explain delays outside those phase boundaries. --timeline joins both when it reconstructs a pod.

The output preserves the precision it actually captured. A populated Event eventTime can carry microseconds, while the older firstTimestamp and lastTimestamp fields are whole-second values; --timeline doesn’t invent fractional precision for the latter.

Reconstructing a PipelineRun after its pods are deleted

The payoff is a timeline that remains readable after the pods are gone. A three-TaskRun Tekton PipelineRun produced three pods on a kind cluster. One deliberately blocked for about thirteen seconds against a non-routable address before giving up, giving the timeline a known slow phase to find. All three pods were then deleted directly:

1
$ kubectl delete pod -l tekton.dev/pipelineRun=multi-taskrun-run-slow
2
$ kubectl get pods -l tekton.dev/pipelineRun=multi-taskrun-run-slow
3
No resources found in default namespace.

kubectl has nothing left to say about any of these three pods. A direct query against Postgres confirms the same thing from the storage side: all three pod rows carry a non-null deleted_at, with their phases and events still attached:

1
         owner          |                   child                    |                      pod                       | deleted | phases | events
2
------------------------+--------------------------------------------+------------------------------------------------+---------+--------+--------
3
 multi-taskrun-run-slow | multi-taskrun-run-slow-first-task-run      | multi-taskrun-run-slow-first-task-run-pod      | t       |      3 |      7
4
 multi-taskrun-run-slow | multi-taskrun-run-slow-second-task-run     | multi-taskrun-run-slow-second-task-run-pod     | t       |      4 |     18
5
 multi-taskrun-run-slow | multi-taskrun-run-slow-third-task-run-slow | multi-taskrun-run-slow-third-task-run-slow-pod | t       |      3 |      7
6
(3 rows)

With no matching live pods in the PipelineRun’s namespace, --timeline reconstructs the stored history alone:

1
$ profiler-cli lookup --kind tekton.dev/v1.pipelinerun \
2
    --name multi-taskrun-run-slow --namespace default --timeline \
3
    --exporter-url http://127.0.0.1:19090
4

5
Owner historical timeline: default/multi-taskrun-run-slow
6
(reconstructed from durable storage — works even if every pod below is now deleted)
7
══════════════════════════════════════════════════════════════════════
8

9
Child: multi-taskrun-run-slow-third-task-run-slow
10
  Pod: multi-taskrun-run-slow-third-task-run-slow-pod  [deleted]  node=pod-startup-profiler-control-plane
11
  ──────────────────────────────────────────────────────────────────
12
    schedule                            0.335s
13
    init_container (prepare)            0.000s
14
    main_container (step-blocking-step)   13.000s <-- SLOWEST PHASE IN PIPELINE
15
  ──────────────────────────────────────────────────────────────────
16
    TOTAL                              14.000s
17
    ...
18

19
══════════════════════════════════════════════════════════════════════
20
FLAGGED SLOW: main_container (step-blocking-step) in Child multi-taskrun-run-slow-third-task-run-slow / pod multi-taskrun-run-slow-third-task-run-slow-pod took 13.000s (longest phase across the whole owner chain)

All three TaskRuns reconstruct with their pods labeled [deleted]; the excerpt shows the deliberately blocked 13-second step as the owner chain’s longest phase. A second read returned the same bytes, and a fresh three-pod run reproduced the same slow-phase flag after deletion.

When no container ever starts

The deleted PipelineRun shows that completed startup history survives deletion. A pod blocked on an unbound PersistentVolumeClaim never records a container phase, so there is no started/finished pair to time. The exporter used to drop that shape entirely. Now --timeline renders no phases recorded and shows the captured Event log.

No container started, so this timeline has no syscall or probe records. The CLI keeps its standard merged-timeline heading, but only captured Events follow:

1
Pod: pvc-failure-demo  [exists]  node=?
2
──────────────────────────────────────────────────────────────────
3
  no phases recorded — this pod never reached container start; see the event log below for why
4
──────────────────────────────────────────────────────────────────
5
  TOTAL                               0.000s
6

7
  MERGED TIMELINE (K8s Events + eBPF syscalls/probes, interleaved by real timestamp)
8
  2026-08-30T04:13:26Z  T+  0.000s  [K8S EVENT]  ExternalProvisioning   PersistentVolumeClaim/rook-ceph-stuck-pvc (x2 repeats)
9
      Waiting for a volume to be created either by the external provisioner
10
      'rook-ceph.rbd.csi.ceph.com' or manually by the system administrator. If volume creation
11
      is delayed, please verify that the provisioner is running and correctly registered.
12
  2026-08-30T04:13:26.572306Z  T+  0.572s  [K8S EVENT]  FailedScheduling       Pod/pvc-failure-demo
13
      0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found
14
  2026-08-30T04:13:26.578279Z  T+  0.578s  [K8S EVENT]  FailedScheduling       Pod/pvc-failure-demo
15
      0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found

That output names the provisioner handling the claim and records the scheduler’s immediate blocker: an unbound immediate PersistentVolumeClaim. It identifies the next component to inspect and explains why the pod can’t be scheduled. A phase-durations-only view has no row for this pod, regardless of how long it waits.

This capture stores only two repeats. A stored count can understate a persistent failure because the exporter refreshes the Event set only when it observes the pod; later Event changes can be missed if no further pod observation occurs. This run alone doesn’t show that happened here.

The previous case retained phase data; this pod never reached container start. The same lookup --timeline interface covers both shapes.

Open questions

This test doesn’t answer whether the database keeps up under sustained production load.

Part 6 follows a flagged slow phase into eBPF syscall evidence, then fixes a bug in assigning those syscalls to the right pod.

Capture it before deletion

What actually gets stored

Reconstructing a PipelineRun after its pods are deleted

When no container ever starts

Open questions

References