# Recovering macOS builds when the controller or agent stops. Killing the Mac agent during `xcodebuild build` left its VM clone on disk. After restart, the agent deleted that clone and admitted another attempt. The build started twice and finished once. Killing the controller mid-build caused no second attempt: the agent kept working and deleted the VM. [Part 3](/blog/tekton-macos-build-vm-lifecycle) recorded each attempt in a host journal before updating cluster status. Recovery reads the last recorded transition to decide whether to repeat the build, finish deletion or return an undelivered result. The slot remains reserved until VM deletion is confirmed. The controller runs as a Deployment in the cluster, while the agent runs as a LaunchAgent on the Mac, with one VM slot per agent. If the agent stays down, Tekton can time out while its VM still exists. Recovery must enforce cleanup independently of the process that ran the build. ## The controller can stop without losing the build The controller creates the MacBuild idempotently by name and leaves VM allocation to the agent. After restart, it resumes tracking the build from cluster status. With the controller pod force-deleted mid-build, a replacement pod was running in under a second, and the PipelineRun `Succeeded` on one attempt. With the Deployment scaled to zero for ~1m, the agent finished the build, published the artifact and journaled `Retired` while the controller was down: ```text 02:34:31.738 scale controller to 0 02:35:09.149 agent journaled Retired with the controller down 02:35:21.578 scale controller to 1 ``` A fresh controller read the finished status and wrote the CustomRun `True/TestsPassed` 0.75s after the scale-up. That interval uses controller log timestamps because API stamps have whole-second precision. ## An agent restart can mean a second build That recovery path works because the controller does not own the guest process. An agent restart loses the original `tart exec` session, and the restarted agent cannot reconnect to it. I chose to stop and delete the old VM before allowing another attempt. Recovery records an interrupted build as `Interrupted`. An outcome already recorded at `Retiring` is preserved while cleanup finishes. The agent owns retries because it knows whether the previous attempt's VM has been deleted. The controller rejects Tekton `retries > 0` to keep a second retry mechanism from repeating the same job. Process cleanup also depends on how the agent was started. Its LaunchAgent configuration leaves `AbandonProcessGroup` unset, so [launchd kills the remaining processes in its group](https://github.com/apple-oss-distributions/launchd/blob/main/man/launchd.plist.5) when the agent dies. In the mid-build case, the agent was killed with `kill -9`. Within 1s, both `tart run` and `tart exec` had stopped, but the clone remained on disk. With `KeepAlive` also unset, the agent stayed down until `launchctl kickstart` restarted it. Recovery then deleted the clone before admitting another attempt, as the journal shows: ```text #120 a1 Booting -> Running (GuestReady) tart exec ready after 13.91s 02:41:40.304 driver kill -9 agent pid 18107 02:41:58.395 driver launchctl kickstart (restart) #121 a1 Running -> Retiring (AgentRestarted) found Running at agent start; killed 0 tart process(es) outcome=Interrupted #122 a1 Retiring -> Retired (VMDeleted) deleted (or already absent) at agent start #124 a2 Requested -> Admitted (SlotFree) slot 1/1, deadline 2026-10-05T02:56:06.653639Z ... #130 a2 Retiring -> Retired (VMDeleted) stopped and deleted in 6.03s ``` In this abridged excerpt, `a1` and `a2` identify attempts and `#NNN` is the journal sequence. Attempt 2 was admitted just after attempt 1 reached `Retired`. A second recovery case, covered below, held the slot until a failed VM deletion succeeded, then admitted another attempt. Deleting the VM before retrying prevents the old allocation from overlapping the new one. It cannot undo work the first attempt already performed. Execution is at least once, so publishing or signing must tolerate repetition. Rollback of external side effects was not tested. Retries are bounded by `maxAttempts`: 2 attempts by default, configurable from 1 to 3. Each retry uses the original deadline and needs at least 120s remaining before it can start. The CustomRun result includes the completed attempt's id (`...#2` in this case), identifying which attempt produced the result. Only an `Interrupted` outcome permits a retry. Build and test failures do not trigger retries. Recovery of a failing build was not exercised in these runs. The journal showed no overlapping admitted allocations in the tested cases. In both retry cases launchd had already removed the old processes, so recovery killed none. Recovery also kills surviving processes before it permits another attempt. That cleanup ran in a live case, but no retry followed, so these runs did not test both actions together. In the other restart case, the agent died 0.1s after recording `Retiring` with outcome `Succeeded`. Recovery finished deleting the VM and preserved that outcome without running the build again. The CustomRun stays `Unknown/DeletingVM` during cleanup, as in [Part 1](/blog/tekton-customrun-macos-builds). The mid-build kill and this case each ran once, under launchd. ## The VM is gone; Tekton still needs the result A finished build can still need recovery when its result has not reached Tekton. The agent syncs journal transitions before submitting status to the API. A crash between those writes can leave the journal ahead of the cluster. Terminal records include the status snapshot needed to submit it again. Recovery resubmits that saved status only if the cluster still identifies the same attempt with an older journal sequence. An admission policy refused terminal `macbuilds/status` writes. The journal reached `Retired` while the API still showed `Running/Collecting`. After the agent was killed and the policy removed, the restarted agent read the terminal record from the journal and submitted the status again: ```text #218 a1 Collecting -> Retiring (TestsPassed) 2/2 tests passed outcome=Succeeded ... ERROR status ... refused, will retry: ... forbidden: ValidatingAdmissionPolicy 'mac-ci-deny-terminal-status' ... #219 a1 Retiring -> Retired (VMDeleted) stopped and deleted in 5.01s kill -9 agent pid 7205 INFO recovery: status behind journal (record 219); delivering it again ``` Recovery decided to redeliver 0.08s after restart, and the CustomRun became `True/TestsPassed` within the same second. This used an injected policy rather than network loss. If writes remain refused past the controller deadline, Tekton receives a timeout even though the host journal records `Succeeded`. ## Tekton can time out before the VM is gone Redelivery needs the agent to return. If it remains unavailable, the controller has to bound how long Tekton waits without treating silence as evidence of VM deletion. The agent renews a Kubernetes [Lease](https://kubernetes.io/docs/concepts/architecture/leases/) every 5s, with a 20s duration. The agent that claimed the MacBuild is its holder. An expired holder lease tells the controller that renewals have stopped. It stops no process on the Mac. Normal completion waits for `VMDeleted=True`. If the holder's lease has expired and cleanup remains unconfirmed, the controller waits until `timeoutSeconds` plus a 45s grace period has elapsed. It then ends the CustomRun as `False/CustomRunTimedOut` with result `vmRetirement=unconfirmed`. The CustomRun deadline is measured from the CustomRun's start. The host execution deadline is measured from the agent's first admission, so the two deadlines can differ. On timeout, the controller requests MacBuild deletion, but the agent's finalizer keeps the object until the agent returns and finishes cleanup. Cancellation leaves the same uncertainty about cleanup when the agent's lease has expired. The controller then reports `CustomRunCancelled` with VM deletion unconfirmed. Ending the CustomRun does not delete the VM. A separate LaunchAgent on the Mac runs the agent executable in `watchdog` mode, checking VM deadlines every 15s and deleting overdue VMs. The build agent writes a per-VM deadline file before `tart clone`, so the watchdog can find overdue VMs without reading the journal or taking the agent's lock. It allows 60s after the host execution deadline before deleting a VM. Without that separate process, cleanup waits for agent recovery. To check cleanup without the build agent, `SIGSTOP` paused that agent while its guest processes continued. In the `nohup` cases, killing the agent left those processes alive because launchd did not manage them. The watchdog's record and CustomRun result for the paused-agent case show separate completion times. This excerpt is abridged, with the agent identifier normalized: ```text (abridged) {"time":"2026-10-05T02:56:40.151497Z","vm":"...","deadline":"2026-10-05T02:55:36.40012Z","action":"deleted","detail":"killed 0 tart process(es); stopped and deleted in 1.18s"} False/CustomRunTimedOut: MacBuild did not finish within timeout 100s + 45s grace; VM retirement unconfirmed: agent agent-a lease expired (last renewed 2026-10-05T02:54:20Z); the MacBuild is kept until the agent returns result vmRetirement = unconfirmed ``` All three CustomRuns ended before their VMs were removed. The final column measures cleanup from the host execution deadline: | Case | CustomRun terminal | VM removed | CustomRun earlier by (±1s, whole-second stamp) | After host execution deadline | |-------------------------------------------------|--------------------|------------------------|------------------------------------------------|----------------------------------| | Agent frozen (SIGSTOP), watchdog on | 02:56:21 | 02:56:40.151, watchdog | 19.2s | 63.75s | | Killed, agent started with `nohup`, watchdog on | 02:59:43 | 03:00:11.609, watchdog | 28.6s | 73.52s | | Killed, agent started with `nohup`, no watchdog | 03:07:32 | 03:08:27.910, recovery | 55.9s | 100.19s (still running at 91.6s) | Expected watchdog cleanup is 60s grace plus up to 15s until its next check, then deletion. The two deletions took 1.18s and 0.82s, giving an approximate range of 61 to 76s. Both observed totals fell inside it. Agent-driven deletion took 5 to 8s, so slower deletion extends the range. These timings cover an awake Mac. Sleep was not tested. Without the watchdog, cleanup waited for the agent restart at 03:08. In both `nohup` cases, compilation finished, but no tests ran. Recovery recorded `Failed`, using `WatchdogDeadline` after watchdog cleanup and `DeadlinePassed` without it. ## A VM that cannot be confirmed gone keeps its slot A watchdog can attempt deletion while the agent is unavailable, but deletion can still fail. Recovery cannot release a slot until it confirms the VM is gone. If deletion fails, the attempt enters `Quarantined`, a state that keeps the slot reserved while cleanup remains unresolved. With `tart delete` made to fail, the MacBuild stayed `Pending` and identified the VM: ```text #261 a1 Running -> Retiring (AgentRestarted) found Running at agent start; killed 0 tart process(es) outcome=Interrupted #262 a1 Retiring -> Quarantined (DeleteFailed) could not delete at agent start, slot quarantined: tart delete mb-default-... INFO MacBuild ... Pending: attempt 1 not confirmed stopped; its VM mb-default-...-c49f62f4 is quarantined #263 a1 Quarantined -> Retired (Resolved) clone directory gone (deleted on retry); slot released ``` Once deletion succeeded and the slot was released, the agent admitted attempt 2, which completed the build and tests. Quarantine costs availability: with one slot, another build cannot start while the old VM remains unaccounted for. The diagram adds that recovery state to the cleanup transitions from [Part 3](/blog/tekton-macos-build-vm-lifecycle): ```mermaid stateDiagram-v2 [*] --> Requested [*] --> Quarantined Running --> Retiring Retiring --> Retired Retiring --> Quarantined Quarantined --> Retired ``` Only an injected delete failure was tested. A clone with the agent's `mb-` prefix and no open attempt is quarantined as `UnknownVM` and left to an operator. The next MacBuild waited until the operator deleted it. ## Cancellation and competing claims need separate checks Cleanup ordering assumes that one agent owns the attempt. A host lock and fenced API writes enforce different parts of that ownership: | Mechanism | What it prevents | Boundary | |----------------------|--------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------| | Host lock | `flock` on the state directory prevents a second agent from writing the same journal; a paused agent keeps the lock | Separate state directories have separate locks | | Fenced status writes | A JSON patch `test` checks `resourceVersion` for the first claim, then the attempt id for later claims and status writes | Rejects a stale status write but cannot stop a process acting on the host | An API probe rejected a stale attempt's status write. A live stale writer was not exercised, so that check does not establish that a disconnected process cannot repeat an external side effect. The remaining cases checked competing claims, brief loss of agent renewals, cluster unavailability and cancellation: | Case | Observed outcome (API stamps whole seconds) | VM and allocations afterwards | |-----------------------------------|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------| | Kind node paused 75.2s | Agent kept the VM; artifact upload succeeded on retry; `Succeeded`, attempt 1 | `Retired`; whole kind node paused, so Kubernetes timers also stopped | | Agent hung (SIGSTOP 45s) | `HostUnreachable` condition on the CustomRun while the agent was frozen; build continued, same attempt `Succeeded` | `Retired`; 0 overlapping | | Two agents, two samples | The loser's claim on the MacBuild fails, journaled as `ClaimLost`; it clones nothing; either can win | One VM; the two agents used separate state directories, so the claim, not the host lock, separates them | | Two controller replicas | Both tried to create the MacBuild; `AlreadyExists` left one object (one sample) | One attempt | | PipelineRun cancelled during boot | Cancel took effect 3.8s later, when the clone and start step returned | `Retired`; `CustomRunCancelled` | | Owner deleted, agent dead | `Retired` on return, `Cancelled`, not retried | MacBuild held by finalizer until return | ## What this covers and what it does not These recovery cases ran on one Mac connected to a kind cluster, with one slot per agent and cold clones. Every case above ended with the VM removed and no overlapping admission. Most paths ran once, with two runs for the competing-agent case. Kubernetes timestamps have whole-second precision. Cancellation with an expired holder lease and downstream handling of `vmRetirement` were not exercised. Mac sleep or reboot, a one-sided partition, multi-slot hosts and a launchd job with `AbandonProcessGroup` set were not tested. ## A terminal result does not release the host's slot A controller restart can recover from cluster status. An agent restart needs the host journal to decide whether to finish cleanup, submit a saved result or retry an interrupted build. Tekton may stop waiting before VM cleanup finishes. The host slot is reusable only after VM deletion is confirmed. That rule can leave the only slot blocked until cleanup succeeds or an operator removes an unknown VM. [Part 5](/blog/starting-macos-builds-in-seconds) keeps a fresh VM booted ahead of a request to reduce startup time. That VM occupies capacity before a build arrives and is still deleted after use. The next question is whether the faster start justifies the slot and memory it consumes.