# Tracking a macOS build until its VM is deleted

In the cancellation run, the PipelineRun reported `Cancelled` while its macOS VM still existed. The agent still had to stop and delete that VM. A terminal PipelineRun result did not mean VM cleanup had finished.

[Part 2](/blog/tekton-macos-source-and-artifacts) checked the files moving between Linux and the guest. The host also needs a record of which VM belongs to each build and whether deletion has completed. That record stays on the Mac, where the agent can read it even when it cannot reach the cluster.

## The host keeps the build attempt's lifecycle record

A `MacBuild` disappears when deleted and cannot be read if the Mac loses its connection to the cluster. The host therefore keeps its own lifecycle journal through execution and cleanup. A ConfigMap or Lease would retain the dependency on cluster access. Neither alternative was tested.

The build outcome (`Succeeded`, `Failed` or `Cancelled`) is recorded separately from the lifecycle state. Recording a failed outcome still leaves the agent responsible for deleting its VM. In this lifecycle, states that hold a VM pass through `Retiring` before reaching `Retired`.

At this stage, the lifecycle uses nine states: `Requested`, `Rejected`, `Admitted`, `Cloning`, `Booting`, `Running`, `Collecting`, `Retiring` and `Retired`. The diagram shows their main transitions. [Part 4](/blog/tekton-macos-build-recovery) adds `Quarantined` for recovery when VM deletion remains unconfirmed.

```mermaid
stateDiagram-v2
  [*] --> Requested
  Requested --> Rejected
  Requested --> Admitted
  Requested --> Retired
  Admitted --> Retired
  Admitted --> Cloning
  Cloning --> Booting
  Cloning --> Retiring
  Booting --> Running
  Booting --> Retiring
  Running --> Collecting
  Running --> Retiring
  Collecting --> Retiring
  Retiring --> Retiring
  Retiring --> Retired
  Rejected --> [*]
  Retired --> [*]
```

A job deleted while waiting for a slot or fetching source can go straight to `Retired` without creating a VM. A failed VM delete leaves the attempt in `Retiring` for another try.

## The host records the transition before the cluster

The agent appends each transition as one JSON line and calls `sync_data()` on the journal before updating MacBuild status. A crash between those writes leaves the journal ahead of cluster status: the host has a record that the cluster has not yet received.

In the successful build, the journal recorded the outcome at `Retiring` and confirmed VM deletion at `Retired`. Excerpts below use normalized names and abbreviated digests. The journal transcript is abridged:

```text
#18 None       -> Requested  (Seen)
#19 Requested  -> Admitted   (SlotFree)       slot 1/1
#20 Admitted   -> Cloning    (SourceVerified) mac-ci/source@sha256:0f4cde83...effe493f
#21 Cloning    -> Booting    (Cloned)         mac-ci-base -> clone
#22 Booting    -> Running    (GuestReady)     tart exec ready after 22.56s
#23 Running    -> Collecting (TestsFinished)  xcodebuild test exited 0
#24 Collecting -> Retiring   (TestsPassed)    outcome=Succeeded artifactDigest=sha256:39fdda0b...8e91
#25 Retiring   -> Retired    (VMDeleted)      stopped and deleted in 6.37s
```

The `Succeeded` outcome at sequence 24 did not release the VM. Sequence 25 recorded deletion. On normal completion, the controller waits for the agent's `VMDeleted=True` before reporting a terminal CustomRun result.

These one-slot runs did not test power-loss recovery or concurrent journaling. Journal-write failure was not exercised either: the agent logs that error and continues execution without advancing the recorded state or queuing its status update. Durable recording is therefore not an enforced prerequisite for the build to continue.

## The image digest does not verify the VM disk

Before taking a VM slot, the agent checks whether the host can satisfy the requested environment. Rejection at this point avoids creating a VM that would then need cleanup. The source digest identifies the input. The request also names the image digest and Xcode version that will build it.

The agent compares the request with a host catalog, then checks the cached image and the base VM's configuration. Tart's OCI cache stores the image in a directory named by its digest. Hashing that directory's manifest checks the image digest, and hashing the base VM's `config.json` checks it against the config layer named in the manifest:

| Check                                              | Value on this host (Tart 2.40.1)    |
|----------------------------------------------------|-------------------------------------|
| Cache directory name                               | `sha256:324ea565...8fab`            |
| SHA-256 of that directory's `manifest.json`        | `324ea565...8fab`, equal            |
| Config layer in the manifest (162 layers in total) | `sha256:848163d2...4ff4`, 510 bytes |
| SHA-256 of the base VM's `config.json`             | `848163d2...4ff4`, equal            |

These checks tie the base configuration to the pulled image. They do not hash the base VM's disk, about 86 GB, which was changed after the pull when an SSH key was installed into it. Re-pulling or rebuilding the base from the pinned image on a schedule would limit drift. Hashing the disk would check its contents at the cost of reading all of it, which may be acceptable for a small base or an infrequent check. Neither mitigation was tested.

The verified configuration covers the hardware model, machine ID and sizing. The check runs before cloning because the clone's config differs from the base: in the captured clone, only `macAddress` differed, changing its file hash to `76962fd4...` from `848163d2...`.

The manifest does not carry the Xcode version. Before boot, the agent trusts the catalog's version. After boot, the guest's `xcodebuild -version` must report the requested Xcode. The MacBuild status records the checked digests and guest version (excerpt):

```yaml
status:
  environment:
    imageDigest: sha256:324ea565...8fab
    configDigest: sha256:848163d2...4ff4
    guestXcode: Xcode 27.0
```

Two requests checked rejection before cloning: an all-zero image digest, and Xcode 26.4 against a base with Xcode 27.0. Both produced `EnvironmentMismatch`, created no VM and ended the PipelineRun as `Failed` with its next task skipped. The digest case went from `Requested` to `Rejected`. The Xcode case returned this CustomRun condition (base name normalized):

```text
False / EnvironmentMismatch
requested Xcode 26.4 but base VM mac-ci-base has Xcode 27.0
```

Neither case exercised the guest's post-boot Xcode mismatch check.

## Seven PipelineRuns, no build VM left behind

Each case used a fresh PipelineRun on local kind, with the agent running as a LaunchAgent on one Mac and using one VM slot. The macOS CustomRun sat between the Linux `prepare` and `verify-artifact` tasks from [Part 2](/blog/tekton-macos-source-and-artifacts):

| Case                  | PipelineRun | CustomRun reason       | VM after the run |
|-----------------------|-------------|------------------------|------------------|
| Pass                  | `Succeeded` | `TestsPassed` (2 of 2) | Deleted          |
| Compile error         | `Failed`    | `BuildFailed`          | Deleted          |
| Failing test          | `Failed`    | `TestsFailed` (1 of 2) | Deleted          |
| Cancel mid-build      | `Cancelled` | `CustomRunCancelled`   | Deleted          |
| Fresh run             | `Succeeded` | `TestsPassed` (2 of 2) | Deleted          |
| Image digest mismatch | `Failed`    | `EnvironmentMismatch`  | None created     |
| Xcode mismatch        | `Failed`    | `EnvironmentMismatch`  | None created     |

The compile error was `xcodebuild build` exiting 65 on a type error in the app source, so no tests ran and no artifact was published. The failing test was `testToggleFailure()`: `xcodebuild test` exited 65, and the agent still published the artifact collected before the tests.

In the cancellation case, the PipelineRun reported `Cancelled` several seconds before the VM was gone, as in [Part 0](/blog/mac-build-host-private-eks) and [Part 1](/blog/tekton-customrun-macos-builds). The controller reported `CustomRunCancelled` after cleanup in this run. That reason alone is not a general guarantee of deletion. With an expired host lease, the controller can finish a cancelled or timed-out CustomRun with cleanup unconfirmed. [Part 4](/blog/tekton-macos-build-recovery) tests the timeout paths and describes the cancellation behavior. Cancellation with an expired lease was not exercised.

After every VM case, the clone directory was absent and `tart list` showed only the base VM and its two cache entries. These are single runs on one Mac.

Five attempts reached `Running`. The pass, failing-test and fresh runs also reached `Collecting`. Compile error and cancellation went straight from `Running` to `Retiring`. The two environment mismatches ended at `Rejected`. Failures during `Cloning` or `Booting` were not exercised.

The cancellation path removed its MacBuild. For the other six cases, journal and status agreed on attempt, state and sequence number. All seven attempts ended in a terminal state: five `Retired` and two `Rejected`.

## Does a new VM inherit anything from the last one?

Confirming deletion of the old clone does not establish what files the next guest begins with. Before checkout, the agent lists the guest's work directory and any marker file from an earlier run, and fails if it finds either. It then writes its own marker. All five VM runs began with an empty listing, including the fresh run immediately after cancellation.

An empty listing is useful only if the check can detect files that exist. In one clone, the check printed nothing until a work directory and marker file were created by hand. It then printed both paths. A second clone of the same base printed nothing (names normalized):

```text
== VM marker-a, before marker
== VM marker-a, after writing marker and work dir
/Users/admin/mac-ci-marker-control
/Users/admin/work
== VM marker-b (fresh clone), same check
(end)
```

The positive control shows that the empty result was not simply a check that could never find those paths. The fresh clone lacked the two paths checked. This does not rule out other state carrying over from the base. The agent's `DirtyWorkspace` rejection is covered by a unit test and was not triggered live.

## Recovery needs to read the journal

In these runs, the journal recorded the build outcome at `Retiring` and confirmed VM deletion at `Retired`. Recovery has to decide what to do when the agent stops between those records, or when cluster status has not caught up with the host. [Part 4](/blog/tekton-macos-build-recovery) follows the controller and agent failures that exercise those decisions.