# Starting macOS builds in seconds instead of half a minute

Cloning the VM for a macOS build takes 0.02s. The first `xcodebuild` command runs about 26s after the request. Most of that wait is VM startup, until the guest answers `tart exec`. Guest preparation and source checkout follow.

The Mac agent from [Part 3](/blog/tekton-macos-build-vm-lifecycle) and [Part 4](/blog/tekton-macos-build-recovery) now boots a fresh VM before a build request arrives and keeps it ready in a pool. The next eligible job uses that VM, skipping the wait for startup. After the job finishes, the VM is deleted and never returned to the pool. On an idle Mac, direct MacBuild requests reach the first build command in 4.7s instead of 26.0s (medians).

Keeping a VM ready occupies one of the Mac's two VM slots and adds about 10 GiB to the combined resident memory (RSS) of its VM processes. In separate Tekton PipelineRun measurements, the first build command starts at 30.40s cold and 9.27s warm (medians), about 21s sooner. The saving comes from moving VM startup ahead of the request; it does not imply faster compilation.

Six-job batches also had a lower median completion time with the pool enabled. But the slot using the warm VM always finished before the slot running only cold VMs. The startup saving alone does not explain the batch result.

## What the first build command waits for

The cold path starts with a 0.02s clone. The agent then watches `tart run` for an early process exit for 5s. The agent's 5s watch is included in the cold-path timing and does not establish guest readiness. Readiness probing follows: the guest takes a further 16.3s (median) to answer `tart exec`.

Guest preparation adds 2.8s across several `tart exec` round trips to check the clock, Xcode version and workspace, and write a marker file. Source checkout follows, taking 1.2s.

I chose fresh clones booted ahead of time so each build receives a VM that has never run another build. This avoids state left by previous builds, though the clone still inherits the contents of the base image.

Reusing a VM would require checking that cleanup removed the previous build's state. Restoring a suspended VM is another option: [Tart](https://github.com/openai/tart) provides `tart suspend` and `tart run --suspendable`, but suspend and restore were not tested in these measurements.

Parts 3 and 4 ran one VM slot. This part uses two, with a pool of one leaving the second slot available for a build.

> **License note:** [Apple's macOS 27 agreement](https://www.apple.com/legal/sla/docs/macOS27.pdf), section 2B(iii), permits up to two additional virtualized macOS instances on an Apple-branded Mac you own or control that already runs macOS, for specified purposes including software development and testing during development. The following paragraph restricts service-bureau, time-sharing, terminal-sharing, relay and similar uses, except as expressly permitted in section 3. Section 3 separately permits developer-service leasing, including CI, under stated conditions. A hosted CI arrangement needs assessment under the applicable agreement.

A MacBuild, the [Part 1](/blog/tekton-customrun-macos-builds) resource requesting a build, takes the oldest Ready VM at admission only if it requests the pool's base image and has not opted out. The pool refills only when no MacBuild is waiting for a slot, so preparing the next VM does not take priority over queued work. The journal shows a claim below (abridged, with illustrative names):

```text
#3561 a1 Requested -> Admitted (SlotFree) warm VM mac-ci-pool-1dade0f61754 from the pool, slot 1/2, ...
#3562 a1 Admitted -> Cloning (PoolClaimed) took unused pool VM mac-ci-pool-1dade0f61754 (booted in 26.65s, idle 42.6s)
#3563 a1 Cloning -> Booting (PoolReady) commit ccfede49... from mac-ci/source@sha256:8d11de5d...; warm VM, no clone or boot
#3564 a1 Booting -> Running (GuestReady) tart exec ready after 0.71s
```

## Measuring startup on one Mac

Three moments could count as the start of a build: the clone returning, `tart exec` first answering inside the guest, and the first build command running. I used the third, when the guest starts doing the requested work. The agent stamps the guest's own clock as the first line of the build log and maps it to the host clock using one `tart exec` round trip. The offset uses the midpoint between the host timestamps before and after that call. The collector records half the round-trip duration as timing uncertainty; the converted timestamp is an estimate.

The timings cover four conditions: cold or warm (no warm pool or one pre-booted VM), and idle or loaded (a second build already running in its guest when the measured one is applied). Each has 30 runs in three groups of 10, with at least 20s between one run finishing and the next apply.

Every measured run was applied when a slot or the Ready VM was free, so queue time is 0.06s (median, p95 at most 0.09s) in every condition. All runs applied a MacBuild with `kubectl apply` on one M3 Max Mac with kind on Colima, and built a small macOS app with two XCTest tests (build about 16s to 19s, test 5s to 7s). Idle means no second build was running. The agent, local cluster, measurement processes and other host applications still ran.

## Waiting for VM readiness takes 21.5s of the 26.0s

Median / p95, 30 runs per column, successful runs only. The p95 is the 29th of 30 sorted values.

| Phase                                     | cold, idle          | cold, loaded        | warm, idle        | warm, loaded      |
|-------------------------------------------|---------------------|---------------------|-------------------|-------------------|
| apply to Admitted                         | 0.06s / 0.07s       | 0.06s / 0.08s       | 0.06s / 0.09s     | 0.06s / 0.09s     |
| Admission to first successful `tart exec` | 21.52s / 25.81s     | 24.75s / 27.58s     | 0.74s / 1.12s     | 0.50s / 1.11s     |
| guest preparation                         | 2.76s / 6.60s       | 2.77s / 5.20s       | 2.58s / 2.91s     | 2.65s / 2.77s     |
| checkout                                  | 1.15s / 1.34s       | 1.34s / 2.25s       | 1.15s / 1.25s     | 1.20s / 1.45s     |
| **apply to first build command**          | **25.98s / 29.73s** | **29.32s / 33.03s** | **4.69s / 6.04s** | **4.64s / 5.65s** |
| `xcodebuild build`                        | 16.51s / 18.54s     | 19.04s / 23.81s     | 17.51s / 20.11s   | 16.18s / 23.32s   |
| `xcodebuild test`                         | 5.49s / 6.08s       | 6.52s / 8.47s       | 6.74s / 8.04s     | 6.63s / 8.81s     |
| collect and publish                       | 1.47s / 1.65s       | 1.44s / 2.15s       | 1.54s / 2.52s     | 1.54s / 2.60s     |

Booting ahead of time removes most of the readiness wait. Guest preparation and checkout still happen after admission, accounting for most of the remaining 4.7s.

| Comparison, median difference | First build command | Interval (95%)     |
|-------------------------------|---------------------|--------------------|
| warm vs cold, idle            | -21.29s             | -23.85s to -20.68s |
| warm vs cold, loaded          | -24.69s             | -25.51s to -22.85s |
| cold, loaded vs idle          | +3.35s              | +0.47s to +4.40s   |
| warm, loaded vs idle          | -0.05s              | -0.37s to +0.39s   |

The intervals come from 10,000 bootstrap resamples of individual runs within each condition, taking the difference of medians each time. Resampling individual runs does not preserve their groups, so the intervals do not account for possible dependence among runs within a measurement group. They describe variation in the captured samples and exclude clock uncertainty and systematic differences between groups.

A second build adds about 3s to median cold startup, while median warm startup changes little in these samples. With no second build running, the groups' median cold startup times were 25.5s, 26.0s and 29.0s. The median warm startup times differed by only 0.4s between groups. Both variations were smaller than the roughly 21s startup saving from the warm VM.

## Compilation and tests were not consistently faster

Compared with cold VMs, median compilation time on warm VMs was 1.00s longer when no other build was running, and 2.87s shorter when another build was running. Median test time was 1.26s longer without another build and 0.11s longer with one.

The 95% bootstrap intervals for the compilation differences were +0.34s to +1.54s without another build, and -3.55s to -2.00s with one. Positive values mean compilation took longer on warm VMs; negative values mean it took less time.

When a job takes the ready VM and the other slot is free, the pool boots a replacement. In idle warm runs, that replacement boot overlapped a median 79% of the time from compilation starting to tests finishing. This extra work may explain the longer compilation and test times when no other build was running, but the measurements did not establish that cause.

Even so, warm jobs reached VM deletion a median 20.54s sooner than cold jobs when no other build was running (95% bootstrap interval -21.91s to -19.27s).

## The faster job did not determine when the batch finished

A ready VM removes most of one job's startup wait. The batch still has to wait for the other five jobs and VM cleanup, so its completion time depends on what happens on both slots.

Each batch submitted six MacBuilds together in one `kubectl apply`. Ten batches ran without a pool, and ten started with one VM already booted and waiting. The two setups alternated in groups. All 120 jobs succeeded.

| Metric                                                                        | No warm pool     | One pre-booted VM |
|-------------------------------------------------------------------------------|------------------|-------------------|
| median batch completion time (apply to last VM deleted), 10 batches per setup | 205.5s           | 182.5s            |
| range of batch completion times                                               | 192.8s to 245.5s | 175.4s to 199.2s  |
| mean queue wait                                                               | 69.5s            | 53.4s             |
| first job, apply to first build command                                       | 28.0s to 39.0s   | 4.1s to 4.9s      |
| warm jobs                                                                     | 0 of 60          | 10 of 60          |

Median batch completion was 23.0s sooner with the pool, though the ranges overlap: the slowest batch with the pool, 199.2s, ended after two without it at 192.8s and 194.6s. Only 10 of the 60 jobs with the pool were warm. With both slots busy and jobs waiting, the pool never refills, so each batch gets only the VM that was Ready at the start.

Each slot runs three jobs in sequence. The batch ends when the slower slot finishes its third job and deletes the VM. In all 20 batches, that slot ran only cold VMs. In the ten batches with the pool, the slot using the warm VM finished 15s to 28s earlier.

![Gantt chart of batch pool0-4: six cold jobs on two slots, ending at 192.8s and 207.4s](burst-pool0-4.svg)

*Batch pool0-4, no warm pool, completion time 207.4s. Grey is queue wait, red is waiting for the VM to answer, purple guest preparation, blue build and test, green collect and delete. Both charts use the same 0s to 210s scale.*

![Gantt chart of batch pool1-4: one warm job and five cold jobs on two slots, ending at 156.5s and 183.2s](burst-pool1-4.svg)

*Batch pool1-4, one pre-booted VM, completion time 183.2s. Job 01 is the warm job.*

To locate that difference, the table follows the three jobs on the slot that finished last in each batch. All three used cold VMs. Each job's time is split into boot (from admission until `tart exec` answers), work (guest preparation, checkout, compilation and tests), and tail (collecting results, stopping the VM and deleting it).

Each row is an average across 10 batches per setup. Adding the row differences gives a 24.2s difference in average batch completion time. The batch-results table above uses medians, giving a 23.0s difference.

| Job on the slower slot | No warm pool: slot time (boot, work, tail) | One pre-booted VM: slot time (boot, work, tail) | Difference |
|------------------------|--------------------------------------------|-------------------------------------------------|------------|
| first                  | 75.45s (30.75s, 31.83s, 12.86s)            | 60.63s (22.67s, 31.21s, 6.75s)                  | 14.82s     |
| second                 | 67.65s (25.74s, 32.54s, 9.37s)             | 61.65s (23.66s, 31.23s, 6.76s)                  | 6.00s      |
| third                  | 64.68s (24.69s, 31.66s, 8.33s)             | 61.30s (23.65s, 30.56s, 7.08s)                  | 3.38s      |

The largest difference is in the first job on the slower slot: 8.1s of boot, 6.1s of tail and 0.6s of work. Without the pool, the first two cold VMs boot at the same moment. With it, the first cold job boots next to a warm VM that is already building, and the two slots run out of step.

Across the ten pool-enabled batches, 22 cold jobs booted without another cold boot in progress and 28 overlapped another cold boot. Their average slot times were 61.5s and 62.5s, respectively. The ten warm jobs averaged 36.9s. Without the pool, the 60 cold jobs averaged 67.4s. These counts are jobs within batches, not independent batches.

For those same cold-job groups, median boot time was 25.3s with another cold boot in progress (28 jobs), versus 22.2s without overlap (22 jobs). This is an association; the runs did not isolate the effect of starting two boots together.

The overlap comparison does not explain the full 8.1s boot gap or the tail difference. The causes of the cold-job differences remain unresolved. These results cover one Mac and one batch size, with pool preparation between batches excluded from completion time.

## A warm start consumes idle capacity

| Cost                                                                            | Measured                                                                              |
|---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
| Combined RSS of Virtualization.framework VM processes, pool VM Ready vs no pool | median +9.97 GiB (range +4.42 to +10.34 GiB)                                          |
| Free-memory percentage reported by `memory_pressure`                            | median -2 percentage points (range -9 to 0 percentage points)                         |
| Swap                                                                            | within 8MB in every window pair                                                       |
| Clone to Ready                                                                  | about 25s (median 25.09s, range 19.65s to 30.24s, 87 pool VMs)                        |
| Slot-seconds per job, idle runs                                                 | about 95 for a warm job (58.5s before the claim, 36.2s as a build), 57 for a cold job |

The memory figures compare seven switches between no warm pool and one Ready VM, with no builds running. Each side was sampled every 10s for 180s. The combined RSS of the host's Virtualization.framework VM processes, including Colima, was about 10 GiB higher with the pool VM ready. Four of the seven windows began with the pool VM already running and gave +4.42, +7.93, +9.97 and +9.76 GiB. The other three gave about +10.3 GiB. The cause of the lower values is not established.

The warm job's 36.2s of slot time excludes the pool VM's occupied slot before the claim. Including preparation and waiting gives about 95 slot-seconds, a sum of medians rather than paired runs. Ready VMs waited a median 34.6s, much of it set by the 20s gap between runs. The spacing between requests contributes to this cost: the pool occupies capacity while waiting for the next build.

A pool of one takes half of the host's two slots while it waits. That capacity buys a faster start for the next eligible request. A stream that keeps both slots busy leaves no room to prepare another warm VM: each measured batch consumed its initial Ready VM and ran the remaining jobs cold. The decision therefore depends on whether faster starts between batches justify the memory and occupied slot. These timings do not establish a general throughput gain.

The earlier start still has to use a fresh VM for each build. The reuse check covered a subset of runs: 14 batches, all 20 Tekton runs and 40 direct runs with their load builds. These runs used 171 VM names, each belonging to exactly one build.

Of those VMs, 37 came from the pool. None was claimed twice, all 37 had a clean workspace before checkout, and no more than two VMs ran at a time.

## The saving survives the Tekton path

The same comparison through Tekton used 20 PipelineRuns of the pipeline from [Part 2](/blog/tekton-macos-source-and-artifacts): a Linux `prepare` task, the MacBuild CustomRun and a Linux `verify-artifact` task. Five cold, ten warm and five cold again ran on an idle Mac, with 20s between one run finishing and the next apply. All 20 succeeded and all 10 warm runs claimed a pool VM. Times start at the `kubectl apply` of the PipelineRun.

| Median / p95                    | cold, Tekton (10) | warm, Tekton (10) | cold, direct (30) | warm, direct (30) |
|---------------------------------|-------------------|-------------------|-------------------|-------------------|
| apply to MacBuild Admitted      | 4.64s / 5.39s     | 4.72s / 5.45s     | 0.06s / 0.07s     | 0.06s / 0.09s     |
| apply to first build command    | 30.40s / 34.47s   | 9.27s / 10.26s    | 25.98s / 29.73s   | 4.69s / 6.04s     |
| apply to VM deleted             | 60.30s / 65.56s   | 40.81s / 41.96s   | 56.95s / 60.80s   | 36.41s / 39.91s   |
| PipelineRun start to completion | 65s               | 45.5s             |                   |                   |

The warm PipelineRun reaches the first build command 21.13s sooner (interval -24.33s to -19.98s), against 21.29s on the direct path. The Tekton path took about 4.6s longer than direct execution before MacBuild admission: +4.58s (+4.28s to +4.91s) cold and +4.66s (+3.96s to +5.21s) warm.

The `prepare` task takes 4s to 5s by whole-second API timestamps, which fits most of that time, though the split from the controller hop was not measured. The same 4.6s appear on both sides (10 runs each), so the pool's 21s remain. PipelineRun duration also includes the `verify-artifact` task.

## What this covers and what it does not

These timings are for one M3 Max with two VM slots, one small app and a locally cached image. Direct and Tekton runs were collected in groups rather than interleaved run by run. The Tekton comparison covered idle runs only. For a longer build, I expect the startup saving to represent a smaller share of total duration. Its size and the effect of pool refill on completion time were not measured.

A pool larger than one, an idle-age limit, resuming a suspended VM, an uncached image pull, other hardware, other macOS and Xcode versions, a loaded host on the Tekton path and a fleet of Macs were not tested.

A fresh VM booted ahead of time removes most of the startup wait for the next eligible request, including through Tekton. It costs memory and a slot while waiting. The batch results improved too, but their cold-slot differences remain unexplained, and sustained demand leaves no opportunity to refill the pool. I would use these results to decide whether faster starts between batches justify that reserved capacity. [Part 6](/blog/should-you-run-macos-jobs-from-tekton) considers whether the wider integration is worth operating.