# Running macOS builds from Tekton with a CustomRun A passing `xcodebuild` command does not tell Tekton whether its disposable VM has been deleted. The build outcome and VM cleanup are separate facts. On the normal completion path, this controller keeps the CustomRun running until the Mac agent confirms deletion. Cancellation and lost-agent recovery need separate rules. The Mac agent connected in [Part 0](/blog/mac-build-host-private-eks) can receive build requests, but a TaskRun still needs a Pod. Its steps depend on Tekton's injected entrypoint and files shared inside that Pod. The agent runs guest commands through `tart exec`. Moving execution outside the Pod also moves control of the remote process and its cleanup to the integration. The macOS build therefore needs an external job with its own status and cleanup owner, rather than a TaskRun step executing on another machine. ## CustomRun gives the controller an external job to track I chose CustomRun, [Tekton's extension point for custom tasks](https://tekton.dev/docs/pipelines/customruns/), so a dedicated controller could track the external build and its cleanup while the Mac remained an outbound API client. A Go controller watches the CustomRun and creates a `MacBuild` describing the build. The Rust agent watches that object and executes the requested build inside a fresh Tart VM. It records the build outcome and cleanup status on the `MacBuild`. The controller has to define status, respond to cancellation and enforce deadlines. The agent has to execute the build and confirm cleanup. Disposable VMs could also be managed from an ordinary remote-execution step. CustomRun makes the external job's status visible to its own controller. Part 0 showed the later EKS deployment. These local cases established the earlier execution contract, with the Go controller running on the Mac with a kubeconfig for kind. The diagram uses an illustrative cluster name: ```mermaid flowchart LR subgraph cluster["kind cluster mac-ci-local (Linux node)"] PR[PipelineRun] --> TR[TaskRun: Linux Pod] PR --> CR[CustomRun] MB[(MacBuild)] end subgraph mac["Mac host (external API clients)"] CTRL[Go controller] AG[Rust agent, LaunchAgent] VM[disposable Tart VM] end CTRL -- watches --> CR CTRL -- creates --> MB MB -- watched by --> AG AG -- "tart exec: xcodebuild" --> VM AG -- "status, VMDeleted" --> MB MB -- status --> CTRL CTRL -- "condition and results" --> CR ``` ## A TaskRun expects more than a command In this one-step Pipeline, the PipelineRun references a TaskRun, which owns the Pod. Tekton replaces the step's command with its entrypoint binary. A `prepare` init container copies that binary into a shared volume. This excerpt is abbreviated, and the image reference is shortened: ```text initContainer: prepare image: command: /ko-app/entrypoint init /ko-app/entrypoint /tekton/bin/entrypoint step-uname container: step-uname image: alpine:3.22 command: /tekton/bin/entrypoint args: -wait_file /tekton/downward/ready -wait_file_content -post_file /tekton/run/0/out -termination_path /tekton/termination -step_metadata_dir /tekton/run/0/status -results arch -entrypoint /bin/sh -- -c "uname -sm; ..." ``` The original `/bin/sh -c ...` is now an argument to Tekton's binary. [Tekton's container contract](https://tekton.dev/docs/pipelines/container-contract/) describes this replacement. The paths and flags above are what v1.15.3 generated on this Linux cluster. Step ordering uses files in shared volumes and a file mounted from a Pod annotation through the downward API. Tekton reads the step result from its termination message and copies it to the TaskRun as `arch=aarch64`. `tart exec` runs the guest command without supplying the init container, shared Pod files or termination-message path. ## A node selector needs a matching worker The first failure was placement. The test TaskRun specified `xcodebuild -version`, a macOS node selector and a 120s timeout: ```yaml spec: timeout: 2m podTemplate: nodeSelector: kubernetes.io/os: darwin ``` The cluster's only node was `linux` on `arm64`. The Pod stayed `Pending`, and the scheduler recorded the mismatch: ```text Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. ... ``` After 120s the TaskRun reported `False` with reason `TaskRunTimeout`, and the Pod was gone. The step never ran. The kubelet [sets `kubernetes.io/os` from Go's `GOOS`](https://kubernetes.io/docs/reference/labels-annotations-taints/). The [Kubernetes worker-node documentation](https://kubernetes.io/docs/concepts/windows/intro/) covers Linux and Windows. [EKS Hybrid Nodes](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-os.html) lists supported Linux distributions. The selector matches an existing worker label. It does not register the external Mac as a worker. ## A Darwin binary would solve only one problem A node advertising `darwin` could clear this selector mismatch. The next dependency is the injected binary. Tekton's v1.15.3 entrypoint image lists linux/amd64, linux/arm64, linux/ppc64le, linux/s390x and two windows/amd64 builds, with no darwin entry. The extracted linux/arm64 binary had the same SHA-256 on the host and in a fresh macOS 27.0 guest. Executing it in the guest produced these errors: | Invocation in the guest | Output | Exit | |-----------------------------------------------------------------------------------------|----------------------------------------|------| | `/tmp/entrypoint -help`, from `sh` | `cannot execute binary file` | 126 | | `/tmp/entrypoint -help`, from `zsh` | `exec format error` | 126 | | `/tmp/entrypoint -entrypoint xcodebuild -- -version`, from `sh` (the shape Tekton uses) | `cannot execute binary file` | 126 | | `os.execv` from `python3` | `OSError: [Errno 8] Exec format error` | 1 | Both are arm64, but the file is a Linux ELF executable. macOS loads Mach-O, so `execve` returns `ENOEXEC`. These invocations used Tekton's published Linux entrypoint binary. A build of its Go source targeting darwin was not tested. A compatible binary would solve only the executable-format problem. A provider would still need to implement the Pod interfaces for TaskRun execution. ## The controller returns test results to Tekton A PipelineRun ran a Linux `prepare` TaskRun, then delegated the macOS task through a CustomRun referencing the `MacBuild` kind. The agent ran the tests in the VM, and the controller returned their counts as string results on the CustomRun. The CustomRun reported `TestsPassed` and two passing tests. This excerpt is abbreviated and uses an illustrative API group: ```yaml spec: customRef: apiVersion: mac-ci.example.com/v1alpha1 kind: MacBuild timeout: 10m0s status: conditions: - type: Succeeded status: "True" reason: TestsPassed message: 2/2 tests passed results: - {name: testsTotal, value: "2"} - {name: testsPassed, value: "2"} - {name: testsFailed, value: "0"} ``` The PipelineRun finished `Succeeded` in 58s. The `MacBuild` showed `VMDeleted` `True`. Deleting the PipelineRun afterwards removed the CustomRun and `MacBuild`. The agent released its finalizer. ## A build verdict does not mean the VM is gone Tekton v1.15.3 reads the CustomRun's `Succeeded` condition and named string results. It leaves CustomRun status to the custom controller. On the normal completion path, the controller maps the agent's build verdict, but holds the CustomRun at `Unknown` until the agent confirms `VMDeleted=True`. Part 0's EKS trace recorded the PipelineRun as `Cancelled` before VM cleanup completed. Those timestamps cross cluster and Mac clocks whose offsets were not measured. A terminal PipelineRun outcome alone does not confirm external cleanup. Tekton requests cancellation by setting `spec.status: RunCancelled`. The controller's cancellation path deletes the `MacBuild` and keeps the CustomRun at `Unknown` with reason `Cancelling` while the child still exists. The agent's finalizer keeps that deleting object present while it stops and deletes the VM. On a subsequent reconcile, a `NotFound` response for the child lets the controller report `False` with reason `CustomRunCancelled`. Tekton requires that exact reason to count the custom task as cancelled. The controller interprets the child's absence in the context of the cancellation request. It relies on the agent releasing its finalizer only after cleanup, rather than independently checking the VM. Absence alone cannot distinguish completed cleanup from a child whose finalizer was removed outside that protocol. These local cases tested the agent's half of that path by deleting a `MacBuild` during execution. The controller's PipelineRun cancellation path was reviewed in source, but was not exercised here. Waiting for cleanup also needs a limit when the agent stops responding. [Part 4](/blog/tekton-macos-build-recovery) extends the controller to end a timed-out or cancelled CustomRun when the agent's lease has expired. It reports `vmRetirement=unconfirmed` because it cannot confirm that the VM is gone. ## The VM was gone after success, failure, and deletion The successful integration above exercised the PipelineRun path. The following four cases exercised `MacBuild` directly, using a pinned commit of a small macOS app with two XCTest tests and one VM slot on kind. They did not test failure or cancellation through a PipelineRun. The agent ran as a launchd LaunchAgent in the user's GUI session. In this guest, `tart exec` used the auto-logged-in user's GUI session for `xcodebuild test`. Running without that session was not tested. | Case | `MacBuild` outcome | Tests | VM afterwards | |-----------------|-----------------------------------------------------------------------------|---------------|---------------------| | Pass | `Succeeded`, `TestsPassed` | 2 of 2 passed | Deleted | | Failing test | `Failed`, `TestsFailed`: `testToggleFailure()`, `xcodebuild test` exited 65 | 1 of 2 failed | Deleted | | Deleted mid-run | Object deleted during `xcodebuild build` | None reported | Stopped and deleted | | Bad commit | `Failed`, `AgentError`: git "not our ref" | None | None created | After each case, `tart list` showed no build clone. `VMDeleted` was `True` for the pass, failing-test and bad-commit cases. For the bad commit, the `VMDeleted` condition's message reads "no VM was created". The failure happened during source fetch, before VM allocation. In the deletion case, stopping and deleting the VM took 6.39s. The object became `NotFound` 9.10s after deletion was requested. The agent released its finalizer after the VM was gone. Each row describes one run on one Mac and one sample app. They establish the observed outcomes and cleanup ordering. [Part 5](/blog/starting-macos-builds-in-seconds) measures timing across repeated runs. ## A timeout field still needs someone to enforce it [Tekton's CustomRun contract](https://tekton.dev/docs/pipelines/customruns/) distinguishes fields it passes to the controller from behavior the controller must implement. The division below follows the v1.15.3 source: | Interface | Tekton supplies | Controller responsibility | |----------------------------------|----------------------------------------------------------------|-----------------------------------------------------------------------------------------------| | `Succeeded` and `status.results` | Reads the condition and named results. | Write `Unknown` while running, then a terminal condition and results. | | `spec.timeout` | Copies the task timeout without enforcing a CustomRun timeout. | Enforce the deadline, clean up and report the outcome. | | `spec.retries` | Copies retries without creating another CustomRun. | If retries are supported, retry internally and remain `Unknown` until attempts are exhausted. | | Ownership | The PipelineRun owns the CustomRun. | Give child build objects an owner. | | `customSpec` | Supplies an embedded custom-task spec. | Support that spec or report an error in status. | In v1.15.3, `CustomRun.GetTimeout()` defaults an unset timeout to 60m. This implementation has an agent deadline and a controller backstop, but neither was reached in these runs. This baseline did not implement retries or `customSpec` handling. [Part 4](/blog/tekton-macos-build-recovery) later adds bounded agent retries. Tekton permits custom-task controllers to [ignore retries](https://github.com/tektoncd/pipeline/blob/v1.15.3/docs/customruns.md#L252-L261). Callers should rely on retries only when the controller explicitly supports them. Unsupported `customSpec` requests need an error in status. ## Other routes still need a cleanup owner CustomRun is one way to give the external job a cleanup owner. A Linux step can connect to the Mac, a custom controller can own the job, or a virtual-node provider can represent the VM as a Pod. The choice depends on which component will track the remote process and confirm cleanup. Only the CustomRun integration was exercised here. The alternatives were reviewed in documentation and source; their cancellation behavior was not tested.[^source-versions] [orka-tekton](https://github.com/macstadium/orka-tekton) uses an exit trap to delete the VM when `delete-vm` is true, with a sample `finally` teardown task. [Konflux's controller](https://github.com/konflux-ci/multi-platform-controller) uses a TaskRun finalizer to deallocate the host. Neither mechanism alone confirms remote cleanup. For plain SSH, [OpenSSH bug 396](https://lists.mindrot.org/pipermail/openssh-bugs/2018-November/019727.html) describes a remote command left running after the client exits when no PTY is allocated. A remote-execution step therefore needs an explicit cleanup protocol. Konflux removes its finalizer even if deallocation fails. The orka-tekton source has no explicit cancel handling beyond its teardown mechanisms. As I read Agoda's post, the provider suits CI systems that drive a running VM from outside the Pod. [Agoda describes](https://medium.com/agoda-engineering/how-we-integrated-native-macos-workloads-with-kubernetes-b4d3c14881a0) running GitLab's Kubernetes executor this way, with the VM as the build environment and helper containers in Docker. That does not supply the TaskRun contract traced above. Agoda's provider omits init containers. In the v1.4.1 source I reviewed, it does not run the first container's `command` and `args`, so this TaskRun would need Tekton-specific changes to run there. [Part 6](/blog/should-you-run-macos-jobs-from-tekton) compares the execution models and other ways to run Mac builds. ## The agent must complete the cleanup The macOS stage exposes parameters and string results. Steps, sidecars and shared Pod volumes remain features of ordinary TaskRuns. Linux stages can still run ordinary TaskRuns in the same Pipeline, as the `prepare` task did here. The external path needs controller-enforced deadlines, explicit retry behavior, scoped API access and recovery when the agent or controller disappears. [Part 2](/blog/tekton-macos-source-and-artifacts) follows how source and artifacts cross that boundary without a shared workspace. [Part 3](/blog/tekton-macos-build-vm-lifecycle) explains how the agent admits a build, allocates its VM and stops and deletes it before releasing the job. CustomRun gives Tekton an external job to track. Reliable execution and confirmed cleanup still depend on the protocol between the controller and the Mac agent. [^source-versions]: Source review used orka-tekton at `91f0a8ac`, multi-platform-controller at `3f45c507`, and macOS-vz-kubelet v1.4.1.