# A circuit breaker for Kubernetes admission webhooks

A slow webhook can block API requests even when its mutation is optional. With `failurePolicy: Fail`, Kubernetes rejects a matching request when the webhook times out.

`failurePolicy: Ignore` changes that result. Kubernetes still calls the webhook and waits for the call to fail, but it lets the request continue afterward. That is the important limitation: `Ignore` restores admission availability, not webhook latency.

I built a small controller that changes one approved webhook from `Fail` to `Ignore` after repeated failed health checks. It changes the policy back after the webhook recovers. The local run completed that cycle. The recovery code is not safe enough to deploy.

## When is it safe to fail open?

`failurePolicy` applies when a webhook call fails, times out, or cannot be completed. It does not override a webhook that replies with `allowed: false`. A healthy webhook can still reject a request while its failure policy is `Ignore`. Kubernetes documents that behavior in [Dynamic Admission Control](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy).

If it is always safe to admit an object without the mutation, use `Ignore` from the start. Kubernetes [recommends that design](https://kubernetes.io/docs/concepts/cluster-administration/admission-webhooks-good-practices/#fail-open-validate-final-state), with separate validation for rules that must still be enforced.

This controller is for the narrower case: the webhook is required in normal operation, but an operator has approved admitting objects without its mutation during an outage. That means accepting two risks: some requests fail before the controller opens the circuit, and some objects are admitted without the mutation afterward.

If an operator accepts those risks, the target should say so directly:

```yaml
allowAdmissionWithoutMutation: true
```

The controller can check that a target is explicitly listed. It cannot decide whether skipping a mutation is safe. That decision belongs to the operator. A mutation can be required for security, routing, storage, or another operational reason.

## The delayed-webhook test

I ran the controller and a TLS webhook on a local kind cluster. The webhook matched ConfigMap creates:

```yaml
webhooks:
  - name: slow-mutator.webhook-circuit-breaker.test
    failurePolicy: Fail
    timeoutSeconds: 1
    clientConfig:
      service:
        namespace: webhook-circuit-breaker
        name: slow-mutator
        path: /mutate
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["configmaps"]
```

The test webhook does not change the ConfigMap. For the API-server request used here, it waits for a configured delay and then returns `Allowed: true`. I used a two-second delay, which is longer than the one-second admission timeout. This isolates what Kubernetes does when a webhook call fails under `Fail` and `Ignore`; it does not show what a missing real mutation would do to an application.

In this run, the controller sends an HTTPS `GET` to `/healthz` once a second, with a 250 ms deadline. Three failed checks open the circuit. Five successful checks close it. The test applies the same delay to `/healthz` and `/mutate`, so the probe sees the injected fault. That does not make `/healthz` a general test of admission health: the API server can reach and exercise a webhook differently.

## `Ignore` admits requests, but still waits

The four ConfigMap creates produced these client-side times:

| Webhook condition      | Failure policy | Result   | Elapsed time |
|------------------------|----------------|----------|-------------:|
| Healthy                | `Fail`         | Created  |        40 ms |
| Delayed                | `Fail`         | Rejected |     1,049 ms |
| Delayed                | `Ignore`       | Created  |     1,045 ms |
| Healthy after recovery | `Fail`         | Created  |        43 ms |

The delayed request succeeded under `Ignore`, but it still took about one second. Kubernetes called the webhook, waited for the configured timeout, then continued because the policy was `Ignore`. The controller did not remove the webhook from the request path.

These are single wall-clock measurements around `kubectl create`. They include client and API-server work. They show the sequence, not a latency distribution, throughput limit, or the exact time required to open the circuit.

The registration changed from `Fail` to `Ignore`, then back to `Fail`. The request and registration captures show those three policy states. They do not show the controller's saved recovery record.

## How the controller opens the circuit

The controller needs the old policy so that it can put it back later. It saves that value in a ConfigMap, then patches the webhook registration:

```go
base := cur.DeepCopy()
original := cur.Webhooks[i].FailurePolicy
rec := Record{Original: original, OriginalWasNil: original == nil,
	OpenedAt: r.Now(), ResourceVersion: cur.ResourceVersion}

if err := r.Store.Put(ctx, t.Key(), rec); err != nil {
	return err
}

cur.Webhooks[i].FailurePolicy = failPolicy(admissionv1.Ignore)
if err := r.Client.Patch(ctx, &cur, client.MergeFrom(base)); err != nil {
	return err
}
```

That ordering creates the first serious problem. If the ConfigMap write succeeds but the webhook patch fails, the controller has a record saying the circuit is open even though the webhook is still `Fail`. On the next check, it finds the record and stops trying to open the circuit. Requests can keep failing.

The record is useful after a restart, but it is not proof that the policy patch succeeded. The controller needs an explicit state for “record saved, policy not yet changed”, or it needs a different design.

## How recovery can get stuck

When the webhook appears healthy, the controller reads the current policy. If it is still `Ignore`, it restores the saved value and then deletes the record:

```go
if policy(cur.Webhooks[i].FailurePolicy) != "Ignore" {
	return nil // another writer changed the policy
}

base := cur.DeepCopy()
if rec.OriginalWasNil {
	cur.Webhooks[i].FailurePolicy = nil
} else {
	cur.Webhooks[i].FailurePolicy = rec.Original
}
if err := r.Client.Patch(ctx, &cur, client.MergeFrom(base)); err != nil {
	return err
}

return r.Store.Delete(ctx, t.Key())
```

This has the reverse failure. The patch can restore `Fail`, then the ConfigMap delete can fail. The controller restarts with a stale record. It sees `Fail`, treats it as an external change, and keeps the record instead of clearing it.

There is a second ownership problem. The controller only checks whether the current policy is `Ignore`. It cannot tell its own `Ignore` from an administrator's `Ignore`. It also does not save the webhook configuration UID, so deleting and recreating a configuration with the same name creates ambiguity.

The patch itself can also overwrite another writer's change. `client.MergeFrom(base)` can replace the complete webhook list. The fresh read helps, but it does not protect an update made after that read. [controller-runtime's optimistic-lock merge patch](https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.25.1/pkg/client#MergeFrom) is one option: it turns a concurrent update into a conflict, which the controller can retry from a fresh read.

## Why I would not deploy this

The controller's ClusterRole can patch any mutating webhook configuration and update ConfigMaps across the cluster. The target list limits the program, not its Kubernetes permissions. A real deployment should restrict the allowed webhook configurations and keep state access inside the controller namespace.

The probe accepts only Service-backed webhooks with a PEM-encoded `caBundle` set explicitly. Kubernetes permits other valid configurations, including an omitted CA bundle that uses the API server's system roots. This controller counts those unsupported configurations as probe failures. It could open the circuit because it cannot probe the target, not because the webhook is down.

The probe also ignores `clientConfig.service.port` and always uses 443. A webhook on another port looks unhealthy even when it is working.

One replica holds the probe counters. Adding replicas is not safe without leader election or shared counters. If the controller opens the circuit and then stops without restarting, nothing restores `Fail`. Restoring the policy also does not fix objects that were created while the mutation was skipped.

The unit tests use a fake Kubernetes client and a fake prober. They cover the counter logic, but not TLS, API-server defaults, or the two-write failures above.

## Changing the policy is easy; recovering safely is hard

For an always-optional mutation, static `Ignore` is simpler. For a temporary exception, the request and registration captures show `Fail`, then `Ignore`, then `Fail`. They do not show the saved record being written or deleted. The policy change did not remove the one-second webhook wait.

Changing `Fail` to `Ignore` is easy. Safely owning that change, recovering from an interrupted update, and knowing when to undo it are the hard parts.