Cover image credit: Photo by Unsplash on Denny Müller
Disk I/O QoS for Kubernetes with cgroup v2 io.weight
Table of Contents
Kubernetes can request and limit CPU, memory, and ephemeral-storage capacity. It has no native resource model for disk I/O bandwidth, IOPS, or latency QoS.
CPU has requests and limits. Memory has requests, limits, and OOM eviction. ResourceQuota can account for requests.ephemeral-storage/limits.ephemeral-storage, but that’s disk capacity, not disk performance. A backup job writing two gigabytes to the same device your database reads from can stay well inside its storage quota while still saturating the shared device, and neither the scheduler nor ResourceQuota sees that contention at all. Kubernetes’ own resource-management docs describe capacity accounting for storage; nothing performance-related. cgroup v2 has had mechanisms in this area for years; Kubernetes itself has not exposed them as native Pod resource semantics.
The mechanism I went in planning to use was io.latency: a cgroup v2 interface that takes a target completion latency for one cgroup and, once that cgroup misses its target, throttles peer cgroups whose own latency target is higher (via queue-depth throttling and, for some I/O, artificial delay). That’s a mechanism for protecting a workload toward a configured latency target, not a hard SLA, but it maps almost directly onto “protect the database from the backup job.”
It wasn’t there. Not disabled, not zero-valued: find /sys/fs/cgroup -name "io.latency" across every cgroup on the node returned nothing at all. The node runs Debian 13, kernel 6.8.0-117-generic, inside Docker Desktop’s Linux VM on macOS, and that VM’s kernel doesn’t build in CONFIG_BLK_CGROUP_IOLATENCY support. (io.latency’s config option is separate from iocost’s CONFIG_BLK_CGROUP_IOCOST, a different facility, not tested here either.)
So this post measures the fallback. I built a daemon around io.weight, ran two fio jobs against the same block device with and without it running, and recovered 52% of the protected workload’s lost throughput. That number is real, and it’s also the weaker half of the story, because io.weight is a ratio rather than a reservation and the benchmark shows exactly what that costs.
What io.weight actually is
Every non-root cgroup v2 group with the io controller enabled exposes io.weight as a plain text file, default 100, range 1 to 10000. Writing default 500 into it sets a proportional share of I/O service time against every device the cgroup touches; under contention, I/O service is distributed roughly by weight ratio between the competing cgroups, and the resulting bandwidth/IOPS split depends on request size, workload, and device. The two workloads below differ by 256x in request size specifically to surface that.
That answers “who gets more when everyone’s competing.” It doesn’t answer “can this workload maintain a target latency under contention,” which is the problem io.latency is designed to address. The substitution isn’t cosmetic: io.weight shifts a ratio and doesn’t reserve a floor, so everything measured below is a proportional-share mechanism standing in for a target-latency one. If you’re building this for real, check io.latency on your actual node image before you design around it. It isn’t universal, even now.
The cgroup path is not where the docs suggest
My starting assumption was the commonly cited /sys/fs/cgroup/kubepods.slice/.... On this cluster no such directory exists at that level. With the systemd cgroup driver and kubelet’s cgroupRoot set to /kubelet, the live hierarchy is:
/sys/fs/cgroup/kubelet.slice/ kubelet.service kubelet-kubepods.slice/ io.max, io.weight, io.stat, io.prio.class, cgroup.subtree_control kubelet-kubepods-besteffort.slice/ kubelet-kubepods-besteffort-pod<uid_with_underscores>.slice/ kubelet-kubepods-burstable.slice/ kubelet-kubepods-burstable-pod<uid_with_underscores>.slice/ <- write targetGuaranteed pods sit directly under kubelet-kubepods.slice/, while Burstable and BestEffort pods get an extra QoS-class slice layer in between. UID dashes become underscores in the leaf name.
The important part isn’t the exact path. It’s that the path is derived from cgroupRoot, QoS class, and pod UID, which is why it’s a flag in the daemon rather than a hardcoded constant: this depends on the cgroup driver and kubelet’s configuration, and a cgroupfs-driver cluster wouldn’t follow this systemd slice naming at all.
func PodSlicePath(cgroupRoot, qosClass, uid string) (string, error) { if uid == "" { return "", fmt.Errorf("cgroup: empty pod UID") } base, err := kubepodsBase(cgroupRoot) if err != nil { return "", err }
normalizedUID := strings.ReplaceAll(uid, "-", "_") lowerQOS := strings.ToLower(qosClass)
switch lowerQOS { case "guaranteed": leaf := fmt.Sprintf("kubelet-kubepods-pod%s.slice", normalizedUID) return base + "/" + leaf, nil case "burstable", "besteffort": qosSlice := fmt.Sprintf("kubelet-kubepods-%s.slice", lowerQOS) leaf := fmt.Sprintf("kubelet-kubepods-%s-pod%s.slice", lowerQOS, normalizedUID) return base + "/" + qosSlice + "/" + leaf, nil default: return "", fmt.Errorf("cgroup: unknown QOS class %q", qosClass) }}Per-device weight returns EIO
The first write form I tried was per-device, one ratio per block device, which is the form the interface documentation describes:
$ echo "253:16 200" > .../low-priority.../io.weightsh: 7: echo: echo: I/O errorexit code: 1$ cat .../low-priority.../io.weightdefault 10The shell reports I/O error and the file retains its previous value, so the write was rejected outright rather than silently ignored. The cgroup-wide form, on the same file, in the same cgroup, seconds later:
$ echo 150 > .../kubelet-kubepods-burstable-podX.../io.weight$ cat .../kubelet-kubepods-burstable-podX.../io.weightdefault 150The scheduler is the wrong place to look. In cgroup v2 the file named io.weight is registered by the iocost controller (CONFIG_BLK_CGROUP_IOCOST); BFQ exposes its own separate file, io.bfq.weight. So vdb running mq-deadline rather than BFQ (/sys/block/vdb/queue/scheduler shows none [mq-deadline]) doesn’t by itself explain the rejection. The likelier candidate is -EOPNOTSUPP, which the per-device path returns when the policy isn’t enabled for that device: blk-iocost.c has no EIO return at all, and sh: echo: I/O error is the shell’s strerror output rather than a captured errno. I didn’t capture the raw errno, so I can’t map it to a specific return path.
What I can say is what reproduced: the per-device write fails and the cgroup-wide write succeeds, twice, in separate sessions. The daemon writes the cgroup-wide form exclusively, because it’s the only form I was able to apply successfully on this kernel and device.
The daemon
It polls two pods, reads an io-qos.demo/tier annotation (high/low, no PriorityClass wiring), maps tier to weight, and writes it.
The write-then-verify pattern below exists directly because of the EIO above: a write can fail depending on form, so the daemon reads the value back rather than trusting that the write call succeeded.
func (d *Daemon) reconcileOne(ctx context.Context, t Target) error { pod, err := d.Client.CoreV1().Pods(t.Namespace).Get(ctx, t.Name, metav1.GetOptions{}) if err != nil { return fmt.Errorf("get pod: %w", err) }
if pod.Status.Phase != corev1.PodRunning { log.Printf("qosd: %s/%s: phase=%s, waiting", t.Namespace, t.Name, pod.Status.Phase) return nil }
tier, ok := pod.Annotations[TierAnnotation] if !ok { return fmt.Errorf("missing annotation %s", TierAnnotation) } weight, ok := TierWeight[tier] if !ok { return fmt.Errorf("unknown tier %q", tier) }
if applied, ok := d.applied[string(pod.UID)]; ok && applied == weight { return nil }
slicePath, err := cgroup.PodSlicePath(d.CgroupRoot, string(pod.Status.QOSClass), string(pod.UID)) if err != nil { return fmt.Errorf("resolve cgroup path: %w", err) }
written, readback, err := cgroup.WriteWeight(slicePath, weight) if err != nil { return fmt.Errorf("write weight: %w", err) } ...}func WriteWeight(slicePath string, weight int) (written string, readback string, err error) { line, err := WeightLine(weight) if err != nil { return "", "", err }
weightFile := slicePath + "/io.weight" if err := os.WriteFile(weightFile, []byte(line), 0644); err != nil { return line, "", fmt.Errorf("cgroup: write %s: %w", weightFile, err) }
data, err := os.ReadFile(weightFile) if err != nil { return line, "", fmt.Errorf("cgroup: read back %s: %w", weightFile, err) }
return line, string(data), nil}go test ./... covers path construction for all three QoS classes, weight formatting at the range boundaries, and write/read-back against a temp dir. It ran as a plain process inside nginx-lab-control-plane, cross-compiled for linux/arm64 and docker cp’d in, pointed at /etc/kubernetes/admin.conf. No DaemonSet, no image build.
The test
Two Burstable pods, low-priority-writer and high-priority-reader, running fio inside Alpine containers against a hostPath on the same device (/dev/vdb1). Low priority: --rw=write --bs=1M --size=2G --numjobs=4, heavy sequential. High priority: --rw=randwrite --bs=4k --numjobs=1 --size=256M, database-shaped.
fio has to run inside the pods rather than on the node, and that’s not a stylistic choice. I/O issued from the node’s own shell doesn’t run inside any pod’s cgroup, so the weights under test wouldn’t apply to it at all. That meant a separate Alpine image with fio built into it, loaded into the kind cluster.
All three stages use --ioengine=psync rather than a queued engine. Same engine across all three runs, so it moves the absolute numbers, not the comparison.
kubectl exec high-priority-reader -- fio --name=highprio --directory=/data \ --rw=randwrite --bs=4k --ioengine=psync --numjobs=1 --size=256M \ --time_based --runtime=60s --output-format=json --group_reporting| Stage | IOPS | Bandwidth (KB/s) | clat mean (µs) | clat p99 (µs) | % of solo ceiling |
|---|---|---|---|---|---|
| 1. Solo baseline | 163,061 | 652,244 | 4.5 | 9.0 | 100% |
| 2. Contended, no daemon | 71,255 | 285,019 | 11.4 | 7.6 | 43.7% |
| 3. Protected, daemon running | 118,970 | 475,879 | 6.5 | 6.9 | 73.0% |
Before the daemon, both pods sat at the kernel default:
$ cat .../kubelet-kubepods-burstable-pod938cca59_..._.slice/io.weightdefault 100$ cat .../kubelet-kubepods-burstable-pode24d2cb2_..._.slice/io.weightdefault 100Daemon log, after reconciling both:
2026/08/09 04:15:39 qosd: default/low-priority-writer uid=938cca59-498a-41ef-86d0-80c39c1d8a83 tier=low qos=Burstable path=/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod938cca59_498a_41ef_86d0_80c39c1d8a83.slice/io.weight wrote="default 10" readback="default 10"2026/08/09 04:15:39 qosd: default/high-priority-reader uid=e24d2cb2-c787-4862-8a73-7f3e4d93ba92 tier=high qos=Burstable path=/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pode24d2cb2_c787_4862_8a73_7f3e4d93ba92.slice/io.weight wrote="default 500" readback="default 500"sequenceDiagram
participant Low as low-priority-writer<br/>(2G sequential write)
participant Dev as /dev/vdb1
participant High as high-priority-reader<br/>(4k randwrite)
Note over Low,High: weight 100/100
Low->>Dev: heavy sequential I/O
High->>Dev: competes equally
Note over High: 71,255 IOPS
Note over Low,High: weight 10/500
Low->>Dev: throttled share
High->>Dev: proportionally favored
Note over High: 118,970 IOPS
What the numbers say, and what they don’t
Distance from the solo ceiling went from 56.3 points contended to 27.0 points protected, which is 52% of the contention-induced gap closed. That’s real. It’s also half.
The remaining 27 points are the cost of the substitution. A 50:1
weight ratio changes who wins the contest for bandwidth without guaranteeing anyone a floor, so under sustained sequential writes from the low-priority pod the high-priority pod still pays real tax even with 50 times the competing weight. That’s what a proportional-share mechanism does when the workload underneath it doesn’t stop, and it’s the specific failure mode io.latency addresses by targeting a number instead of a ratio.
Latency moved further, relatively, than IOPS: 4.5µs solo, 11.4µs contended, 6.5µs protected. The contended row’s own p99 (7.6µs) sits below its mean, which looks like an error until you check the tail: p50 for that run is 1.18µs and max is 740ms, an extremely heavy-tailed distribution where a handful of massive outliers (queued behind the low-priority pod’s sequential writes) drag the mean far above the 99th-percentile bucket. Both numbers are real, straight from the same fio run; the mean is dominated by rare, huge stalls that the percentile itself doesn’t capture.
The p99 column has a second oddity the table makes plain: the solo run has the worst tail of the three, 9.0µs against 7.6µs contended and 6.9µs protected. p99 gets better as contention is added. With one run per stage and a heavy tail in all three, the 99th percentile here isn’t measuring what the mean is measuring, and I wouldn’t argue anything from that column. The mean is the number that tracks the weight change.
The low-priority pod didn’t lose anything under weight 10. It improved on every axis: 614 to 820 write IOPS, 629 to 840 MB/s, mean latency 6.5ms down to 4.8ms. Both metrics move together because its block size is fixed at 1M, which makes IOPS and bandwidth the same measurement in different units. Aggregate throughput across both pods went from 914 to 1,316 MB/s, up 44%. A proportional-share mechanism redistributes a roughly fixed amount of service; it doesn’t hand both competitors a third more of everything, and that the supposedly-starved pod came out ahead is a reason to treat the recovery figure as observational rather than as arbitration doing its job.
All three runs are ext4 buffered writeback on a virtio block device inside Docker Desktop’s VM, and 163k IOPS for 4K random writes is writeback cache absorption rather than physical device throughput. None of the absolute numbers transfer to real storage. The workload, device, filesystem, and fio parameters were held constant between stages 2 and 3; the daemon changed the cgroup weights. The runs were sequential rather than randomized or repeated, so this establishes an observed 52% recovery in this experiment, not an isolated, controlled measurement of the weight change as the sole variable: page cache and writeback state carry over between sequential runs on the same device. That’s still the number worth trusting more than the six-figure IOPS counts around it, which are cache-absorption artifacts first and device behavior second.
io.max as a second lever
The same controller exposes io.max, keyed by $MAJ:$MIN with rbps/wbps/riops/wiops fields, which caps a cgroup outright instead of ranking it against others:
$ echo "253:16 wiops=200" > .../low-priority.../io.max$ cat .../low-priority.../io.max253:16 rbps=max wbps=max riops=max wiops=200It applied cleanly, which makes it the only per-device write form on this kernel that worked at all. A production version could combine both, using io.max as a hard ceiling on device I/O for known offenders and io.weight to arbitrate everything else, with the cap sized against the workload’s own latency sensitivity, since the kernel documents that I/O is delayed once a cgroup hits its io.max limit, and a tight cap on a latency-sensitive workload trades one failure mode for another. A cap and a ratio solve different problems, and neither one alone gives you both.
Scope
Single kind node, no cross-node coordination tested. Neither io.latency nor iocost was benchmarked, since both are absent on this kernel. No network-attached storage; whether any of this reaches an NFS or iSCSI volume is a separate question this doesn’t touch. Tier assignment is annotation-driven, reading a fixed annotation off two named pods rather than watching scheduling.k8s.io/v1 cluster-wide.
This is an observational experiment, not a controlled benchmark: one node, one virtual block device, one workload pair, one sequential execution order, one measurement per stage. It demonstrates that changing io.weight coincided with a substantial improvement for this workload on this kernel and device. It doesn’t quantify the general performance benefit of io.weight, and it doesn’t rule out sequential-run cache/writeback drift as a partial contributor to the specific numbers above.
What it would take to go further
The path resolution logic already treats cgroupRoot as a flag rather than a constant, which was the right call given how much it varies. Making tier assignment PriorityClass-driven instead of annotation-driven is a smaller change than it looks, since the daemon already reads pod.Status.QOSClass off the live object, so adding a PriorityClass watch is additive rather than a redesign.
The bigger open question is io.latency. Any real deployment needs a capability check for CONFIG_BLK_CGROUP_IOLATENCY before committing to it, because some nodes will have it built in and some won’t. A daemon that assumes universal availability fails the same way this build’s first attempt at per-device weights did: quietly, and only on specific kernels. (iocost, CONFIG_BLK_CGROUP_IOCOST, is a separate mechanism and shouldn’t be treated as interchangeable with io.latency.)
Which is the actual lesson here, and it’s about interface names rather than about my benchmark. io.weight gives you arbitration, io.max gives you a ceiling, and io.latency gives you a target. Those are three different control semantics wearing one label, and calling all of them “I/O QoS” hides the distinction until a benchmark forces it back out. A production controller shouldn’t start by picking a policy. It should start by checking which of the three the node underneath it can actually make.