Reading your own page cache back

2026-01-27 · about six minutes

The benchmark reported 6.1 GB/s from a device rated at 3.4. Nobody questioned it for three weeks, because the number was good.

This is the most embarrassing category of measurement error: the one where the result is better than reality and therefore attracts no scrutiny. A slow number gets investigated within the hour. A fast number gets put in a slide.

How it happened

The harness generated a working set, wrote it out, and then immediately began its read phase. The working set was eleven gigabytes; the machine had sixty-four. Every byte the read phase requested was already resident in the page cache, still warm from the write phase that had just produced it.

# the missing lines
sync
echo 3 > /proc/sys/vm/drop_caches

# and the check that would have caught it
grep -E '^(Cached|Dirty)' /proc/meminfo

Dropping caches between phases brought the number to 3.2 GB/s, which is what the device does. The gap was entirely DRAM.

Why the obvious fixes are not enough

Dropping caches is necessary and not sufficient. Three further things bit us in the same investigation:

Readahead. A sequential access pattern gets prefetched aggressively. If your production pattern is random, a sequential benchmark is measuring a feature you will never receive.

The device's own cache. Consumer drives absorb bursts into a fast write region and quietly relocate them later. A benchmark shorter than that region measures the burst, not the drive. Ours needed to run for eleven minutes before the curve fell off a cliff.

Filesystem-level short-circuits. On a compressing filesystem a buffer of zeroes is not I/O at all. Fill test data from a pseudorandom stream with a fixed seed, and record the seed.

If a storage benchmark beats the device's rating, it is measuring something else. There is no exception to this.

The check I now run first

Before believing any storage number, compare it to the datasheet. If it is higher, stop and find the cache. If it is lower, the investigation is at least pointed at something real — and, in my experience, ends up being the scheduler about as often as the disk.

← all notes