The compaction tail nobody budgets for

2026-04-02 · about seven minutes

Capacity plans for log-structured stores are written in terms of steady-state write amplification. The tail latency that wakes people up is not a steady-state property.

The usual model says: pick a level multiplier, multiply out the rewrites, divide the device throughput by the result, publish that as the sustainable ingest rate. It is a fine average and a poor prediction, because compaction is not smooth. Work arrives in lumps sized by whichever level just filled, and the lumps are correlated with exactly the traffic that filled them.

Where the lump comes from

A level fills, a compaction is scheduled, and it holds device bandwidth for its whole duration. If foreground writes continue at the rate that caused the fill, the memtable flush queue grows behind it. Once that queue hits its bound the engine stalls writers — and the stall is not proportional to the overload, it is proportional to the remaining compaction.

# p99.9 write latency, same average ingest, two schedulers
                    burst    steady
  1 worker          2140ms     310ms
  2 workers          760ms     240ms
  4 workers          290ms     235ms
  4 + rate limiter   180ms     190ms

Note the last row. Adding a rate limiter made the steady case marginally worse and the bursty case dramatically better. That trade is almost always correct, and almost never the default.

Three things that helped

Bound the queue by bytes, not by files. A file count limit lets a small number of very large flushes sail past the check and then stall everything at once.

Give compaction its own bandwidth budget. Not a priority — a budget. Priorities let background work starve or dominate depending on arrival order; a token bucket does neither.

Start earlier than the trigger. Beginning compaction at seventy percent of the level's capacity, at a reduced rate, converts one large lump into several small ones for the same total bytes written.

Write amplification tells you what the device must do. It tells you nothing about when.

The measurement mistake

For a long time I looked at compaction throughput graphs and concluded the system was healthy, because the area under the curve matched the model. Area under the curve is precisely the quantity that hides this problem. The graph worth watching is the flush queue depth, and the number worth alerting on is how long it has been non-zero — not how deep it got.

← all notes