Nobody agrees on what fsync promises

2026-06-14 · about nine minutes

The write-rename-fsync dance is the most copied durability pattern in server software, and roughly half the copies are missing a step that only matters on one filesystem, on one kind of power loss.

The pattern looks settled. Write the new contents to a temporary file in the same directory, flush it, rename it over the target, and you have an atomic replacement. Every review I have sat in treats this as boilerplate. It is not boilerplate, because the rename is a directory operation and the directory has its own dirty state.

fd = open(tmp, O_WRONLY | O_CREAT | O_EXCL, 0600);
write(fd, buf, len);
fsync(fd);                 /* contents are durable   */
close(fd);
rename(tmp, target);       /* metadata change only   */
dirfd = open(dir, O_RDONLY | O_DIRECTORY);
fsync(dirfd);              /* the step people forget */
close(dirfd);

Without that last flush you have a guarantee that the data survives and no guarantee at all that anyone can find it. After a crash the directory entry may still point at the old inode, or at neither.

What actually differs

I ran the same torture harness against four configurations on a machine with a real power-cut relay rather than a simulated one. The interesting column is the last: whether omitting the directory flush was ever observably wrong.

"Rarely observable" is the trap. Two of these four will pass a naive test suite forever while remaining unspecified. The ext4 data=ordered behaviour in particular is a widely relied-upon accident of implementation, and the documentation has never promised it.

The part that surprised me

Error handling is worse than ordering. On Linux, if a writeback error occurs and is reported to one file descriptor, the error may be cleared before your descriptor ever calls fsync. A successful flush can therefore follow a failed write that you will never be told about. The fix accepted upstream some years ago narrowed the window; it did not close the whole class.

Treat a failed fsync as unrecoverable for that file. Do not retry and do not assume the previous contents are intact.

Practically this means a storage engine cannot treat flush failure as a transient condition. The only safe response is to stop accepting writes, mark the file suspect, and recover from a log — which is exactly the code path that is never exercised and therefore never works.

What I do now

Flush the file, flush the parent directory, and on any flush failure fail the whole subsystem loudly rather than retrying. Then write a test that cuts power for real, because the simulated version passes on everything.

← all notes