Advisionary vs Mandatory Locking

In September 2025 I was working on getting OpenTofu's unit tests to run on Windows in CI (#1201). Most failures were boring: missing .exe extensions, \r\n line endings, path separators. One wasn't. TestApply_lockedStateWait passed on Linux and macOS and failed on Windows every time, and the reason turned out to be a real bug hiding behind a difference in how operating systems lock files.

What the test does

OpenTofu uses a local state file, terraform.tfstate, and locks it so two tofu apply runs don't write to it at the same time. The test checks the "wait for the lock" path:

1. A helper process (statelocker) locks the state file.
2. The test runs tofu apply -lock-timeout=4s, so OpenTofu should keep retrying for up to four seconds.
3. After 500 ms, the helper releases the lock.

The expected result is that apply waits, gets the lock, and succeeds. On Windows it failed right away, without waiting at all.

Two kinds of locks

The two platforms lock files in very different ways.

On Unix, OpenTofu uses POSIX fcntl record locks:

flock := &syscall.Flock_t{
    Type:   syscall.F_RDLCK | syscall.F_WRLCK,
    Whence: int16(io.SeekStart),
    Start:  0,
    Len:    0, // the whole file
}
return syscall.FcntlFlock(f.Fd(), syscall.F_SETLK, flock)

These locks are advisory. The kernel keeps track of who holds them, but it only enforces them against processes that also ask for the lock. A process that just calls open() and read() ignores the lock completely. It's a convention: it works as long as everybody follows it. (Linux used to support mandatory locks through a special mount option, but almost nobody used it, and it was removed in Linux 5.15.)

On Windows, OpenTofu calls LockFileEx:

return lockFileEx(
    syscall.Handle(f.Fd()),
    _LOCKFILE_EXCLUSIVE_LOCK|_LOCKFILE_FAIL_IMMEDIATELY,
    0,              // reserved
    0,              // bytes low
    math.MaxUint32, // bytes high
    ol,
)

This lock is mandatory. Once one process holds an exclusive lock on a byte range, the OS rejects reads and writes to that range from every other process, whether or not they "participate". The read fails with ERROR_LOCK_VIOLATION: "The process cannot access the file because another process has locked a portion of the file."

The bug

Here is roughly what Filesystem.Lock did before acquiring the lock:

1. Call createStateFiles(), which opens the state file and reads it to keep a copy for the .backup file.
2. Then try to lock the file, retrying until the timeout expires.

On Linux, step 1 succeeds even while another process holds the lock, because the lock is advisory and a plain read doesn't care. Step 2 then gets resource temporarily unavailable, and the retry loop does its job.

On Windows, step 1 is a read from a locked file, so it fails. Lock returns that error right away and never reaches the retry loop. The lock timeout is never used.

So the code had a hidden assumption: you can read a file someone else has locked. That's true on Unix and false on Windows. It also meant that on Unix we could read state for the backup from a file another process was in the middle of changing.

The fix

The fix (#3240, issue #3239) splits createStateFiles in two. Opening the file stays where it was. Reading it for the backup moves into a new prepareBackupFile, which runs only after the lock is acquired:

func (s *Filesystem) Lock(_ context.Context, info *LockInfo) (string, error) {
    // ... acquire the lock, retrying until timeout ...

    if err := s.prepareBackupFile(); err != nil {
        return "", fmt.Errorf("failed to write backup file: %w", err)
    }

    s.lockID = info.ID
    return s.lockID, s.writeLockInfo(info)
}

The diff is small. The rule behind it is the important part: take the lock first, then touch the resource. On Windows that's required for the code to work at all. On Unix it's what makes the code correct, because the backup now always reflects the state we actually hold the lock on.

Other Windows surprises

The same locking model explains a few other test fixes from that month:

• Unlocking means closing. On Windows, the low-level flock.Unlock does nothing: the lock goes away only when the file handle is closed, and the state manager closes that handle in its own Unlock. Some tests persisted state without ever locking, so the handle stayed open, and deleting the workspace failed with "The process cannot access the file because it is being used by another process" (#3284).
• Leaked handles block cleanup. Windows won't delete files that are still open, so a few tests had to call runtime.GC() to let finalizers close leftover handles before t.TempDir() cleanup (#3282).
• Signals aren't portable. The statelocker helper used to be controlled with Unix signals. Signals don't work the same way on Windows, so I switched it to communicate over pipes instead (#3206). The remote backend's lock-timeout tests also rely on signals, so for now they're skipped on Windows (#3288).

Takeaways

• Advisory locks only protect code that asks for them. Any code path that reads or writes the file before taking the lock is unprotected, and nothing will warn you.
• Mandatory locks make ordering bugs visible. Windows turned a silent race into a loud failure. Running tests on a second OS is cheap compared to finding that bug in production.
• Lock first, then read. If you need a lock to protect a file, every read of that file, including "just making a backup", belongs after the lock is held.