etcd Write-Ahead Log: Entry to Disk and Replay

Raft can only call an entry committed once it is durable on the disks of a quorum, and etcd's durability primitive is the write-ahead log. The WAL is an append-only sequence of length-prefixed, CRC-protected records on disk: before any raft entry is treated as committed and before the apply loop is allowed to execute it, that entry is encoded into a record, appended, and (when the entry demands it) flushed to stable storage with an fsync. The same file format is what lets a node that lost power come back up, replay its log, and reconstruct the exact raft state it had before the crash. WAL is the struct that owns this: an open file directory, an encoder for the tail segment, a running CRC, and the index of the last entry it has seen.

This spoke follows a single raft entry from the moment the ready loop hands it over to the moment it is safely fsynced, then follows the reverse path a restarting server takes to replay it. The analysis is generated from Symvanta's code graph of etcd pinned at commit c34dc7e. One thing to keep in mind while reading the call counts: the server never calls this package directly. The ready loop calls a small Storage wrapper in server/storage/storage.go, and that wrapper is what calls WAL.Save, WAL.SaveSnapshot, and WAL.ReleaseLockTo, so the WAL's own suite accounts for most of the recorded callers of every method below.

The moving parts

• WAL
• WAL.Save
• encoder.encode
• WAL.sync
• WAL.cut
• NewDecoderAdvanced
• decoder.decodeRecord
• Record.Validate
• WAL.ReadAll

The write side and the read side are near mirror images. On the way to disk, WAL.Save is the entry point, encoder.encode turns a walpb.Record into framed, CRC-tagged bytes, and WAL.sync is the fsync that makes those bytes survive a crash. On the way back, NewDecoderAdvanced constructs the reader over the on-disk segments, decoder.decodeRecord pulls one record at a time, and Record.Validate checks each record's stored CRC against the running checksum so a torn or corrupted tail is caught rather than silently replayed. WAL.cut is the housekeeping that keeps segments bounded, and NewDecoder is the strict wrapper over NewDecoderAdvanced that refuses to continue past a CRC mismatch, the entry point both a replaying server and the offline etcdutl inspection tools decode through.

How it works

Here is what happens to a raft entry between the ready loop calling Save and the same entry being replayed after a restart.

1. WAL.Save receives the raft HardState and the slice of entries the ready loop just pulled off the Raft library. It short-circuits when there is nothing to write, then computes mustSync from raft.MustSync, which decides whether this batch requires a durable flush or can ride along in the OS page cache.
2. WAL.saveEntry runs once per entry. It marshals the raft entry into a walpb.Record tagged as an entry record and hands it to the encoder; Save then writes the hard state as a state record after the entries.
3. encoder.encode frames the record. It folds the record's payload into the running CRC (seeded when the encoder is built by newEncoder), stamps that checksum into the record, writes a length prefix, and pads each record to an 8-byte boundary so the decoder can detect a partial trailing write. The CRC chaining is what ties the integrity of each record to every record before it.
4. WAL.sync is the durability barrier. When mustSync is set and the current segment still has room, Save calls sync, which flushes the encoder's page writer and then invokes fileutil.Fdatasync on the tail segment to force the appended bytes to stable storage, timing the flush and logging a warning when it runs long. Only after this call returns is the entry genuinely durable, which is the precondition Raft needs before it treats the entry as committed.
5. WAL.cut rotates the log. When Save sees the tail segment has grown past SegmentSizeBytes, it closes the current file, opens a preallocated next segment, and starts it with a fresh CRC record via WAL.saveCrc so the chain continues cleanly into the new file. Bounded segments are what make compaction and lock release cheap later.
6. Snapshotter.SaveSnap is the compaction half. Periodically the server writes a full snapshot of applied state through the snap package, and WAL.SaveSnapshot records a small snapshot marker in the log itself; WAL.ReleaseLockTo then unlocks and lets old segments below that snapshot be reclaimed, so the log does not grow without bound.
7. WAL.ReadAll is the restart path. A recovering server opens the WAL at its most recent snapshot with Open, and ReadAll walks the segments from that point forward, dispatching on each record's type to rebuild the metadata, the last HardState, and the slice of entries the node had persisted.
8. decoder.decodeRecord does the low-level reading under ReadAll. It reads a length, reads the framed record, and calls Record.Validate to verify the stored CRC against the running checksum built with crc.New. A mismatch at the very end of the log is the expected signature of a crash mid-append and is tolerated as a truncated tail; a mismatch anywhere else is real corruption and stops recovery. Once replay finishes, the entries feed straight back into the same apply loop that produced them, and the node is exactly where it left off.

Where it connects

The WAL sits directly under the ready loop that the sibling spoke on Raft consensus describes. raftNode.start calls Save on every Ready batch of entries before those entries are treated as committed, and it does so through the Storage wrapper in server/storage/storage.go. That wrapper is one of only two production callers among the 37 the graph records for WAL.Save: the other is AppendAndCommitEntries in the server's bootstrap path, and the remaining 35 belong to the package's own suite and its fixtures. The durability that Save provides is the precondition that makes the whole "committed means safe" contract in that spoke true. In the other direction, the server consumes the WAL once per process lifetime, at startup, through openWALFromSnapshot, which calls ReadAll before the server begins serving clients; the offline etcd-dump-logs tool reads the log through the same method.

On the etcd module map, this package is its own module: wal, 311 symbols, with WAL.tail (the accessor for the segment currently being appended to) as its highest-PageRank member, which is what you get when every append, fsync, and rotation in the package routes through one file handle. Its outbound traffic goes almost entirely to schema (13 calls) and etcdserver (8), the two places a log record's meaning is decided. The snapshot machinery that compaction depends on clusters separately as snap, and the offline etcdutl tools that decode a WAL without a running server form another module again, so the same decoder serves a recovering server and a command-line reader with no code in common between them.

By the numbers

The write path is dominated by one method: WAL.Save is 43 lines (wal.go 995 to 1037) and everything else on the hot path (saveEntry, encode, sync) is small and called from it. The read path is lopsided the other way: ReadAll is a 124-line function (wal.go 472 to 595) because it has to handle every record type, snapshot matching, and the torn-tail case, while decodeRecord beneath it is 68 lines. The two decoder constructors make the integrity policy explicit in the type system: NewDecoderAdvanced takes a continueOnCrcError flag, and NewDecoder is the one-line wrapper that hard-codes it to false, so the default everywhere in the running server is to stop on corruption rather than guess.