etcd Raft Consensus: How a Put Becomes a Committed, Applied Write

etcd's promise is that a successful write is durable and linearizable: once a client's Put returns, every replica in the cluster agrees the key changed, and no later read is allowed to miss it. That guarantee is not a property of the storage engine, it is a property of the path a write takes through Raft before it is ever permitted to touch storage. EtcdServer.Put writes nothing. It wraps the client's PutRequest in an InternalRaftRequest, proposes it to the local Raft node, and blocks until the cluster has committed that entry and the apply loop has executed it. The key only changes inside applierV3backend.Put, which runs later, on a different goroutine, after consensus is already reached.

This spoke traces that whole path, from the gRPC handler to the applied mutation, as far as the static call graph follows it. The analysis is generated from Symvanta's code graph of etcd pinned at commit 6006f40, and one artifact of the graph is worth stating up front: the hop where a proposal crosses into the Raft library and returns as a committed entry is a Go channel boundary, not a direct call, so several of the symbols below have zero recorded callers even though they execute on every write. The consensus algorithm itself lives in an external module, go.etcd.io/raft/v3 (pinned at v3.7.0), which etcd's server drives through a ready loop.

The moving parts

• EtcdServer.Put
• EtcdServer.processInternalRaftRequestOnce
• exceedsRequestLimit
• raftNode.start
• EtcdServer.apply
• EtcdServer.applyEntryNormal
• applierV3backend.Put
• storeTxnWrite.Put
• Read.LinearizableReadLoop

The two ends of the write path are the pair to hold in your head. EtcdServer.Put is a thin 14-line handler: build the internal request, propose, wait, return. applierV3backend.Put is the function that finally writes the key, and the graph records it with zero callers, which is exactly what a handler dispatched dynamically off a committed log entry looks like. Between them, processInternalRaftRequestOnce is the gatekeeper that assigns each proposal an ID, admits or rejects it, and parks the calling goroutine on a wait channel, while raftNode.start is the long-running loop that talks to the Raft library and turns its output into work for the rest of the server. Read.LinearizableReadLoop is the read-side mirror of all this: it exists so that reads can be linearizable without going through the log themselves.

How it works

Here is the canonical path of a client Put, from the gRPC handler to the applied mutation and back.

1. EtcdServer.Put receives the decoded PutRequest, wraps it in an InternalRaftRequest (the union type every mutating operation shares), and calls raftRequest, which brackets the proposal with a traceutil.Trace so a slow write gets logged and then delegates to processInternalRaftRequestOnce.
2. EtcdServer.processInternalRaftRequestOnce stamps the request with a unique ID from idutil.Generator.Next, classifies it with getRequestType, and runs the admission check exceedsRequestLimit, which compares the server's committed and applied Raft indexes so a client cannot propose faster than the node can apply. It registers a wait keyed on the request ID, marshals the request, and proposes it to the Raft node.
3. raftNode.start is where the proposal leaves etcd's own code. The Raft library replicates the entry to peers over the transport, and once a quorum has acknowledged it, the library returns it as a committed entry on its Ready channel. This loop reads each Ready, persists the entries and hard state to the write-ahead log, sends outbound messages to peers, and packages the committed entries as a toApply batch for the server. Because this hop is a channel handoff rather than a call, the static graph shows no edge from the propose side to the apply side.
4. EtcdServer.apply drains that batch inside the server's own run loop (reached through applyAll and applyEntries). It walks the committed entries in index order, which is the point where Raft's total ordering becomes etcd's execution order.
5. EtcdServer.applyEntryNormal handles each ordinary entry: it unmarshals the bytes back into an InternalRaftRequest, checks whether the entry has already been applied (so replay after a crash is idempotent), and dispatches it through the applierV3.Apply interface rather than a direct call.
6. applierV3backend.Put is the applier method the dispatch lands on for a Put. This is the first moment in the entire path that the request is actually a write against storage rather than a proposal about a write.
7. storeTxnWrite.Put does the mutation. The applier opens a write transaction with store.Write, and storeTxnWrite.Put assigns the key its new main revision, writes the versioned value into the bolt-backed store, and updates the in-memory treeIndex. When the transaction ends and the apply index advances, the wait registered back in step 2 fires and the original Put call unblocks with its response.
8. Read.LinearizableReadLoop is how reads stay consistent with all of the above without paying for a log entry. A linearizable read requests a confirmed read index from the leader, records the current AppliedIndex, and if the applied index is behind, waits on ApplyWait until the store has caught up to that index before serving from mvcc. It batches concurrent reads so one round trip to the leader can release many waiters at once.

Where it connects

This path is the spine that the hub page's clusters hang off. The request and option vocabulary a client uses to build a Put lives in the Etcd Cluster Operations module, the largest on the etcd module map; the admission check in step 2 leans on getCommittedIndex, the same committed-index accessor the map places in the Store Configuration Utilities cluster; and the applied mutation in step 7 lands in the mvcc data structures that anchor the Data Structure Management cluster. The apply loop is deliberately the only writer into that storage, which is why applierV3backend.Put and storeTxnWrite.Put both read as zero-caller nodes: nothing calls them by name, the log dispatches to them.

The half of the story this spoke skips over is step 3's phrase "persists the entries to the write-ahead log." Before Raft is allowed to call an entry committed and before the apply loop is allowed to execute it, that entry has to be durable on disk, so a power loss cannot lose an acknowledged write. That durability contract, the encoding, the fsync discipline, the segment rotation, and the replay that rebuilds this exact in-memory state after a restart, is the subject of the sibling spoke on the write-ahead log.

By the numbers

The propose side of the flow is compact: processInternalRaftRequestOnce spans 76 lines and, per the graph, depends on 13 symbols including idutil.Generator.Next, getRequestType, exceedsRequestLimit, and parseProposeCtxErr, the helper that translates a context or proposal error into the error a client sees. The apply side is where the length lives: applyEntryNormal runs 59 lines and raftNode.start, the ready loop, is a single 169-line function (raft.go lines 174 to 342) that fans out to the external go.etcd.io/raft/v3 v3.7.0 API (Ready, Advance, IsEmptyHardState) alongside etcd's own WAL Save, snapshotter SaveSnap, and transport Send. The clean split between a thin proposal front end and a fat apply back end is the structural signature of a consensus system: almost nothing happens where the request arrives, and everything happens where the log is replayed.