etcd Architecture: How It Actually Works

etcd is a distributed, reliable key-value store built on the Raft consensus protocol: the coordination store that Kubernetes and a long list of other distributed systems rely on to hold cluster state consistently across machines. Clients talk to it over gRPC (Range, Put, Delete, Txn, Watch, LeaseGrant, and friends), and every write that changes the store's state has to be proposed to and committed by Raft before it lands in the underlying storage engine. Symvanta's Louvain community detection organized the codebase's indexed symbols into 148 functional modules (modularity Q=0.92, a clean separation of concerns for a codebase this size, with zero circular dependencies between modules). The largest module that belongs to the server rather than to its harnesses is etcdserver (962 symbols), and its hub, raftRequest, is the funnel every mutating client request passes through on its way to a Raft proposal. The modules around it exist to make that funnel safe: mvcc holds the versioned data, wal makes an entry durable before Raft is allowed to call it committed, schema gives the bolt database its bucket layout, and the two client/v3 modules carry the request vocabulary and the connection every caller reaches the cluster through.

Module map

The diagram below shows the 10 largest of the 148 detected modules once test clusters are excluded, sized by symbol count, with arrows weighted by how many calls cross between them. Five of the biggest Louvain communities in the repository are excluded on that rule: the tests/framework/integration cluster at 1,103 symbols (the largest community in the codebase), the 699-symbol linearizability model under tests/robustness/model, the two clusters of the e2e framework (696 and 557 symbols), and the tests/common suite they share (269). Each of them exercises the server without being part of it.

etcdserver carries the heaviest traffic in both directions. It makes 39 calls into mvcc and 36 into schema, and it receives 20 from embed, 17 from rafthttp, and 8 from wal. The storage packages form a tighter triangle underneath: mvcc into schema (7 calls), schema back into mvcc (16), and wal into schema (13). On the client side the two client/v3 modules trade 37 calls one way and 13 the other, the request builders and the connection layer leaning on each other.

Where to start reading

In a 148-module codebase, finding the handful of symbols everything else depends on is exactly the kind of task that trips up AI coding agents working from a file tree and grep instead of a call graph. These 10 are blended from the global PageRank ranking and the hub of each module in the diagram above, so load-bearing symbols from smaller modules keep their seats alongside the generic utilities that get called more often: start here to orient in the call graph.

• raftRequest
• ensureCompare
• ContextError
• String
• tail
• NewConfig
• Revision
• enqueueResponse
• mustClientFromCmd
• togRPCError

Two entries from the raw ranking are dropped here, cleanup and NewTmpBackendFromCfg, throwaway-backend fixtures in server/storage/mvcc/kv_test.go and server/storage/backend/testing/betesting.go that etcd's storage suite calls from nearly every package. Four of the survivors (String, tail, Revision, NewConfig) are bare Go names the graph groups across receivers and packages, so every link above points at one exact definition.

A few are worth calling out individually. raftRequest is EtcdServer.raftRequest (server/etcdserver/v3_server.go), the funnel every mutating v3 RPC reaches: it delegates to processInternalRaftRequestOnce on its first line, then reworks the traceutil.Trace that comes back so a slow proposal is logged against the request's own start time. ensureCompare is clientv3.Cmp.ensureCompare (client/v3/compare.go), the step that normalizes the comparison a Txn guards on before the transaction goes out over the wire, and it anchors the whole Op and OpOption builder surface around it. ContextError sits at the other end of the client: it turns a cancelled or expired context into the error a caller actually sees, so a dead connection reads as a client-side timeout rather than a server fault. String is rafthttp.streamType.String, the label the two peer-stream flavors carry through logs and metrics. tail is WAL.tail, the accessor for the log segment currently being appended to: Save reaches through it to find the write offset, and sync reaches through it to pick the file descriptor it fsyncs. Revision is authStore.Revision (server/auth/store.go), the auth revision stamped into every request header so the server can tell a proposal built against a stale permission set from a current one. togRPCError (server/etcdserver/api/v3rpc/util.go) sits on the boundary every client-facing RPC crosses, converting etcd's internal error values into the gRPC status codes clients see. mustClientFromCmd is etcdctl's client constructor, which is why the CLI packages rank as high as they do.

Key subsystems

etcdserver

The server core. raftRequest and processInternalRaftRequestOnce turn a client mutation into a Raft proposal and block until it has been applied, the apply loop that executes committed entries in index order lives beside them, and the membership accessors around both (MemberID, AuthStore, ToMemberDir) answer who this node is and who else is in the cluster.

mvcc

The multi-version key-value store. NewStore opens the bolt-backed store, newTreeIndex builds the in-memory index that maps a key to the revisions it was written at, and the tombstone and revision helpers around them decide what a read at a given revision returns. Its highest-PageRank member is a teardown fixture from the package's own suite (cleanup), which is what happens when a package is exercised from as many directions as etcd exercises this one; the thematic center is the store, the tree index, and the compaction path.

client/v3

The client library's request vocabulary. NewOp and the OpOption builders assemble every Get, Put, Delete, and Txn a client sends, and Cmp.ensureCompare normalizes the comparison a transaction guards on before it goes out over the wire. At 621 symbols this is the largest module outside the server, and it is the one piece of etcd that ships into every consumer of the store.

client/v3 connection

The client library's connection layer. newClient builds the gRPC connection a Config describes, translateEndpoint turns an endpoint string into a dial target and its credential requirement, the retry interceptor decides which failures are safe to replay, and ContextError maps a dead context onto the error the caller sees. Louvain splits it from the request vocabulary because the two halves call each other far more than either calls anything else: 37 calls one way, 13 back.

schema

The bucket layout of the bolt database. UnsafeReadConsistentIndex reads the applied Raft index etcd stamps into the database on every commit, and the alarm, auth, lease, and membership buckets alongside it carry the server state that must survive a restart. Validate and UnsafeMigrate are what let one binary open a database another version wrote.

rafthttp

Peer-to-peer transport. Transport.Send hands outbound Raft messages to the right peer, each peer holds a streamWriter for the long-lived connection and a pipeline for one-shot posts, and urlPicker rotates through a peer's advertised URLs when one stops answering. It is the only module on the map whose whole job is talking to other machines.

wal

The write-ahead log. Create and parseWALName own the segment files, encode frames each record with a chained CRC, and sync is the fsync barrier a Raft entry clears before the cluster is allowed to call it committed. The write-ahead log walkthrough follows one entry through this module to disk and back out again on restart.

embed

Server configuration. NewConfig builds the Config an embedded or standalone etcd starts from, and Validate, InitialClusterFromName, and the advertise-URL accessors decide whether that configuration describes a cluster this node can actually join. Its 20 calls into etcdserver are the boot sequence handing a validated config to the server it is about to start.

auth

The authentication store: users, roles, and the range-permission cache that answers whether a key is in scope for the caller. Revision is the auth revision every request header carries, so a proposal built against a stale permission set is rejected instead of applied.

cache

A watch-backed read cache that sits in front of a cluster. Cache.Watch fans one upstream watch out to many local watchers, demux keeps the local snapshot current from that stream, and newRingBuffer holds a bounded window of recent events so a lagging watcher resyncs from history instead of opening a second upstream watch. Its hub, enqueueResponse, is the non-blocking delivery step that returns false when a watcher buffer is full, which is how a slow consumer gets marked lagging instead of stalling the fan-out.

Canonical request flow

etcd has no plain HTTP surface of its own (the client API is gRPC), so the flow traced here is a client Put reaching consensus, as far as the graph's static call edges follow it:

1. v3rpc.kvServer.Put (server/etcdserver/api/v3rpc/key.go:90) is the gRPC entry point. It validates the request with checkPutRequest, calls the server's Put, routes any failure through togRPCError so the client gets a gRPC status code rather than an internal error value, and stamps the cluster and revision header on the way out.
2. etcdserver.EtcdServer.Put (server/etcdserver/v3_server.go:295) is 14 lines and writes nothing. It wraps the request in an InternalRaftRequest and hands it to raftRequest.
3. etcdserver.EtcdServer.raftRequest (server/etcdserver/v3_server.go:1012) calls processInternalRaftRequestOnce on its first line, then reworks the traceutil.Trace that comes back on the result (GetStartTime, SetStartTime, InsertStep, LogIfLong) so a slow proposal gets logged against the request's own start time.
4. etcdserver.EtcdServer.processInternalRaftRequestOnce (server/etcdserver/v3_server.go:1058-1133) checks exceedsRequestLimit first (line 1062), which compares the server's applied and committed Raft indexes so a client cannot propose faster than the server can keep up. It then stamps the request with a unique ID (line 1067), resolves the caller's identity through AuthInfoFromCtx (line 1072), classifies the request with getRequestType (line 1086), marshals it (line 1093), registers a wait keyed on the request ID (line 1106), and proposes the bytes to the Raft node (line 1113). A proposal that times out is translated by parseProposeCtxErr (line 1129) before the error reaches the caller.

The traced chain stops there. Submitting the proposal to Raft and the eventual write into etcd's mvcc storage cross a channel boundary: the bytes go into the Raft library, and the apply loop reads the committed entry back off a Ready channel later, on a different goroutine. The static call graph has no edge to follow across that gap. The apply side exists in the graph as its own connected run, EtcdServer.run to applyEntries to EtcdServer.apply to applyEntryNormal, with no edge joining it to the propose side. One more seam sits at the end of that run: apply.applierV3backend.Put, the function that finally writes the key, has no inbound call edge from applyEntryNormal at all, because the apply loop dispatches through the applierV3 interface instead of naming it. The Raft consensus spoke follows the whole path, both sides of the channel included.

Health signals

Symvanta detected 0 dependency cycles across 148 modules (modularity Q=0.92): every module's call traffic resolves without a circular dependency between modules. Symvanta also detected 11 sets of mutually recursive symbols, ten of them two-symbol pairs. Six of the eleven are the same shape, and it is the shape etcd's data model dictates: a Txn can nest sub-transactions, so anything that walks a transaction has to recurse into its own children. Permission checking does it (apply: checkTxnPermission and checkTxnReqsPermission), so does request validation (v3rpc: checkRequestOp and checkTxnRequest), quota accounting (server/storage/quota.go: costTxn and costTxnReq), proxy translation (grpcproxy: requestOpToOp and TxnRequestToOp), key-prefix namespacing (namespace: prefixOps and prefixOp), and the client's own conversion to the wire format (clientv3: toTxnRequest and toRequestOp). The largest set is the four-symbol group in etcdctl's watch command (NewWatchCommand, parseWatchArgs, watchInteractiveFunc, watchCommandFunc), where interactive mode re-enters the command parser it was launched from.