Prometheus Architecture: How It Actually Works

Prometheus is a single Go binary that scrapes targets, writes samples into a local time series database, and answers PromQL queries against it. Symvanta's graph of the repo at d8eeedd detects 83 functional modules (modularity Q=0.79), and the weight sits in storage: the largest cluster holds 1,363 symbols around labels.FromStrings, and it sends 520 calls into the head block and chunk pool cluster beside it and takes 744 back. The scrape loop, service discovery, remote write, and the HTTP API all arrange themselves around that pair.

The clearest structural signal is where the cycles are. All 8 dependency cycles live in the web UI (web/ui/react-app, web/ui/mantine-ui, and the codemirror-promql module). The Go packages that make up the server have none.

Module map

The diagram shows the 10 largest of the 83 detected modules, with edges weighted by how many calls cross between them. The biggest holds 1,363 symbols and its hub is labels.FromStrings, which puts label construction at the center of the codebase: nothing reaches a block or a chunk without building a label set first. Module names are checked by hand against the files their members live in; the symbol counts, hubs, and edge weights are what the graph computed.

Where to start reading

These are the most depended-upon symbols by PageRank over the call graph, followed by the HTTP surface registered in web/web.go. Three of them are one-word Go names that several types define independently, so each link points at the one definition the ranking scored: Labels.Get in model/labels, Head.MinTime in tsdb, and the bstream accessor in tsdb/chunkenc.

• labels.FromStrings
• labels.Labels.Get
• tsdb.Head.MinTime
• labels.EmptyLabels
• histogram.FloatHistogram.UsesCustomBuckets
• labels.decodeSize
• strutil.SanitizeLabelName
• histogram.IsCustomBucketsSchema
• labels.NewMatcher
• v1.createYAMLNode
• chunkenc.bstream.bytes
• labels.NewFastRegexMatcher
• GET /graph
• GET /federate
• GET /consoles/*filepath
• GET /user/*filepath
• GET /-/healthy
• HEAD /-/healthy
• GET /-/ready
• HEAD /-/ready
• GET /-/quit
• GET /-/reload

Key subsystems

Label sets and TSDB database

The largest cluster in the repo. labels.FromStrings and the string-packed Labels representation in model/labels/labels_stringlabels.go sit with the database lifecycle in tsdb/db.go (DefaultOptions) and the compactor beside it (DefaultPostingsDecoderFactory in tsdb/compact.go). Every read and every write builds a label set before it opens a block, so label construction and database setup land in one cluster.

Scrape loop and appendables

The scrape loop and the appendable interfaces every scrape writes through, with staleness handling from model/value.IsStaleNaN. 1,018 symbols, and the ingest path shows in its edge weights: 129 calls into the label and database cluster, 89 into label string encoding, 62 into label access and config parsing.

Head block and chunk pool

The in-memory head block (DefaultHeadOptions in tsdb/head.go), the chunk pool that hands out and recycles chunk objects (chunkenc.NewPool), and label hashing (Labels.Hash). Its pair of edges with the label and database cluster is the heaviest coupling on the map: 744 calls out and 520 back.

Label access and config parsing

labels.EmptyLabels and the label accessors in model/labels, clustered with the loaders that read them: ScrapeConfig.Validate in config/config.go, rulefmt.ParseFile, and the PromQL parser constructor. Config and rule files are label-shaped, so the parsers land next to the accessors.

PromQL functions and native histograms

The PromQL function table in promql/functions.go (simpleFloatFunc) clustered with the native histogram arithmetic in model/histogram (FloatHistogram.UsesCustomBuckets, compactBuckets, floatBucketIterator). Histogram functions are where the query engine and the histogram model meet, 868 symbols in all.

Write-ahead log segments

tsdb/wlog: segment naming and listing (SegmentName, listSegments) alongside the label symbol table in model/labels. 691 symbols, and it calls into the head and chunk pool cluster 250 times, the third heaviest edge on the map.

Service discovery core

The discovery registry and metrics plumbing every service discovery provider registers through (NewMetricRegisterer in discovery/util.go, NewRefreshMetrics), plus strutil.SanitizeLabelName, which turns provider metadata into valid label names. 629 symbols, and its heaviest edges leave for the providers (6 into the AWS discovery cluster, 3 each into configuration management and the scrape loop). It reaches the label and database cluster once, which is how isolated this subsystem is from storage.

Canonical request flow

The sequence worth reading first is the database opening. Repair index files an old release left behind, clear temporary directories and checkpoints, build the chunk pool, take the directory lock, construct the compactor, open the write-ahead log, then create the head block and replay the log into it. Every step below is a call edge out of tsdb.open, in the order the work happens.

1. tsdb.Open (tsdb/db.go:902) is the public entry point: it validates the options, derives the block ranges from them, and hands the rest to the unexported open.
2. tsdb.open (tsdb/db.go:986) runs every step below in one function and returns a *DB that is ready to serve.
3. tsdb.repairBadIndexVersion (tsdb/repair.go:30) rewrites the index and meta.json versions on blocks written by Prometheus 2.1, before anything tries to read them.
4. tsdbutil.RemoveTmpDirs (tsdb/tsdbutil/remove_tmp_dirs.go:27) deletes leftover temporary directories, once under the WAL directory and once under the data directory.
5. wlog.DeleteTempCheckpoints (tsdb/wlog/checkpoint.go:89) removes checkpoint directories from a truncation the process died in the middle of.
6. chunkenc.NewPool (tsdb/chunkenc/chunk.go:356) builds the pool that hands out and recycles chunk objects, and the DB struct takes it at construction.
7. tsdbutil.NewDirLocker (tsdb/tsdbutil/dir_locker.go:44) takes the lock file in the data directory so a second process cannot open the same database.
8. tsdb.NewLeveledCompactorWithOptions (tsdb/compact.go:208) constructs the compactor over those block ranges and the chunk pool.
9. wlog.NewSize (tsdb/wlog/wlog.go:300) opens the write-ahead log at the configured segment size, and opens a second log for out-of-order samples when that window is set or a WBL is already on disk.
10. tsdb.NewHead (tsdb/head.go:284) creates the in-memory head block over both logs.
11. tsdb.Head.Init (tsdb/head.go:729) replays the write-ahead log into the head. This is where a restart spends its time.
12. wlog.WL.Repair (tsdb/wlog/wlog.go:400) runs only when that replay returns a read error: the log is truncated at the last good record, and the WBL takes the same path through its own error type.
13. tsdb.DB.run (tsdb/db.go:1260) starts the background loop that reloads blocks, signals compaction, and mmaps head chunks for the life of the process.

Health signals

Symvanta detected 8 dependency cycles across 83 modules (modularity Q=0.79). The largest cycle spans 16 files in the graph area. 17 sets of mutually recursive symbols were also detected, the largest being parser (16 symbols). Every one of those cycles sits in the browser code: the React app under web/ui/react-app, the Mantine app under web/ui/mantine-ui, and the codemirror-promql editor module. The Go server packages carry none, and the recursion they do carry is the PromQL lexer and evaluator calling back into themselves, which is what a hand-written parser looks like in a graph.