vLLM Architecture: How It Actually Works

vLLM is an inference engine for large language models: a continuous-batching scheduler feeds a paged KV cache and tensor-parallel model execution, behind an OpenAI-compatible HTTP server. Symvanta's graph at 63ac04a detects 500 functional modules at modularity Q=0.85, and the biggest clusters are the plumbing every model file imports: the logger and platform interface at 4130 symbols and the tensor-parallel state accessors at 3540.

That shape is the structural signal. Hundreds of model implementations and kernels all call the same logging, platform, and distributed-state helpers, so the utility clusters grow huge while the engine itself stays compact: the V1 scheduler cluster holds 1435 symbols and the engine core client 1032. The graph found no module-level dependency cycles across the 500 modules.

Module map

The diagram shows the 10 largest of the 500 detected modules, with edges weighted by how many calls cross between them. The biggest holds 4130 symbols and its hub is _VllmLogger.warning_once, the log-once helper in vllm/logger.py, which is what a hub looks like when every file in a repo calls the same wrapper. The heaviest edge pair on the map runs between the two utility clusters: 404 calls one way, 377 back. Module names are checked by hand against the files their members live in; the symbol counts, hubs, and edge weights are what the graph computed.

Where to start reading

These are the most depended-upon symbols by PageRank over the call graph, followed by the OpenAI-compatible server's HTTP surface. Seven entries from the raw ranking are left out below: the _VllmLogger once-helpers, _should_log_with_scope, and _print_warning_once are logging wrappers every file calls, _nvmlGetFunctionPointer lives in the vendored NVML bindings, RemoteVLLMServer.url_for is the test harness's server wrapper, and random_uuid is a bare utility. They rank high without pointing anywhere useful.

• get_tp_group
• AutoWeightsLoader.load_weights
• LLM.generate
• GroupCoordinator.is_first_rank
• MultiModalRegistry.register_processor
• GET /health
• GET /version
• POST /tokenize
• POST /detokenize
• GET /v1/models
• POST /v1/chat/completions
• POST /v1/completions
• POST /v1/embeddings
• POST /pooling
• POST /score

Key subsystems

Logger and platform interface

_VllmLogger.warning_once from vllm/logger.py sits with the Platform interface (is_rocm, is_cuda, fp8_dtype, get_device_capability) in one 4130-symbol cluster, the largest in the repo. Every model implementation logs once and asks what hardware it is on, and those two habits pull the whole model zoo toward this cluster.

Tensor parallel state

get_tp_group and the accessors around it in vllm/distributed/parallel_state.py, plus init_logger and default_weight_loader. 3540 symbols. Its 404 calls into the logger cluster and the 377 coming back are the heaviest edge pair on the map.

Multimodal processing info

BaseProcessingInfo (vllm/multimodal/processing/context.py), the TokenizerLike protocol, and InputProcessingContext: the interface every model uses to declare its HF config, processor, and tokenizer. 1790 symbols.

LLM entrypoint and test harness

LLM.generate from vllm/entrypoints/llm.py clusters with the test helpers that drive it (multi_gpu_test, check_logprobs_close, create_new_process_for_each_test from tests/utils.py). 1683 symbols, and its hub ranks in the PageRank top 10 because the test suite calls LLM.generate from everywhere.

V1 scheduler and config factories

The engine's brain: Scheduler.schedule and Scheduler.add_request from vllm/v1/core/sched/scheduler.py, Request.num_tokens, KV cache spec registration, and the create_vllm_config / create_scheduler test factories. 1435 symbols. The canonical flow below starts here.

Engine core client

AsyncMPClient from vllm/v1/engine/core_client.py, the msgpack encoder, and AsyncLLM.generate: the async client side of the engine core process boundary. 1032 symbols.

Shared infrastructure

The rest of the top 10: GroupCoordinator ranks (1884 symbols, hub GroupCoordinator.is_first_rank), SamplingParams and serving utils (1522, hub random_uuid), EngineArgs and CLI parsing (1046, hub LLM.get_default_sampling_params), and Quantization config and debug logging (954, hub _VllmLogger.debug_once).

Canonical request flow

The sequence worth reading first is one engine step: the loop that turns queued requests into sampled tokens. Schedule a batch, allocate KV cache slots inside the schedule pass, hand the batch to the executor, compute the grammar bitmask for guided decoding, sample tokens, then fold the model output back into scheduler state. Every step below is a call edge out of EngineCore.step and its Scheduler.schedule pass, in the order the work happens.

1. EngineCore.step (vllm/v1/engine/core.py:580)
2. Scheduler.schedule (vllm/v1/core/sched/scheduler.py:438)
3. KVCacheManager.allocate_slots
4. Executor.execute_model
5. Scheduler.get_grammar_bitmask
6. Executor.sample_tokens
7. Scheduler.update_from_output

Health signals

Symvanta detected 0 dependency cycles across 500 modules (modularity Q=0.85). 13 sets of mutually recursive symbols were also detected, the largest being tool_parsers (5 symbols).