vLLM Architecture: How It Actually Works

vLLM is an inference engine for large language models: a continuous-batching scheduler feeds a paged KV cache and tensor-parallel model execution, behind an OpenAI-compatible HTTP server. Symvanta's graph at fe76112 detects 423 functional modules (modularity Q=0.85), and the largest by a wide margin is the plumbing every model file imports: the tensor-parallel state accessors and the Platform interface, 3832 symbols in one cluster.

That shape is the structural signal. Hundreds of model implementations and kernels reach for the same distributed-state, platform, and logging helpers, so the utility clusters grow huge while the engine itself stays compact: the V1 scheduler cluster holds 1571 symbols and the engine core client 787. Three of the ten largest clusters take their hub from vllm/logger.py. The graph found no module-level dependency cycles across the 423 modules.

Module map

The diagram shows the 10 largest of the 423 detected modules, with edges weighted by how many calls cross between them. The biggest holds 3832 symbols and its hub is get_tp_group, the tensor-parallel group accessor in vllm/distributed/parallel_state.py. The heaviest edge pair on the map runs between that cluster and the log-once helpers: 338 calls one way, 247 back. The LLM.generate cluster ranks eighth by size (1153 symbols) and is left out as scaffolding, because most of its members are the asset and output-comparison helpers under tests/. Module names are checked by hand against the files their members live in; the symbol counts, hubs, and edge weights are what the graph computed.

Where to start reading

These are the load-bearing entry points: the PageRank ranking over the call graph, blended with the hub symbol of each of the largest clusters, followed by the OpenAI-compatible server's HTTP surface. Seven entries are left out below. _print_warning_once, _VllmLogger.warning_once, and init_logger are the logging wrappers in vllm/logger.py that every file in the repo calls, _nvmlGetFunctionPointer lives in the vendored NVML bindings at vllm/third_party/pynvml.py, RemoteVLLMServer.url_for is the harness server wrapper in tests/utils.py, hf_api is a cached Hugging Face client accessor in vllm/transformers_utils/repo_utils.py, and random_uuid is a bare request-id utility. They rank high without pointing anywhere useful. LLM.generate is listed even though the blend drops it with its scaffolding cluster: it is the public offline API and it ranks seventh in the raw PageRank order.

• get_tp_group
• LLM.generate
• ModelConfig.is_encoder_decoder
• get_kv_cache_torch_dtype
• AsyncMPClient.call_utility_async
• _parse_gemma4_args
• GET /health
• GET /version
• POST /tokenize
• POST /detokenize
• GET /v1/models
• POST /v1/chat/completions
• POST /v1/completions
• POST /v1/embeddings
• POST /pooling
• POST /score

Key subsystems

Tensor parallel state and platform interface

get_tp_group and the accessors around it in vllm/distributed/parallel_state.py cluster with the Platform interface from vllm/platforms/interface.py (is_rocm, is_cuda, fp8_dtype, get_device_capability). 3832 symbols, the largest module in the repo. Every model implementation asks which distributed group it belongs to and what hardware it runs on, and those two habits pull the whole model zoo into one cluster.

V1 scheduler and KV cache specs

The engine's brain: Scheduler.schedule and Scheduler.add_request from vllm/v1/core/sched/scheduler.py, Request.num_tokens, and register_all_kvcache_specs from vllm/v1/core/single_type_kv_cache_manager.py. 1571 symbols. Its hub is _print_warning_once from vllm/logger.py, a log wrapper that ranks first here because the scheduler path calls it more than anything else in the cluster. The canonical flow below starts in this module.

Model config and GPU input batch

ModelConfig.is_encoder_decoder from vllm/config/model.py clusters with the V1 GPU worker state it shapes: InputBatch.req_ids (vllm/v1/worker/gpu_input_batch.py), BlockTable.map_to_kernel_blocks (vllm/v1/worker/block_table.py), and the sampling kernel apply_top_k_top_p_triton (vllm/v1/sample/ops/topk_topp_triton.py). 1392 symbols. Model shape decides batch layout, so the config accessors and the batch structures land together.

KV cache dtype and attention metadata

get_kv_cache_torch_dtype from vllm/utils/torch_utils.py sits with the shape and dtype questions an attention backend asks before it builds metadata: ModelConfig.use_mla and ModelConfig.get_head_size (vllm/config/model.py), split_decodes_and_prefills (vllm/v1/attention/backends/utils.py), and MambaStateDtypeCalculator._mamba_state_dtype. 929 symbols.

SamplingParams and offline entrypoint utils

Handles per-request sampling settings and the offline batch entrypoint: SamplingParams.from_optional (vllm/sampling_params.py), ModelConfig.get_diff_sampling_param, and the OfflineInferenceMixin run helpers in vllm/entrypoints/offline_utils.py. 1572 symbols. Its hub is random_uuid from vllm/utils/__init__.py, the request-id helper the serving path calls everywhere.

EngineArgs and CLI parsing

Provides functions for turning command-line flags into a running engine: FlexibleArgumentParser.add_argument and parse_args from vllm/utils/argparse_utils.py, then EngineArgs.add_cli_args and EngineArgs.create_engine_config from vllm/engine/arg_utils.py. The dataclass-reflection helper pair is_init_field and get_field sits here too. 1398 symbols.

Engine core client

AsyncMPClient from vllm/v1/engine/core_client.py, the msgpack codec in vllm/v1/serial_utils.py, and AsyncLLM.generate (vllm/v1/engine/async_llm.py): the async client side of the engine core process boundary. 787 symbols. This is the cluster that carries a request across the process split between the API server and the engine.

Tool call and reasoning parsers

_parse_gemma4_args from vllm/parser/gemma4.py and get_json_schema_from_tools from vllm/tool_parsers/utils.py: the code that reads a model's tool-call and reasoning output back into structured arguments. 755 symbols, the one cluster in the top 10 that lives in the serving layer.

Shared infrastructure

Two logging clusters round out the diagram, and between them they are why the map looks the way it does. Log-once helpers (1381 symbols, hub _VllmLogger.warning_once) holds the warning_once / info_once / debug_once family and the _should_log_with_scope rank gate. Logger factory and shared utils (1380 symbols, hub init_logger) holds logger construction plus small shared functions like cdiv and resolve_obj_by_qualname. Both are called from everywhere, which is what pins them near the top by size.

Canonical request flow

The sequence worth reading first is one engine step: the loop that turns queued requests into sampled tokens. Pick a batch, allocate KV cache blocks for it inside the schedule pass, dispatch the model without blocking, build the grammar bitmask for structured output, sample, then fold the model output back into scheduler state. Every step below is a call edge out of EngineCore.step and its Scheduler.schedule pass, in the order the work happens.

1. EngineCore.step (vllm/v1/engine/core.py:583) returns an empty result when the scheduler holds no requests, then drives one schedule, execute, sample, and update cycle.
2. Scheduler.schedule (vllm/v1/core/sched/scheduler.py:484) spends a token budget across running and waiting requests. Each request simply gets tokens assigned until its computed count catches up with its token count, which is the one mechanism that covers chunked prefill, prefix caching, and speculative decoding.
3. KVCacheManager.allocate_slots (vllm/v1/core/kv_cache_manager.py:347) allocates the blocks a scheduled request needs and returns None when the pool is short, which is the signal that makes the scheduler preempt the lowest-priority request and try again.
4. Executor.execute_model (vllm/v1/executor/abstract.py:212) broadcasts the scheduler output to the workers over collective_rpc with non_block=True, so the engine can do the next step while the forward pass runs.
5. Scheduler.get_grammar_bitmask (vllm/v1/core/sched/scheduler.py:1720) collects the scheduled requests that use structured output and asks the structured-output manager for their bitmask rows, returning None when the batch has none.
6. Executor.sample_tokens (vllm/v1/executor/abstract.py:232) runs when the executor deferred sampling: execute_model hands back None in that mode, and this call ships the grammar bitmask to the workers and samples the batch.
7. Scheduler.update_from_output (vllm/v1/core/sched/scheduler.py:1744) reads sampled tokens, logprobs, and KV-connector results back, returns deferred-free blocks to the pool, and builds the EngineCoreOutputs the client receives.

Health signals

Symvanta detected 0 dependency cycles across 423 modules (modularity Q=0.85). 14 sets of mutually recursive symbols were also detected, the largest being tool_parsers (5 symbols). That group is the streaming XML tool-call parser's state machine (StreamingXMLToolCallParser.setup_parser, _start_element, _end_element, and the two reset helpers), which is what mutual recursion looks like when it is deliberate.