Airflow Architecture: How It Actually Works

Apache Airflow is a workflow orchestrator: a scheduler turns DAG definitions into task runs, executors run them, and provider packages connect those tasks to outside services. Symvanta indexed the monorepo (airflow-core, task-sdk, providers, chart, and the dev/breeze tooling) and grouped its symbols into 500 functional modules (modularity Q=0.92), flagging a display cap at that number, so a repo this size carries more clusters than the map prints. The largest, 2,436 symbols anchored on provide_session, is the core's SQLAlchemy session plumbing, the decorator that hands a database session to any function whose caller did not supply one; the Google provider base hook follows at 2,001 symbols, and a cluster joining LoggingMixin to the Amazon base hook at 1,747. Four of the ten largest clusters hub inside task-sdk, which is where Airflow 3 moved the DAG-authoring API that operators, hooks, and task groups are written against.

Module map

The diagram below shows the 10 largest of the 500 detected modules, sized by symbol count, with arrows weighted by how many calls cross between them, the kind of connectivity data a code graph captures that similarity search alone can't. The two heaviest arrows both run into shared plumbing: Core config and CLI bootstrap into Airflow core session plumbing (193 calls), and Google provider base hook into LoggingMixin and AWS base hook (186). One drawn cluster stands apart: Breeze developer CLI console (1,195 symbols) is the repo's own developer tooling under dev/breeze, and its single outbound edge lands in a Breeze parameter cluster too small to draw. The repo's fifth-largest cluster is left off the diagram on purpose: 1,385 symbols hubbed on render_chart in chart/tests/chart_utils/helm_template_generator.py, which renders the Helm chart so the chart tests can assert against the Kubernetes objects it produces.

Where to start reading

These are 12 of the codebase's most depended-upon symbols, blended from the global PageRank ranking and the hub of each of the ten largest modules, followed by a sample of the HTTP surface the graph indexed. Two raw PageRank entries are dropped. render_chart tops the raw ranking as the entry point the Helm chart suite runs every assertion through, and validate_template_fields ranks because the Amazon provider's operator tests call it from 33 files; both live under a tests/ directory and neither is code an application runs. Start here to see how the pieces connect.

• provide_session
• GoogleBaseHook.get_credentials_and_project_id
• create_session
• LoggingMixin.log
• get_console
• BaseHook.get_connection
• BaseSetupTeardownContext.update_context_map
• BaseOperator.__init__
• CommsDecoder.send
• TaskGroup.child_id
• mask_secret
• aws_template_fields
• POST /auth/token
• GET /auth/me
• GET /assets
• POST /assets/events
• GET /assets/{asset_id}
• GET /config
• GET /dag_stats
• GET /dagWarnings
• POST /clearTaskInstances
• POST /clearDagRuns

Three of these are worth calling out. provide_session and create_session are the same file's two halves: create_session opens a SQLAlchemy session and commits or rolls it back, and provide_session is the decorator that hands one to any function whose caller did not pass a session keyword. The graph records callers of provide_session in 35 files, reaching it from airflow-core models, dependency checks, and the Amazon, Google, Edge, and Standard providers. BaseOperator.__init__ is 219 lines of argument validation that every operator in every provider package inherits, which is why a constructor ranks alongside the session helpers. mask_secret registers a value with the secrets masker in the task process and forwards it to the supervisor over the same comms channel, so a connection extra or a Variable is redacted in both processes' log output.

Key subsystems

Airflow core session plumbing

The core's database access layer, 2,436 symbols around provide_session in airflow-core/src/airflow/utils/session.py, plus the LoggingMixin accessors and the ORM model properties (DagRun.state, DagVersion.get_latest_version) that ride on the same sessions. It is the largest cluster on the map and the busiest destination: Core config and CLI bootstrap sends 193 calls into it, Task SDK DAG authoring context 63, and Log context and scheduler queueing 42.

Google provider base hook

GoogleBaseHook in providers/google is the base class every Google Cloud hook inherits: credential resolution, client options, quota project checks, and the fallback_to_default_project_id decorator that fills in a project id when a caller omits one. At 2,001 symbols it is the second largest cluster, and 186 of its calls land in the logging and AWS hook cluster, the heaviest single edge out of any provider.

Core config and CLI bootstrap

Configuration loading, CLI action wrappers, and process setup: create_session, providers_configuration_loaded, action_cli, get_hostname, and Variable.get. 1,841 symbols. This is the code that runs before anything schedules, and its 193 calls into the session cluster are the heaviest edge on the whole map.

LoggingMixin and AWS base hook

Louvain puts LoggingMixin.log and the Amazon provider's AwsGenericHook.conn in one 1,747-symbol cluster, because every boto3 session, connection config, and region lookup in providers/amazon logs through the mixin on its way. It receives the map's second heaviest edge (186 calls from the Google base hook) and sends 69 back.

Log context and scheduler queueing

A second 1,201-symbol logging cluster, holding LoggingMixin.__init__ and _set_context (the logger a task instance writes through), the Google credential provider, and SchedulerJobRunner._executable_task_instances_to_queued. The split from the cluster above is real: one gathers around reading the logger, this one around configuring it per task instance.

Breeze developer CLI console

Breeze is Airflow's own development environment, and its console layer under dev/breeze/src/airflow_breeze/utils/ is 1,195 symbols: get_console, console_print, run_command, theming, and the cache-file helpers. It is tooling for working on Airflow, and it barely touches the rest of the graph: one outbound edge, into a Breeze build-parameter cluster the diagram does not draw.

Task SDK BaseHook and Connection

BaseHook.get_connection in task-sdk/src/airflow/sdk/bases/hook.py is the single call every provider hook makes to resolve a connection id, and Connection.get, Connection.extra_dejson, and get_field unpack what comes back. 1,067 symbols. It sends 78 calls into the logging and AWS cluster and 11 into the Google base hook, which is the shape of provider hooks resolving credentials.

Task SDK DAG authoring context

The DAG-authoring context machinery: DAG.task, DAG.get_task, get_current_context, and the setup/teardown context map hubbed on BaseSetupTeardownContext.update_context_map. 916 symbols, and 63 of its calls reach the core session cluster.

Task SDK BaseOperator constructors

BaseOperator.__init__ plus the argument validators it calls (validate_key, validate_instance_args) and the provider base-operator constructors that chain up to it, such as DataplexCatalogBaseOperator.__init__ and ManagedKafkaBaseOperator.__init__. 907 symbols, almost all of it inheritance converging on one initializer.

Task SDK supervisor comms

How a running task talks to its supervisor process: CommsDecoder.send frames a request as length-prefixed msgpack, writes it to the supervisor over stdin, and blocks for the reply, with _FrameMixin.as_bytes and CommsDecoder._from_frame on either side of the wire. 881 symbols. User task code never speaks to the Task Execution API server itself, which the module's docstring gives two reasons for: it halves the concurrent HTTP connections on that server, and it keeps the per-try identity token out of user code.

Canonical request flow

The sequence worth reading first is one pass of the scheduler loop, the code that turns DAG definitions into queued task instances. Guard the critical section against stray commits, create the DAG runs that are due, start the queued ones, fetch the running runs to examine, schedule their task instances, resolve each run's serialized DAG for its callbacks, check how much executor capacity is free, enqueue task instances inside the critical section, then heartbeat the executors. Every step below is a call edge out of _run_scheduler_loop or its _do_scheduling pass, listed in the order the calls appear in the source.

1. SchedulerJobRunner._run_scheduler_loop is the loop itself: an unbounded itertools.count that times every pass and breaks once the configured num_runs is reached.
2. SchedulerJobRunner._do_scheduling makes the pass's decisions and returns how many task instances it queued.
3. prohibit_commit wraps the work in a guard that raises if anything commits without going through the guard.
4. SchedulerJobRunner._create_dagruns_for_dags creates the runs whose next_dagrun_create_after has passed.
5. SchedulerJobRunner._start_queued_dagruns moves queued runs to running, within the per-DAG concurrency limits.
6. DagRun.get_running_dag_runs_to_examine bulk-fetches the active runs in one query.
7. SchedulerJobRunner._schedule_all_dag_runs walks those runs and collects the callbacks they produce.
8. DBDagBag.get_dag_for_run resolves each run's serialized DAG, behind an lru_cache so repeated runs of one DAG cost one lookup.
9. BaseExecutor.slots_available is summed across executors; a zero total skips the critical section entirely.
10. SchedulerJobRunner._critical_section_enqueue_task_instances takes the lock and moves task instances from scheduled to queued.
11. BaseExecutor.heartbeat then runs on every executor, whether or not it received work this pass.

Health signals

Symvanta detected 7 dependency cycles across 500 modules (modularity Q=0.92). The largest cycle spans 19 files in the FlexibleForm area. 28 sets of mutually recursive symbols were also detected, the largest being serialization (21 symbols). All 7 cycles sit in the repo's TypeScript: six in the React web UI under airflow-core/src/airflow/ui/ (its form, graph, connections, and pagination components), one in the simple auth manager's login UI. The Python packages under airflow-core/, task-sdk/, and providers/ carry none.