Airflow Architecture: How It Actually Works

Apache Airflow is a workflow orchestrator: a scheduler turns DAG definitions into task runs, executors run them, and provider packages connect those tasks to outside services. The repo is a monorepo where the providers outweigh the core. Symvanta's graph at 3160334 detects 500 functional modules at modularity Q=0.96, and the single biggest cluster is the Google provider base hook at 1842 symbols, ahead of the core's session plumbing at 1704 and the AWS base hook at 1400.

The modularity itself is the clearest structural signal. Q=0.96 means the clusters barely call each other: provider packages call into the core's session and serialization plumbing and almost never into another provider. The AWS base hook, 1400 symbols, has exactly one outbound edge on the whole map. The largest dependency cycle sits far from the Python core: it spans 277 files of generated openapi-gen TypeScript client code in the web UI.

Module map

The diagram shows the 10 largest of the 500 detected modules, with edges weighted by how many calls cross between them. The biggest holds 1842 symbols and its hub is GoogleBaseHook.fallback_to_default_project_id, a decorator that fills in the default GCP project id when a caller does not pass one. Module names are checked by hand against the files their members live in; the symbol counts, hubs, and edge weights are what the graph computed.

Where to start reading

These are the most depended-upon symbols by PageRank over the call graph, followed by a sample of the HTTP surface the graph indexed. Four entries from the raw ranking are left out below: render_chart is the Helm chart test harness in chart/tests, AirflowClient._make_request is the e2e test suite's HTTP client, and request and inner are bare names that generated API clients and decorator wrappers define over and over, so the graph groups them by name without pointing anywhere useful.

• GoogleBaseHook.get_credentials_and_project_id
• AwsGenericHook.conn
• GoogleBaseHook.fallback_to_default_project_id
• provide_session
• TaskGroup.group_id
• BaseDatabricksHook._do_api_call
• FabAirflowSecurityManagerOverride.session
• _get_plugins
• POST /auth/token
• GET /auth/me
• GET /assets
• POST /assets/events
• GET /assets/{asset_id}
• GET /config
• GET /dag_stats
• GET /dagWarnings
• POST /clearTaskInstances
• POST /clearDagRuns

Key subsystems

Google provider base hook

GoogleBaseHook in providers/google is the base class every Google Cloud hook inherits: credential handling, client options, quota project checks, retry counts. At 1842 symbols it is the largest cluster in the repo, and its outbound edges go to the Google credentials provider (15 calls) and GCS Data Extraction (6).

Session and LoggingMixin

The core's plumbing. provide_session and create_session from airflow-core/src/airflow/utils/session.py wrap every database access in a SQLAlchemy session, and LoggingMixin gives every Airflow object its logger. 1704 symbols, and the heaviest edge on the map lands here: 138 calls in from the serialization cluster, 56 back.

AWS provider base hook

AwsGenericHook hands boto3 sessions, connection config, and region resolution to every Amazon operator. 1400 symbols and nearly sealed off: its only outbound edge on the map is a single call into an FTP test hook.

Helm chart tests

The fourth-largest module in the repo is a test suite. render_chart in chart/tests/chart_utils/helm_template_generator.py renders the Helm chart, and the surrounding 1380 symbols validate the resulting Kubernetes objects against their schemas. Its hub tops the raw PageRank ranking because every chart test calls it.

Generated TS API clients

1324 symbols of openapi-gen output: request, sendRequest, and getFormData in the React UI and the edge3 provider's plugin UI. Nobody wrote this cluster by hand, and the largest dependency cycle sits in the same generated output: 277 files under openapi-gen/queries.

Serialization and API decorators

DAG serialization (SerializedMappedOperator), the API access decorators (requires_access_dag, action_logging), and secrets plumbing (get_fernet) share one 721-symbol cluster. Its 138 calls into the session cluster are the heaviest edge on the map, which is what serializing to and from the metadata database looks like.

Shared infrastructure

Four more clusters round out the shared infrastructure on the map: TaskGroup and task SDK comms (680 symbols, hub TaskGroup.group_id), the Kubernetes provider hook (391, hub generic_api_retry), the Google credentials provider (370, hub get_credentials_and_project_id), and the Plugins manager (356, hub _get_plugins).

Canonical request flow

The sequence worth reading first is one pass of the scheduler loop: the code that turns DAG definitions into queued task instances. Guard the critical section against stray commits, create DAG runs that are due, walk the running DAG runs, resolve each run's serialized DAG, check executor capacity, enqueue task instances inside the critical section, then heartbeat the executors. Every step below is a call edge out of SchedulerJobRunner._run_scheduler_loop and its _do_scheduling pass, in the order the work happens.

1. SchedulerJobRunner._run_scheduler_loop (airflow-core/src/airflow/jobs/scheduler_job_runner.py:1767)
2. SchedulerJobRunner._do_scheduling (airflow-core/src/airflow/jobs/scheduler_job_runner.py:1967)
3. prohibit_commit
4. SchedulerJobRunner._create_dagruns_for_dags
5. DagRun.get_running_dag_runs_to_examine
6. DBDagBag.get_dag_for_run
7. BaseExecutor.slots_available
8. SchedulerJobRunner._critical_section_enqueue_task_instances
9. BaseExecutor.heartbeat

Health signals

Symvanta detected 6 dependency cycles across 500 modules (modularity Q=0.96). The largest cycle spans 277 files in the queries area. 31 sets of mutually recursive symbols were also detected, the largest being serialization (21 symbols).