{
  "generator": {
    "name": "Symvanta",
    "url": "https://symvanta.com"
  },
  "page": {
    "url": "https://symvanta.com/architecture/airflow",
    "title": "Airflow Architecture: How It Actually Works",
    "description": "How the Airflow codebase is structured: module map, load-bearing symbols, and dependency structure, generated by Symvanta.",
    "datePublished": "2026-08-10",
    "dateModified": "2026-08-10"
  },
  "repository": {
    "name": "apache/airflow",
    "url": "https://github.com/apache/airflow",
    "commit": "3160334",
    "license": "Apache-2.0"
  },
  "graph": {
    "modules": 500,
    "dependencyCycles": 6,
    "modularityQ": 0.96,
    "largestCycleFiles": 277,
    "mutuallyRecursiveGroups": 31
  },
  "summaryMarkdown": "Apache Airflow is a workflow orchestrator: a scheduler turns DAG definitions\ninto task runs, executors run them, and provider packages connect those tasks\nto outside services. The repo is a monorepo where the providers outweigh the\ncore. Symvanta's graph at `3160334` detects 500 functional modules at\nmodularity Q=0.96, and the single biggest cluster is the Google provider base\nhook at 1842 symbols, ahead of the core's session plumbing at 1704 and the AWS\nbase hook at 1400.\n\nThe modularity itself is the clearest structural signal. Q=0.96 means the\nclusters barely call each other: provider packages call into the core's\nsession and serialization plumbing and almost never into another provider. The\nAWS base hook, 1400 symbols, has exactly one outbound edge on the whole map.\nThe largest dependency cycle sits far from the Python core: it spans 277\nfiles of generated `openapi-gen` TypeScript client code in the web UI.",
  "subsystems": [
    {
      "name": "Google provider base hook",
      "descriptionMarkdown": "`GoogleBaseHook` in `providers/google` is the base class every Google Cloud\nhook inherits: credential handling, client options, quota project checks,\nretry counts. At 1842 symbols it is the largest cluster in the repo, and its\noutbound edges go to the Google credentials provider (15 calls) and GCS Data\nExtraction (6)."
    },
    {
      "name": "Session and LoggingMixin",
      "descriptionMarkdown": "The core's plumbing. `provide_session` and `create_session` from\n`airflow-core/src/airflow/utils/session.py` wrap every database access in a\nSQLAlchemy session, and `LoggingMixin` gives every Airflow object its logger.\n1704 symbols, and the heaviest edge on the map lands here: 138 calls in from\nthe serialization cluster, 56 back."
    },
    {
      "name": "AWS provider base hook",
      "descriptionMarkdown": "`AwsGenericHook` hands boto3 sessions, connection config, and region\nresolution to every Amazon operator. 1400 symbols and nearly sealed off: its\nonly outbound edge on the map is a single call into an FTP test hook."
    },
    {
      "name": "Helm chart tests",
      "descriptionMarkdown": "The fourth-largest module in the repo is a test suite. `render_chart` in\n`chart/tests/chart_utils/helm_template_generator.py` renders the Helm chart,\nand the surrounding 1380 symbols validate the resulting Kubernetes objects\nagainst their schemas. Its hub tops the raw PageRank ranking because every\nchart test calls it."
    },
    {
      "name": "Generated TS API clients",
      "descriptionMarkdown": "1324 symbols of `openapi-gen` output: `request`, `sendRequest`, and\n`getFormData` in the React UI and the edge3 provider's plugin UI. Nobody wrote\nthis cluster by hand, and the largest dependency cycle sits in the same\ngenerated output: 277 files under `openapi-gen/queries`."
    },
    {
      "name": "Serialization and API decorators",
      "descriptionMarkdown": "DAG serialization (`SerializedMappedOperator`), the API access decorators\n(`requires_access_dag`, `action_logging`), and secrets plumbing (`get_fernet`)\nshare one 721-symbol cluster. Its 138 calls into the session cluster are the\nheaviest edge on the map, which is what serializing to and from the metadata\ndatabase looks like."
    },
    {
      "name": "Shared infrastructure",
      "descriptionMarkdown": "Four more clusters round out the shared infrastructure on the map: TaskGroup\nand task SDK comms (680 symbols, hub `TaskGroup.group_id`), the Kubernetes\nprovider hook (391, hub `generic_api_retry`), the Google credentials provider\n(370, hub `get_credentials_and_project_id`), and the Plugins manager (356,\nhub `_get_plugins`)."
    }
  ],
  "startReading": {
    "symbols": [
      {
        "name": "GoogleBaseHook.get_credentials_and_project_id"
      },
      {
        "name": "AwsGenericHook.conn"
      },
      {
        "name": "GoogleBaseHook.fallback_to_default_project_id"
      },
      {
        "name": "provide_session"
      },
      {
        "name": "TaskGroup.group_id"
      },
      {
        "name": "BaseDatabricksHook._do_api_call"
      },
      {
        "name": "FabAirflowSecurityManagerOverride.session"
      },
      {
        "name": "_get_plugins"
      }
    ],
    "endpoints": [
      {
        "method": "POST",
        "path": "/auth/token"
      },
      {
        "method": "GET",
        "path": "/auth/me"
      },
      {
        "method": "GET",
        "path": "/assets"
      },
      {
        "method": "POST",
        "path": "/assets/events"
      },
      {
        "method": "GET",
        "path": "/assets/{asset_id}"
      },
      {
        "method": "GET",
        "path": "/config"
      },
      {
        "method": "GET",
        "path": "/dag_stats"
      },
      {
        "method": "GET",
        "path": "/dagWarnings"
      },
      {
        "method": "POST",
        "path": "/clearTaskInstances"
      },
      {
        "method": "POST",
        "path": "/clearDagRuns"
      }
    ]
  },
  "requestFlow": [
    {
      "position": 1,
      "symbol": "SchedulerJobRunner._run_scheduler_loop (airflow-core/src/airflow/jobs/scheduler_job_runner.py:1767)",
      "sourceUrl": null,
      "markdown": "`SchedulerJobRunner._run_scheduler_loop (airflow-core/src/airflow/jobs/scheduler_job_runner.py:1767)`"
    },
    {
      "position": 2,
      "symbol": "SchedulerJobRunner._do_scheduling (airflow-core/src/airflow/jobs/scheduler_job_runner.py:1967)",
      "sourceUrl": null,
      "markdown": "`SchedulerJobRunner._do_scheduling (airflow-core/src/airflow/jobs/scheduler_job_runner.py:1967)`"
    },
    {
      "position": 3,
      "symbol": "prohibit_commit",
      "sourceUrl": null,
      "markdown": "`prohibit_commit`"
    },
    {
      "position": 4,
      "symbol": "SchedulerJobRunner._create_dagruns_for_dags",
      "sourceUrl": null,
      "markdown": "`SchedulerJobRunner._create_dagruns_for_dags`"
    },
    {
      "position": 5,
      "symbol": "DagRun.get_running_dag_runs_to_examine",
      "sourceUrl": null,
      "markdown": "`DagRun.get_running_dag_runs_to_examine`"
    },
    {
      "position": 6,
      "symbol": "DBDagBag.get_dag_for_run",
      "sourceUrl": null,
      "markdown": "`DBDagBag.get_dag_for_run`"
    },
    {
      "position": 7,
      "symbol": "BaseExecutor.slots_available",
      "sourceUrl": null,
      "markdown": "`BaseExecutor.slots_available`"
    },
    {
      "position": 8,
      "symbol": "SchedulerJobRunner._critical_section_enqueue_task_instances",
      "sourceUrl": null,
      "markdown": "`SchedulerJobRunner._critical_section_enqueue_task_instances`"
    },
    {
      "position": 9,
      "symbol": "BaseExecutor.heartbeat",
      "sourceUrl": null,
      "markdown": "`BaseExecutor.heartbeat`"
    }
  ],
  "diagrams": [
    {
      "url": "https://symvanta.com/architecture/airflow/module-map.svg",
      "encodingFormat": "image/svg+xml"
    }
  ]
}