Compose Stack¶

Airflow is deployed as a Docker Compose stack on Marines. We are aware This is not the topology Airflow recommends for operational production, but it is fine for our volume and gives us full control without the Kubernetes nightmare, at least for now.

Warning: Technical Overhead

This page describes the technical details of running Airflow in Docker Compose. If you are just interested in the DAGs and how they work, you can skip this page. If you do not understand how the services below work together, it is not a problem; the most important thing is to understand the DAGs and the data flow.

Services¶

The stack is defined in /home/airflow/docker-compose.yml. All Airflow services share a common YAML anchor (x-airflow-common) that injects the environment variables, volume mounts, and base image into each service.

graph TD
    subgraph "Docker Compose stack"
        PG[(postgres:13)]
        RD[(redis:7.2)]
        WEB[airflow-webserver<br/>port 8080]
        SCH[airflow-scheduler]
        WRK[airflow-worker<br/>Celery]
        TRG[airflow-triggerer]
        INT[airflow-init<br/>one-shot]
        CLI[airflow-cli<br/>debug profile]
        FLW[flower<br/>flower profile<br/>port 5555]
    end
    WEB --> PG
    SCH --> PG
    WRK --> PG
    WRK --> RD
    SCH --> RD
    INT --> PG
    INT --> RD

postgres¶

Property	Value
Image	`postgres:13`
Purpose	Airflow metadata database (DAG run history, task instances, Variables, Connections, XComs)
Port	Internal only (not exposed to host)
Data persistence	Named volume `postgres-db-volume`
Health check	`pg_isready -U airflow` every 10 s

The metadata database is on a named Docker volume, meaning it survives docker compose down && docker compose up. DAG run history, Airflow Variables (including credentials and the submitted_job job number), and Connections are all stored here and are not lost on restart.

redis¶

Property	Value
Image	`redis:7.2-bookworm`
Purpose	Celery task broker. The scheduler enqueues task instances here; workers pick them up
Port	6379 (internal)
Health check	`redis-cli ping` every 10 s

airflow-webserver¶

Property	Value
Port	`8080:8080`
Purpose	Web UI and REST API
External URL	`https://airflow.marines-data.eu` (proxied)
Health check	`curl http://localhost:8080/health` every 30 s

The webserver is where you monitor DAG runs, inspect task logs, manage Variables and Connections, and manually trigger or clear tasks.

airflow-scheduler¶

The scheduler reads DAG files from the dags/ folder, determines which task instances are due to run, and enqueues them to Redis for workers to execute.

Property	Value
Health check	`curl http://localhost:8974/health` every 30 s
Health check server	Enabled via `AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK=true`

The scheduler exposes a lightweight HTTP health check on port 8974 (internal). This is an Airflow feature for monitoring systems to detect a stuck scheduler without requiring database access.

airflow-worker¶

The Celery worker picks up task instances from the Redis queue and executes them. This is where your DAG code actually runs.

Property	Value
Health check	`celery inspect ping`
User	`0:0` (root, see known issues)
Signal handling	`DUMB_INIT_SETSID=0` for correct warm shutdown

airflow-triggerer¶

The triggerer runs deferrable operators and sensors. When a sensor uses mode='reschedule' (releasing the worker slot between polls), the triggerer handles the polling so that a Celery worker slot is not consumed while waiting.

airflow-init¶

A one-shot service that runs once at stack startup:

Checks available memory (minimum 4 GB) and CPUs (minimum 2)
Runs Airflow DB migrations (_AIRFLOW_DB_MIGRATE=true)
Creates the admin web user (_AIRFLOW_WWW_USER_CREATE=true)
Exits with code 0; other services wait on condition: service_completed_successfully

airflow-cli¶

Available only under the debug Docker Compose profile:

docker compose --profile debug run --rm airflow-cli dags list

airflow-cli is your best friend for debugging DAGs and inspecting the Airflow environment. It has the same environment variables and volume mounts as the other services, so you can run any Airflow CLI command in the same context as the scheduler and workers.

The advantage of using it within docker compose is that you do not need to install Airflow on your host machine, and you can run commands in the same environment as the scheduler and workers. For example, you can run airflow tasks test to execute a task instance in the same environment as the worker.

flower¶

Celery monitoring UI on port 5555, available under the flower profile:

docker compose --profile flower up

Executor model¶

We use CeleryExecutor with Redis as the broker and PostgreSQL as the result backend.

Scheduler
  └─ enqueues task instance to Redis
       └─ Celery worker picks up task
            └─ executes Python/Bash/SSH code
            └─ writes result to Postgres result backend
            └─ Scheduler reads state and advances the DAG

This differs from the simpler LocalExecutor (which runs tasks as subprocesses in the scheduler) and from KubernetesExecutor (which spawns pods). CeleryExecutor means the worker is a separate process and can be scaled horizontally if needed, though we currently run a single worker.

Some relevant environment variables set in docker-compose.yml:

AIRFLOW__CORE__EXECUTOR=CeleryExecutor
AIRFLOW__CELERY__BROKER_URL=redis://:@redis:6379/0
AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow

Email alerting¶

SMTP is configured to use OVH's mail server. Failure alert emails are sent from our own domain *@marines-data.eu.

AIRFLOW__SMTP__SMTP_HOST=pro3.mail.ovh.net
AIRFLOW__SMTP__SMTP_PORT=587
AIRFLOW__SMTP__SMTP_STARTTLS=True
AIRFLOW__SMTP__SMTP_SSL=False
AIRFLOW__SMTP__SMTP_USER=...@marines-data.eu
AIRFLOW__SMTP__SMTP_MAIL_FROM=...@marines-data.eu

Technical details¶

Containers run as root

All Airflow service containers run as user: "0:0" (root). This is a temporary workaround for a file-permission issue with the bind-mounted /mnt/md0 volume. Not a priority for now...

numpy.ndarray size changed RuntimeWarning

When running DAGs or CLI commands, a RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility may appear. This is a binary compatibility warning between the numpy version and a compiled extension in the environment. It does not affect our results, don't be worried if you see it in logs.