Deploying DAGs

databaas dag deploy uploads a repo's Airflow code to the environment's DAG bucket, where Airflow picks it up. It runs in CI, with the environment's DAG deployment key from the repository secrets. It needs no databaas login.

pip install "databaas-utils[deploy]"
databaas dag validate -t prod   # with Airflow and the DAGs' dependencies installed
databaas dag deploy -t prod

CI only, for now

The command authenticates with a long-lived S3 key, which only CI has. Run on a laptop, it stops with an error asking for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Deploying with your own login is coming.

Configuration

databaas.yml at the repo root. The command finds it from the working directory or any parent; --config PATH points at another one.

bundle:
  name: marine-pipeline   # lower-case letters, digits and hyphens
targets:
  dev:
    mode: development
  prod:
    mode: production
artifacts:
  dags: airflow/          # the directory that is deployed

Key
`bundle.name`	Names the deployment in the bucket. Starts with a letter.
`targets.<name>.mode`	`development` or `production`. At most one production target.
`targets.<name>.default`	Optional. `true` lets you leave out `-t`.
`artifacts.dags`	The directory to upload, relative to `databaas.yml`.

Other keys are ignored.

Commands

Command	Does
`databaas dag validate -t TARGET`	Parse the target's DAGs with Airflow, exactly as they would be deployed
`databaas dag deploy -t TARGET`	Make the target's prefix hold exactly the `artifacts.dags` directory
`databaas dag deploy -t TARGET --dry-run`	List what would be uploaded and deleted, change nothing
`databaas dag destroy -t TARGET`	Delete everything under the target's prefix. Production needs `--force`.

All three take --dev-key KEY and --config PATH. deploy and destroy also take --bucket NAME and --endpoint-url URL. deploy -v prints every upload; validate -v prints Airflow's output.

Validating before a deploy

databaas dag validate stages the directory exactly as deploy would, renamed packages and _databaas_target.json included, and parses it with Airflow's DagBag. The staged directory is appended to sys.path, as in the cluster, and from databaas_target import target resolves to this package. It fails on:

a file that does not import, including Airflow's own checks such as a duplicate dag id or one longer than 250 characters;
no DAGs at all;
on a development target, a dag id without the dev_<dev-key>_ prefix. Such a DAG would replace the production DAG of the same id; build ids with target.dag_id(...).

Problems are reported against the file in the repo, not the staged copy. In GitHub Actions they also appear as annotations on the file.

It needs no key, but it runs the DAG files, so Airflow and the DAGs' own dependencies must be installed in the same Python as databaas-utils. Without Airflow it says so. Pin the Airflow version the environment runs, and install it with its constraints file.

What a deploy does

Each target is one prefix in the environment's metadata bucket:

s3://<bucket>/dags/prod/<bundle>/                production
s3://<bucket>/dags/dev/<dev-key>/<bundle>/       development

A deploy works like aws s3 sync --delete. It uploads files that changed and deletes files that are no longer in the directory. It never touches anything outside its own prefix. __pycache__, *.pyc, .git, .venv and .DS_Store are not uploaded, and symlinks are refused.

Every deploy writes a _databaas_target.json next to the code. It records the mode, dev key, bundle, repo, branch, commit, CI run and time. target_for and scoped() read it, so a dev deploy's DAG ids become dev_<dev-key>_<name>. See Dev deployments.

The directory is copied to a temporary folder first, so the checkout is never changed. That matters for development targets, which are adjusted before upload as described below.

Development targets

The dev key is the branch name as a slug: lower case, with anything other than letters, digits and - replaced by -. A branch longer than 40 characters is shortened, with a hash of the full name added. CI provides the branch: GITHUB_HEAD_REF on a GitHub pull request, BITBUCKET_BRANCH on Bitbucket. Elsewhere it comes from git. --dev-key overrides it, and is turned into a slug the same way.

Airflow puts every deployed directory on one shared sys.path. If two open branches both ship a package called marine_pipeline, either copy can win an import. So a dev deploy renames the top-level packages and modules in artifacts.dags to <name>_<dev_key> and rewrites the imports of them. On branch feature/x:

In the repo	Deployed
`marine_pipeline/`	`marine_pipeline_feature_x/`
`helpers.py`	`helpers_feature_x.py`
`from marine_pipeline.tasks import run`	`from marine_pipeline_feature_x.tasks import run`

dags/ and dotfiles keep their names. Relative imports are left as they are, and so are directories without Python files, such as dbt/. Entries in .airflowignore are renamed too. If a file does not parse, the deploy stops before anything is uploaded.

Production deploys upload the directory exactly as it is.

Credentials

Variable
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`	The environment's DAG deployment key. Required.
`AWS_ENDPOINT_URL`	The S3 endpoint, e.g. `https://s3.nl-ams.scw.cloud`. Or `--endpoint-url`.
`DATABAAS_DAGS_BUCKET`	The environment's metadata bucket. Or `--bucket`.
`AWS_DEFAULT_REGION`	Optional, default `nl-ams`.

Only these variables are read. ~/.aws and a databaas login are ignored, so a deploy uses exactly the key the pipeline was given. A key from another environment is refused by the bucket with a plain 403.

CI

GitHub Actions, deploying pull requests as dev targets, main as production, and removing a dev target when its pull request closes:

# .github/workflows/deploy-dags.yaml
on:
  pull_request:
    types: [opened, synchronize, reopened, closed]
  push:
    branches: [main]

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.SCW_DAGS_ACCESS_KEY }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.SCW_DAGS_SECRET_KEY }}
  AWS_ENDPOINT_URL: https://s3.nl-ams.scw.cloud
  AWS_DEFAULT_REGION: nl-ams
  DATABAAS_DAGS_BUCKET: databaas-demo-metadata
  AIRFLOW_CONSTRAINTS: https://raw.githubusercontent.com/apache/airflow/constraints-3.1.8/constraints-3.12.txt

jobs:
  deploy:
    if: github.event.action != 'closed'
    runs-on: ubuntu-latest
    concurrency: { group: "dags-${{ github.head_ref || 'prod' }}", cancel-in-progress: false }
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: |
          target=${{ github.event_name == 'push' && 'prod' || 'dev' }}
          uvx --from 'databaas-utils[deploy]' --with apache-airflow==3.1.8 \
            -c "$AIRFLOW_CONSTRAINTS" --with-requirements requirements.txt \
            databaas dag validate -t "$target"
          uvx --from 'databaas-utils[deploy]' databaas dag deploy -t "$target"

  cleanup:
    if: github.event.action == 'closed'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uvx --from 'databaas-utils[deploy]' databaas dag destroy -t dev

Pull requests from forks get no secrets, so their deploy stops at the key check.

requirements.txt holds what the DAG files import at parse time. Airflow's constraints file keeps its dependencies at versions it works with; without it, a newer SQLAlchemy can break the Airflow import. Steps that build files into the DAG directory, such as dbt parse for a Cosmos manifest, go before both commands. deploy uploads the directory; only validate runs your DAG files.

Bitbucket Pipelines work the same, with the key and AIRFLOW_CONSTRAINTS in repository variables:

# bitbucket-pipelines.yml
image: python:3.12

pipelines:
  pull-requests:
    "**":
      - step:
          script:
            - pip install 'databaas-utils[deploy]' apache-airflow==3.1.8 -r requirements.txt -c "$AIRFLOW_CONSTRAINTS"
            - databaas dag validate -t dev
            - databaas dag deploy -t dev
  branches:
    main:
      - step:
          script:
            - pip install 'databaas-utils[deploy]' apache-airflow==3.1.8 -r requirements.txt -c "$AIRFLOW_CONSTRAINTS"
            - databaas dag validate -t prod
            - databaas dag deploy -t prod

Bitbucket cannot start a pipeline when a pull request closes. Remove a dev target with databaas dag destroy -t dev --dev-key <dev-key>, for example from a manual pipeline.

Coming: deploying through Lakekeeper

DAG code will move behind Lakekeeper, like the rest of the lakehouse. A deploy will then be signed by Lakekeeper rather than made with an S3 key, so people can deploy their own dev target after databaas login and CI can deploy production as a machine account, without long-lived keys in repository secrets. Who may deploy where will be a Lakekeeper grant.

Until then, databaas dag deploy with the environment's DAG deployment key is the supported way to deploy, from CI.