Skip to content

Changelog

All notable changes to this project are documented here. The format follows Keep a Changelog and this project adheres to Semantic Versioning.

Semver covers the names exported from databaas_utils.__all__ and databaas_utils.airflow.__all__, the Airflow connection field contract, and the Iceberg index schema. Anything underscore-prefixed or absent from __all__ is not covered.

[0.6.0] - Unreleased

Added

  • databaas dag deploy deploys a repo's Airflow code from CI. It reads bundle.name, targets and artifacts.dags from the repo's databaas.yml and syncs that directory to dags/prod/<bundle>/ or dags/dev/<dev-key>/<bundle>/ in the environment's DAG bucket, the layout the platform's DataBaasS3DagBundle reads. Files that changed are uploaded and files that were removed are deleted, never outside the target's prefix, and a _databaas_target.json is written for target_for and scoped(). --dry-run lists the changes without making them.
  • No login needed. The command authenticates only with the environment's DAG deployment key in AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY. It reads the endpoint from AWS_ENDPOINT_URL and the bucket from DATABAAS_DAGS_BUCKET, or from --endpoint-url and --bucket. Without the key it stops and says so. That is expected on a laptop: deploying with your own login, through Lakekeeper, comes later.
  • Development targets are named after the branch (GITHUB_HEAD_REF, BITBUCKET_BRANCH, else git), as the same slug the per-repo workflows computed, and --dev-key overrides it. Before upload, a dev deploy renames the repo's top-level packages and modules to <name>_<dev_key> and rewrites their imports, so two open branches cannot shadow each other on Airflow's shared sys.path. This is done on a staged copy; the checkout is not changed.
  • databaas dag validate parses a target's DAGs with Airflow's DagBag before a deploy, staged exactly as deploy would upload them and with the bundle appended to sys.path as in the cluster. It fails on import errors, on no DAGs, and on a dev target's dag id without the dev_<dev-key>_ prefix, reports problems against the repo's own files, and adds GitHub annotations in Actions. It needs Airflow and the DAGs' dependencies installed, and no key. It replaces the per-repo validate scripts and their copy of the databaas_target shim.
  • databaas dag destroy deletes a target's prefix, for cleanup when a pull request closes. Production needs --force.
  • deploy extra (PyYAML, boto3); all includes it.

[0.5.3] - 2026-10-08

Fixed

  • Using a second Iceberg table no longer breaks the first with 403 Forbidden. pyiceberg's FsspecFileIO gives every table the same S3 client and re-points its signer at whichever table loaded last, so table A's reads and writes were signed for table B and refused. Its per-thread filesystem cache also re-registers the signer on that shared client from each worker thread, and a request in flight at that moment went out unsigned; that made overwrite and other multi-threaded operations fail intermittently. rest_catalog now loads a LakekeeperCatalog whose tables use a SigningFileIO with S3 clients of their own, signer attached before first use. Lakekeeper's loadTable response names the stock FsspecFileIO, which the catalog swaps for SigningFileIO.

Changed

  • The integration stack pulls MinIO from cgr.dev/chainguard/minio. minio/minio and minio/mc now refuse anonymous pulls on both Docker Hub and quay.io, so CI could not start the stack.

[0.5.2] - 2026-10-08

Added

  • databaas profile delete NAME logs the profile out, removes its keychain entry and drops it from config.toml. Deleting the default profile leaves no default; the command names the remaining profiles to pick from with databaas profile use. On macOS it also removes the profile's pre-0.5.2 databaas-utils entry when it can, and says so when only the Python installation that saved it may.

Fixed

  • Logging in again on macOS no longer fails with -25244 once Python has changed. macOS lets only the binary that created a Keychain entry replace it, and through keyring that binary was the Python interpreter, so a uv Python upgrade, another virtualenv or another Python version could not save the rotated refresh token. On macOS the refresh token is now written through /usr/bin/security, whose identity does not change, and the access token is cached in a 0600 file under <cache_dir>/sessions. DATABAAS_TOKEN_STORE=keyring on macOS selects this store too.
  • databaas logout no longer crashes when the issuer cannot be reached. Revoking the refresh token was meant to be best effort, but fetching the issuer's OIDC metadata first could raise, so logging out of a removed environment or while offline failed.

Changed

  • Log in once more on macOS after upgrading. The new store uses its own Keychain service, databaas-refresh-token. Entries under the old databaas-utils service are no longer read; delete them in Keychain Access if you want them gone.

[0.5.1] - 2026-10-06

Fixed

  • A login saved by another Python installation no longer crashes with a keyring traceback. On macOS only the program that created a Keychain entry may replace or remove it, so a second virtualenv logging in to the same profile hit PasswordSetError (-25244). Saving now raises a ConfigurationError naming the profile and the ways out: databaas logout --profile NAME from the other installation, deleting the entry in Keychain Access, another --profile, or DATABAAS_TOKEN_STORE=file.
  • databaas logout no longer reports success when the Keychain refuses to remove the entry. A refusal for an entry that still exists is now the same clear error; a missing entry is still a no-op.

[0.5.0] - 2026-10-06

Added

  • Package documentation. A docs/ site (mkdocs) with guides for notebooks, Airflow DAGs and laptop login, and a generated API reference. Setting up the Airflow connection is documented at https://docs.databaas.eu/pipelines/service-account; the Zitadel app for local login is provisioned by the platform. Build it with uv run --group docs --all-extras mkdocs build.
  • Docstrings on the whole public API, with arguments, raised errors and examples, so editor hover and help() are useful. Ruff's pydocstyle rules (Google convention) now fail CI on an undocumented public name.

Changed

  • databaas login URL derives the Zitadel and catalog addresses from the portal's base URL (auth.<host>, catalog.<host>) and defaults the warehouse to databaas-lakehouse. The client id comes from the portal's /.well-known/databaas-client.json, which now only needs client_id; --client-id skips that lookup, and --issuer, --catalog-url and --warehouse still override.
  • --project-id is optional. Without it, databaas login requests no urn:zitadel:iam:org:project:id:<id>:aud scope: Zitadel already puts the app's own project in the token's audience, which is what Lakekeeper and Trino check. A saved profile may have an empty project_id.
  • The warehouse defaults to databaas-lakehouse everywhere it is read from configuration: databaas login (all forms), saved profiles, LakehouseSettings.from_mapping() and from_env() (LAKEKEEPER_WAREHOUSE is now optional), and the Airflow connection's Warehouse field. Set it only to use another warehouse. The LakehouseSettings constructor still takes it explicitly.
  • databaas login and databaas whoami show your email again. Zitadel leaves it out of the access token, so it is now read from the issuer's userinfo endpoint (UserLogin.userinfo()); the user id is shown when that is unavailable.
  • databaas_utils.notebook is now databaas_utils.user. Its helpers authenticate as the person running the code, on a laptop as much as in a notebook, and the old name hid that. databaas_utils.notebook stays as an alias, so existing imports keep working.

[0.4.0] - Unreleased

Added

  • Notebook access as the logged-in JupyterHub user. JupyterHubToken is a TokenProvider that fetches the user's access token from the Databaas hub's GET /hub/api/oauth-token, authenticating with the JUPYTERHUB_API_TOKEN every singleuser pod carries. It caches the token until shortly before its own exp — each hub call rotates the refresh token at Zitadel — and hands pyiceberg and the S3 signer a live reference, so a notebook keeps working past the one-hour token lifetime without re-creating its catalog. LakehouseSettings.from_jupyterhub() builds settings from the pod's environment (CATALOG_URI, LAKEKEEPER_WAREHOUSE); region and endpoint come from Lakekeeper's table config. Replaces the databaas_auth helper previously copied into the notebooks image.
  • databaas_utils.notebook — get_catalog(), get_store(), get_index(), get_zarr_group() and get_settings() for notebooks, mirroring databaas_utils.airflow's helpers. Inside JupyterHub they use the hub token; anywhere else the databaas login session. profile= forces a login profile.

[0.3.0] - Unreleased

Added

  • Local, interactive login as your own Zitadel identity — no service account needed. databaas login (a new databaas console script) opens a browser for Authorization Code + PKCE, or --device for the OAuth2 device-code flow on machines without a reachable browser. The session is cached across Python processes: the access token in memory, the refresh token in the OS keychain via a new login extra (keyring, filelock). databaas whoami / token / logout manage it; databaas profile list / use switch between named environments saved in ~/.config/databaas/config.toml.
  • LakehouseSettings.from_profile() — the Python-side entry point, building settings from a saved profile and a UserLogin credential. New Profile, load_profile, UserLogin and login() exports.
  • New local extra (databaas-utils[local]) for the analyst install: iceberg, obstore, zarr and login in one.
  • databaas login also registers the caller with Lakekeeper (POST /management/v1/user), since the platform's own pre-provisioning webhook is currently broken — see the README's admin runbook for the Zitadel app this needs.

Fixed

  • A custom TokenProvider (including the new UserLogin and the existing ServiceAccountKey) is no longer frozen into a single static bearer when building a pyiceberg catalog or a pylakekeeper client. Both now hold a live reference to the provider and call auth_header() fresh on every request, so a session whose access token is short-lived (an hour, for a user login) keeps working past that hour instead of failing once the frozen token expired.

0.2.1 - 2026-09-18

Fixed

  • __version__ now actually matches the installed version. It was a hand-maintained literal, separate from pyproject.toml's version — bumped there for 0.2.0, never bumped here, so the published 0.2.0 wheel reported __version__ == "0.1.0". Derived from installed package metadata instead, so the two cannot drift apart again.

0.2.0 - 2026-09-18

Added

  • ServiceAccountKey — Private Key JWT (RFC 7523) authentication for Zitadel service accounts, for consoles that only offer a downloadable JSON key for a machine user, not a plain client secret. Signs a JWT assertion with the key's private key and exchanges it for a real access token via the urn:ietf:params:oauth:grant-type:jwt-bearer grant; the resulting token is a normal Zitadel-issued JWT, indistinguishable to Lakekeeper from one obtained via client_credentials. ServiceAccountKey.from_json() builds one directly from the key file's own contents, its raw JSON text, or a path to it. Takes priority over client_id/token when a connection's service_account_key extra is set.
  • New [jwt] extra (PyJWT[crypto]), pulled into [airflow] and [all]. Kept out of core dependencies: it pulls in cryptography, a real compiled dependency most callers — anyone using a client secret or a static token — never need.

Fixed

  • A Zitadel Personal Access Token does not work against Lakekeeper, and is not a supported credential form here. A PAT is an opaque credential meant for calling Zitadel's own APIs, not a JWT a resource server's OIDC validator can parse — Lakekeeper rejects it with a plain AuthenticationFailed, before any permission check. ServiceAccountKey (above) is the fix for the console configurations that made a PAT look like the only available option.

0.1.0 - 2026-09-16

First release. Extracted from Infoplaza's lakehouse_core, which shipped as a directory synced into a DAG bundle rather than as a package.

Added

  • LakehouseSettings, with credentials as a sum type (OAuthClientCredentials / StaticToken / any TokenProvider) rather than six nullable fields validated against each other.
  • CatalogClient and rest_catalog, wired for Lakekeeper's remote signer.
  • GenericTableStore — blob access under generic tables, signed rather than credentialed, with put/get/open/list/delete.
  • FileIndex — an Iceberg record of what landed in the generic tables.
  • open_zarr_group — zarr groups backed by the signed object store.
  • scoped / slug for deployment-scoped naming, and the Airflow layer: LakehouseHook (a registered databaas_lakehouse connection type with a working Test button), get_catalog / get_store / get_index / get_zarr_group, and the databaas_target deployment shim.

Fixed

  • Remote signing is now actually enabled. The predecessor set s3.remote-signing-enabled, which pyiceberg does not implement, and left py-io-impl unset — so the catalog resolved to PyArrowFileIO, which cannot sign, and reached S3 with whatever ambient credentials the process carried. rest_catalog now sets py-io-impl and s3.signer, and leaves s3.signer.uri to the server's loadTable response.
  • Plaintext endpoints work. obstore's Rust object_store rejects an http:// endpoint with BadScheme before any I/O unless allow_http is set; signed_store now sets it when, and only when, the endpoint is already plaintext.

Changed

  • An unresolvable connection now raises ConnectionNotConfigured. The predecessor returned None, which callers turned into a local filesystem sink — so a typo'd conn_id redirected a production write to an ephemeral pod disk and the task went green. Local storage is now an explicit local_root=.
  • FileRecord no longer requires product or forecast_hour, and gained an attributes map. The predecessor's schema encoded weather-forecast knowledge a general-purpose index should not demand.