databaas-utils

Read Iceberg tables and store files or zarr arrays in the Databaas lakehouse, from a notebook, a laptop, or an Airflow DAG.

Object storage is reached by asking Lakekeeper to sign each request, so no S3 keys are involved in reading or writing data. You authenticate as yourself (notebooks, laptop) or as a service account (DAGs), and Lakekeeper decides what you can see. Deploying DAG code is the exception for now: CI does it with the environment's DAG deployment key.

Pick your starting point

I am in...	Read	Import from
A Databaas JupyterHub notebook	Notebooks	`databaas_utils.user`
A notebook or script on my laptop	Laptop login	`databaas_utils.user`
An Airflow DAG	Airflow DAGs	`databaas_utils.airflow`
CI deploying a repo's DAGs	Deploying DAGs	`databaas dag deploy`
Something else (a container, Superset, CI)	Reference	`databaas_utils`

Install

The core install is small. Everything heavier is an extra:

Extra	Adds	Use it when
`iceberg`	pyiceberg, pyarrow, s3fs	you read or write Iceberg tables
`obstore`	`obstore-databaas`	you upload or download files
`zarr`	zarr (+ `obstore`)	you use zarr groups
`login`	keyring, filelock	you sign in with `databaas login`
`jwt`	PyJWT	your service account authenticates with a JSON key
`deploy`	PyYAML, boto3	CI runs `databaas dag deploy`
`local`	`iceberg`, `obstore`, `zarr`, `login`	you are an analyst on a laptop
`airflow`	`iceberg`, `obstore`, `jwt`	your DAG environment

pip install "databaas-utils[local]"

Do not install upstream obstore alongside this

obstore-databaas and upstream obstore share the module name obstore. Whichever installs last wins, and only obstore-databaas can do signed access. If signed access fails with RemoteSigningUnavailable, run pip list | grep obstore.

A first look

from databaas_utils.user import get_catalog

catalog = get_catalog()
catalog.load_table("open_meteo.daily_summary").scan(limit=10).to_arrow()

Docs in your editor

Every public function and class has a docstring with arguments, errors raised and an example, so hovering in VS Code, PyCharm or a notebook (Shift+Tab, or help(...)) shows the same content as this site. The package ships type information (py.typed).

Stability

Names exported from databaas_utils.__all__ and databaas_utils.airflow.__all__ follow semantic versioning. Anything underscore-prefixed, or not in those lists, may change in a patch release. See the changelog.