ai-data-registry

Git-native, PR-driven data platform. Fork, add a data pipeline as a workspace, open a PR. Automated validation, extraction, and catalog federation handle the rest.

How It Works

Each workspace is an isolated pipeline with its own language, deps, and compute backend. DuckLake federates all outputs into one queryable global catalog via zero-copy file registration.

Getting Started

Two ways to use this project:

Creating your own registry? See docs/template-setup.md for template initialization, S3 configuration, and backend setup.

Contribute a Workspace (Fork)

Prerequisites: Pixi (brew install pixi or curl -fsSL | bash)

Create a Workspace

Then add your [tool.registry] config, write your extract script, and open a PR. See workspaces/test-minimal/ for a working example.

Workspace Contract

Every workspace pixi.toml needs:

Key rules: Write Parquet to $OUTPUT_DIR/, never to S3 directly. Do not hardcode OUTPUT_DIR in task env. No credentials in code (use $WORKSPACE_SECRET_*).

Compute Backends

Need something else? Open an issue. Infrastructure is maintainer-managed.

PR Validation

Layers 1-2 and 4 work on fork PRs without secrets. Layer 3 gracefully skips when S3 credentials are unavailable.

After validation, a maintainer can trigger full extraction to a staging prefix:

Staging data is auto-cleaned when the PR closes.