64 lines
1.7 KiB
Markdown
64 lines
1.7 KiB
Markdown
# data-platform
|
|
|
|
A [Dagster](https://dagster.io/) + [dbt](https://www.getdbt.com/) data platform, managed with
|
|
[uv](https://github.com/astral-sh/uv) and deployed via Docker Compose.
|
|
|
|
## Stack
|
|
|
|
| Layer | Tool |
|
|
| -------------- | ---------------------------- |
|
|
| Orchestration | Dagster (webserver + daemon) |
|
|
| Transformation | dbt-core + dbt-postgres |
|
|
| Storage | PostgreSQL 16 |
|
|
| Package/venv | uv |
|
|
| Secrets | `.env` file |
|
|
|
|
## Project layout
|
|
|
|
```
|
|
data_platform/ # Dagster Python package (assets, definitions)
|
|
dbt/ # dbt project (models, seeds, tests)
|
|
profiles.yml # reads credentials from env vars
|
|
dagster_home/ # dagster.yaml + workspace.yaml
|
|
Dockerfile # single image used by both dagster services
|
|
docker-compose.yaml # postgres + dagster-webserver + dagster-daemon
|
|
.env.example # copy to .env and fill in credentials
|
|
pyproject.toml # uv-managed dependencies
|
|
```
|
|
|
|
## Getting started
|
|
|
|
```bash
|
|
# 1. Install uv (if not already)
|
|
curl -Lsf https://astral.sh/uv/install.sh | sh
|
|
|
|
# 2. Clone and enter the project
|
|
cd ~/git/data-platform
|
|
|
|
# 3. Create your credentials file
|
|
cp .env.example .env
|
|
# Edit .env with your passwords
|
|
|
|
# 4. Install dependencies into a local venv
|
|
uv sync
|
|
|
|
# 5. Generate the dbt manifest (needed before first run)
|
|
uv run dbt parse --profiles-dir dbt --project-dir dbt
|
|
|
|
# 6. Start all services
|
|
docker compose up -d --build
|
|
|
|
# 7. Open the Dagster UI
|
|
# http://localhost:3000
|
|
```
|
|
|
|
## Local development (without Docker)
|
|
|
|
```bash
|
|
uv sync
|
|
source .venv/bin/activate
|
|
|
|
# Run the Dagster UI locally
|
|
DAGSTER_HOME=$PWD/dagster_home dagster dev
|
|
```
|