feat: initial project setup
This commit is contained in:
64
README.md
64
README.md
@@ -1 +1,63 @@
|
||||
# data-platform
|
||||
# data-platform
|
||||
|
||||
A [Dagster](https://dagster.io/) + [dbt](https://www.getdbt.com/) data platform,
|
||||
managed with [uv](https://github.com/astral-sh/uv) and deployed via Docker Compose.
|
||||
|
||||
## Stack
|
||||
|
||||
| Layer | Tool |
|
||||
|---|---|
|
||||
| Orchestration | Dagster (webserver + daemon) |
|
||||
| Transformation | dbt-core + dbt-postgres |
|
||||
| Storage | PostgreSQL 16 |
|
||||
| Package/venv | uv |
|
||||
| Secrets | `.env` file |
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
data_platform/ # Dagster Python package (assets, definitions)
|
||||
dbt/ # dbt project (models, seeds, tests)
|
||||
profiles.yml # reads credentials from env vars
|
||||
dagster_home/ # dagster.yaml + workspace.yaml
|
||||
Dockerfile # single image used by both dagster services
|
||||
docker-compose.yaml # postgres + dagster-webserver + dagster-daemon
|
||||
.env.example # copy to .env and fill in credentials
|
||||
pyproject.toml # uv-managed dependencies
|
||||
```
|
||||
|
||||
## Getting started
|
||||
|
||||
```bash
|
||||
# 1. Install uv (if not already)
|
||||
curl -Lsf https://astral.sh/uv/install.sh | sh
|
||||
|
||||
# 2. Clone and enter the project
|
||||
cd ~/git/data-platform
|
||||
|
||||
# 3. Create your credentials file
|
||||
cp .env.example .env
|
||||
# Edit .env with your passwords
|
||||
|
||||
# 4. Install dependencies into a local venv
|
||||
uv sync
|
||||
|
||||
# 5. Generate the dbt manifest (needed before first run)
|
||||
uv run dbt parse --profiles-dir dbt --project-dir dbt
|
||||
|
||||
# 6. Start all services
|
||||
docker compose up -d --build
|
||||
|
||||
# 7. Open the Dagster UI
|
||||
# http://localhost:3000
|
||||
```
|
||||
|
||||
## Local development (without Docker)
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
source .venv/bin/activate
|
||||
|
||||
# Run the Dagster UI locally
|
||||
DAGSTER_HOME=$PWD/dagster_home dagster dev
|
||||
```
|
||||
Reference in New Issue
Block a user