paracelsus
Paracelsus
Your own catalog: readme_own_catalog.md — what is portable, and the four things that need your attention.
Live board: fiya-chris-and-ai.github.io/paracelsus/board/ — the same output this repo produces, served as a static snapshot.
“Sola dosis facit venenum” — the dose makes the poison.
New here? explainer.html is the plain-language overview — what Paracelsus
does, how the risk math works, how to start it, and where its limits are. Open it directly in a
browser, no server or repo context needed. The Quickstart below is for running it yourself.
Paracelsus is a risk-scoring tool for a DataHub data catalog. It replaces naive “blast radius” scoring (how many downstream assets could this break?) with a regulatory drug-safety methodology applied to data: Risk = Hazard x Exposure x Receptor-sensitivity x Uncertainty-factor. Hazard is what actually changed in the schema. Exposure is measured usage, not raw successor count — a table with 24 downstream links that nobody queries is lower risk than one with 3 links under heavy real load. Receptor-sensitivity weights who’s downstream (a finance dashboard is not a scratch table). And Uncertainty-factor is the inversion that gives the tool its name: missing evidence about a change raises the score instead of silently dropping it, the opposite of what naive blast-radius tools do when their inputs are absent. An unconfirmed dose is not a zero-risk dose. Every score ships with a plain-language recommended action, so a finding ends in a decision, not just a number.
Prerequisites
Five things need to be on your machine before anything in this repo will run. make preflight
(see below) checks all five for you — but here’s what each one is for, in case you need to
install it first. The reference machine for this project is Apple Silicon macOS; install
pointers are given for Intel Mac and Linux too since a juror’s machine may not match.
| Tool | What it’s for | macOS | Linux |
|---|---|---|---|
| Docker (Desktop or Engine) | Runs the DataHub quickstart stack (GMS, frontend, Kafka, Elasticsearch, etc. via datahub docker quickstart) |
Docker Desktop | Docker Engine |
| pipx | Installs the datahub CLI in its own isolated environment, separate from this project’s .venv |
brew install pipx && pipx ensurepath |
python3 -m pip install --user pipx && python3 -m pipx ensurepath |
| Python 3.11 | Needed twice, independently: pipx uses it to build the datahub CLI’s own environment, and this project’s own .venv (make venv) is built with it too. See “Known fragility” below. |
brew install python@3.11 |
sudo apt install python3.11 or pyenv install 3.11 |
datahub CLI |
The DataHub command-line tool this repo shells out to (datahub docker quickstart, datahub datapack load, etc.) |
Install via pipx, see next section — this step is not automated by anything in this repo | same |
uv / uvx (Astral) |
Runs the mcp-server-datahub MCP server as a subprocess (uvx --python 3.11 mcp-server-datahub@latest), invoked from engine/write_back.py and engine/mcp_smoke_test.py |
curl -LsSf https://astral.sh/uv/install.sh \| sh |
same — see astral.sh/uv docs |
uv is more self-sufficient than the other two Python dependencies above: it can fetch its own
Python versions on demand and doesn’t rely on a system-wide python3.11 at all.
Quickstart
Run these in order from the paracelsus/ directory. Each step’s own doc-comment in the
Makefile explains what it does and why — this list mirrors that, in the order that actually
works.
# 1. Verify Docker, pipx, Python 3.11, the datahub CLI, and uv/uvx are all present and working.
# Run this FIRST, before anything else — a missing tool here is the #1 cause of a juror not
# being able to start the project. Uses only the system python3, no project deps required.
make preflight
# 2. Create this project's own .venv and install its Python dependencies.
make venv
# 3. One-time: install the datahub CLI itself via pipx. Nothing in this repo does this for you —
# if you skip it, every later step that shells out to `datahub` will fail.
pipx install --python python3.11 acryl-datahub
# 4. Bring up DataHub (token auth on), load the showcase-ecommerce datapack, fix up usage stats,
# mint an access token. Safe to re-run against an already-running stack.
make up
# 5. Verify the catalog actually loaded correctly. First waits for DataHub's async indexing to
# settle (it can under-report the dataset count for up to ~2 minutes after step 4 -- see
# Troubleshooting below), then checks the three data-availability conditions the risk engine
# needs (see engine/verify_data_availability.py).
make status
# 6. Run the risk pipeline: Detect -> Expose -> Assess over the loaded catalog. No manual seeding
# step needed -- see "How the demo numbers reproduce" below. Safe to run as many times as you
# like, in any order, from any number of terminals at once; every run produces the same numbers.
make risk
# 7. One-time: define this project's structured properties in DataHub. Safe to re-run.
make write-setup
# 8. Write the risk assessment back into DataHub as structured properties, risk-band tags, and
# a document per asset.
make write
# 9. Serve the triage board.
make board
Once make board is running, open http://localhost:8090 to see the triage board.
How the demo numbers reproduce
make risk diffs the live catalog against seed/schema_baseline_seeded.json — a committed,
frozen fixture, not a file this project ever writes to. That fixture is the live catalog’s real
schema with the five [GEMOCKT/KURATIERT] demo incidents’ pre-refactor field values overlaid on
top (see seed/demo_defects_seed.json for exactly which fields and why). make risk’s own
current-catalog snapshot goes to state/schema_baseline.json instead — gitignored scratch/log
state that nothing reads back as input. Every run, on any machine, in any order, concurrently or
not, diffs the same frozen baseline against the same live catalog and finds the same five
confirmed transitions: 44.47 / 36.92 / 23.70 / 14.83 / 7.35. There is nothing to run once,
nothing to recover from, and no manual seeding step in the Quickstart above on purpose.
If the seed manifest or the underlying catalog ever changes and the fixture needs regenerating,
engine/seed_demo_defects.py’s own docstring has the exact recipe (fetch a fresh live snapshot,
apply the seed, commit the result) — not a step in this Quickstart, since it should not need to
happen on a normal clone.
Honesty boundary: what’s real, what’s curated
The underlying catalog is real: this project runs against DataHub’s own showcase-ecommerce
demo datapack — genuine schemas, lineage, and usage statistics, loaded via
datahub datapack load. Five specific assets carry deliberately seeded, disclosed defects
(fifteen field-level schema changes, marked [GEMOCKT/KURATIERT] throughout
seed/demo_defects_seed.json), because the real catalog on its own has no natural schema drift
for the detector to find and triage. A handful of the seeded fields are fictional placeholders
that never existed in the live catalog — they’re marked as such directly in that file’s own
comments, not silently invented. Every score the demo shows for those five assets is still a
real, mechanically-computed output of the exact same detection pipeline every other asset runs
through — what’s curated is the input (a plausible defect story), not the scoring logic. The
demo never claims a curated data point is organic. What’s curated is that a defect exists on
five specific assets — the seed decides something broke. What is not curated is what separates
their scores: the identical null_spike defect class, planted on two of those five assets, scores
36.92 on snowflake CUSTOMERS and 14.83 on postgres addresses — and postgres addresses is the
asset a naive hazard ranker puts second in the entire catalog, on 24 downstream assets. That gap
comes entirely from real inputs the seed file never touches: exposure from real lineage traversal,
real hop distance, and real query volume on the unmodified catalog; receptor sensitivity from real
tags and domains; the uncertainty surcharge from real metadata gaps. The seed decides that
something broke. The catalog decides whether it matters. The other 22 assets carry no seeded
defect and score exactly 0.0 — that’s not a stable-sort artifact to explain away, it’s the model
working as intended: zero confirmed transitions means zero risk, full stop. See
seed/demo_defects_seed.json’s own _comment and _comment_2026_08_09_expansion fields for the
full reasoning behind which five assets were chosen and why.
Troubleshooting
Auth setting looks wrong / METADATA_SERVICE_AUTH_ENABLED env var override doesn’t work.
This is a hardcoded literal in the committed docker/docker-compose.yml, not a ${VAR} default
— DataHub’s upstream quickstart compose hardcodes it to 'false', and this repo’s copy patches
it directly to 'true'. Setting the environment variable does nothing; if auth seems off, open
docker/docker-compose.yml and check the METADATA_SERVICE_AUTH_ENABLED line directly.
Reading usageFeatures directly returns nothing. That aspect is write-only in this DataHub
version — verified independently across every read path GMS exposes. Don’t be alarmed; it’s
expected. engine/import_usage_stats.py (run automatically as part of make up) re-emits the
same real datapack values through datasetUsageStatistics, DataHub’s standard externally-readable
usage aspect, which is queryable and is what make status and make risk actually read.
import_usage_stats.py needs a different Python than the rest of this project. This project’s
own .venv (make venv) deliberately does not carry the full acryl-datahub SDK — it conflicts
with the mcp package’s pydantic requirement. make up already handles this correctly by
invoking that one script with the pipx-installed datahub CLI’s own interpreter instead of
.venv’s; you only need to know this if you’re calling it manually.
Anything hits a timeout / connection error talking to GMS — most often, Docker Desktop’s VM ran
out of disk. engine/datahub_client.py names the two most likely causes directly in the error
message, in the order to check them. The first is the more serious one and easy to miss because
nothing about the visible symptom points at disk:
Docker Desktop ships a VM with its own disk limit, separate from the host machine’s — 16 GB by
default. The DataHub stack alone needs about 16.3 GB (roughly 11.8 GB of images, 2.8 GB of
containers, 1.7 GB of volumes), just over that default. When the VM’s disk fills, the Kafka broker
inside it exits with ERROR Shutdown broker because all log dirs in /var/lib/kafka/data have
failed / Caused by: java.io.IOException: No space left on device — and every pipeline call
after that dies as a plain requests.exceptions.ReadTimeout against localhost:8080, because GMS
can no longer produce events to a broker that’s gone. Nothing in that visible error mentions disk.
Checking the host’s own disk does not help either: df -h / on the Mac/Linux host reports the
host’s free space and can show tens of gigabytes free the entire time the VM’s filesystem is
completely full — the VM’s disk and the host’s disk are two different things, and this failure
mode is only visible from inside the VM (docker run --rm alpine df -h /).
Fix: Docker Desktop → Settings → Resources → Advanced → raise “Disk usage limit” to at least
40 GB, then Apply & Restart, then make up again. make preflight checks this automatically
before pulling any images, so a fresh clone should catch it before it happens — this section is
for anyone who hits the timeout anyway, e.g. because the limit was lowered again after preflight
last ran.
The second, less serious cause: DataHub is still starting up or still indexing. Check
docker ps --filter name=datahub — every container should say “healthy”, not “starting”; if
they’re all healthy, the catalog may just not be done indexing yet (make status waits for this
automatically). Timeout defaults to 60s; override with DATAHUB_TIMEOUT_SEC in .env if your
machine or connection is slower than the reference machines this was built and verified on.
Known fragility: Python 3.11 discovery on non-reference machines
This project’s reference machine is Apple Silicon macOS, and its make venv target originally
hardcoded the Homebrew keg path for Python 3.11 (/opt/homebrew/opt/python@3.11/bin/python3.11).
That breaks on Intel Mac (different Homebrew prefix, /usr/local) and on Linux entirely. The
venv target now prefers a plain python3.11 on PATH and only falls back to the hardcoded
Homebrew path if that’s what’s actually present — the reference-machine behavior still works,
it’s just no longer the only path that works. engine/preflight.py uses the same
PATH-first-then-Homebrew-fallback logic, and if neither is found it says so plainly
(“Python 3.11 not found”) with an OS-specific next step, instead of a bare stack trace.
License
Apache License 2.0. See LICENSE.