Companion to README.md, which covers the demo catalog. This file covers pointing
Paracelsus at a DataHub instance of your own — what works out of the box, and the four things that
will need your attention.
Nothing in the scoring path is tied to our demo catalog. Paracelsus talks to DataHub only through the GraphQL API and the MCP server, and it discovers datasets by enumerating whatever the catalog contains.
git clone https://github.com/fiya-chris-and-AI/paracelsus.git
cd paracelsus
make preflight && make venv
Pointing it at your instance is then a two-line change in .env:
DATAHUB_GMS_URL=https://your-datahub.example.com/api/gms
DATAHUB_GMS_TOKEN=<a personal access token from your instance>
Then make risk. Steps 4 and 5 of the Quickstart (make up, which brings up a local DataHub and
loads the demo datapack) are not needed — you already have a catalog.
We have not run this against a second catalog. The claim above is about architecture, not about a run we performed. Four things will need attention, and we would rather name them than let you discover them:
action_threshold in engine/weights.yaml is anchored to a written reference case from our
catalog — a specific, documented defect that we decided should sit exactly at the line between
“act” and “don’t”. Everything else follows from it: Critical is 2× the threshold, Monitor is ⅓×,
and the safety margin printed next to every score is action_threshold / risk.
That anchor is ours, not yours. On your catalog the relative ordering of findings is
meaningful immediately — that is the part the method produces. The labels are not, until you
pick your own reference case: one asset, one defect, where your team agrees this is exactly worth
acting on. Set action_threshold to that asset’s score and the bands mean something again.
We deliberately did not make this automatic. A threshold that calibrates itself is a threshold nobody has to defend, and the entire point of this tool is that every number can be defended in a sentence out loud.
engine/import_usage_stats.py exists to work around a DataHub 1.7.0 behaviour: the
usageFeatures aspect is write-only — it can be ingested but not read back through any of the
seven read paths we tried. We filed this as
datahub#19014. Our workaround re-emits
the datapack’s own query counts as datasetUsageStatistics, which is readable.
If your catalog already carries datasetUsageStatistics — most production instances with usage
ingestion configured do — skip that step entirely. Exposure will read your real query volumes.
If your catalog has no usage data at all, Paracelsus still runs, and this is where its central idea does the work: missing usage evidence is not treated as “unused, therefore safe”. It raises the uncertainty factor. You will see higher scores and wider safety margins, which is the correct answer to “we cannot see who touches this”.
The detector compares the live schema against state/schema_baseline.json, which it maintains
itself. On your catalog, the first make risk establishes that baseline and will legitimately
report nothing — there is no “before” to compare against yet. Real findings appear from the second
run onward, as your schemas actually drift.
Do not carry over seed/schema_baseline_seeded.json. That fixture exists so the demo can show
findings in three minutes instead of three weeks; it describes our catalog’s assets and means
nothing on yours.
engine/preflight.py:262 — EXPECTED_DATASET_COUNT = 67 is the size of our demo catalog. It is
a readiness check for DataHub’s asynchronous indexing, not a limit on the engine. On your
catalog make status will keep reporting “still indexing?”; set it to your dataset count or
skip that check.engine/weights.yaml — max_hops_counted: 4. Successors further away than four hops stop
contributing to exposure. That was tuned to our catalog’s real lineage depth (degrees 2 to 7);
a deeper graph may want a higher number.Dataset enumeration paginates through GraphQL search a page at a time, which is fine for catalogs
in the low hundreds. A catalog with thousands of datasets needs scroll/cursor pagination in
engine/datahub_client.py::search_dataset_urns. We knew this and chose not to build it — see the
comment at that function.