Rifqi Hafizuddin Claude Opus 4.8 commited on
Commit Β·
61e6fa3
1
Parent(s): 5f4818a
/docs REPO_STATUS: FK inference (FK-less dedorch catalog) + shared Fernet-key gotcha
Browse files- REPO_STATUS.md +10 -1
REPO_STATUS.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
|
| 3 |
**Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
|
| 4 |
**Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only β no roadmap or to-dos.
|
| 5 |
-
**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** Β§8/Β§12 updated β dedorch cutover done, `data_catalog` model reconciled. **Cross-repo update 2026-06-29:** Β§2/Β§8/Β§11/Β§12 re-verified against
|
| 6 |
the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
|
| 7 |
own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
|
| 8 |
**`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet β those skills
|
|
@@ -199,6 +199,8 @@ unless `SKIP_INIT_DB=true`.
|
|
| 199 |
**Catalog shape** (the jsonb in `data_catalog`):
|
| 200 |
`Catalog β Source[ {source_id, source_type β schema|tabular|unstructured, name, location_ref} β Table[ {table_id, name, row_count, foreign_keys[]} β Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]`. PII columns have `sample_values: null` so real values never enter prompts.
|
| 201 |
|
|
|
|
|
|
|
| 202 |
**QueryIR shape** (`src/query/ir/models.py`):
|
| 203 |
`{ source_id, table_id, joins[], select[], filters[], group_by[], order_by[], limit }`.
|
| 204 |
Joins are single-level equi-joins to a related table **in the same source**, FK-backed,
|
|
@@ -358,6 +360,13 @@ records-based report; floor: β₯1 `analyze_*` success). Wiring Go β Python is
|
|
| 358 |
values are always parameterized.
|
| 359 |
- **Settings aliases:** `.env` uses double-underscore names (`azureai__api_key__4o`); `Settings`
|
| 360 |
exposes them as `azureai_api_key_4o`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 361 |
- **Never-throw seams** are pervasive (tool invoker, query service, executors, state/binding reads,
|
| 362 |
record persistence, report summary). Failures degrade into soft output rather than raising β good
|
| 363 |
for UX, but they can mask real breakage (e.g. a binding silently fail-opening to the full catalog).
|
|
|
|
| 2 |
|
| 3 |
**Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
|
| 4 |
**Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only β no roadmap or to-dos.
|
| 5 |
+
**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** Β§8/Β§12 updated β dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** Β§8/Β§13 β dedorch catalogs ship no FKs β Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Cross-repo update 2026-06-29:** Β§2/Β§8/Β§11/Β§12 re-verified against
|
| 6 |
the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
|
| 7 |
own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
|
| 8 |
**`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet β those skills
|
|
|
|
| 199 |
**Catalog shape** (the jsonb in `data_catalog`):
|
| 200 |
`Catalog β Source[ {source_id, source_type β schema|tabular|unstructured, name, location_ref} β Table[ {table_id, name, row_count, foreign_keys[]} β Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]`. PII columns have `sample_values: null` so real values never enter prompts.
|
| 201 |
|
| 202 |
+
> β οΈ **dedorch catalogs ship empty `foreign_keys`** (Go's introspection drops FK constraints), yet the IR validator only allows FK-backed joins β so every cross-table question failed validation until 2026-07-02. `src/catalog/fk_inference.py` (wired into `CatalogStore.get`) now infers the obvious `<base>_id β <table>.id` edges at read time: conservative (single unambiguous target, matching `data_type`, schema sources only) and **self-disabling** once any real FK is present. It's a **stopgap** β the durable fix is Go emitting real FKs during introspection.
|
| 203 |
+
|
| 204 |
**QueryIR shape** (`src/query/ir/models.py`):
|
| 205 |
`{ source_id, table_id, joins[], select[], filters[], group_by[], order_by[], limit }`.
|
| 206 |
Joins are single-level equi-joins to a related table **in the same source**, FK-backed,
|
|
|
|
| 360 |
values are always parameterized.
|
| 361 |
- **Settings aliases:** `.env` uses double-underscore names (`azureai__api_key__4o`); `Settings`
|
| 362 |
exposes them as `azureai_api_key_4o`.
|
| 363 |
+
- **Shared Fernet key across repos (gotcha).** User DB credentials in `databases` are written +
|
| 364 |
+
encrypted by **Go** and decrypted by Python; both read the **same** env var
|
| 365 |
+
`dataeyond__db__credential__key` (Go: `configs/app.yaml` β `credentials.fernet_key`). The two
|
| 366 |
+
deployments MUST hold the **identical value** or Python's decrypt throws
|
| 367 |
+
`cryptography.fernet.InvalidToken` β whose `str()` is **empty**, so it logged as `error=""` and
|
| 368 |
+
masqueraded as a DB-connection failure (the executor now logs `repr(e)` to expose it). Tell-apart:
|
| 369 |
+
a valid-but-wrong key β `InvalidToken`; a malformed key β a non-empty `ValueError` at cipher build.
|
| 370 |
- **Never-throw seams** are pervasive (tool invoker, query service, executors, state/binding reads,
|
| 371 |
record persistence, report summary). Failures degrade into soft output rather than raising β good
|
| 372 |
for UX, but they can mask real breakage (e.g. a binding silently fail-opening to the full catalog).
|