Rifqi Hafizuddin Claude Opus 4.8 commited on
Commit
61e6fa3
Β·
1 Parent(s): 5f4818a

/docs REPO_STATUS: FK inference (FK-less dedorch catalog) + shared Fernet-key gotcha

Browse files
Files changed (1) hide show
  1. REPO_STATUS.md +10 -1
REPO_STATUS.md CHANGED
@@ -2,7 +2,7 @@
2
 
3
  **Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
4
  **Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only β€” no roadmap or to-dos.
5
- **Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** Β§8/Β§12 updated β€” dedorch cutover done, `data_catalog` model reconciled. **Cross-repo update 2026-06-29:** Β§2/Β§8/Β§11/Β§12 re-verified against
6
  the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
7
  own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
8
  **`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet β€” those skills
@@ -199,6 +199,8 @@ unless `SKIP_INIT_DB=true`.
199
  **Catalog shape** (the jsonb in `data_catalog`):
200
  `Catalog β†’ Source[ {source_id, source_type ∈ schema|tabular|unstructured, name, location_ref} β†’ Table[ {table_id, name, row_count, foreign_keys[]} β†’ Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]`. PII columns have `sample_values: null` so real values never enter prompts.
201
 
 
 
202
  **QueryIR shape** (`src/query/ir/models.py`):
203
  `{ source_id, table_id, joins[], select[], filters[], group_by[], order_by[], limit }`.
204
  Joins are single-level equi-joins to a related table **in the same source**, FK-backed,
@@ -358,6 +360,13 @@ records-based report; floor: β‰₯1 `analyze_*` success). Wiring Go β†’ Python is
358
  values are always parameterized.
359
  - **Settings aliases:** `.env` uses double-underscore names (`azureai__api_key__4o`); `Settings`
360
  exposes them as `azureai_api_key_4o`.
 
 
 
 
 
 
 
361
  - **Never-throw seams** are pervasive (tool invoker, query service, executors, state/binding reads,
362
  record persistence, report summary). Failures degrade into soft output rather than raising β€” good
363
  for UX, but they can mask real breakage (e.g. a binding silently fail-opening to the full catalog).
 
2
 
3
  **Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
4
  **Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only β€” no roadmap or to-dos.
5
+ **Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** Β§8/Β§12 updated β€” dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** Β§8/Β§13 β€” dedorch catalogs ship no FKs β†’ Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Cross-repo update 2026-06-29:** Β§2/Β§8/Β§11/Β§12 re-verified against
6
  the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
7
  own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
8
  **`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet β€” those skills
 
199
  **Catalog shape** (the jsonb in `data_catalog`):
200
  `Catalog β†’ Source[ {source_id, source_type ∈ schema|tabular|unstructured, name, location_ref} β†’ Table[ {table_id, name, row_count, foreign_keys[]} β†’ Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]`. PII columns have `sample_values: null` so real values never enter prompts.
201
 
202
+ > ⚠️ **dedorch catalogs ship empty `foreign_keys`** (Go's introspection drops FK constraints), yet the IR validator only allows FK-backed joins β€” so every cross-table question failed validation until 2026-07-02. `src/catalog/fk_inference.py` (wired into `CatalogStore.get`) now infers the obvious `<base>_id β†’ <table>.id` edges at read time: conservative (single unambiguous target, matching `data_type`, schema sources only) and **self-disabling** once any real FK is present. It's a **stopgap** β€” the durable fix is Go emitting real FKs during introspection.
203
+
204
  **QueryIR shape** (`src/query/ir/models.py`):
205
  `{ source_id, table_id, joins[], select[], filters[], group_by[], order_by[], limit }`.
206
  Joins are single-level equi-joins to a related table **in the same source**, FK-backed,
 
360
  values are always parameterized.
361
  - **Settings aliases:** `.env` uses double-underscore names (`azureai__api_key__4o`); `Settings`
362
  exposes them as `azureai_api_key_4o`.
363
+ - **Shared Fernet key across repos (gotcha).** User DB credentials in `databases` are written +
364
+ encrypted by **Go** and decrypted by Python; both read the **same** env var
365
+ `dataeyond__db__credential__key` (Go: `configs/app.yaml` β†’ `credentials.fernet_key`). The two
366
+ deployments MUST hold the **identical value** or Python's decrypt throws
367
+ `cryptography.fernet.InvalidToken` β€” whose `str()` is **empty**, so it logged as `error=""` and
368
+ masqueraded as a DB-connection failure (the executor now logs `repr(e)` to expose it). Tell-apart:
369
+ a valid-but-wrong key β†’ `InvalidToken`; a malformed key β†’ a non-empty `ValueError` at cipher build.
370
  - **Never-throw seams** are pervasive (tool invoker, query service, executors, state/binding reads,
371
  record persistence, report summary). Failures degrade into soft output rather than raising β€” good
372
  for UX, but they can mask real breakage (e.g. a binding silently fail-opening to the full catalog).