Spaces:
Sleeping
Sleeping
ktsn-ud commited on
Commit ·
8779d33
1
Parent(s): dafa8fa
codex: 更新点の洗い出し
Browse files
docs/project_schema_change_points.md
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Project Schema / Data Source Migration – Change Points
|
| 2 |
+
|
| 3 |
+
## Background
|
| 4 |
+
- `schemas/projects.py` now defines DB-backed models whose core fields are `description` (企画内容), `prSummary` (PRコメント), `prDetail` (PRコメント詳細) together with `notes`, `images`, and `urls`.
|
| 5 |
+
- Batch scripts under `scripts/` must keep generating search assets from the database export (`projects.json`) instead of the legacy CSV.
|
| 6 |
+
- The API layer (`app/main.py`) serves the DB-derived models directly, while the search engine (`app/search/engine.py`) consumes the statically generated artefacts.
|
| 7 |
+
|
| 8 |
+
## Config (`config/files.json`)
|
| 9 |
+
- `config/files.json:11-12` still references `projects.original_csv`; this key can be retired once the CSV path is unused.
|
| 10 |
+
- Add an explicit output mapping (e.g. `projects.output_json`) so scripts resolve the generated JSON path without relying on hard-coded fallbacks. Update all callers to reference the new key.
|
| 11 |
+
|
| 12 |
+
## Batch Scripts (`scripts/`)
|
| 13 |
+
### `scripts/2_create_projects_data.py`
|
| 14 |
+
- Lines `20-32` call `ProjectsRepository.list_projects()` and then pass the resulting Pydantic models straight into `json_dumps`; convert them to serialisable dictionaries via `model_dump(mode="json")` (or equivalent) before writing.
|
| 15 |
+
- Ensure `ProjectsRepository` and `app/repositories/query.sql` surface `prSummary`, `prDetail`, and `notes` so the exported JSON aligns with `schemas.projects.Project`.
|
| 16 |
+
|
| 17 |
+
### `scripts/3_build_synonyms_from_sudachi.py`
|
| 18 |
+
- The loader at `scripts/3_build_synonyms_from_sudachi.py:65-112` instantiates `Project` objects and iterates `target_fields`. Verify that the regenerated `projects.json` exposes `name`, `circleName`, `circleNameKana`, `description`, `prSummary`, and `prDetail`; missing keys will degrade vocabulary extraction for synonyms.
|
| 19 |
+
|
| 20 |
+
### `scripts/4_prepare_bm25f_meta.py`
|
| 21 |
+
- The token loop at `scripts/4_prepare_bm25f_meta.py:52-104` depends on the same field names. Confirm that `projects.json` no longer contains legacy keys such as `organization`/`title`, otherwise IDF statistics will be incorrect.
|
| 22 |
+
|
| 23 |
+
### `scripts/5_prepare_tf_token.py`
|
| 24 |
+
- Update the `tf_token.Fields` construction at `scripts/5_prepare_tf_token.py:108-115` to use `name`, `circleName`, `circleNameKana`, `description`, `prSummary`, and `prDetail`. This keeps the TF schema consistent with the updated models.
|
| 25 |
+
- Continue serialising with `model_dump()` at `scripts/5_prepare_tf_token.py:117-127` once the field names match.
|
| 26 |
+
|
| 27 |
+
### `scripts/7_prepare_circle_names.py`
|
| 28 |
+
- The mapper at `scripts/7_prepare_circle_names.py:68-74` still relies on `Project.organization` / `Project.reading`. Replace these with `circleName` / `circleNameKana` so the generated circle name and substring assets remain accurate.
|
| 29 |
+
|
| 30 |
+
## API Layer (`app/main.py`)
|
| 31 |
+
- `app/main.py:94-105` currently dumps `ProjectSummary` instances directly. Convert to `model_dump()` (or `jsonable_encoder`) before gzipping to avoid serialisation errors.
|
| 32 |
+
- `ProjectSummary.image` should pick the `Image` entry whose `order == 0`. If multiple images share `order == 0`, use the *last* one encountered to mirror repository behaviour.
|
| 33 |
+
- Ensure the summary payload exposes `notes` (renamed from `note`) and the PR fields `prSummary` / `prDetail` as returned by the repository.
|
| 34 |
+
|
| 35 |
+
## Search Engine (`app/search/engine.py`)
|
| 36 |
+
- In `SearchEngine.initialize` (`app/search/engine.py:164-175`) we populate kana norms into `self.circleNameKana_norms`, but the boosting logic reads `self.reading_norms` (`app/search/engine.py:545`). Assign the kana norms into `self.reading_norms` so substring boosts still consider kana.
|
| 37 |
+
- Debug responses at `app/search/engine.py:643-645` reference `organization` / `title`; switch these to `circleName` / `name` so diagnostics stay meaningful with the new schema.
|
| 38 |
+
- Rebuild all search assets (`projects.json`, `tf_token.json`, `bm25_meta.json`, `substring_index.json`) after applying the script updates above.
|
| 39 |
+
|
| 40 |
+
## Open Questions / Follow-ups
|
| 41 |
+
1. None – requirements above clarify the PR field names, notes rename, image selection rule, and config key addition.
|