Spaces:
Running
Running
| # Analiza struktury GrantForge — Hybrid Grant Intelligence System | |
| Data: 2026-06-23 | Ścieżka wdrożenia: `grantforge-patch/backend/` | |
| ## 1. Modele grantów i naborów | |
| | Plik | Rola | | |
| |------|------| | |
| | `core/grants/models.py` | `Grant` ORM (pełne pola HGIS), `GrantVersion`, `ResearchJob`, `HumanVerificationItem` | | |
| | `core/grants/wyszukiwarka_mapper.py` | Mapowanie rekordów JSON WYSZUKIWARKA → payload API (PKD, regiony, linki, program_year) | | |
| | `core/grants/wyszukiwarka_import.py` | Upsert DB, wersjonowanie przy imporcie, enrichment; **nie** seeduje snapshotów research | | |
| | `core/grants/eligibility_fields.py` | Parser PKD, beneficjentów, regionów, rozmiarów firm | | |
| | `core/grants/catalog_visibility.py` | Jedno źródło prawdy: `resolve_catalog_hidden`, `is_catalog_visible`, wiek >3 lat | | |
| | `core/grants/catalog_filters.py` | Re-eksport widoczności + `build_catalog_description` | | |
| | `core/grants/completeness.py` | `compute_completeness`, `grant_dict_from_row` | | |
| | `core/grants/versioning.py` | `detect_field_changes`, `record_changes`, `get_recent_versions` | | |
| | `core/grants/enrichment.py` | Uzupełnianie pól z surowego JSON (regulamin, linki, PKD) | | |
| | `core/grants/research_snapshot.py` | Pure helpers: hash/diff snapshotów dla Live Research | | |
| ## 2. Endpointy wyszukiwania i rekomendacji | |
| | Plik | Endpointy / funkcje | | |
| |------|---------------------| | |
| | `endpoints/grants_nabory_enhanced.py` | `GET /api/grants/nabory` — filtry + `hybrid=true` | | |
| | `endpoints/grants_catalog.py` | `POST /catalog/search`, stats — `use_hybrid` | | |
| | `endpoints/grants_intelligence.py` | `/intelligence/dashboard`, `/recommendations`, `/hybrid-search`, `/research/*`, `/verification/*` | | |
| | `endpoints/grants_loader.py` | Rejestracja routerów grants | | |
| | `core/grants/catalog_service.py` | `nabory_search`, `search_catalog`, `get_catalog_stats`, filtry PKD | | |
| | `core/grants/hybrid_search.py` | FTS + Pinecone (gdy klucz) lub `fts_only` bez klucza | | |
| | `core/grants/regulation_backfill.py` | Backfill `regulation_url` z raw_data + raport kompletności + próbka HTTP | | |
| | `core/grants/catalog_vector_index.py` | Indeksowanie grantów do Pinecone (`grants_catalog` namespace) | | |
| | `core/grants/recommendations.py` | `recommend_grants_for_company`, `score_grant_for_company` + `why_it_fits` | | |
| ## 3. Agenci LangGraph | |
| | Plik | Rola | | |
| |------|------| | |
| | `agents/generator_agent.py` | Wzorzec StateGraph (generator wniosków) | | |
| | `agents/researcher.py` | `researcher_node` — katalog PostgreSQL | | |
| | `agents/grant_research_agent.py` | HGIS: `analyze_changes` → `enrich_metadata` → `summarize` → `recommend` | | |
| | `agents/matcher.py` | Dopasowanie grantów do profilu firmy (istniejący) | | |
| ## 4. Scraping / sync / research | |
| | Plik | Rola | | |
| |------|------| | |
| | `rag_pipeline/scraper.py` | Scraping treści programów | | |
| | `rag_pipeline/refresh_job.py` | `ContentRefreshJob` + hash diff URL | | |
| | `rag_pipeline/change_detector.py` | Hash diff treści (plik JSON) | | |
| | `rag_pipeline/vector_store.py` | Pinecone store | | |
| | `rag_pipeline/hybrid_retriever.py` | RRF FTS + vector | | |
| | `core/grants/live_research.py` | `detect_grant_changes`, `establish_research_baselines`, kolejka `research_jobs` | | |
| | `core/grants/source_research.py` | Organic refresh: dwa snapshoty agregatora + `detect_url_content_changes` | | |
| | `core/grants/hybrid_search_status.py` | Jawny status FTS vs Pinecone (`get_hybrid_search_status`) | | |
| | `core/grants/credibility.py` | `score_source_credibility` | | |
| | `core/grants/sync_service.py` | `sync_wyszukiwarka_catalog`, live scrape | | |
| | `core/grants/catalog_freshness.py` | Sprawdzanie URL | | |
| | `core/grants/human_verification.py` | Human Verification Queue | | |
| | `scripts/full_catalog_cron.py` | Cron katalogu | | |
| | `scripts/sync_grants_cron.py` | Sync grantów | | |
| | `scripts/import_wyszukiwarka.py` | CLI import | | |
| | `bootstrap_wyszukiwarka.py` | Hook bootstrap przy starcie | | |
| ## 5. Admin / diagnostyka | |
| | Plik | Rola | | |
| |------|------| | |
| | `endpoints/admin_diagnostics.py` | Diagnostyka admin (istniejący) | | |
| | `endpoints/grants_intelligence.py` | Dashboard HGIS (4 warstwy) | | |
| | `grantforge_bootstrap.py` | Migracja HGIS przy starcie | | |
| | `scripts/migrate_hgis_schema.py` | Idempotentna migracja kolumn/tabel | | |
| | `scripts/verify_hgis.py` | Gating verification (6 kroków) | | |
| | `scripts/verify_catalog.py` | Weryfikacja katalogu | | |
| | `scripts/test_intelligence_local.py` | Lokalne endpointy bez serwera HTTP | | |
| ## 6. Testy (shipped) | |
| | Plik | Zakres | | |
| |------|--------| | |
| | `tests/test_hgis.py` | completeness, versioning, recs, agent, filtry (fixture PARP + verify DB) | | |
| | `tests/fixtures/real_grant_sample.json` | Rzeczywisty rekord PARP z `dotacje-latest.json` | | |
| | `tests/test_research_snapshot.py` | Unit snapshot helpers | | |
| | `tests/test_wyszukiwarka_pipeline.py` | Import JSON → baseline → file mutation → detect | | |
| | `tests/test_catalog_filters.py` | Widoczność, catalog_hidden | | |
| | `tests/test_catalog_pipeline.py` | Pipeline katalogu | | |
| ## 7. Przepływ danych (skrót) | |
| ``` | |
| dotacje-latest.json | |
| → wyszukiwarka_mapper (map) | |
| → enrichment + catalog_visibility | |
| → wyszukiwarka_import (upsert, NO snapshot seed) | |
| → establish_research_baselines (detect_grant_changes) | |
| → run_organic_aggregator_refresh (older JSON → newer JSON, real field diffs) | |
| → [cron] detect_grant_changes + detect_url_content_changes | |
| → versioning + research_jobs + HVQ | |
| → grant_research_agent (LangGraph) | |
| → nabory_search / hybrid_search / recommendations | |
| ``` | |
| ## 8. Ograniczenia środowiska (ujawnione w verify) | |
| | Obszar | Stan w sandbox | | |
| |--------|----------------| | |
| | `regulation_url` w DB | ~26% widocznych rekordów (limit źródła JSON ~23% surowych) | | |
| | `PINECONE_API_KEY` | Brak → `search_mode=fts_only`; semantic wymaga klucza (`hybrid_search_status`) | | |
| | Organic change detection | `dotacje-2026-06-20.json` → `dotacje-latest.json`: 149 zmian w verify | | |
| | Mutacje importu | Tylko przez edycję pliku JSON (brak `mutations=` w API) | | |
| | Neo4j / pełny server HTTP | Opcjonalne; verify używa direct function calls | | |
| ## 9. Zmienne środowiskowe (HGIS) | |
| | Zmienna | Domyślnie | Opis | | |
| |---------|-----------|------| | |
| | `ENABLE_LIVE_RESEARCH` | `true` | Warstwa 2 | | |
| | `ENABLE_GRANT_RESEARCH_AGENT` | `true` | LangGraph agent | | |
| | `ENABLE_HYBRID_SEARCH` | `true` | Hybrydowe wyszukiwanie | | |
| | `ENABLE_PINECONE_SEMANTIC` | `true` | Wymaga `PINECONE_API_KEY` | | |
| | `HUMAN_VERIF_QUEUE` | `true` | Kolejka weryfikacji | | |
| | `ENABLE_IMPORT_MUTATIONS` | `false` | Dev-only patch importu | |