Spaces:
Build error
Build error
Commit History
-
fix: wire 3 hardcoded AI models into the existing model_selector tiering (R-2/R-3/R-4)
Root cause: config/model_selector.py's get_model() (low/medium/high variants per task) has existed since Phase 31 but was called nowhere in the codebase -- confirmed via grep, zero call sites outside its own file and tests. Three consumers hardcoded a fixed model regardless of the machine's detected runtime_profile tier: - ai/nlp_extractor.py: always en_core_web_sm (get_model('ner')'s medium-tier value -- correct only by coincidence on medium tier) - ai/shadow_draft_detector.py: always all-MiniLM-L6-v2 (same coincidental medium-tier match) - processing/entity_resolver_v2.py: always paraphrase-multilingual-MiniLM-L12-v2 -- get_model('embeddings')'s HIGH-tier value, unconditionally, on every tier including 'low'. This is the most severe of the three: the 2GB-RAM 'low' tier -- the exact case runtime_profile.py exists to protect -- was loading the single heaviest embedding model available, every time. Trigger condition: always, independent of any misconfi…3066feb
-
fix(F-10): give scraper-pipeline concurrency its own I/O-bound resource dimension
Decision (owner-confirmed): scraper concurrency should NOT follow the same low/medium/high CPU/RAM tiers as the AI investigators -- the investigator pool is CPU-bound, the scraper pipeline is I/O-bound. Added RuntimeProfile.scraper_workers: max(2, min(10, cpu_count * 2)), overridable via BHARATGRAPH_SCRAPER_WORKERS=N. Surfaced in GET /runtime. processing/pipeline.py now reads PROFILE.scraper_workers instead of the hardcoded max_workers=6. Verified: py_compile passes on both files. Formula sanity-checked across a realistic CPU range. Spied on the real ThreadPoolExecutor constructor and confirmed the env-var override reaches it end-to-end.a2f923e
-
Add header documentation for pipeline.py
Unverified signatureAdded a description header for the BharatGraph data pipeline.4b4af4b
-
feat(phase-32): rebuild AliasGraph in pipeline after entity resolution
After resolve_dataset() merges duplicate politicians and companies, the pipeline now rebuilds the AliasGraph: 1. Bulk-adds all canonical records (name -> id) 2. Adds all alias names found during resolution 3. Saves to data/processed/alias_graph.json On next API restart, the startup lifespan loads this file so /resolve/alias returns results immediately for all entities that have been through the pipeline.aef2ed8
-
fix(ascii): replace all non-ASCII chars in Python source files
CI check requires all .py files (except transliteration/multilingual_ner/ translator/languages) to be pure ASCII. Pre-existing violations fixed: U+03BB lambda -> lambda_val U+2014 em-dash -> -- U+2013 en-dash -> -- U+20B9 rupee Rs -> Rs. U+2192 arrow -> -> U+2500 box-h -> - U+2502 box-v -> | U+2081 subscript -> 1fa87992
-
fix(ARCH-2): add deprecation notice to entity_resolver.py -- canonical version is entity_resolver_v2.py
ccc58b9
-
fix(test): rewrite normalise_indian_name -- add Sh. to honorifics, while-loop for stacked honorifics, fix broken function body
9195906 -
fix(test): add 'sh' abbreviation to _HONORIFICS; change to while-loop for stacked honorifics (Late Shri X -> X) -- fixes 1 failing pytest
db0ae09
-
fix(api+graph+pipeline): M-04+M-05+M-08+M-09 -- misc medium fixes
M-04: /debug/env endpoint was publicly accessible with no auth gate. It revealed NEO4J_URI prefix and credential configuration status. Fix: added same DEBUG_MODE env-var gate as /debug/neo4j. M-05: Wikidata enrichment used exact name match (=) -- any difference in spacing, middle name, honorific, or transliteration caused a silent miss. With Indian names this is extremely common. Fix: changed to CONTAINS match in both directions for fuzzy matching. M-08: --parallel flag had action='store_true', default=True making it always True whether passed or not. The flag had zero effect. Confusing for operators who expected --parallel to mean something. Fix: removed --parallel; --sequential now inverts the default. M-09: startNode(rel).id and endNode(rel).id in Cypher can return null if a node has no 'id' property. This created edges with null source/target silently corrupting graph output. Fix: added isinstance str check to filter null src/tgt before appending.5d9b8a9 -
fix(scrapers+pipeline): H-03+L-02+H-12 -- MyNeta elections and git bloat
H-03: STATE_PAGES only had 3 stale elections (lok_sabha_2024, delhi_2020, tamil_nadu_2021). 10+ major 2022-2024 elections were completely missed: UP 2022, Gujarat 2022, Karnataka 2023, Rajasthan 2023, MP 2023, Telangana 2023, Maharashtra 2024, Jharkhand 2024, etc. Fix: expanded to 14 elections covering all recent major elections. L-02: delhi_2020 was defined in STATE_PAGES but fetch_sample_data() only iterated ['lok_sabha_2024', 'tamil_nadu_2021'] -- delhi_2020 was always skipped. Fix: iterate list(self.STATE_PAGES.keys()) instead. H-12: pipeline.save() included the full 'raw' key (all scraper output) in the JSON committed to git by the GitHub Action. With 20 scrapers x 50 records each plus nested objects, these files grew large and accumulated in git history indefinitely. Fix: save() now only writes summary+links; raw is kept in memory only.1ffe057 -
fix(api+pipeline): C-05+C-06 -- unsearchable nodes and hardcoded ICIJ names
C-05: DataGovDocument and CrimeReport nodes were loaded into Neo4j by the graph loader but had no LABEL_QUERIES entry, SOURCE_MAP entry, or fulltext index definition. They were permanently invisible to search. Fix: added both types to LABEL_QUERIES and SOURCE_MAP. C-06: ICIJ scraper only searched 5 hardcoded names (Adani, Reliance, Tata, Birla, Ambani). OpenSanctions searched 5 names. Every other politician, company, or sanctioned person was completely missed. Fix: expanded both lists to include known high-risk Indian entities.2f11b48
-
refactor(pipeline): upgrade EntityResolver import to v2
Imports EntityResolverV2 as EntityResolver so all existing pipeline.py code continues to work without further changes. Threshold raised 0.65 -> 0.72 because v2 has better signals (Jaro-Winkler + embeddings) so a higher threshold is safe without missing valid matches.16258d8 -
feat(processing): add EntityResolverV2 -- multi-signal identity engine
Replaces the Jaccard-only resolver with a 4-signal pipeline: - Jaro-Winkler (0.30) -- character-level typo and transliteration - Jaccard token (0.20) -- word-order variations - Embedding cosine (0.50) -- multilingual semantic similarity - Exact PAN/CIN/GSTIN match -- overrides all (confidence=1.0) When sentence-transformers is unavailable, rescales to Jaro-Winkler (0.60) + Jaccard (0.40) -- no crash, graceful fallback. Indian name normalisation: - Removes 17 honorific patterns: Sh., Smt., Dr., Late, Col., etc. - Normalises company suffixes: Private Limited -> Pvt Ltd - Strips M/s prefix Backward compatible: EntityResolver = EntityResolverV2 alias so pipeline.py imports work without modification. Also adds pure-Python Jaro-Winkler (no jellyfish dependency).b1074d8 -
feat(processing): add AliasGraph -- alias-to-canonical-id lookup
Persists and serves alias -> canonical_id mappings built by EntityResolverV2. All lookups normalise to lowercase so 'RAHUL KUMAR', 'Rahul Kumar', and 'R. Kumar' all hit the same slot. Supports bulk_add(), merge(), save()/load(), and stats(). Pure ASCII.b3a84d6 -
feat(processing): add canonical_id.py -- SHA-256 ID generation
Single source of truth for stable entity IDs, matching make_id() in graph/loader.py for full compatibility. Helper functions for politician, company, contract, NGO ID generation. Pure ASCII.63af5ed -
fix(encoding): strip non-ASCII chars from pipeline.py and loader.py
Both files contained UTF-8 em dashes (U+2014) and arrows (U+2192) in comments which cause UnicodeDecodeError on GitHub Actions runners that read files without explicit encoding= argument. Replaced all non-ASCII punctuation with ASCII equivalents: U+2014 (em dash) -> -- U+2192 (arrow) -> -> All other logic is unchanged.06270b4
-
fix(pipeline): BUG-13 — entity resolver now runs on international data
BUG-13: entity resolution only ran on myneta+mca records. ICIJ, OpenSanctions, and IBBI entities were never resolved against Indian politicians/companies so cross-dataset evidence chains were never built. Fixed: politicians pool now includes opensanctions + icij; companies pool now includes ibbi. All cross-dataset matches now produce DIRECTOR_OF links.297af77
-
feat: Procurement DNA + Cartel Detection + full pipeline fix
pipeline.py: expanded from 5 to all 20 scrapers, parallel ThreadPoolExecutor - loksabha, sebi, ed, cvc, njdg, electoral_bond, icij, opensanctions, wikidata, ibbi, ngo_darpan, cppp, ncrb, lgd — all now included - parallel=True by default (6 workers), fallback-safe per scraper - per_source count in summary output admin.py: POST /admin/pipeline triggers full pipeline in background - GET /admin/pipeline/status shows progress and last result - auto-loads into Neo4j after pipeline completes - works on HF Space with no local DB connection needed ai/forensics/bid_dna.py: TF-IDF cosine similarity on bid documents - SIMILARITY_THRESHOLD=0.72 flags near-identical bids from different vendors - cover-bid detection via price clustering (std dev analysis) ai/forensics/cartel_detector.py: 3-method cartel detection - award rotation: equal share pattern across vendors - co-bidding network: same vendors appearing together repeatedly api/routes/procurement.py: GET /procurement/bid-…eece913
-
fix: complete pipeline - pull correct scraper versions into branch
- processing/pipeline.py: working, 23 records, saves to data/processed/ - processing/cleaner.py + entity_resolver.py: restored from main - scrapers/pib_scraper.py: pulled HTML version from main (was old RSS) - scrapers/gem_scraper.py: pulled fixed version from main (was old hardcoded key) - scrapers/base_scraper.py: pulled load_dotenv version from main - .gitignore: *.pyc added9c33ffa
-
fix: add __init__.py to all packages, create venv setup
- Added __init__.py to scrapers, processing, graph, ai, api, blockchain, config, tests folders - Fixes ModuleNotFoundError when running scrapers - Added main.py as project entry point - Updated .gitignore to exclude venv/ and __pycache__/ - All scripts must now run from project root with: python -m scrapers.Xd76057f
-
feat: add requirements.txt and base_scraper.py - Day 2
1c02240