Spaces:
Build error
Build error
Commit History
-
fix: loader counter-accuracy -- 20 methods counted attempts, not successes
Root cause: 20 of 22 load_* methods called self._run(query, params) without capturing its return value, then unconditionally incremented their success counter right after. _run() returns None on 3 distinct conditions: a genuine Cypher/Neo4j failure (caught by its own broad exception handler), the D-2 sample-data-rejection gate silently refusing to write a record, or dry_run mode. In all 3 cases, the counter still incremented as if the write had succeeded -- already directly observed in this project's own D-2 verification, where 'Politicians loaded: 1' logged even though the D-2 gate had correctly rejected the write underneath it. Only load_affidavits and load_wikidata_enrichments (2 of 22) already checked the return value correctly. Fix: all 20 now capture result = self._run(...) and only increment count when result is not None. Also moved the rels_created stat increment in load_politician_company_links inside the same guard, same reasoning. One deliberate trade-off, stated precisely…719c3d0
-
fix: wire source_type into the Party/Ministry/Scheme stub-node helpers (P-1/D8)
Root cause: the previous commit wired source_type into all 20 primary load_* methods, but explicitly deferred 5 smaller stub-node creation sites: _create_party_and_link, _link_contract_to_ministry, _link_audit_to_scheme, and the inline Party/Company/Ministry ON CREATE SET stubs in load_electoral_bonds and load_tenders. Party/Ministry/Scheme ARE documented in NODE_SCHEMAS with the same source_type field and the same 'default OFFICIAL' language as every other node type -- this was a real, if smaller, gap, not a lower-priority skip. Fix: added source_type='OFFICIAL' to all 5 -- 3 plain SET clauses (re-applied every call, harmless since the value is always the same constant) and 2 ON CREATE SET clauses (correctly fires only the first time each stub node is created, consistent with how those clauses already handle source='electoral_bond'/'cppp'). Verified: invoked all 5 code paths for real (not just read the diff) by calling load_politicians/load_contracts/load_audit_reports/ load_elector…df73861 -
feat: wire source_type into graph/loader.py's actual write queries (P-1/D8)
Root cause: D1 added source_type to graph/schema.py's NODE_SCHEMAS and SETUP_QUERIES, but scoped that phase as schema-only groundwork -- no load_* method actually wrote the field. Every real node write across the 20 methods that create primary nodes from scraped records was missing it entirely. Scope decision, stated precisely: added source_type='OFFICIAL' to all 20 load_* methods that create primary nodes (load_politicians through load_datagov_documents). Explicitly did NOT touch: - load_wikidata_enrichments: uses MATCH, not MERGE, to enrich EXISTING Politician nodes -- stamping OFFICIAL here would misrepresent provenance for a node this method didn't create. - load_politician_company_links: its MERGE calls are inference- fallback stubs to anchor a DIRECTOR_OF relationship when no real Politician/Company node exists yet, not sourcing an authoritative record -- the relationship itself is already correctly labeled source='entity_resolution'. - The small st…19a0456
-
feat(D1): add source_type provenance field and Dataset/UserDataset schema
Phase D1 scope (PHASE_ROADMAP.md): 'graph/schema.py gains a source_type property (OFFICIAL/UNOFFICIAL/COMMUNITY/USER_CREATED) and a Dataset/UserDataset schema.' Added SOURCE_TYPES as an actual importable constant (not just documentation text) plus validate_source_type(), so future code (D7's dataset manager, D8's UI badges) has something real to import and check against rather than a string convention nobody enforces. Added source_type to all 8 existing node types' property lists. Added Dataset and UserDataset as two distinct node labels, not one label distinguished by a property -- matching D7's explicit isolation requirement ('My Datasets view for user-imported data, isolated storage... never merged into or presented as official data'). Separate labels let any query exclude user-imported data structurally, by label, rather than depending on every query author remembering to filter a property. Added 4 corresponding constraints/indexes to SETUP_QUERIES, matching the file's existing…a2b4761
-
fix(D-2): refuse to write fabricated sample-data records into the live graph
Decision (owner-confirmed): fail loudly rather than write-and-badge. NEW FINDING, not in the original audit -- 5 of GraphLoader's load_* methods (load_politicians, load_companies, load_contracts, load_audit_reports, load_press_releases) read r.get("_source", ...) (underscore-prefixed), but every scraper that sets this field uses the key "source" with no underscore. The real source label -- including every sample-data fallback's label -- was silently discarded and replaced with a generic hardcoded default for EVERY record these 5 methods ever loaded. Fixed first, since the D-2 gate below reads the same "source" param and would have been a silent no-op otherwise. GraphLoader._run() -- the single choke point every load_* method funnels through -- now refuses to execute any query whose "source" param contains "sample" (case-insensitive), logging a warning and incrementing self.stats["sample_data_rejected"] instead. Verified: py_compile passes. End-to-end test with a fake Neo4j driver/se…1ae0333
-
fix(loader): load_affidavits() query was missing every $param placeholder
Root cause: the method's own docstring says 'B-05 FIX: was missing -- affidavit data scraped but never written to graph', describing a real prior bug. The fix that shipped for that bug never actually worked: the MERGE/SET query text had every single value slot empty ('id: }', every 'SET a.field = ,' with nothing after the '='), while the params dict passed alongside it had correct values for every field. The query text simply never referenced any of them with $. Mechanism of silence: GraphLoader._run() catches all exceptions by design (so one bad record doesn't abort a whole batch load) and returns None on failure. load_affidavits() only increments its counter when _run() returns non-None. Because the query fails to parse on every call, the counter never advances, and the method logs '[Loader] Affidavits loaded: 0' -- a message that reads identically to 'ran successfully, nothing to load' rather than 'failed every time'. Blast radius: zero Affidavit nodes have ever been created by th…9147d29
-
fix(B-08,NEW-1): remove duplicate fulltext from SETUP_QUERIES; modern CREATE FULLTEXT INDEX syntax
B-08: globalSearch fulltext index existed in three places: FULLTEXT_INDEX_QUERY (informational, never called), SETUP_QUERIES (8-type CALL syntax, externally run), and loader.py setup_schema() (20-type authoritative version). Running SETUP_QUERIES externally tried to create a conflicting narrower index, producing silent DatabaseError: index globalSearch already exists. Removed from SETUP_QUERIES. loader.py is the single source of truth. NEW-1: FULLTEXT_INDEX_QUERY used deprecated CALL db.index.fulltext.createNodeIndex removed in Neo4j 5.x and AuraDB Free. Replaced with modern DDL: CREATE FULLTEXT INDEX globalSearch IF NOT EXISTS FOR (n:...) ON EACH [...]7b8375b
-
fix(ascii): replace all non-ASCII chars in Python source files
CI check requires all .py files (except transliteration/multilingual_ner/ translator/languages) to be pure ASCII. Pre-existing violations fixed: U+03BB lambda -> lambda_val U+2014 em-dash -> -- U+2013 en-dash -> -- U+20B9 rupee Rs -> Rs. U+2192 arrow -> -> U+2500 box-h -> - U+2502 box-v -> | U+2081 subscript -> 1fa87992 -
fix(B-05 step3, NEW-1): add load_affidavits() to GraphLoader; modern fulltext in setup_schema()
B-05: load_affidavits() was completely missing from GraphLoader. The myneta_scraper scrapes affidavit data but there was no method to write it to Neo4j as Affidavit nodes with FILED_AFFIDAVIT relationships. Added full MERGE-based loader with all fields. NEW-1: setup_schema() updated from deprecated CALL db.index.fulltext.createNodeIndex to modern CREATE FULLTEXT INDEX IF NOT EXISTS DDL syntax. Includes Affidavit in the 21-type label list.6b02e4a -
fix(B-07): CONTESTED_IN relationship target was Scheme, corrected to Constituency
Schema had CONTESTED_IN: Politician -> Scheme (wrong). Politicians contest elections in Constituencies, not government welfare schemes. This caused Cypher traversals for election history to either fail or return wrong nodes.7087c8a -
fix(B-05 step2): add Affidavit uniqueness constraint and year index to SETUP_QUERIES
Without a uniqueness constraint, MERGE (a:Affidavit {id: $id}) could create duplicate nodes on repeated pipeline runs. Without the year index, queries filtering by election year would do full label scans across all Affidavit nodes.ff2a14c -
fix(B-05 step1): add Affidavit node and FILED_AFFIDAVIT to schema
The Affidavit node type and FILED_AFFIDAVIT relationship were entirely absent from NODE_SCHEMAS and RELATIONSHIP_SCHEMAS. Investigators and risk routes all query (p:Politician)-[:FILED_AFFIDAVIT]->(a:Affidavit) but the nodes never existed in the graph. All affidavit queries returned zero rows.63bc051
-
fix(BUG-C4): add scraped_at field to all seed records so live feed shows data after /admin/seed
e055b14
-
fix(BUG-30): increase SHA-256 ID truncation 20->32 chars (80->128-bit collision resistance)
493af54
-
fix(api+graph+pipeline): M-04+M-05+M-08+M-09 -- misc medium fixes
M-04: /debug/env endpoint was publicly accessible with no auth gate. It revealed NEO4J_URI prefix and credential configuration status. Fix: added same DEBUG_MODE env-var gate as /debug/neo4j. M-05: Wikidata enrichment used exact name match (=) -- any difference in spacing, middle name, honorific, or transliteration caused a silent miss. With Indian names this is extremely common. Fix: changed to CONTAINS match in both directions for fuzzy matching. M-08: --parallel flag had action='store_true', default=True making it always True whether passed or not. The flag had zero effect. Confusing for operators who expected --parallel to mean something. Fix: removed --parallel; --sequential now inverts the default. M-09: startNode(rel).id and endNode(rel).id in Cypher can return null if a node has no 'id' property. This created edges with null source/target silently corrupting graph output. Fix: added isinstance str check to filter null src/tgt before appending.5d9b8a9
-
fix(encoding): strip non-ASCII chars from pipeline.py and loader.py
Both files contained UTF-8 em dashes (U+2014) and arrows (U+2192) in comments which cause UnicodeDecodeError on GitHub Actions runners that read files without explicit encoding= argument. Replaced all non-ASCII punctuation with ASCII equivalents: U+2014 (em dash) -> -- U+2192 (arrow) -> -> All other logic is unchanged.06270b4
-
fix(loader): BUG-2/3/17/19/21 — 7 new loaders, fulltext, kwargs, SHA-256
BUG-2 (CRITICAL): 7 datasets were scraped but never loaded. Added loaders: load_icij_entities, load_sanctioned_entities, load_court_cases, load_local_bodies, load_crime_reports, load_wikidata_enrichments, load_datagov_documents. All 7 called in load_from_pipeline_output(). BUG-3: fulltext index in setup_schema() now covers all 20 node types (was only 8). Added: NGO, ElectoralBond, InsolvencyOrder, Tender, RegulatoryOrder, EnforcementAction, ParliamentQuestion, VigilanceCircular, ICIJEntity, SanctionedEntity, CourtCase, LocalBody. BUG-17: _run() now accepts BOTH positional dict and **kwargs so new Phase 28 loaders (which used keyword args) work without errors. BUG-19: make_id() upgraded from MD5 to SHA-256 (20 hex chars) for lower collision risk across large datasets. BUG-21: GraphLoader.__init__ now accepts driver= kwarg so admin.py can pass the existing FastAPI driver instead of opening a second Neo4j connection.8078bc3
-
fix(BUG-18/19): timeline audit events use entity_name not entity_id; fulltext index covers all 16 node types
e728463
-
fix(BUG-17): DIRECTOR_OF links match canonical node IDs via toLower name lookup
51ce6bb -
fix(seed): DIRECTOR_LINKS keys politician_name→name_a — was creating zero relationships
23e6d00
-
fix(seed): add DIRECTOR_OF + WON_CONTRACT relationships to sample data
- Added 7 companies (was 5), added source field to all records - Added SAMPLE_DIRECTOR_LINKS: Amit Shah→Adani, Modi→ONGC, Thakur→TCS - seed() now calls load_politician_company_links() - Searching 'Amit Shah' or 'Anurag Thakur' will now show connected grapha66284c
-
fix(loader): GeM amount_inr fallback + schema session bug
graph/loader.py: load_contracts() now falls back to amount_inr/1e7 when amount_crore is absent. GeM scraper normalizes to amount_inr but older records lack amount_crore — values were silently becoming 0. Schema session bug was already fixed in PR#54.8f27281 -
fix(graph): correct :AUDITS relationship to :FLAGS in ministry_audit_flags query
graph/queries.py ministry_audit_flags used [:AUDITS] but no loader creates an AUDITS relationship. The CAG loader creates [:FLAGS]. Corrected to match actual graph schema.ed51cce
-
fix(data): schema session bug and seed data field mismatches
graph/loader.py: setup_schema() session bug fixed. The for query in SETUP_QUERIES loop was running outside the with self.driver.session() block, reusing a closed session. Moved entire loop inside the with block. Added logging for full-text index creation result instead of silent pass. graph/seed.py: field names aligned with loader expectations. Politicians: total_assets_crore renamed to total_assets. Politicians: added criminal_case_count alongside criminal_cases. Contracts: added seller_name and seller_name_raw from company IDs so load_contracts creates WON_CONTRACT relationships correctly. Audit reports: irregularity_amount_crore renamed to amount_crore.344b0c4
-
fix(config+admin+main): 4 bugs resolved
- config/settings.py: NEO4J_URI default was bolt://localhost:7687 causing HF Space to always try connecting to localhost (fails) Changed to empty string — env var must be set explicitly - graph/queries.py: same bolt://localhost default — same fix - api/routes/admin.py: load_companies() was called with SAMPLE_CONTRACTS instead of SAMPLE_COMPANIES — seed was loading wrong data as companies - api/main.py: load_dotenv() was never called — environment variables from .env and HF Space secrets were not loaded at API startupb70b0c6
-
fix(search+multilingual+sources): global search, fulltext index, sources API
- search.py: fulltext fast-path via globalSearch index (instant results) Falls back to 8-label scan when index not ready - multilingual.py: expanded from 3 to 8 labels, all name/title fields Deduplication fix — checks by id not by object identity - app.js: normalises multilingual {id,type} → {entity_id,entity_type} before rendering ResultCard — fixes blank multilingual results - sources.py: GET /sources shows count per dataset from Neo4j GET /sources/{dataset} shows sample records from that dataset All 20 scrapers listed with URL and data type - loader.py: creates globalSearch fulltext index on setup_schema() - schema.py: FULLTEXT_INDEX_QUERY documented for reference - api/main.py: sources router registered5e26080
-
fix(data): add sample data seed — politicians, companies, contracts, audit reports
Root cause: Neo4j database is empty. Pipeline has not been run yet. Fix: seed.py loads 10 politicians, 5 companies, 3 contracts, 3 audit reports. admin router exposes POST /admin/seed to trigger from HF Space. Search for Modi, Gandhi, Adani, Tata, Infosys will now return results.b1d8714
-
fix(graph): remove bolt://localhost fallback URI in loader
NEO4J_URI fallback was bolt://localhost:7687 which caused DNS failure when .env loaded correctly but getenv saw the default before dotenv ran. Now raises ValueError if URI is not set, forcing explicit configuration.fc50d3e
-
feat(graph): add Cypher query library for corruption detection
8 pre-built queries covering key investigative patterns: - politician_company_contracts: THE CORE - politician->company->contract - repeated_contract_winners: concentration risk detection - ministry_audit_flags: most CAG-flagged ministries - scheme_irregularities: largest financial irregularities by scheme - politicians_with_cases: filter by declared criminal cases - politician_profile: full graph profile for any named politician - high_value_contracts: contracts above crore threshold - node_counts + relationship_counts: health checks QueryRunner class: connects to Neo4j, runs named queries print_query_library(): lists all queries with risk levelsfb18fa1 -
feat(graph): add GraphLoader - loads all scraped data into Neo4j
- GraphLoader connects to Neo4j AuraDB using .env credentials - setup_schema(): runs all constraints and indexes from schema.py - load_politicians(): MERGE nodes, creates Party + MEMBER_OF rel - load_companies(): MERGE by CIN or name hash - load_contracts(): MERGE + creates WON_CONTRACT + AWARDED_BY rels - load_audit_reports(): MERGE + creates FLAGS -> Scheme rel - load_press_releases(): MERGE PIB articles as nodes - load_politician_company_links(): creates DIRECTOR_OF rels from entity_resolver cross-dataset matches - load_from_pipeline_output(): main entry - loads a full pipeline JSON - --dry-run flag: test without writing to database7cdf407 -
feat(graph): add Neo4j schema - nodes, relationships, constraints
Node types (7): Politician, Company, Contract, AuditReport, PressRelease, Ministry, Party, Scheme Relationship types (6): MEMBER_OF, DIRECTOR_OF, WON_CONTRACT, AWARDED_BY, FLAGS, AUDITS Core corruption pattern in schema: (Politician)-[:DIRECTOR_OF]->(Company)-[:WON_CONTRACT]->(Contract) (AuditReport)-[:FLAGS]->(Scheme) SETUP_QUERIES: 7 uniqueness constraints + 3 indexes for fast lookup6ade5c9
-
fix: add __init__.py to all packages, create venv setup
- Added __init__.py to scrapers, processing, graph, ai, api, blockchain, config, tests folders - Fixes ModuleNotFoundError when running scrapers - Added main.py as project entry point - Updated .gitignore to exclude venv/ and __pycache__/ - All scripts must now run from project root with: python -m scrapers.Xd76057f
-
feat: add requirements.txt and base_scraper.py - Day 2
1c02240