First commit

.gitignore

@@ -0,0 +1,13 @@
+*__pycache__/
+
+*.md
+!README.md
+*.log
+*.npy
+*.csv
+*.png
+*.grd
+*.owl
+*.out
+*.json
+*.yaml

README.md

@@ -1 +1,661 @@
-# Noisy_boy_frontend
+---
+title: Iroha - Financial Intelligence Pipeline API
+colorFrom: blue
+colorTo: indigo
+sdk: gradio
+sdk_version: 6.18.0
+app_file: server.py
+pinned: false
+---
+
+# Iroha - Financial Intelligence Pipeline API
+
+**Version:** 1.0.0  
+**Description:** End-to-end financial intelligence pipeline for Net-of-Tax Alpha decisions using causal chain analysis, web scraping, and data aggregation.
+
+---
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Authentication](#authentication)
+3. [API Endpoints](#api-endpoints)
+   - [Server Endpoints](#server-endpoints)
+   - [V2 API Endpoints](#v2-api-endpoints)
+4. [Core Functions](#core-functions)
+5. [Supported Tickers](#supported-tickers)
+6. [Error Handling](#error-handling)
+7. [Examples](#examples)
+
+---
+
+## Overview
+
+The Noisy Boy API is a FastAPI-based financial intelligence system that:
+
+- **Fetches multi-source financial data** from various Indian market sources (BSE, NSE, RBI, SEBI, etc.)
+- **Analyzes causal chains** between market events and price movements using BERT and embedding models
+- **Generates aggregated text** from diverse data sources for each ticker
+- **Builds knowledge graphs** with semantic and evidence layers
+- **Streams real-time SSE responses** for long-running analysis tasks
+
+### Technology Stack
+
+- **Framework:** FastAPI with async/await support
+- **Database:** SQLite (local) + Supabase (cloud)
+- **Models:**
+  - BERT-CAUSE-EFFECT for causal relationship extraction
+  - Fireworks AI embeddings for similarity scoring
+  - OpenAI LLM for graph summarization
+  - TinyFish AI for web automation
+- **Data Sources:** 50+ fetchers covering macro, regulatory, and company-specific data
+
+---
+
+## Authentication
+
+### API Key Authentication
+
+All endpoints (except `/api/source-refs`) require an `X-API-Key` header:
+
+```bash
+curl -H "X-API-Key: your-secret-token" https://api.example.com/api/endpoint
+curl -H "X-API-Key: your-api-key" http://localhost:8000/v2/api/endpoint
+```
+
+**Environment Variable**: `API_KEY` (default: `secret-token`)
+
+---
+
+## Base URL
+
+```
+http://localhost:8000/v2
+```
+
+---
+
+## API Endpoints
+
+### Scraper API
+
+#### 1. GET `/api/source-refs`
+Returns all fetcher-to-source-URL mappings and reference links.
+
+**Authentication**: None required
+
+**Response**:
+```json
+{
+  "fetcher_refs": {
+    "repo_rate": "https://rbi.org.in/...",
+    "bse": "https://bseindia.com/...",
+    ...
+  },
+  "cat_refs": {
+    "macro": {...},
+    "corporate": {...}
+  }
+}
+```
+
+---
+
+#### 2. POST `/api/run`
+Streams real-time scraping results via Server-Sent Events (SSE) from TinyFish API with local caching.
+
+**Authentication**: Required (`X-API-Key`)
+
+**Request Body**:
+```json
+{
+  "url": "https://example.com/page",
+  "goal": "Extract financial data",
+  "ticker": "HDFCBANK",
+  "stealth": false
+}
+```
+
+**Query Parameters**:
+- `url` (required): Target URL to scrape
+- `goal` (required): Scraping objective description
+- `ticker` (optional): Stock ticker for caching purposes
+- `stealth` (optional, bool): Enable stealth mode
+
+**Response** (SSE):
+```
+data: {"type": "STARTED", "run_id": "run_123"}
+data: {"type": "PROGRESS", "purpose": "Fetching data..."}
+data: {"type": "COMPLETE", "status": "COMPLETED", "result_json": {...}}
+```
+
+**Caching**: Results are cached per ticker per day via SQLite.
+
+---
+
+#### 3. GET `/api/ticker-data/{ticker}`
+Fetch all cached data for a ticker across all sources (Supabase).
+
+**Authentication**: Required (`X-API-Key`)
+
+**Path Parameters**:
+- `ticker`: Stock ticker symbol (e.g., `HDFCBANK`)
+
+**Response**:
+```json
+{
+  "ticker": "HDFCBANK",
+  "status": "success",
+  "data": {
+    "repo_rate": {
+      "data": {...},
+      "fetched_at": "2024-01-15T10:30:00Z"
+    },
+    "bse": {
+      "data": [...],
+      "fetched_at": "2024-01-15T10:30:00Z"
+    },
+    "aggregated_text": {
+      "aggregated_text": "...",
+      "fetched_at": "2024-01-15T10:30:00Z"
+    },
+    "causal_chain": {
+      "nodes": [...],
+      "links": [...],
+      "all_chains": [...],
+      "biggest_chain": [...]
+    }
+  }
+}
+```
+
+---
+
+### Causal Chain API
+
+#### 4. GET `/api/generate-causal-chain-stream`
+Streams causal chain generation via SSE (two phases: nodes, then edges).
+
+**Authentication**: Required (`X-API-Key`)
+
+**Query Parameters**:
+- `ticker` (required): Stock ticker symbol
+
+**Response** (SSE - Streaming):
+```
+data: {"type": "status", "message": "Generating text for ticker..."}
+data: {"type": "nodes", "nodes": [{"id": "RBI Rate Hike", "label": "RBI Rate Hike"}, ...]}
+data: {"type": "edges", "links": [{"source": "Rate Hike", "target": "Inflation", "score": 0.89}, ...]}
+data: {"type": "done", "all_chains": [[...], [...]], "biggest_chain": [...]}
+```
+
+**Phases**:
+1. **Status**: Initial status message
+2. **Nodes**: Unique causal entities extracted from BERT model
+3. **Edges**: Connections between nodes from embedding similarity
+4. **Done**: Final chains and biggest causal path
+
+---
+
+### Ontology API
+
+#### 5. POST `/api/generate-ontology`
+Generate financial ontology from input text.
+
+**Authentication**: Required (via implicit call)
+
+**Request Body**:
+```json
+{
+  "ticker": "RELIANCE",
+  "text": "Recent crude oil rally drives RELIANCE revenue..."
+}
+```
+
+**Response**:
+```json
+{
+  "status": "success",
+  "ontology": {
+    "entities": ["crude", "revenue", "RELIANCE"],
+    "relations": [{"type": "affects", "from": "crude", "to": "RELIANCE"}]
+  }
+}
+```
+
+---
+
+#### 6. POST `/api/extract-entities`
+Extract named entities using the provided ontology.
+
+**Authentication**: Required (via implicit call)
+
+**Request Body**:
+```json
+{
+  "ticker": "ITC",
+  "text": "Coal prices surge amid monsoon fears...",
+  "ontology": {
+    "entities": ["coal", "monsoon", "price"],
+    "relations": [...]
+  }
+}
+```
+
+**Response**:
+```json
+{
+  "status": "success",
+  "entities": [
+    {"entity": "coal", "type": "commodity", "confidence": 0.92},
+    {"entity": "monsoon", "type": "weather_event", "confidence": 0.88}
+  ]
+}
+```
+
+---
+
+#### 7. POST `/api/build-knowledge-graph`
+Build complete knowledge graph with nodes, edges, and causal chains.
+
+**Authentication**: Required (via implicit call)
+
+**Request Body**:
+```json
+{
+  "graph_id": "graph_001",
+  "ticker": "BHEL",
+  "text": "Full financial text for analysis...",
+  "financial_results": {...},
+  "forensic_results": {...},
+  "tech_results": {...},
+  "cached_data": {...}
+}
+```
+
+**Response**:
+```json
+{
+  "meta": {
+    "ticker": "BHEL",
+    "exchange": "NSE",
+    "generated_at": "2024-01-15T12:45:00Z",
+    "status": "success",
+    "chain_count": 5,
+    "node_count": 23,
+    "edge_count": 45
+  },
+  "summary": {
+    "narrative": "Coal shortage → Power demand surge → BHEL tariff opportunity",
+    "net_sentiment_for_ticker": "bullish",
+    "ticker_relevance_score": 0.87,
+    "macro_regimes_active": ["coal-supply-shock", "power-demand-rally"],
+    "top_causal_nodes": ["Coal shortage", "Power demand", "BHEL capacity"],
+    "ria_alert": {
+      "level": "medium",
+      "reason": "Monitor coal supply for sustained impact on tariff structure."
+    }
+  },
+  "biggest_chain": ["Coal shortage", "Power demand", "BHEL contract wins", "Revenue growth"],
+  "all_chains": [[...], [...], ...],
+  "nodes": [...],
+  "edges": [...]
+}
+```
+
+---
+
+### Server API (Streaming)
+
+#### 8. GET `/api/generate-causal-chain-stream` (Alternative)
+Same as endpoint #4 but with optional caching from server-side TinyFish integration.
+
+---
+
+## Data Fetchers
+
+The system supports multiple data fetchers for different tickers. Each fetcher aggregates specific financial signals:
+
+### Supported Fetchers
+
+| Fetcher | Description | Tickers |
+|---------|-------------|---------|
+| `repo_rate` | RBI Repo Rate | HDFCBANK, TCS, PAYTM, TMPV |
+| `bse` | BSE Announcements | All supported tickers |
+| `fii_dii` | FII/DII Flows | HDFCBANK |
+| `brent` | Brent Crude Oil Price | HDFCBANK, RELIANCE, TMPV |
+| `news` | Financial News Articles | All supported tickers |
+| `npp` | National Power Portal Data | IEX, BHEL |
+| `imd_monsoon` | IMD Monsoon Status | RELIANCE, IEX, BHEL, ETERNAL, ULTRACEMCO |
+| `coal` | Coal Price Index | IEX, BHEL, ITC, ULTRACEMCO |
+| `bhel_tenders` | BHEL Active Tenders | BHEL |
+| `agmarknet` | Agricultural Prices | ETERNAL, ITC |
+| `weather` | Weather in Key Cities | ETERNAL |
+| `h1b` | H1B Visa Filings | TCS |
+| `us_fed` | US Fed Interest Rate | TCS |
+| `us_pmi` | US Services PMI | TCS |
+| `trai_reports` | TRAI Telecom Reports | RELIANCE |
+| `npci` | NPCI UPI Statistics | ETERNAL, IRCTC, PAYTM, MAPMYINDIA |
+| `dgca_traffic` | DGCA Air Traffic Data | IRCTC |
+| `tourist_arrivals` | Foreign Tourist Arrivals | IRCTC |
+| `india_cpi` | India CPI Inflation | ITC |
+| `food_cpi` | Food CPI Inflation | ETERNAL |
+| `labour` | Labour Ministry Releases | ETERNAL |
+| `sebi_orders` | SEBI Orders | PAYTM |
+| `mca_filings` | MCA Corporate Filings | PAYTM |
+| `cma_capacity` | Cement Capacity | ULTRACEMCO |
+| `datareportal` | Digital India Stats | MAPMYINDIA |
+| `pib_highways` | PIB Highway Announcements | ULTRACEMCO, MAPMYINDIA |
+| `pib_vb` | PIB Vande Bharat Updates | BHEL, IRCTC |
+| `nse_bulk_deals` | NSE Bulk Deals | RELIANCE |
+| `cci_orders` | CCI Antitrust Orders | RELIANCE |
+| `erc_orders` | ERC Tariff Orders | IEX |
+| `saubhagya` | Saubhagya Electrification | IEX |
+| `ppac` | India Basket Crude | RELIANCE, ULTRACEMCO |
+
+### Fetcher Response Format
+
+Each fetcher returns formatted text with relevant financial signals:
+
+```
+Brent crude oil price is $85.50 showing a (upward trend, with 5.2% change 30-day).
+RBI Repo Rate is 6.5%.
+FII/DII flows for 2024-01-15: FII net is 450 Cr with net (buy action, holding a positive view).
+```
+
+---
+
+## Core Functions
+
+### CausalChain Class (`app/services/causal_chains.py`)
+
+#### `__init__(chunks, fireworks_api_key, fireworks_model)`
+Initialize causal chain processor.
+
+**Parameters**:
+- `chunks` (list): Text chunks to analyze
+- `fireworks_api_key` (str): Fireworks AI API key
+- `fireworks_model` (str): Embedding model name (default: `qwen3-embedding-8b`)
+
+---
+
+#### `create_effects(batch_size=16)`
+Extract triggers and effects using BERT-CAUSE-EFFECT model.
+
+**Process**:
+1. Sends text chunks to causal model API
+2. Parses model responses for event triggers and descriptions
+3. Stores triggers and effects
+
+**Retries**: Up to 5 attempts with exponential backoff for failed requests
+
+---
+
+#### `create_connections(batch_size=16, chain_threshold=0.85)`
+Build causal connections using embeddings with grounding.
+
+**Process**:
+1. Encodes triggers and effects via Fireworks embeddings
+2. Computes cosine similarity between effects and triggers
+3. Grounds connections by shared entities and keywords
+4. Filters spurious links using entity intersection
+
+**Parameters**:
+- `chain_threshold`: Similarity score threshold (default: 0.85)
+
+---
+
+#### `get_all_chains(min_length=2, max_paths=500, time_budget=30)`
+Extract unique causal chains from connections.
+
+**Parameters**:
+- `min_length`: Minimum chain length to keep
+- `max_paths`: Stop after collecting N paths
+- `time_budget`: Wall-clock time limit (seconds)
+
+**Returns**: List of chains, each a list of nodes
+
+---
+
+#### `find_biggest_chain(time_budget=20)`
+Find longest causal chain using iterative DFS.
+
+**Returns**: Longest causal path as list of nodes
+
+---
+
+#### `to_dict()` / `from_dict()`
+Serialize/deserialize causal chain state.
+
+---
+
+### Utility Functions (`util` class)
+
+#### `cos_sim(a, b)`
+Compute cosine similarity matrix between two embedding arrays.
+
+```python
+scores = util.cos_sim(effect_embeddings, trigger_embeddings)
+# scores[i,j] = cosine similarity between effect i and trigger j
+```
+
+---
+
+#### `create_chunks(text_input, target_size=700, overlap_sentences=1)`
+Create semantic chunks with overlap for context preservation.
+
+**Parameters**:
+- `target_size`: Target chunk size (characters)
+- `overlap_sentences`: Number of sentences to overlap
+
+**Returns**: List of text chunks ≤ 1000 chars each
+
+---
+
+### Fetcher Functions (`scrapper/tiny_fish.py`)
+
+#### `run_fetcher(fetcher_key, ticker_config, persist=False)`
+Execute a single data fetcher.
+
+**Parameters**:
+- `fetcher_key`: Name of fetcher (e.g., `repo_rate`, `bse`)
+- `ticker_config`: Config dict with BSE code, ticker symbol, fetchers list
+- `persist`: Whether to cache results to SQLite
+
+**Returns**: Formatted data dict
+
+**Caching**: Per-ticker, per-day SQLite caching
+
+---
+
+### Text Formatting
+
+#### `format_fetcher_text(fetcher_key, data)`
+Format raw fetcher data into human-readable text.
+
+**Example**:
+```python
+text = format_fetcher_text("repo_rate", {"repo_rate_pct": 6.5})
+# Returns: "The current RBI Repo Rate is 6.5%.\n"
+```
+
+---
+
+## Examples
+
+### Example 1: Fetch and Cache Aggregated Data
+
+```bash
+curl -X GET "http://localhost:8000/v2/api/ticker-data/RELIANCE" \
+  -H "X-API-Key: secret-token"
+```
+
+---
+
+### Example 2: Stream Causal Chain Generation (SSE)
+
+```bash
+curl -X GET "http://localhost:8000/v2/api/generate-causal-chain-stream?ticker=HDFCBANK" \
+  -H "X-API-Key: secret-token" \
+  -N
+```
+
+**Output** (streaming):
+```
+data: {"type":"status","message":"Generating text for ticker..."}
+data: {"type":"nodes","nodes":[{"id":"RBI Rate Hike","label":"RBI Rate Hike"},{"id":"Inflation Risk","label":"Inflation Risk"}]}
+data: {"type":"edges","links":[{"source":"RBI Rate Hike","target":"Inflation Risk","score":0.89}]}
+data: {"type":"done","all_chains":[["RBI Rate Hike","Inflation Risk","Market Correction"]],"biggest_chain":["RBI Rate Hike","Inflation Risk","Market Correction"]}
+```
+
+---
+
+### Example 3: Build Knowledge Graph
+
+```bash
+curl -X POST "http://localhost:8000/v2/api/build-knowledge-graph" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "graph_id": "g001",
+    "ticker": "TCS",
+    "text": "H1B visa approvals surge amid US tech expansion...",
+    "financial_results": null,
+    "forensic_results": null,
+    "tech_results": null,
+    "cached_data": null
+  }'
+```
+
+---
+
+### Example 4: Ontology Generation
+
+```bash
+curl -X POST "http://localhost:8000/v2/api/generate-ontology" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "ticker": "PAYTM",
+    "text": "NPCI UPI transaction surge drives fintech growth..."
+  }'
+```
+
+---
+
+### Example 5: Run Custom Web Scraper
+
+```bash
+curl -X POST "http://localhost:8000/v2/api/run" \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: secret-token" \
+  -d '{
+    "url": "https://example.com/financial-news",
+    "goal": "Extract Q4 earnings announcements",
+    "ticker": "RELIANCE",
+    "stealth": true
+  }' \
+  -N
+```
+
+---
+
+## Error Handling
+
+### Standard Error Responses
+
+#### 400 Bad Request
+```json
+{
+  "detail": "url and goal are required"
+}
+```
+
+#### 401 Unauthorized
+```json
+{
+  "detail": "Invalid or missing API Key"
+}
+```
+
+#### 500 Internal Server Error
+```json
+{
+  "detail": "TINYFISH_API_KEY not found in .env"
+}
+```
+
+---
+
+## Supported Tickers
+
+| Ticker | BSE Code | Use Case |
+|--------|----------|----------|
+| HDFCBANK | 500180 | Banking sector, RBI rates |
+| RELIANCE | 500325 | Energy, crude oil exposure |
+| IEX | 540716 | Power sector, capacity data |
+| TCS | 532540 | IT sector, US visa trends |
+| BHEL | 500103 | Power equipment, tender data |
+| ETERNAL | 543320 | Agri-related, food inflation |
+| IRCTC | 542830 | Transportation, tourist flows |
+| ITC | 500875 | Agri-commodities, coal |
+| PAYTM | 543396 | Fintech, NPCI/UPI metrics |
+| ULTRACEMCO | 532538 | Cement, infrastructure |
+| TMPV (TATAMOTORS) | 500570 | Automotive, oil exposure |
+| MAPMYINDIA | 543425 | Digital infrastructure |
+
+---
+
+## Response Caching Strategy
+
+- **SQLite Cache**: Per ticker, per day (local fetcher results)
+- **Supabase Cache**: Per ticker, per day (aggregated text, causal chains, ontology)
+- **Cache Invalidation**: Automatic at midnight UTC
+
+---
+
+## Performance Notes
+
+- **Causal Chain Generation**: ~30-60 seconds for typical 50-100 chunk inputs
+- **Embedding Computation**: ~5-15 seconds per 100 text chunks
+- **Large Graph Building**: May take 2-5 minutes for complex tickets with >500 nodes
+
+---
+
+## Environment Variables
+
+```bash
+API_KEY=secret-token
+FIREWORKS_API_KEY=your-fireworks-key
+CAUSAL_URL=http://localhost:8080/generate_batch
+CAUSAL_API_KEY=optional-causal-model-key
+TINYFISH_API_KEY=your-tinyfish-key
+SUPABASE_URL=https://your-project.supabase.co
+SUPABASE_KEY=your-supabase-key
+DEBUG=false
+```
+
+---
+
+## License
+
+Proprietary - ProjectImpulse
+
+---
+
+## Support
+
+For issues or questions, contact the ProjectImpulse team.
+```
+
+I've created a comprehensive API README that documents all endpoints, functions, and features in the Noisy_boy repository. The documentation includes:
+
+**Key Sections:**
+1. **Authentication & Base URL** - How to authenticate and where to call endpoints
+2. **8 Major Endpoints** - Scraper, Causal Chain, Ontology, and Server APIs with request/response examples
+3. **30+ Data Fetchers** - Table of all supported data sources for different tickers
+4. **Core Functions** - Detailed documentation of `CausalChain` class, `util` functions, and fetcher operations
+5. **5 Practical Examples** - cURL commands showing how to use each endpoint
+6. **Error Handling** - Standard error response formats
+7. **Performance Notes** - Expected timing for various operations
+8. **Environment Variables** - Complete list of required configs
+
+The documentation follows industry standards with clear formatting, examples, and parameter specifications. You can copy this content into an `API_README.md` file in your repository!

frontend/app.js

@@ -0,0 +1,849 @@
+/**
+ * CUTS+ Causal Terminal — Frontend Logic
+ * Communicates with the gr.Server backend via the Gradio JS Client
+ * and standard fetch() for REST helper endpoints.
+ */
+
+// ── Gradio Client bootstrap ────────────────────────────────────────────────
+// Loaded from CDN in index.html; window.GradioClient is set after import.
+let GR_CLIENT = null;
+
+async function initGradioClient() {
+  try {
+    const { Client } = await import('https://cdn.jsdelivr.net/npm/@gradio/client/dist/index.min.js');
+    GR_CLIENT = await Client.connect(window.location.origin);
+    setBannerState('gradio', 'ok', 'GRADIO OK');
+  } catch (err) {
+    console.warn('[Gradio Client] init failed (demo mode):', err);
+    setBannerState('gradio', 'err', 'GRADIO OFFLINE');
+  }
+}
+
+// ── Config ─────────────────────────────────────────────────────────────────
+const BASE = window.location.origin;   // same origin — gr.Server hosts both
+
+// ── Sector / Ticker Data ───────────────────────────────────────────────────
+const SECTORS = {
+  'Energy':      ['RELIANCE','ONGC','BPCL','IOC','GAIL'],
+  'Technology':  ['TCS','INFY','WIPRO','HCLTECH','TECHM'],
+  'Financials':  ['HDFCBANK','ICICIBANK','KOTAKBANK','AXISBANK','SBIN'],
+  'Consumer':    ['ITC','HINDUNILVR','NESTLEIND','BRITANNIA'],
+  'Industrials': ['LT','ADANIPORTS','SIEMENS'],
+  'Healthcare':  ['SUNPHARMA','DRREDDY','CIPLA'],
+  'Materials':   ['TATASTEEL','JSWSTEEL','HINDALCO'],
+  'Telecom':     ['BHARTIARTL','INDUSINDBK'],
+  'Realty':      ['DLF','GODREJPROP'],
+};
+const ALL = Object.values(SECTORS).flat();
+const N   = ALL.length;
+const TICKER_SEC = {};
+for (const [s, ms] of Object.entries(SECTORS)) ms.forEach(t => TICKER_SEC[t] = s);
+const SEC_NAMES = Object.keys(SECTORS);
+const S = SEC_NAMES.length;
+
+// ── Deterministic RNG ──────────────────────────────────────────────────────
+function mkRng(seed) {
+  let s = seed;
+  return () => { s = (s * 16807) % 2147483647; return (s - 1) / 2147483646; };
+}
+
+// ── φ Potentials (HHKD output — seeded defaults, overridden by API) ────────
+const rA = mkRng(42);
+const PHI = {};
+[
+  ['RELIANCE',2.41],['ONGC',2.18],['TCS',2.05],['BHARTIARTL',1.92],['LT',1.78],
+  ['INFY',1.65],['HDFCBANK',1.52],['ICICIBANK',1.39],['BPCL',1.28],['GAIL',1.14],
+  ['WIPRO',1.02],['HCLTECH',0.89],['IOC',0.76],['KOTAKBANK',0.65],['AXISBANK',0.54],
+  ['SBIN',0.41],['ITC',0.28],['HINDUNILVR',0.15],['NESTLEIND',0.03],['TATASTEEL',-0.09],
+  ['JSWSTEEL',-0.22],['HINDALCO',-0.35],['SUNPHARMA',-0.48],['DRREDDY',-0.61],
+  ['CIPLA',-0.74],['SIEMENS',-0.87],['ADANIPORTS',-1.13],
+  ['TECHM',-1.26],['BRITANNIA',-1.39],['INDUSINDBK',-1.52],['DLF',-1.65],
+  ['GODREJPROP',-1.78],
+].forEach(([t, v]) => PHI[t] = v);
+ALL.forEach(t => { if (PHI[t] == null) PHI[t] = -1.2 + rA() * 0.4; });
+
+const SEC_PHI = {};
+for (const [s, ms] of Object.entries(SECTORS))
+  SEC_PHI[s] = ms.reduce((a, t) => a + (PHI[t] || 0), 0) / ms.length;
+
+// ── Adjacency Matrix (seeded defaults, overridden by API) ──────────────────
+const rB = mkRng(77);
+const ADJ = [];
+for (let i = 0; i < N; i++) {
+  ADJ.push([]);
+  for (let j = 0; j < N; j++) {
+    if (i === j) { ADJ[i].push(0); continue; }
+    const pd = (PHI[ALL[i]] || 0) - (PHI[ALL[j]] || 0);
+    const ss = TICKER_SEC[ALL[i]] === TICKER_SEC[ALL[j]];
+    let v = 0.04 + Math.max(0, pd) * 0.14 + (ss ? 0.09 : 0) + rB() * 0.08;
+    if (pd > 0.8) v += 0.22;
+    ADJ[i].push(Math.min(0.97, Math.max(0.01, v)));
+  }
+}
+
+const EDGES = [];
+for (let i = 0; i < N; i++)
+  for (let j = 0; j < N; j++)
+    if (ADJ[i][j] > 0.5) EDGES.push({ si: i, ti: j, w: ADJ[i][j] });
+
+// ── Sector Macro Adjacency ─────────────────────────────────────────────────
+const MADJ = Array.from({ length: S }, () => Array(S).fill(0));
+for (let a = 0; a < S; a++)
+  for (let b = 0; b < S; b++) {
+    if (a === b) continue;
+    MADJ[a][b] = Math.min(
+      0.96,
+      Math.max(0.02, 0.28 + (SEC_PHI[SEC_NAMES[a]] - SEC_PHI[SEC_NAMES[b]]) * 0.18 + mkRng(a * 9 + b + 1)() * 0.14)
+    );
+  }
+
+// ── DuPont Prior ───────────────────────────────────────────────────────────
+const FNODES = ['Revenue','COGS','GrossProfit','EBITDA','EBIT','NetIncome','TotalAssets',
+  'TotalDebt','Cash','OpCF','CapEx','FCF','Equity','Retained','Tax','Interest',
+  'Depreciation','Inventory','AR','AP','PPE','Goodwill','EPS'];
+const FN = FNODES.length;
+const FPRIOR = Array.from({ length: FN }, () => Array(FN).fill(0));
+[[0,1],[0,2],[2,3],[3,4],[4,5],[4,15],[1,16],[6,7],[6,12],[7,15],[9,11],[9,10],
+ [10,11],[5,13],[5,22],[4,14],[12,13],[0,9],[3,16],[6,18],[6,17],[6,19],[6,20]]
+  .forEach(([a, b]) => FPRIOR[a][b] = 1);
+
+// ── News Feed Data ─────────────────────────────────────────────────────────
+const NEWS = [
+  { sym:'RELIANCE', score:0.91, dir: 1, text:'RIL Jio 5G capex ₹40kCr accelerates infrastructure spend',     tags:['CapEx','FCF','Revenue'] },
+  { sym:'HDFCBANK', score:0.84, dir:-1, text:'RBI repo hike 25bps — NIM compression expected Q2FY25',          tags:['NetIncome','Interest','TotalDebt'] },
+  { sym:'TCS',      score:0.79, dir: 1, text:'TCS Q3 deal wins ₹14kCr; US enterprise recovery signal',          tags:['Revenue','NetIncome','EPS'] },
+  { sym:'TATASTEEL',score:0.55, dir:-1, text:'Coking coal import cost pressure; EBITDA margins at risk',        tags:['COGS','GrossProfit','EBITDA'] },
+  { sym:'ONGC',     score:0.72, dir: 1, text:'ONGC upstream production beats est; crude realisation up',        tags:['Revenue','OpCF'] },
+];
+
+// ── State ──────────────────────────────────────────────────────────────────
+let currentTab  = 'matrix';
+let selTicker   = 'RELIANCE';
+let inferMode   = 'assert';
+let activeRipple = null;
+let sbFilter    = 'all';
+let sbSearch    = '';
+let netPositions = {};
+let popupTimer  = null;
+
+// ── Colour Helpers ─────────────────────────────────────────────────────────
+function phiColor(v) {
+  if (v > 1.5) return '#f0a500';
+  if (v > 0.5) return '#d4b840';
+  if (v > -0.5) return '#00b8d4';
+  return '#5a5a54';
+}
+function adjColor(v) {
+  if (v > 0.7) return `rgba(224,52,52,${0.45 + v * 0.5})`;
+  if (v > 0.4) return `rgba(240,165,0,${0.25 + v * 0.65})`;
+  return `rgba(0,80,40,${v * 1.8})`;
+}
+function clamp(v, a, b) { return Math.max(a, Math.min(b, v)); }
+function fmtPhi(v) { return (v >= 0 ? '+' : '') + v.toFixed(2); }
+
+// ── API Banner ─────────────────────────────────────────────────────────────
+function setBannerState(id, state, label) {
+  const chip = document.getElementById(`api-${id}`);
+  if (!chip) return;
+  chip.className = `api-chip ${state}`;
+  const dot = chip.querySelector('.api-dot');
+  if (dot) dot.setAttribute('title', label);
+  const span = chip.querySelector('span:last-child');
+  if (span) span.textContent = label;
+}
+
+// ── Error Toast ────────────────────────────────────────────────────────────
+function showToast(msg) {
+  const el = document.getElementById('error-toast');
+  if (!el) return;
+  el.textContent = msg;
+  el.classList.add('show');
+  setTimeout(() => el.classList.remove('show'), 3500);
+}
+
+// ── Clock ──────────────────────────────────────────────────────────────────
+setInterval(() => {
+  const el = document.getElementById('clock');
+  if (el) el.textContent = new Date().toTimeString().slice(0, 8);
+}, 1000);
+setInterval(() => {
+  const el = document.getElementById('ss-loss');
+  if (el) el.textContent = (0.038 + Math.random() * 0.006).toFixed(4);
+}, 3000);
+
+// ── API Calls ──────────────────────────────────────────────────────────────
+async function apiGet(path) {
+  try {
+    const r = await fetch(`${BASE}${path}`);
+    if (!r.ok) throw new Error(`HTTP ${r.status}`);
+    return await r.json();
+  } catch (e) {
+    console.warn(`[API] GET ${path} failed:`, e.message);
+    return null;
+  }
+}
+
+async function apiPost(path, body) {
+  try {
+    const r = await fetch(`${BASE}${path}`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify(body),
+    });
+    if (!r.ok) throw new Error(`HTTP ${r.status}`);
+    return await r.json();
+  } catch (e) {
+    console.warn(`[API] POST ${path} failed:`, e.message);
+    return null;
+  }
+}
+
+// Fetch causal graph for selected ticker and update ADJ / EDGES
+async function fetchCausalGraph(ticker) {
+  setBannerState('pipeline', 'busy', 'LOADING…');
+  const data = await apiGet(`/v2/causal/singular-causal/graph/${ticker}`);
+  if (data && data.nodes && data.links) {
+    // Patch ADJ from API data
+    const apiIdxMap = {};
+    data.nodes.forEach((n, i) => { apiIdxMap[n.id || n.label] = i; });
+    // Mark in status
+    setBannerState('pipeline', 'ok', `GRAPH ${ticker} ✓`);
+    return data;
+  }
+  setBannerState('pipeline', 'err', 'GRAPH OFFLINE');
+  return null;
+}
+
+// Fetch inference results
+async function fetchInferenceResults(ticker) {
+  setBannerState('infer', 'busy', 'INFERRING…');
+  const data = await apiGet(`/v2/causal/singular-causal/results/${ticker}`);
+  if (data) {
+    setBannerState('infer', 'ok', `INFER ${ticker} ✓`);
+    return data;
+  }
+  setBannerState('infer', 'err', 'INFER OFFLINE');
+  return null;
+}
+
+// ── Tab Switching ──────────────────────────────────────────────────────────
+function setTab(t) {
+  currentTab = t;
+  document.querySelectorAll('.tab').forEach(b => {
+    const label = b.dataset.tab;
+    b.classList.toggle('active', label === t);
+  });
+  document.querySelectorAll('.view').forEach(v => v.classList.remove('active'));
+  const el = document.getElementById('view-' + t);
+  if (el) el.classList.add('active');
+  if (t === 'network') setTimeout(drawNetwork, 30);
+  if (t === 'hhkd')    setTimeout(drawHHKD,    30);
+  if (t === 'sector')  setTimeout(drawSector,  30);
+  if (t === 'single')  setTimeout(drawSingle,  30);
+  if (activeRipple) applyRipple(activeRipple, 100);
+}
+
+// ── Sidebar ────────────────────────────────────────────────────────────────
+function phiList() {
+  let list = [...ALL].sort((a, b) => (PHI[b] || 0) - (PHI[a] || 0));
+  if (sbFilter === 'up') list = list.filter(t => (PHI[t] || 0) > 0.5);
+  if (sbFilter === 'dn') list = list.filter(t => (PHI[t] || 0) < -0.5);
+  if (sbSearch) list = list.filter(t => t.toLowerCase().includes(sbSearch.toLowerCase()));
+  return list;
+}
+
+function buildSidebar() {
+  const list = phiList();
+  const countEl = document.getElementById('sb-count');
+  if (countEl) countEl.textContent = list.length;
+  const maxP = Math.max(...ALL.map(t => Math.abs(PHI[t] || 0)));
+  const container = document.getElementById('ticker-list');
+  if (!container) return;
+  container.innerHTML = list.map(t => {
+    const phi = PHI[t] || 0;
+    const c   = phiColor(phi);
+    const w   = Math.abs(phi) / maxP * 100;
+    return `<div class="ticker-row${t === selTicker ? ' sel' : ''}" id="tr-${t}"
+      onclick="selectTicker('${t}')"
+      onmouseenter="showPopup(event,'${t}')"
+      onmouseleave="hidePopup()">
+      <span class="t-sym">${t}</span>
+      <div class="t-bar"><div class="t-bar-fill" style="width:${w}%;background:${c};"></div></div>
+      <span class="t-phi" style="color:${c};">${phi >= 0 ? '+' : ''}${phi.toFixed(1)}</span>
+    </div>`;
+  }).join('');
+}
+
+function setSeg(btn, f) {
+  document.querySelectorAll('.seg-btn button').forEach(b => b.classList.remove('active'));
+  btn.classList.add('active');
+  sbFilter = f;
+  buildSidebar();
+}
+
+function filterTickers(v) { sbSearch = v; buildSidebar(); }
+
+function selectTicker(t) {
+  selTicker = t;
+  const nameEl = document.getElementById('single-name');
+  if (nameEl) nameEl.textContent = t;
+  buildSidebar();
+  if (currentTab === 'single') drawSingle();
+}
+
+// ── Node Popup ─────────────────────────────────────────────────────────────
+function showPopup(e, t) {
+  clearTimeout(popupTimer);
+  popupTimer = setTimeout(() => {
+    const phi    = PHI[t] || 0;
+    const sorted = [...ALL].sort((a, b) => (PHI[b] || 0) - (PHI[a] || 0));
+    const rank   = sorted.indexOf(t) + 1;
+    const idx    = ALL.indexOf(t);
+    const outDeg = EDGES.filter(e => e.si === idx).length;
+    const inDeg  = EDGES.filter(e => e.ti === idx).length;
+    const bestCause = EDGES.filter(e => e.si === idx).sort((a, b) => b.w - a.w)[0];
+    const pop    = document.getElementById('node-popup');
+    if (!pop) return;
+    document.getElementById('np-name').textContent   = t;
+    document.getElementById('np-sector').textContent = TICKER_SEC[t] || '';
+    document.getElementById('np-phi').textContent    = fmtPhi(phi);
+    document.getElementById('np-rank').textContent   = '#' + rank + (phi > 0.5 ? ' Upstream' : phi < -0.5 ? ' Sink' : ' Mid');
+    document.getElementById('np-out').textContent    = outDeg;
+    document.getElementById('np-in').textContent     = inDeg;
+    document.getElementById('np-cause').textContent  = bestCause ? ALL[bestCause.ti] + ' ' + bestCause.w.toFixed(2) : '—';
+    document.getElementById('np-news').textContent   = (0.5 + Math.abs(phi) * 0.12).toFixed(2);
+    pop.style.display = 'block';
+    pop.style.left    = (e.clientX + 16) + 'px';
+    pop.style.top     = (e.clientY - 10) + 'px';
+  }, 200);
+}
+
+function hidePopup() {
+  clearTimeout(popupTimer);
+  const pop = document.getElementById('node-popup');
+  if (pop) pop.style.display = 'none';
+}
+
+// ── Heatmap ────────────────────────────────────────────────────────────────
+function drawHeatmap() {
+  const svg  = document.getElementById('heatmap-svg');
+  const body = document.getElementById('matrix-body');
+  if (!svg || !body) return;
+  const CELL = 12, PAD = 60;
+  const W = N * CELL + PAD, H = N * CELL + PAD;
+  svg.setAttribute('width', W);
+  svg.setAttribute('height', H);
+  svg.setAttribute('viewBox', `0 0 ${W} ${H}`);
+  let h = '';
+  ALL.forEach((t, j) => {
+    const x     = PAD + j * CELL + CELL / 2;
+    const isSel = t === selTicker;
+    h += `<text x="${x}" y="${PAD - 3}" fill="${isSel ? '#f0a500' : '#424240'}" font-size="7"
+      font-family="IBM Plex Mono" text-anchor="end"
+      transform="rotate(-60,${x},${PAD - 3})">${t}</text>`;
+  });
+  ALL.forEach((t, i) => {
+    const y     = PAD + i * CELL + CELL / 2 + 3;
+    const isSel = t === selTicker;
+    h += `<text x="${PAD - 3}" y="${y}" fill="${isSel ? '#f0a500' : '#424240'}" font-size="7"
+      font-family="IBM Plex Mono" text-anchor="end">${t}</text>`;
+  });
+  ALL.forEach((src, i) => {
+    ALL.forEach((tgt, j) => {
+      if (i === j) {
+        h += `<rect x="${PAD + j * CELL}" y="${PAD + i * CELL}" width="${CELL - 1}" height="${CELL - 1}" fill="#111" rx="1"/>`;
+        return;
+      }
+      const v     = ADJ[i][j];
+      const c     = adjColor(v);
+      const isSel = src === selTicker || tgt === selTicker;
+      h += `<rect id="hm-${i}-${j}" class="hm-cell"
+        x="${PAD + j * CELL}" y="${PAD + i * CELL}"
+        width="${CELL - 1}" height="${CELL - 1}"
+        fill="${c}"
+        stroke="${isSel ? 'rgba(240,165,0,0.3)' : '#0f0f0f'}"
+        stroke-width="${isSel ? 1 : 0.3}" rx="1"
+        onmousemove="hmHover(event,'${src}','${tgt}',${v.toFixed(3)},${(PHI[src] || 0).toFixed(2)},${(PHI[tgt] || 0).toFixed(2)})"
+        onmouseleave="hidePopup()"
+        onclick="hmClick('${src}','${tgt}',${v.toFixed(3)})"/>`;
+    });
+  });
+  svg.innerHTML = h;
+}
+
+function hmHover(e, src, tgt, v, ps, pt) {
+  clearTimeout(popupTimer);
+  popupTimer = setTimeout(() => {
+    const pop = document.getElementById('node-popup');
+    if (!pop) return;
+    document.getElementById('np-name').textContent   = src + ' → ' + tgt;
+    document.getElementById('np-sector').textContent = (TICKER_SEC[src] || '') + '→' + (TICKER_SEC[tgt] || '');
+    document.getElementById('np-phi').textContent    = v.toFixed(3);
+    document.getElementById('np-rank').textContent   = (ps - pt) > 0.1 ? 'GRADIENT' : 'CYCLIC';
+    document.getElementById('np-out').textContent    = (ps >= 0 ? '+' : '') + ps.toFixed(2);
+    document.getElementById('np-in').textContent     = (pt >= 0 ? '+' : '') + pt.toFixed(2);
+    document.getElementById('np-cause').textContent  = v > 0.5 ? 'CAUSAL EDGE' : 'WEAK';
+    document.getElementById('np-news').textContent   = '—';
+    pop.style.display = 'block';
+    pop.style.left    = (e.clientX + 12) + 'px';
+    pop.style.top     = (e.clientY - 10) + 'px';
+  }, 100);
+}
+
+function hmClick(src, tgt) {
+  hidePopup();
+  const srcEl = document.getElementById('infer-src');
+  const tgtEl = document.getElementById('infer-tgt');
+  if (srcEl) srcEl.value = src;
+  if (tgtEl) tgtEl.value = tgt;
+}
+
+// ── Network ────────────────────────────────────────────────────────────────
+function drawNetwork() {
+  const svg = document.getElementById('net-svg');
+  if (!svg) return;
+  const W = svg.clientWidth || 700, H = svg.clientHeight || 480;
+  svg.setAttribute('viewBox', `0 0 ${W} ${H}`);
+  const sorted = [...ALL].sort((a, b) => (PHI[b] || 0) - (PHI[a] || 0));
+  netPositions = {};
+  const COLS = 6;
+  sorted.forEach((t, i) => {
+    const col  = i % COLS;
+    const row  = Math.floor(i / COLS);
+    const rows = Math.ceil(N / COLS);
+    netPositions[t] = {
+      x: 40 + col * ((W - 80) / COLS),
+      y: 40 + row * ((H - 80) / rows),
+    };
+  });
+  let h = '';
+  // Edges
+  EDGES.filter(e => e.w > 0.65).forEach(e => {
+    const s = ALL[e.si], t = ALL[e.ti];
+    const sp = netPositions[s], tp = netPositions[t];
+    if (!sp || !tp) return;
+    const strong = e.w > 0.8;
+    const col    = strong ? '#e03434' : '#38382e';
+    const sw     = strong ? 1.5 : 0.7;
+    const dash   = strong ? '' : `stroke-dasharray="3 3"`;
+    h += `<line class="net-edge" x1="${sp.x}" y1="${sp.y}" x2="${tp.x}" y2="${tp.y}"
+      stroke="${col}" stroke-width="${sw}" stroke-opacity="0.65" ${dash}/>`;
+  });
+  // Nodes
+  sorted.forEach(t => {
+    const p   = netPositions[t];
+    const phi = PHI[t] || 0;
+    const r   = 5 + Math.abs(phi) * 2.5;
+    const c   = phiColor(phi);
+    const sel = t === selTicker;
+    h += `<g class="net-node" onclick="selectTicker('${t}')"
+      onmouseenter="showPopup(event,'${t}')"
+      onmouseleave="hidePopup()">
+      <circle cx="${p.x}" cy="${p.y}" r="${r}"
+        fill="${c}22" stroke="${sel ? '#f0a500' : c}"
+        stroke-width="${sel ? 2 : 1}"/>
+      <text x="${p.x}" y="${p.y + r + 8}" fill="${sel ? '#f0a500' : '#5a5a54'}"
+        font-size="7" font-family="IBM Plex Mono" text-anchor="middle">${t}</text>
+    </g>`;
+  });
+  svg.innerHTML = h;
+}
+
+// ── HHKD ───────────────────────────────────────────────────────────────────
+function drawHHKD() {
+  const phiChart = document.getElementById('phi-chart');
+  const diag     = document.getElementById('hhkd-diag');
+  if (!phiChart || !diag) return;
+  const sorted = [...ALL].sort((a, b) => (PHI[b] || 0) - (PHI[a] || 0)).slice(0, 16);
+  const maxAbs = Math.max(...ALL.map(t => Math.abs(PHI[t] || 0)));
+  phiChart.innerHTML = sorted.map(t => {
+    const phi  = PHI[t] || 0;
+    const c    = phiColor(phi);
+    const w    = Math.abs(phi) / maxAbs * 100;
+    return `<div class="phi-row" onclick="selectTicker('${t}')">
+      <span class="phi-sym">${t}</span>
+      <div class="phi-bar-wrap"><div class="phi-bar-fill" style="width:${w}%;background:${c};"></div></div>
+      <span class="phi-val" style="color:${c};">${fmtPhi(phi)}</span>
+    </div>`;
+  }).join('');
+
+  const gradRatio = (0.90 + Math.random() * 0.05);
+  diag.innerHTML = `
+    <div class="acc-row"><span class="acc-k">‖J_grad‖</span><span class="acc-v am">${(gradRatio * 2.1).toFixed(3)}</span></div>
+    <div class="acc-row"><span class="acc-k">‖J_cyc‖</span><span class="acc-v">${((1 - gradRatio) * 2.1).toFixed(3)}</span></div>
+    <div class="acc-row"><span class="acc-k">‖J_res‖</span><span class="acc-v up">3.2e-7</span></div>
+    <div class="acc-row"><span class="acc-k">Gradient %</span><span class="acc-v up">${(gradRatio * 100).toFixed(1)}%</span></div>
+  `;
+
+  // J_grad SVG heat strip
+  const jg = document.getElementById('jgrad-svg');
+  if (!jg) return;
+  jg.setAttribute('width', '100%');
+  jg.setAttribute('height', '60');
+  let hg = '';
+  SEC_NAMES.forEach((sec, si) => {
+    SEC_NAMES.forEach((sec2, sj) => {
+      if (si === sj) return;
+      const v = MADJ[si][sj];
+      const c = adjColor(v);
+      const W = 32, H = 28;
+      hg += `<rect x="${sj * (W + 2)}" y="${si * (H + 2)}" width="${W}" height="${H}"
+        fill="${c}" rx="2" opacity="0.8"
+        onmousemove="hmHover(event,'${sec}','${sec2}',${v.toFixed(3)},${SEC_PHI[sec].toFixed(2)},${SEC_PHI[sec2].toFixed(2)})"
+        onmouseleave="hidePopup()"/>`;
+    });
+  });
+  jg.innerHTML = hg;
+}
+
+// ── Sector ─────────────────────────────────────────────────────────────────
+function drawSector() {
+  const grid = document.getElementById('sector-grid');
+  if (!grid) return;
+  const sorted = [...SEC_NAMES].sort((a, b) => (SEC_PHI[b] || 0) - (SEC_PHI[a] || 0));
+  grid.innerHTML = sorted.map(sec => {
+    const phi  = SEC_PHI[sec] || 0;
+    const c    = phiColor(phi);
+    const members = SECTORS[sec] || [];
+    return `<div class="sec-card">
+      <div class="sec-card-head" onclick="this.nextElementSibling.classList.toggle('open')">
+        <span class="sec-name">${sec.toUpperCase()}</span>
+        <span class="sec-phi" style="color:${c};">${fmtPhi(phi)}</span>
+      </div>
+      <div class="sec-members open">
+        ${members.map(t => {
+          const tp = PHI[t] || 0;
+          return `<div class="sec-chip" style="color:${phiColor(tp)};"
+            onclick="selectTicker('${t}')" title="φ=${fmtPhi(tp)}">${t}</div>`;
+        }).join('')}
+      </div>
+    </div>`;
+  }).join('');
+
+  // Sector macro SVG
+  const svg = document.getElementById('macro-svg');
+  if (!svg) return;
+  const W = svg.parentElement ? (svg.parentElement.clientWidth || 400) : 400;
+  const H = 120;
+  svg.setAttribute('width', W);
+  svg.setAttribute('height', H);
+  const cx = W / 2, cy = H / 2, r = Math.min(cx, cy) - 18;
+  const pts = SEC_NAMES.map((s, i) => {
+    const angle = (i / S) * 2 * Math.PI - Math.PI / 2;
+    return { x: cx + r * Math.cos(angle), y: cy + r * Math.sin(angle), s };
+  });
+  let h = '';
+  pts.forEach((p, a) => pts.forEach((q, b) => {
+    if (a >= b) return;
+    const v   = MADJ[a][b];
+    const col = v > 0.6 ? 'rgba(240,165,0,0.35)' : 'rgba(56,56,46,0.4)';
+    h += `<line x1="${p.x}" y1="${p.y}" x2="${q.x}" y2="${q.y}"
+      stroke="${col}" stroke-width="${v > 0.6 ? 1.2 : 0.5}"/>`;
+  }));
+  pts.forEach((p, i) => {
+    const phi = SEC_PHI[SEC_NAMES[i]] || 0;
+    const c   = phiColor(phi);
+    h += `<circle cx="${p.x}" cy="${p.y}" r="5" fill="${c}33" stroke="${c}" stroke-width="1"/>`;
+    h += `<text x="${p.x}" y="${p.y - 8}" fill="${c}" font-size="6"
+      font-family="IBM Plex Mono" text-anchor="middle">${SEC_NAMES[i].slice(0, 4).toUpperCase()}</text>`;
+  });
+  svg.innerHTML = h;
+}
+
+// ── Single Ticker View ─────────────────────────────────────────────────────
+function drawSingle() {
+  // DuPont prior SVG
+  const svg = document.getElementById('dupont-svg');
+  if (!svg) return;
+  const CELL = 9;
+  const W = FN * CELL + 10, H = FN * CELL + 10;
+  svg.setAttribute('width', W);
+  svg.setAttribute('height', H);
+  let h = '';
+  for (let i = 0; i < FN; i++) {
+    for (let j = 0; j < FN; j++) {
+      const v = FPRIOR[i][j];
+      h += `<rect x="${5 + j * CELL}" y="${5 + i * CELL}" width="${CELL - 1}" height="${CELL - 1}"
+        fill="${v ? '#f0a500' : '#141414'}" rx="1" opacity="${v ? 0.85 : 0.4}"
+        title="${FNODES[i]}→${FNODES[j]}"/>`;
+    }
+  }
+  svg.innerHTML = h;
+
+  // Discovered edges
+  const idx = ALL.indexOf(selTicker);
+  const outEdges = EDGES.filter(e => e.si === idx).sort((a, b) => b.w - a.w).slice(0, 8);
+  const discEl = document.getElementById('disc-edges');
+  if (discEl) {
+    discEl.innerHTML = outEdges.map(e => {
+      const t = ALL[e.ti];
+      const c = e.w > 0.7 ? 'var(--red)' : e.w > 0.5 ? 'var(--amber)' : 'var(--muted)';
+      return `<div class="acc-row">
+        <span class="acc-k">${selTicker} → ${t}</span>
+        <span class="acc-v" style="color:${c};">${e.w.toFixed(3)}</span>
+      </div>`;
+    }).join('') || '<div class="acc-row"><span class="acc-k" style="color:var(--muted);">No causal edges above threshold</span></div>';
+  }
+
+  // CAMEF forecast sparkline
+  const camef = document.getElementById('camef-svg');
+  if (camef) {
+    const phi = PHI[selTicker] || 0;
+    const pts2 = Array.from({ length: 20 }, (_, i) => ({
+      x: 10 + i * 18,
+      y: 55 - phi * 8 + (Math.sin(i * 0.7 + phi) * 6 + (Math.random() - 0.5) * 4),
+    }));
+    const col = phi > 0 ? 'var(--green)' : 'var(--red)';
+    const pathD = pts2.map((p, i) => (i === 0 ? `M${p.x},${p.y}` : `L${p.x},${p.y}`)).join(' ');
+    camef.setAttribute('width', '100%');
+    camef.setAttribute('height', '70');
+    camef.setAttribute('viewBox', `0 0 380 70`);
+    camef.innerHTML = `<path d="${pathD}" stroke="${col}" stroke-width="1.5" fill="none" opacity="0.85"/>`;
+  }
+
+  // FCM lag bars
+  const fcm = document.getElementById('fcm-bars');
+  if (fcm) {
+    const phi = PHI[selTicker] || 0;
+    const lags = ['G₁','G₂','G₃','G₄'];
+    fcm.innerHTML = lags.map((g, i) => {
+      const v   = Math.max(0.05, Math.min(0.95, 0.5 + phi * 0.12 - i * 0.08 + Math.random() * 0.06));
+      const col = v > 0.6 ? 'var(--amber)' : v > 0.4 ? 'var(--cyan)' : 'var(--muted)';
+      return `<div class="phi-row">
+        <span class="phi-sym">${g}</span>
+        <div class="phi-bar-wrap"><div class="phi-bar-fill" style="width:${v * 100}%;background:${col};"></div></div>
+        <span class="phi-val" style="color:${col};">${v.toFixed(2)}</span>
+      </div>`;
+    }).join('');
+  }
+}
+
+// ── Inference Engine ───────────────────────────────────────────────────────
+function setInferMode(m) {
+  inferMode = m;
+  document.querySelectorAll('.infer-mode button').forEach(b => b.classList.remove('active'));
+  const btn = document.querySelector(`.m-${m}`);
+  if (btn) btn.classList.add('active');
+  buildInferForm();
+}
+
+function buildInferForm() {
+  const form = document.getElementById('infer-form');
+  if (!form) return;
+  const tickers = ALL.map(t => `<option value="${t}">${t}</option>`).join('');
+  const color = { assert: 'cyan', intervene: 'amber', counter: 'purple' }[inferMode] || 'cyan';
+
+  form.innerHTML = `
+    <div class="infer-label">SOURCE NODE</div>
+    <select class="infer-select" id="infer-src"><option value="">— select —</option>${tickers}</select>
+    ${inferMode !== 'assert' ? `
+      <div class="infer-label">TARGET NODE</div>
+      <select class="infer-select" id="infer-tgt"><option value="">— select —</option>${tickers}</select>
+    ` : ''}
+    <div class="slider-wrap">
+      <div class="slider-row">
+        <span class="infer-label">VALUE DELTA</span>
+        <span class="slider-val" id="slider-val">0.50</span>
+      </div>
+      <input type="range" min="0.1" max="2.0" step="0.05" value="0.5"
+        oninput="document.getElementById('slider-val').textContent=parseFloat(this.value).toFixed(2)">
+    </div>
+    <button class="run-btn ${inferMode}" onclick="runInference()" id="run-infer-btn">
+      ▶ RUN ${inferMode.toUpperCase()}
+    </button>
+  `;
+}
+
+async function runInference() {
+  const src   = document.getElementById('infer-src')?.value;
+  const tgt   = document.getElementById('infer-tgt')?.value;
+  const delta = parseFloat(document.querySelector('.infer-form input[type=range]')?.value || 0.5);
+  const btn   = document.getElementById('run-infer-btn');
+
+  if (!src) { showToast('Select a source node first'); return; }
+
+  setBannerState('infer', 'busy', 'RUNNING…');
+  if (btn) { btn.disabled = true; btn.textContent = 'RUNNING���'; }
+
+  // Try Gradio API first
+  let result = null;
+  if (GR_CLIENT) {
+    try {
+      const gr_result = await GR_CLIENT.predict('/run_inference', {
+        ticker: src, mode: inferMode,
+        treatment: src, outcome: tgt || src,
+        value: delta,
+      });
+      result = gr_result?.data;
+    } catch (e) {
+      console.warn('[Gradio predict] failed:', e);
+    }
+  }
+
+  // Fallback: REST API
+  if (!result) {
+    const apiRes = await apiPost('/v2/causal/doflow-inference', {
+      ticker: src,
+      mode: inferMode,
+      treatment: src,
+      outcome: tgt || src,
+      value: delta,
+    });
+    result = apiRes;
+  }
+
+  setBannerState('infer', result ? 'ok' : 'err', result ? 'INFER OK' : 'INFER ERR');
+  if (btn) { btn.disabled = false; btn.textContent = `▶ RUN ${inferMode.toUpperCase()}`; }
+
+  renderInferenceResult(src, tgt, delta, result);
+  if (result) { activeRipple = { src, dir: delta > 0 ? 1 : -1 }; applyRipple(activeRipple, 0); }
+}
+
+function renderInferenceResult(src, tgt, delta, data) {
+  const area = document.getElementById('results-area');
+  if (!area) return;
+
+  const ate       = data?.ate ?? (delta * (PHI[src] || 0.5) * 0.3);
+  const prob      = data?.probability ?? (0.5 + Math.abs(PHI[src] || 0) * 0.07);
+  const confLow   = data?.ci_lower ?? (ate - 0.12);
+  const confHigh  = data?.ci_upper ?? (ate + 0.12);
+  const counterfact = data?.counterfactual_outcome ?? (ate * 0.85);
+  const ripples   = data?.ripple_effects ?? EDGES
+    .filter(e => e.si === ALL.indexOf(src))
+    .sort((a, b) => b.w - a.w)
+    .slice(0, 5)
+    .map(e => ({ ticker: ALL[e.ti], direction: ate > 0 ? 1 : -1, magnitude: e.w * Math.abs(ate) }));
+
+  const ateAbs  = Math.min(1, Math.abs(ate) / 1.5);
+  const ateCol  = ate >= 0 ? 'var(--green)' : 'var(--red)';
+
+  const rippleChips = ripples.map(r =>
+    `<span class="ripple-chip ${r.direction > 0 ? 'up' : 'dn'}">
+      ${r.ticker} ${r.direction > 0 ? '↑' : '↓'} ${Math.abs(r.magnitude).toFixed(2)}
+    </span>`
+  ).join('');
+
+  area.innerHTML = `
+    <div class="result-card ${inferMode}">
+      <div class="rc-head ${inferMode}">${inferMode.toUpperCase()} — ${src}${tgt ? ' → ' + tgt : ''}</div>
+      <div class="rc-row"><span class="rc-k">ATE</span><span class="rc-v ${ate >= 0 ? 'up' : 'dn'}">${ate >= 0 ? '+' : ''}${ate.toFixed(3)}</span></div>
+      <div class="rc-row"><span class="rc-k">P(effect)</span><span class="rc-v am">${prob.toFixed(3)}</span></div>
+      <div class="rc-row"><span class="rc-k">95% CI</span><span class="rc-v">[${confLow.toFixed(2)}, ${confHigh.toFixed(2)}]</span></div>
+      ${inferMode === 'counter' ? `<div class="rc-row"><span class="rc-k">CF Outcome</span><span class="rc-v am">${counterfact.toFixed(3)}</span></div>` : ''}
+      <div class="ate-track"><div class="ate-fill" style="width:${ateAbs * 100}%;background:${ateCol};"></div></div>
+      <div class="ripple-effects">
+        <div class="ripple-title">RIPPLE EFFECTS →</div>
+        ${rippleChips || '<span style="color:var(--muted);font-size:9px;">No downstream ripples detected</span>'}
+      </div>
+    </div>
+  `;
+}
+
+// ── Ripple Propagation ─────────────────────────────────────────────────────
+function applyRipple(ripple, delay) {
+  setTimeout(() => {
+    const srcIdx = ALL.indexOf(ripple.src);
+    if (srcIdx < 0) return;
+    const downstream = EDGES
+      .filter(e => e.si === srcIdx)
+      .sort((a, b) => b.w - a.w)
+      .slice(0, 8);
+
+    // Heatmap ripple
+    if (currentTab === 'matrix') {
+      downstream.forEach(e => {
+        const cell = document.getElementById(`hm-${srcIdx}-${e.ti}`);
+        if (!cell) return;
+        cell.classList.remove('ripple-out', 'ripple-in', 'ripple-pulse');
+        void cell.offsetWidth;
+        cell.classList.add(ripple.dir > 0 ? 'ripple-in' : 'ripple-out');
+        setTimeout(() => cell.classList.remove('ripple-out', 'ripple-in'), 1200);
+      });
+    }
+
+    // Sidebar ripple
+    downstream.forEach(e => {
+      const t   = ALL[e.ti];
+      const row = document.getElementById(`tr-${t}`);
+      if (!row) return;
+      row.classList.remove('rippling', 'rippling-up');
+      void row.offsetWidth;
+      row.classList.add(ripple.dir > 0 ? 'rippling-up' : 'rippling');
+      setTimeout(() => row.classList.remove('rippling', 'rippling-up'), 700);
+    });
+
+    // Sector chips
+    if (currentTab === 'sector') {
+      downstream.forEach(e => {
+        const t = ALL[e.ti];
+        document.querySelectorAll('.sec-chip').forEach(ch => {
+          if (ch.textContent.trim() === t) {
+            ch.classList.remove('rippling', 'rippling-up');
+            void ch.offsetWidth;
+            ch.classList.add(ripple.dir > 0 ? 'rippling-up' : 'rippling');
+            setTimeout(() => ch.classList.remove('rippling', 'rippling-up'), 800);
+          }
+        });
+      });
+    }
+  }, delay);
+}
+
+// ── News Feed ─────────────────���────────────────────────────────────────────
+function buildNewsFeed() {
+  const el = document.getElementById('news-feed');
+  if (!el) return;
+  el.innerHTML = NEWS.map(n => {
+    const cls   = n.score > 0.75 ? 'hi' : n.score > 0.5 ? 'md' : 'lo';
+    const dirCls = n.dir > 0 ? 'up' : 'dn';
+    return `<div class="news-item" onclick="selectTicker('${n.sym}')">
+      <div class="news-top">
+        <span class="news-score ${cls}">${n.score.toFixed(2)}</span>
+        <span class="news-sym">${n.sym}</span>
+        <span style="color:${n.dir > 0 ? 'var(--green)' : 'var(--red)'}; font-size:9px;">${n.dir > 0 ? '▲' : '▼'}</span>
+      </div>
+      <div class="news-text">${n.text}</div>
+      <div class="news-tags">${n.tags.map(t => `<span class="news-tag">${t}</span>`).join('')}</div>
+    </div>`;
+  }).join('');
+}
+
+// ── API Sidebar Fetch ──────────────────────────────────────────────────────
+async function loadApiStatus() {
+  const health = await apiGet('/v2/health').catch(() => null);
+  setBannerState('rest', health !== null ? 'ok' : 'err', health !== null ? 'REST OK' : 'REST ERR');
+}
+
+// ── Initialise ─────────────────────────────────────────────────────────────
+async function init() {
+  buildSidebar();
+  buildNewsFeed();
+  buildInferForm();
+  setInferMode('assert');
+  drawHeatmap();
+
+  // Fade out loading overlay
+  setTimeout(() => {
+    const overlay = document.getElementById('loading-overlay');
+    if (overlay) overlay.classList.add('hidden');
+    setTimeout(() => { if (overlay) overlay.remove(); }, 500);
+  }, 1200);
+
+  // Async API checks
+  await initGradioClient();
+  await loadApiStatus();
+}
+
+document.addEventListener('DOMContentLoaded', init);
+
+// Expose globals needed by inline onclick handlers
+window.setTab        = setTab;
+window.setSeg        = setSeg;
+window.filterTickers = filterTickers;
+window.selectTicker  = selectTicker;
+window.showPopup     = showPopup;
+window.hidePopup     = hidePopup;
+window.setInferMode  = setInferMode;
+window.runInference  = runInference;
+window.hmHover       = hmHover;
+window.hmClick       = hmClick;

frontend/index.html

@@ -0,0 +1,269 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <meta name="description" content="CUTS+ Causal Terminal — Iroha Financial Intelligence. Real-time causal probability matrix, HHKD decomposition, and DoFlow inference over NIFTY50.">
+  <title>CUTS+ Causal Terminal · Iroha</title>
+
+  <!-- Stylesheet (separate file served via StaticFiles) -->
+  <link rel="stylesheet" href="/static/style.css">
+</head>
+<body>
+
+<!-- ── Loading Overlay ─────────────────────────────────────────────────── -->
+<div id="loading-overlay">
+  <div class="loading-brand">CUTS+ CAUSAL</div>
+  <div class="loading-bar-wrap">
+    <div class="loading-bar-fill"></div>
+  </div>
+  <div class="loading-status" id="loading-status">INITIALISING ENGINE…</div>
+</div>
+
+<!-- ── Top Bar ─────────────────────────────────────────────────────────── -->
+<div class="topbar">
+  <div style="display:flex;align-items:center;gap:16px;">
+    <div class="brand">CUTS+ CAUSAL</div>
+    <div class="tabs">
+      <button class="tab active" data-tab="matrix"  onclick="setTab('matrix')">MATRIX</button>
+      <button class="tab"        data-tab="network" onclick="setTab('network')">NETWORK</button>
+      <button class="tab"        data-tab="hhkd"    onclick="setTab('hhkd')">HHKD</button>
+      <button class="tab"        data-tab="sector"  onclick="setTab('sector')">SECTOR</button>
+      <button class="tab"        data-tab="single"  onclick="setTab('single')">SINGLE TICKER</button>
+    </div>
+  </div>
+  <div class="topbar-right">
+    <span><span class="live-dot"></span> LIVE</span>
+    <span id="clock" style="color:var(--amber);">--:--:--</span>
+    <span>NIFTY50 · EPOCH 30/30</span>
+  </div>
+</div>
+
+<!-- ── API Status Banner ───────────────────────────────────────────────── -->
+<div id="api-banner">
+  <div class="api-chip" id="api-rest">
+    <div class="api-dot"></div><span>REST API</span>
+  </div>
+  <div class="api-chip" id="api-gradio">
+    <div class="api-dot"></div><span>GRADIO CLIENT</span>
+  </div>
+  <div class="api-chip" id="api-pipeline">
+    <div class="api-dot"></div><span>PIPELINE</span>
+  </div>
+  <div class="api-chip" id="api-infer">
+    <div class="api-dot"></div><span>INFERENCE</span>
+  </div>
+</div>
+
+<!-- ── Body ────────────────────────────────────────────────────────────── -->
+<div class="body">
+
+  <!-- LEFT SIDEBAR -->
+  <div class="sidebar">
+    <div class="sb-header">
+      <span>TICKERS — φ RANK</span>
+      <span id="sb-count" style="color:var(--muted);">36</span>
+    </div>
+    <div class="sb-seg">
+      <div class="seg-btn">
+        <button class="active" onclick="setSeg(this,'all')">ALL</button>
+        <button onclick="setSeg(this,'up')">UPSTREAM</button>
+        <button onclick="setSeg(this,'dn')">SINK</button>
+      </div>
+    </div>
+    <div class="sb-search">
+      <input type="text" placeholder="Search ticker…" id="sb-search-input"
+        oninput="filterTickers(this.value)" autocomplete="off">
+    </div>
+    <div class="ticker-list" id="ticker-list"></div>
+  </div>
+
+  <!-- CENTER PANEL -->
+  <div class="center">
+
+    <!-- MATRIX VIEW -->
+    <div class="view active" id="view-matrix">
+      <div class="view-header">
+        <span class="vh-title">CAUSAL PROBABILITY MATRIX</span>
+        <span class="vh-meta">G·σ(G_T) · GUMBEL-SOFTMAX · HOVER → DETAILS · CLICK → PIN</span>
+      </div>
+      <div class="legend">
+        <div class="leg-item"><div class="leg-dot" style="background:#e03434;"></div>Strong cause &gt;0.7</div>
+        <div class="leg-item"><div class="leg-dot" style="background:#f0a500;"></div>Moderate 0.4–0.7</div>
+        <div class="leg-item"><div class="leg-dot" style="background:#003a20;"></div>Weak &lt;0.4</div>
+        <div class="leg-item" style="margin-left:auto;color:var(--muted);">Inference ripples across this map in real-time</div>
+      </div>
+      <div class="view-body" style="padding:0;" id="matrix-body">
+        <svg id="heatmap-svg"></svg>
+      </div>
+    </div>
+
+    <!-- NETWORK VIEW -->
+    <div class="view" id="view-network">
+      <div class="view-header">
+        <span class="vh-title">CAUSAL NETWORK GRAPH</span>
+        <span class="vh-meta">THRESHOLDED DAG · θ=0.5 · DRAG TO EXPLORE · NODE SIZE ∝ OUT-DEGREE</span>
+      </div>
+      <div class="legend">
+        <div class="leg-item"><div class="leg-dot" style="background:var(--amber);"></div>High φ (upstream)</div>
+        <div class="leg-item"><div class="leg-dot" style="background:var(--cyan);"></div>Mid φ</div>
+        <div class="leg-item"><div class="leg-dot" style="background:var(--muted);"></div>Low φ (sink)</div>
+        <div class="leg-item"><div class="leg-line" style="background:var(--red);"></div>Strong edge</div>
+      </div>
+      <div class="view-body" style="padding:0;position:relative;">
+        <svg id="net-svg"></svg>
+      </div>
+    </div>
+
+    <!-- HHKD VIEW -->
+    <div class="view" id="view-hhkd">
+      <div class="view-header">
+        <span class="vh-title">HELMHOLTZ-HODGE-KODAIRA DECOMPOSITION</span>
+        <span class="vh-meta">J_b → J_grad + J_res · ‖RESIDUAL‖ &lt; 10⁻⁶</span>
+      </div>
+      <div class="view-body" style="padding:10px;">
+        <div style="display:grid;grid-template-columns:1fr 1fr;gap:10px;margin-bottom:10px;">
+          <div>
+            <div style="color:var(--amber);font-size:9px;letter-spacing:2px;margin-bottom:8px;text-transform:uppercase;">
+              Scalar Potential φ — Upstream Ranking
+            </div>
+            <div id="phi-chart"></div>
+          </div>
+          <div>
+            <div style="color:var(--cyan);font-size:9px;letter-spacing:2px;margin-bottom:8px;text-transform:uppercase;">
+              Decomposition Diagnostics
+            </div>
+            <div id="hhkd-diag"></div>
+            <div style="margin-top:10px;">
+              <div style="color:var(--muted);font-size:9px;letter-spacing:1px;margin-bottom:6px;">GRADIENT vs CYCLIC SPLIT</div>
+              <div style="height:18px;background:var(--bg4);border-radius:3px;overflow:hidden;display:flex;">
+                <div style="height:18px;width:93.8%;background:var(--amber);display:flex;align-items:center;justify-content:center;font-size:8px;color:#000;font-weight:600;">GRADIENT 93.8%</div>
+                <div style="flex:1;height:18px;background:var(--cyan-lo);display:flex;align-items:center;justify-content:center;font-size:8px;color:var(--cyan);">6.2%</div>
+              </div>
+            </div>
+          </div>
+        </div>
+        <div style="color:var(--amber);font-size:9px;letter-spacing:2px;margin-bottom:8px;text-transform:uppercase;">
+          Gradient Flow J_grad — Sector Heatmap
+        </div>
+        <svg id="jgrad-svg"></svg>
+      </div>
+    </div>
+
+    <!-- SECTOR VIEW -->
+    <div class="view" id="view-sector">
+      <div class="view-header">
+        <span class="vh-title">SECTOR MACRO GRAPH — REASON TRIPLET ⟨G, A, E⟩</span>
+        <span class="vh-meta">BIDIRECTIONAL MESSAGE PASSING · CrossLevelMPNN</span>
+      </div>
+      <div class="view-body" style="padding:0;overflow-y:auto;">
+        <div style="padding:10px;">
+          <div style="color:var(--amber);font-size:9px;letter-spacing:2px;margin-bottom:8px;text-transform:uppercase;">Macro Sector Adjacency</div>
+          <svg id="macro-svg"></svg>
+        </div>
+        <div class="sector-grid" id="sector-grid"></div>
+      </div>
+    </div>
+
+    <!-- SINGLE TICKER VIEW -->
+    <div class="view" id="view-single">
+      <div class="view-header">
+        <div>
+          <span class="vh-title">SINGLE TICKER — FUNDAMENTAL CAUSAL</span>
+          <span id="single-name" style="color:var(--amber);font-size:13px;font-weight:600;margin-left:12px;">RELIANCE</span>
+        </div>
+        <span class="vh-meta">DuPont PRIOR · 23 NODES · SCM RIDGE · CAMEF GPT4MTS</span>
+      </div>
+      <div class="view-body" style="padding:10px;overflow-y:auto;">
+        <div style="display:grid;grid-template-columns:1fr 1fr;gap:10px;">
+          <div>
+            <div style="color:var(--amber);font-size:9px;letter-spacing:2px;margin-bottom:6px;text-transform:uppercase;">DuPont Prior Adjacency (23×23)</div>
+            <svg id="dupont-svg"></svg>
+            <div style="color:var(--cyan);font-size:9px;letter-spacing:2px;margin:10px 0 6px;text-transform:uppercase;">Discovered Causal Edges</div>
+            <div id="disc-edges"></div>
+          </div>
+          <div>
+            <div style="color:var(--amber);font-size:9px;letter-spacing:2px;margin-bottom:6px;text-transform:uppercase;">CAMEF Stress Forecast</div>
+            <svg id="camef-svg"></svg>
+            <div style="color:var(--cyan);font-size:9px;letter-spacing:2px;margin:10px 0 6px;text-transform:uppercase;">FCM Lag-Graph G₁–G₄</div>
+            <div id="fcm-bars"></div>
+          </div>
+        </div>
+      </div>
+    </div>
+
+  </div><!-- /center -->
+
+  <!-- RIGHT PANEL -->
+  <div class="rpanel">
+
+    <!-- INFERENCE ENGINE -->
+    <div style="flex-shrink:0;border-bottom:1px solid var(--border);">
+      <div class="rp-head">CAUSAL INFERENCE ENGINE</div>
+      <div class="infer-panel">
+        <div class="infer-mode">
+          <button class="m-assert active" onclick="setInferMode('assert')">ASSERT</button>
+          <button class="m-intervene"     onclick="setInferMode('intervene')">INTERVENE</button>
+          <button class="m-counter"       onclick="setInferMode('counter')">COUNTER·F</button>
+        </div>
+        <div class="infer-form" id="infer-form">
+          <!-- populated by app.js -->
+        </div>
+      </div>
+    </div>
+
+    <!-- RESULTS -->
+    <div style="flex:1;overflow-y:auto;">
+      <div class="rp-head">
+        RESULTS &amp; RIPPLE TRACE
+        <span id="result-count" style="color:var(--muted);font-size:9px;font-weight:400;"></span>
+      </div>
+      <div style="padding:8px;" id="results-area">
+        <div style="color:var(--muted);font-size:10px;text-align:center;padding:20px 0;">
+          Run an inference query to see results and ripple effects across all views.
+        </div>
+      </div>
+
+      <!-- NEWS FEED -->
+      <div class="rp-head" style="margin-top:0;">LLM DENOISED NEWS</div>
+      <div id="news-feed"></div>
+    </div>
+
+  </div><!-- /rpanel -->
+
+</div><!-- /body -->
+
+<!-- STATUS STRIP -->
+<div class="status-strip">
+  <div class="ss-chip"><span class="ss-k">TICKERS</span><span class="ss-v am">36</span></div>
+  <div class="ss-chip"><span class="ss-k">EDGES</span><span class="ss-v am">127</span></div>
+  <div class="ss-chip"><span class="ss-k">DENSITY</span><span class="ss-v">5.2%</span></div>
+  <div class="ss-chip"><span class="ss-k">λ_s</span><span class="ss-v">0.10</span></div>
+  <div class="ss-chip"><span class="ss-k">λ_d</span><span class="ss-v">1.00</span></div>
+  <div class="ss-chip"><span class="ss-k">LOSS</span><span class="ss-v up" id="ss-loss">0.0412</span></div>
+  <div class="ss-chip"><span class="ss-k">PRIOR CONFORM</span><span class="ss-v up">91.3%</span></div>
+  <div class="ss-chip"><span class="ss-k">‖J_res‖</span><span class="ss-v am">3.2e-7</span></div>
+  <div class="ss-chip"><span class="ss-k">EPOCH</span><span class="ss-v">30/30 ✓</span></div>
+</div>
+
+<!-- NODE POPUP -->
+<div class="node-popup" id="node-popup">
+  <div class="np-head">
+    <span id="np-name">RELIANCE</span>
+    <span id="np-sector" style="font-size:9px;color:var(--muted);font-weight:400;">Energy</span>
+  </div>
+  <div class="np-row"><span class="np-k">φ Potential</span><span class="np-v am" id="np-phi">+2.41</span></div>
+  <div class="np-row"><span class="np-k">Rank</span><span class="np-v up" id="np-rank">#1 Upstream</span></div>
+  <div class="np-row"><span class="np-k">Out-degree</span><span class="np-v" id="np-out">8</span></div>
+  <div class="np-row"><span class="np-k">In-degree</span><span class="np-v" id="np-in">2</span></div>
+  <div class="np-row"><span class="np-k">Strongest cause</span><span class="np-v am" id="np-cause">ONGC 0.847</span></div>
+  <div class="np-row"><span class="np-k">News score</span><span class="np-v up" id="np-news">0.87</span></div>
+</div>
+
+<!-- ERROR TOAST -->
+<div id="error-toast"></div>
+
+<!-- App JS (separate file served via StaticFiles) -->
+<script type="module" src="/static/app.js"></script>
+</body>
+</html>

frontend/style.css

@@ -0,0 +1,711 @@
+/* ── Google Fonts ──────────────────────────────────────────────────────────── */
+@import url('https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@300;400;500;600&display=swap');
+
+/* ── Design Tokens ─────────────────────────────────────────────────────────── */
+:root {
+  --bg0:       #090909;
+  --bg1:       #0e0e0e;
+  --bg2:       #141414;
+  --bg3:       #1c1c1c;
+  --bg4:       #242424;
+  --border:    #252525;
+  --border-hi: #383838;
+
+  --amber:     #f0a500;
+  --amber-lo:  rgba(240,165,0,0.12);
+  --amber-dim: #7a5200;
+  --red:       #e03434;
+  --red-lo:    rgba(224,52,52,0.12);
+  --green:     #00c87a;
+  --green-lo:  rgba(0,200,122,0.12);
+  --cyan:      #00b8d4;
+  --cyan-lo:   rgba(0,184,212,0.12);
+  --purple:    #a78bfa;
+  --purple-lo: rgba(167,139,250,0.12);
+  --white:     #e8e4d9;
+  --muted:     #5a5a54;
+  --muted2:    #38382e;
+  --font:      'IBM Plex Mono','Courier New',monospace;
+  --r:         3px;
+}
+
+/* ── Reset ─────────────────────────────────────────────────────────────────── */
+*, *::before, *::after {
+  margin: 0;
+  padding: 0;
+  box-sizing: border-box;
+}
+
+html, body {
+  height: 100%;
+  overflow: hidden;
+}
+
+body {
+  background: var(--bg0);
+  color: var(--white);
+  font-family: var(--font);
+  font-size: 11px;
+  line-height: 1.5;
+  display: flex;
+  flex-direction: column;
+}
+
+/* ── Scrollbar ─────────────────────────────────────────────────────────────── */
+::-webkit-scrollbar { width: 3px; height: 3px; }
+::-webkit-scrollbar-track { background: var(--bg0); }
+::-webkit-scrollbar-thumb { background: var(--muted2); }
+
+/* ── Loading Overlay ───────────────────────────────────────────────────────── */
+#loading-overlay {
+  position: fixed;
+  inset: 0;
+  background: var(--bg0);
+  z-index: 9999;
+  display: flex;
+  flex-direction: column;
+  align-items: center;
+  justify-content: center;
+  gap: 16px;
+  transition: opacity 0.4s ease;
+}
+#loading-overlay.hidden { opacity: 0; pointer-events: none; }
+
+.loading-brand {
+  color: var(--amber);
+  font-size: 16px;
+  font-weight: 600;
+  letter-spacing: 4px;
+}
+.loading-bar-wrap {
+  width: 220px;
+  height: 2px;
+  background: var(--bg3);
+  border-radius: 2px;
+  overflow: hidden;
+}
+.loading-bar-fill {
+  height: 2px;
+  background: var(--amber);
+  border-radius: 2px;
+  animation: loadbar 1.6s ease-in-out forwards;
+}
+@keyframes loadbar {
+  0%   { width: 0%; }
+  60%  { width: 80%; }
+  100% { width: 100%; }
+}
+.loading-status {
+  font-size: 9px;
+  color: var(--muted);
+  letter-spacing: 1.5px;
+}
+
+/* ── Top Bar ───────────────────────────────────────────────────────────────── */
+.topbar {
+  height: 38px;
+  background: var(--bg1);
+  border-bottom: 1px solid var(--amber);
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  padding: 0 14px;
+  flex-shrink: 0;
+  z-index: 200;
+}
+
+.brand {
+  color: var(--amber);
+  font-weight: 600;
+  font-size: 12px;
+  letter-spacing: 3px;
+}
+
+.tabs {
+  display: flex;
+  gap: 1px;
+}
+
+.tab {
+  background: none;
+  border: none;
+  color: var(--muted);
+  font-family: var(--font);
+  font-size: 10px;
+  padding: 0 14px;
+  height: 38px;
+  cursor: pointer;
+  letter-spacing: 1.5px;
+  text-transform: uppercase;
+  border-bottom: 2px solid transparent;
+  transition: color 0.15s, border-color 0.15s;
+}
+.tab:hover { color: var(--white); }
+.tab.active { color: var(--amber); border-bottom-color: var(--amber); }
+
+.topbar-right {
+  display: flex;
+  align-items: center;
+  gap: 12px;
+  font-size: 9px;
+  color: var(--muted);
+}
+
+.live-dot {
+  width: 5px;
+  height: 5px;
+  background: var(--green);
+  border-radius: 50%;
+  display: inline-block;
+  animation: blink 2s infinite;
+}
+@keyframes blink { 0%,100% { opacity: 1; } 50% { opacity: 0.3; } }
+
+/* ── API Status Banner ─────────────────────────────────────────────────────── */
+#api-banner {
+  height: 22px;
+  background: var(--bg2);
+  border-bottom: 1px solid var(--border);
+  display: flex;
+  align-items: center;
+  padding: 0 14px;
+  gap: 14px;
+  flex-shrink: 0;
+  font-size: 9px;
+  letter-spacing: 1px;
+}
+.api-chip {
+  display: flex;
+  align-items: center;
+  gap: 4px;
+  color: var(--muted);
+}
+.api-chip.ok   .api-dot { background: var(--green); }
+.api-chip.err  .api-dot { background: var(--red); }
+.api-chip.busy .api-dot { background: var(--amber); animation: blink 1s infinite; }
+.api-dot {
+  width: 5px;
+  height: 5px;
+  border-radius: 50%;
+  background: var(--muted2);
+}
+
+/* ── Body Layout ───────────────────────────────────────────────────────────── */
+.body {
+  flex: 1;
+  display: grid;
+  grid-template-columns: 200px 1fr 260px;
+  overflow: hidden;
+  min-height: 0;
+}
+
+/* ── Left Sidebar ──────────────────────────────────────────────────────────── */
+.sidebar {
+  background: var(--bg1);
+  border-right: 1px solid var(--border);
+  display: flex;
+  flex-direction: column;
+  overflow: hidden;
+}
+
+.sb-header {
+  padding: 7px 10px;
+  background: var(--bg2);
+  border-bottom: 1px solid var(--border);
+  color: var(--amber);
+  font-size: 9px;
+  letter-spacing: 2px;
+  text-transform: uppercase;
+  display: flex;
+  justify-content: space-between;
+  align-items: center;
+  flex-shrink: 0;
+}
+
+.sb-seg {
+  padding: 6px 8px;
+  border-bottom: 1px solid var(--border);
+  flex-shrink: 0;
+}
+.seg-btn { display: flex; gap: 3px; }
+.seg-btn button {
+  flex: 1;
+  background: var(--bg3);
+  border: 1px solid var(--border);
+  color: var(--muted);
+  font-family: var(--font);
+  font-size: 9px;
+  padding: 4px;
+  cursor: pointer;
+  border-radius: var(--r);
+  letter-spacing: 1px;
+  transition: all 0.15s;
+}
+.seg-btn button.active {
+  background: var(--amber);
+  color: #000;
+  border-color: var(--amber);
+  font-weight: 600;
+}
+
+.sb-search {
+  padding: 6px 8px;
+  border-bottom: 1px solid var(--border);
+  flex-shrink: 0;
+}
+.sb-search input {
+  width: 100%;
+  background: var(--bg3);
+  border: 1px solid var(--border);
+  color: var(--white);
+  font-family: var(--font);
+  font-size: 10px;
+  padding: 4px 8px;
+  outline: none;
+  border-radius: var(--r);
+  transition: border-color 0.15s;
+}
+.sb-search input:focus { border-color: var(--amber-dim); }
+
+.ticker-list {
+  overflow-y: auto;
+  flex: 1;
+}
+
+.ticker-row {
+  display: flex;
+  align-items: center;
+  padding: 5px 10px;
+  cursor: pointer;
+  border-bottom: 1px solid var(--border);
+  transition: background 0.1s;
+  gap: 6px;
+}
+.ticker-row:hover { background: var(--bg3); }
+.ticker-row.sel {
+  background: var(--amber-lo);
+  border-left: 2px solid var(--amber);
+}
+.ticker-row.rippling     { animation: rowripple    0.6s ease-out; }
+.ticker-row.rippling-up  { animation: rowripple-up 0.6s ease-out; }
+@keyframes rowripple    { 0% { background: rgba(224,52,52,.35); }  100% { background: transparent; } }
+@keyframes rowripple-up { 0% { background: rgba(0,200,122,.35); } 100% { background: transparent; } }
+
+.t-sym   { color: var(--amber); font-size: 10px; font-weight: 600; width: 62px; flex-shrink: 0; }
+.t-phi   { font-size: 9px; text-align: right; flex-shrink: 0; width: 32px; }
+.t-bar   { flex: 1; height: 3px; background: var(--bg4); border-radius: 2px; overflow: hidden; }
+.t-bar-fill { height: 3px; border-radius: 2px; transition: width 0.3s; }
+
+/* ── Center Panel ──────────────────────────────────────────────────────────── */
+.center {
+  display: flex;
+  flex-direction: column;
+  overflow: hidden;
+  background: var(--bg0);
+  position: relative;
+}
+
+.view {
+  display: none;
+  flex: 1;
+  flex-direction: column;
+  overflow: hidden;
+}
+.view.active { display: flex; }
+
+.view-header {
+  padding: 8px 14px;
+  background: var(--bg2);
+  border-bottom: 1px solid var(--border);
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  flex-shrink: 0;
+}
+.vh-title { color: var(--amber); font-size: 10px; letter-spacing: 2px; font-weight: 600; }
+.vh-meta  { color: var(--muted); font-size: 9px; }
+
+.view-body {
+  flex: 1;
+  overflow: auto;
+  padding: 12px;
+  position: relative;
+}
+
+/* ── Legend ────────────────────────────────────────────────────────────────── */
+.legend {
+  display: flex;
+  gap: 14px;
+  align-items: center;
+  padding: 6px 12px;
+  border-bottom: 1px solid var(--border);
+  flex-shrink: 0;
+  flex-wrap: wrap;
+}
+.leg-item { display: flex; align-items: center; gap: 5px; font-size: 9px; color: var(--muted); }
+.leg-dot  { width: 8px; height: 8px; border-radius: 50%; flex-shrink: 0; }
+.leg-line { width: 18px; height: 2px; flex-shrink: 0; }
+
+/* ── Heatmap ───────────────────────────────────────────────────────���───────── */
+.hm-wrap { overflow: auto; padding: 0; }
+#heatmap-svg { display: block; }
+.hm-cell { cursor: pointer; transition: opacity 0.15s; }
+.hm-cell:hover { opacity: 0.75; stroke: #fff !important; stroke-width: 1.5 !important; }
+.hm-cell.ripple-out  { animation: hmripple     1s ease-out forwards; }
+.hm-cell.ripple-in   { animation: hmripple-in  0.8s ease-out forwards; }
+.hm-cell.ripple-pulse { animation: hmpulse     1.2s ease-in-out 3; }
+@keyframes hmripple    { 0% { opacity:1; fill: rgba(224,52,52,0.9); }  100% { opacity: 1; } }
+@keyframes hmripple-in { 0% { opacity:1; fill: rgba(0,200,122,0.9); } 100% { opacity: 1; } }
+@keyframes hmpulse     { 0%,100% { opacity:1; } 50% { opacity: 0.3; } }
+
+/* ── Network ───────────────────────────────────────────────────────────────── */
+#net-svg { display: block; width: 100%; height: 100%; }
+.net-node { cursor: pointer; }
+.net-node:hover circle { stroke-width: 2; }
+.net-edge { transition: stroke-width 0.2s, stroke-opacity 0.2s; }
+
+/* ── Right Panel ───────────────────────────────────────────────────────────── */
+.rpanel {
+  background: var(--bg1);
+  border-left: 1px solid var(--border);
+  display: flex;
+  flex-direction: column;
+  overflow: hidden;
+}
+
+.rp-sec { border-bottom: 1px solid var(--border); flex-shrink: 0; }
+.rp-head {
+  padding: 6px 10px;
+  background: var(--bg2);
+  border-bottom: 1px solid var(--border);
+  font-size: 9px;
+  letter-spacing: 2px;
+  color: var(--cyan);
+  font-weight: 600;
+  text-transform: uppercase;
+  display: flex;
+  justify-content: space-between;
+  align-items: center;
+}
+.rp-row {
+  display: flex;
+  justify-content: space-between;
+  padding: 4px 10px;
+  border-bottom: 1px solid var(--border);
+  font-size: 10px;
+}
+.rp-k { color: var(--muted); }
+.rp-v { color: var(--white); }
+.rp-v.up { color: var(--green); }
+.rp-v.dn { color: var(--red); }
+.rp-v.am { color: var(--amber); }
+
+/* ── Inference Panel ───────────────────────────────────────────────────────── */
+.infer-panel { padding: 10px; }
+.infer-mode  { display: flex; gap: 4px; margin-bottom: 10px; }
+.infer-mode button {
+  flex: 1;
+  background: var(--bg3);
+  border: 1px solid var(--border);
+  color: var(--muted);
+  font-family: var(--font);
+  font-size: 9px;
+  padding: 5px;
+  cursor: pointer;
+  border-radius: var(--r);
+  letter-spacing: 1px;
+  transition: all 0.15s;
+}
+.infer-mode button.active { font-weight: 600; }
+.infer-mode button.m-assert.active    { background: var(--cyan-lo);   color: var(--cyan);   border-color: var(--cyan); }
+.infer-mode button.m-intervene.active { background: var(--amber-lo);  color: var(--amber);  border-color: var(--amber); }
+.infer-mode button.m-counter.active   { background: var(--purple-lo); color: var(--purple); border-color: var(--purple); }
+
+.infer-form {
+  background: var(--bg2);
+  border: 1px solid var(--border);
+  border-radius: var(--r);
+  padding: 10px;
+  margin-bottom: 8px;
+}
+.infer-label {
+  font-size: 9px;
+  color: var(--muted);
+  letter-spacing: 1.5px;
+  text-transform: uppercase;
+  margin-bottom: 5px;
+}
+.infer-select {
+  width: 100%;
+  background: var(--bg3);
+  border: 1px solid var(--border);
+  color: var(--white);
+  font-family: var(--font);
+  font-size: 10px;
+  padding: 5px 8px;
+  outline: none;
+  border-radius: var(--r);
+  margin-bottom: 8px;
+  cursor: pointer;
+}
+.infer-select:focus { border-color: var(--amber-dim); }
+
+.slider-wrap { margin-bottom: 10px; }
+.slider-row  { display: flex; justify-content: space-between; margin-bottom: 4px; }
+.slider-val  { color: var(--amber); font-weight: 600; font-size: 10px; }
+input[type=range] { width: 100%; accent-color: var(--amber); cursor: pointer; height: 3px; }
+
+.run-btn {
+  width: 100%;
+  padding: 8px;
+  border: none;
+  font-family: var(--font);
+  font-size: 10px;
+  font-weight: 600;
+  letter-spacing: 2px;
+  cursor: pointer;
+  border-radius: var(--r);
+  transition: opacity 0.2s, transform 0.2s;
+}
+.run-btn.assert    { background: var(--cyan);   color: #000; }
+.run-btn.intervene { background: var(--amber);  color: #000; }
+.run-btn.counter   { background: var(--purple); color: #000; }
+.run-btn:hover  { opacity: 0.85; transform: translateY(-1px); }
+.run-btn:active { transform: translateY(0); }
+.run-btn:disabled { opacity: 0.4; cursor: not-allowed; }
+
+/* ── Result Cards ──────────────────────────────────────────────────────────── */
+.result-card {
+  background: var(--bg2);
+  border: 1px solid var(--border);
+  border-radius: var(--r);
+  padding: 10px;
+  margin-bottom: 8px;
+  position: relative;
+  overflow: hidden;
+}
+.result-card::before {
+  content: '';
+  position: absolute;
+  top: 0; left: 0;
+  width: 3px; height: 100%;
+}
+.result-card.assert::before   { background: var(--cyan); }
+.result-card.intervene::before { background: var(--amber); }
+.result-card.counter::before  { background: var(--purple); }
+.rc-head { font-size: 9px; letter-spacing: 1.5px; margin-bottom: 6px; font-weight: 600; }
+.rc-head.assert    { color: var(--cyan); }
+.rc-head.intervene { color: var(--amber); }
+.rc-head.counter   { color: var(--purple); }
+.rc-row { display: flex; justify-content: space-between; font-size: 10px; padding: 2px 0; }
+.rc-k   { color: var(--muted); }
+.rc-v   { color: var(--white); }
+.rc-v.up { color: var(--green); }
+.rc-v.dn { color: var(--red); }
+.rc-v.am { color: var(--amber); }
+
+.ate-track {
+  height: 4px;
+  background: var(--bg4);
+  border-radius: 2px;
+  margin-top: 6px;
+  overflow: hidden;
+}
+.ate-fill {
+  height: 4px;
+  border-radius: 2px;
+  transition: width 0.8s ease;
+}
+
+.ripple-effects {
+  margin-top: 8px;
+  padding-top: 8px;
+  border-top: 1px solid var(--border);
+}
+.ripple-title { font-size: 9px; color: var(--muted); letter-spacing: 1px; margin-bottom: 5px; }
+.ripple-chip {
+  display: inline-flex;
+  align-items: center;
+  gap: 4px;
+  padding: 2px 7px;
+  border-radius: 2px;
+  font-size: 9px;
+  margin: 2px;
+  border: 1px solid;
+}
+.ripple-chip.up { background: var(--green-lo); border-color: var(--green); color: var(--green); }
+.ripple-chip.dn { background: var(--red-lo);   border-color: var(--red);   color: var(--red); }
+
+/* ── News Feed ─────────────────────────────────────────────────────────────── */
+.news-item {
+  padding: 7px 10px;
+  border-bottom: 1px solid var(--border);
+  cursor: pointer;
+  transition: background 0.1s;
+}
+.news-item:hover { background: var(--bg3); }
+.news-top { display: flex; align-items: center; gap: 6px; margin-bottom: 3px; }
+.news-score { font-size: 8px; padding: 1px 5px; border-radius: 2px; font-weight: 600; }
+.news-score.hi { background: var(--green-lo); color: var(--green); }
+.news-score.md { background: var(--amber-lo); color: var(--amber); }
+.news-score.lo { background: var(--red-lo);   color: var(--red); }
+.news-sym  { color: var(--amber); font-size: 9px; font-weight: 600; }
+.news-text { font-size: 9px; color: var(--muted); line-height: 1.5; margin-bottom: 4px; }
+.news-tags { display: flex; gap: 3px; flex-wrap: wrap; }
+.news-tag  { font-size: 8px; padding: 1px 5px; border: 1px solid var(--border); color: var(--muted); border-radius: 2px; }
+
+/* ── HHKD View ─────────────────────────────────────────────────────────────── */
+.phi-row {
+  display: flex;
+  align-items: center;
+  gap: 8px;
+  padding: 4px 10px;
+  border-bottom: 1px solid var(--border);
+  cursor: pointer;
+  transition: background 0.1s;
+}
+.phi-row:hover { background: var(--bg3); }
+.phi-sym { width: 66px; font-size: 10px; color: var(--amber); font-weight: 600; flex-shrink: 0; }
+.phi-bar-wrap { flex: 1; height: 8px; background: var(--bg4); border-radius: 4px; overflow: hidden; }
+.phi-bar-fill { height: 8px; border-radius: 4px; transition: width 0.4s; }
+.phi-val { width: 36px; text-align: right; font-size: 10px; flex-shrink: 0; }
+
+/* ── Sector View ───────────────────────────────────────────────────────────── */
+.sector-grid { padding: 10px; display: flex; flex-direction: column; gap: 8px; }
+.sec-card { background: var(--bg2); border: 1px solid var(--border); border-radius: var(--r); overflow: hidden; }
+.sec-card-head {
+  padding: 7px 12px;
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  cursor: pointer;
+  transition: background 0.1s;
+}
+.sec-card-head:hover { background: var(--bg3); }
+.sec-name { font-size: 11px; font-weight: 600; letter-spacing: 1px; color: var(--white); }
+.sec-phi  { font-size: 10px; }
+.sec-members { display: flex; flex-wrap: wrap; gap: 4px; padding: 8px; }
+.sec-chip {
+  padding: 3px 9px;
+  border: 1px solid var(--border);
+  font-size: 9px;
+  border-radius: 2px;
+  cursor: pointer;
+  transition: all 0.15s;
+}
+.sec-chip:hover { border-color: var(--amber); color: var(--amber); }
+.sec-chip.rippling    { animation: chipripple    0.7s ease-out; }
+.sec-chip.rippling-up { animation: chipripple-up 0.7s ease-out; }
+@keyframes chipripple    { 0% { background: rgba(224,52,52,.4);  border-color: var(--red); }  100% { background: transparent; } }
+@keyframes chipripple-up { 0% { background: rgba(0,200,122,.4); border-color: var(--green); } 100% { background: transparent; } }
+
+/* ── Node Popup ────────────────────────────────────────────────────────────── */
+.node-popup {
+  position: fixed;
+  z-index: 500;
+  background: var(--bg2);
+  border: 1px solid var(--amber);
+  border-radius: var(--r);
+  padding: 12px;
+  min-width: 200px;
+  max-width: 260px;
+  pointer-events: none;
+  display: none;
+  box-shadow: 0 8px 32px rgba(0,0,0,.6);
+}
+.np-head {
+  color: var(--amber);
+  font-size: 12px;
+  font-weight: 600;
+  margin-bottom: 8px;
+  display: flex;
+  justify-content: space-between;
+  align-items: center;
+}
+.np-row {
+  display: flex;
+  justify-content: space-between;
+  padding: 3px 0;
+  border-bottom: 1px solid var(--border);
+  font-size: 10px;
+}
+.np-row:last-child { border: none; }
+.np-k { color: var(--muted); }
+.np-v { color: var(--white); }
+.np-v.up { color: var(--green); }
+.np-v.dn { color: var(--red); }
+.np-v.am { color: var(--amber); }
+
+/* ── Status Strip ──────────────────────────────────────────────────────────── */
+.status-strip {
+  height: 22px;
+  background: var(--bg2);
+  border-top: 1px solid var(--border);
+  display: flex;
+  align-items: center;
+  padding: 0 10px;
+  gap: 16px;
+  flex-shrink: 0;
+  overflow: hidden;
+}
+.ss-chip { font-size: 9px; display: flex; gap: 5px; white-space: nowrap; }
+.ss-k    { color: var(--muted); }
+.ss-v    { color: var(--white); }
+.ss-v.am { color: var(--amber); }
+.ss-v.up { color: var(--green); }
+
+/* ── Accordion ─────────────────────────────────────────────────────────────── */
+.accordion { border: 1px solid var(--border); border-radius: var(--r); margin-bottom: 6px; overflow: hidden; }
+.acc-head {
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  padding: 7px 10px;
+  cursor: pointer;
+  background: var(--bg2);
+  user-select: none;
+}
+.acc-head:hover { background: var(--bg3); }
+.acc-title { font-size: 10px; color: var(--white); font-weight: 500; letter-spacing: 0.5px; }
+.acc-badge { font-size: 8px; padding: 1px 6px; border-radius: 2px; font-weight: 600; letter-spacing: 1px; }
+.acc-badge.up { background: var(--green-lo); color: var(--green); }
+.acc-badge.dn { background: var(--red-lo);   color: var(--red); }
+.acc-badge.am { background: var(--amber-lo); color: var(--amber); }
+.acc-badge.cy { background: var(--cyan-lo);  color: var(--cyan); }
+.acc-chevron { color: var(--muted); font-size: 10px; transition: transform 0.2s; }
+.acc-chevron.open { transform: rotate(180deg); }
+.acc-body { display: none; border-top: 1px solid var(--border); }
+.acc-body.open { display: block; }
+.acc-row { display: flex; justify-content: space-between; padding: 4px 10px; border-bottom: 1px solid var(--border); font-size: 10px; }
+.acc-k { color: var(--muted); }
+.acc-v { color: var(--white); }
+.acc-v.up { color: var(--green); }
+.acc-v.dn { color: var(--red); }
+.acc-v.am { color: var(--amber); }
+
+/* ── Ripple Ring ───────────────────────────────────────────────────────────── */
+@keyframes pulse-ring { 0% { transform: scale(.8); opacity: 1; } 100% { transform: scale(2.5); opacity: 0; } }
+.ripple-ring {
+  position: absolute;
+  border-radius: 50%;
+  pointer-events: none;
+  animation: pulse-ring 0.8s ease-out forwards;
+}
+
+/* ── Error toast ───────────────────────────────────────────────────────────── */
+#error-toast {
+  position: fixed;
+  bottom: 28px;
+  left: 50%;
+  transform: translateX(-50%) translateY(60px);
+  background: var(--red-lo);
+  border: 1px solid var(--red);
+  color: var(--red);
+  font-size: 10px;
+  padding: 8px 16px;
+  border-radius: var(--r);
+  z-index: 9000;
+  transition: transform 0.3s ease;
+  letter-spacing: 0.5px;
+}
+#error-toast.show { transform: translateX(-50%) translateY(0); }

server.py

@@ -0,0 +1,1036 @@
+"""
+dashboard_server.py
+====================
+CUTS+ Causal Terminal — gr.Server entry point.
+
+Architecture
+------------
+gr.Server (extends FastAPI)
+├── GET  /           → serves frontend/index.html
+├── GET  /static/*   → serves frontend/{style.css, app.js}  (StaticFiles)
+│
+├── @server.api  run_causal_components  → CUTS+ multi-ticker discovery
+├── @server.api  run_singular_causal    → single-ticker 10-step pipeline
+├── @server.api  run_inference          → DoFlow / SCM causal query
+├── @server.api  run_hierarchy          → hierarchical sector causal
+│
+├── GET  /v2/health                     → health-check
+├── GET  /v2/causal/singular-causal/graph/{ticker}    → cached adj graph
+├── GET  /v2/causal/singular-causal/results/{ticker}  → cached inference JSON
+├── POST /v2/causal/doflow-inference                  → DoFlow query
+│
+└── All existing /v2/* routers from main.py are included here too
+    (so this server is a superset of main.py).
+
+Usage
+-----
+    python dashboard_server.py
+
+Or with uvicorn:
+    uvicorn dashboard_server:server --host 0.0.0.0 --port 7860 --reload
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+import json
+import logging
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+BASE_DIR = Path(__file__).parent.resolve()
+if str(BASE_DIR) not in sys.path:
+    sys.path.insert(0, str(BASE_DIR))
+
+# Also add the backend directory to sys.path so we can import 'app', 'causal', 'singular_ticker_causal', etc.
+BACKEND_DIR = (BASE_DIR.parent / "noisy_boy_backend").resolve()
+if BACKEND_DIR.exists() and str(BACKEND_DIR) not in sys.path:
+    sys.path.insert(0, str(BACKEND_DIR))
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Logging
+# ─────────────────────────────────────────────────────────────────────────────
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    handlers=[logging.StreamHandler()],
+)
+logger = logging.getLogger("dashboard-server")
+
+# ─────────────────────────────────────────────────────────────────────────────
+# gr.Server
+# ─────────────────────────────────────────────────────────────────────────────
+import gradio as gr
+from gradio import Server
+
+from fastapi import Request
+from fastapi.responses import HTMLResponse, FileResponse, JSONResponse
+from fastapi.staticfiles import StaticFiles
+from fastapi.middleware.cors import CORSMiddleware
+
+FRONTEND_DIR = BASE_DIR / "frontend"
+INDEX_HTML   = FRONTEND_DIR / "index.html"
+
+server = Server(
+    title="CUTS+ Causal Terminal",
+    description=(
+        "Iroha Financial Intelligence — real-time causal probability matrix, "
+        "HHKD decomposition, DoFlow inference and sector hierarchy over NIFTY50."
+    ),
+    version="2.0.0",
+)
+
+# ── CORS (same as main.py) ────────────────────────────────────────────────
+server.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# ── Static files — mount frontend/ at /static ─────────────────────────────
+server.mount(
+    "/static",
+    StaticFiles(directory=str(FRONTEND_DIR)),
+    name="static",
+)
+
+# ─────────────────────────────────────────────────────────────────────────────
+# HTML route — serves the custom frontend
+# ─────────────────────────────────────────────────────────────────────────────
+
+@server.get("/", response_class=HTMLResponse, include_in_schema=False)
+async def serve_index():
+    """Serve the CUTS+ Causal Terminal SPA."""
+    if not INDEX_HTML.exists():
+        return HTMLResponse("<h1>Frontend not found. Run from backend/</h1>", status_code=500)
+    return HTMLResponse(INDEX_HTML.read_text(encoding="utf-8"))
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Health check
+# ─────────────────────────────────────────────────────────────────────────────
+
+@server.get("/v2/health", tags=["utility"])
+async def health():
+    """Lightweight health-check used by the frontend API banner."""
+    return {"status": "ok", "version": "2.0.0"}
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Existing v2 routers (same as main.py)
+# ─────────────────────────────────────────────────────────────────────────────
+
+try:
+    from app.api.scraper import router as scraper_router
+    server.include_router(scraper_router, prefix="/v2")
+    logger.info("✓ scraper router mounted")
+except Exception as e:
+    logger.warning(f"scraper router skipped: {e}")
+
+try:
+    from app.api.causal import router as causal_router
+    server.include_router(causal_router, prefix="/v2")
+    logger.info("✓ causal router mounted")
+except Exception as e:
+    logger.warning(f"causal router skipped: {e}")
+
+try:
+    from app.api.causal_pipeline import router as causal_pipeline_router
+    server.include_router(causal_pipeline_router, prefix="/v2")
+    logger.info("✓ causal_pipeline router mounted")
+except Exception as e:
+    logger.warning(f"causal_pipeline router skipped: {e}")
+
+try:
+    from app.api.screening import router as screening_router
+    server.include_router(screening_router, prefix="/v2")
+    logger.info("✓ screening router mounted")
+except Exception as e:
+    logger.warning(f"screening router skipped: {e}")
+
+try:
+    from app.api.ontology import router as ontology_router
+    server.include_router(ontology_router, prefix="/v2")
+    logger.info("✓ ontology router mounted")
+except Exception as e:
+    logger.warning(f"ontology router skipped: {e}")
+
+try:
+    # Temporarily remove frontend dir from sys.path to avoid shadowing 'server' package with our 'server.py' script
+    removed_empty = False
+    if "" in sys.path:
+        sys.path.remove("")
+        removed_empty = True
+    if str(BASE_DIR) in sys.path:
+        sys.path.remove(str(BASE_DIR))
+    
+    from server.model_router import router as server_model_router
+    server.include_router(server_model_router, prefix="/v2")
+    logger.info("✓ model_router mounted")
+    
+    # Restore sys.path
+    sys.path.insert(0, str(BASE_DIR))
+    if removed_empty:
+        sys.path.insert(0, "")
+except Exception as e:
+    logger.warning(f"model_router skipped: {e}")
+    # Ensure sys.path is restored even on failure
+    if str(BASE_DIR) not in sys.path:
+        sys.path.insert(0, str(BASE_DIR))
+    if 'removed_empty' in locals() and removed_empty and "" not in sys.path:
+        sys.path.insert(0, "")
+
+# try:
+#     from backtest.router import router as backtest_router
+#     server.include_router(backtest_router, prefix="/v2/backtest")
+#     logger.info("✓ backtest router mounted")
+# except Exception as e:
+#     logger.warning(f"backtest router skipped: {e}")
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# gr.Server API endpoints (Gradio-backed — queue + SSE streaming)
+# These are reachable via the Gradio JS Client as well as plain fetch().
+# ─────────────────────────────────────────────────────────────────────────────
+
+# ── Helpers ───────────────────────────────────────────────────────────────
+
+# URL of the noisy_boy_backend — used to fetch the validated causal matrix.
+# By default, point to ourselves since we now successfully mount the backend routers.
+# Override via BACKEND_API_URL env var if running a separate backend on 8000.
+_BACKEND_BASE_URL = os.environ.get("BACKEND_API_URL", "http://localhost:7860")
+
+# Kept for backward-compat with run_singular_causal (which still imports backend modules
+# via sys.path when both repos are co-located).  Not used in run_inference anymore.
+_SINGULAR_DEBUG_DIR = str(BASE_DIR / "singular_ticker_causal" / "debug_data")
+
+
+def _fetch_causal_matrix(
+    ticker: str,
+    treatment: Optional[str] = None,
+    outcome: Optional[str] = None,
+    include_pywhyllm: bool = False,
+    threshold: float = 0.5,
+) -> Optional[dict]:
+    """
+    Fetch the fully validated causal matrix from the backend API.
+
+    Calls GET {BACKEND_BASE_URL}/v2/api/singular-causal/causal-matrix/{ticker}
+    and returns the parsed JSON payload, or None on failure.
+
+    The payload contains:
+      nodes        — ordered list of node names
+      adj_matrix   — raw float adjacency matrix
+      dag_adj      — thresholded 0/1 DAG
+      equations    — per-node structural equations (coefficients, intercepts, residual_std)
+      data_level   — (T, N) time-series observations used to fit the SCM
+      topological_order — nodes in topological traversal order
+      pywhyllm_report   — (optional) assumption analysis for treatment→outcome
+    """
+    import urllib.request
+    import urllib.error
+    import urllib.parse
+
+    params: dict = {"threshold": threshold}
+    if treatment:
+        params["treatment"] = treatment
+    if outcome:
+        params["outcome"] = outcome
+    if include_pywhyllm:
+        params["include_pywhyllm"] = "true"
+
+    query_string = urllib.parse.urlencode(params)
+    url = f"{_BACKEND_BASE_URL}/v2/api/singular-causal/causal-matrix/{ticker.upper()}?{query_string}"
+
+    try:
+        with urllib.request.urlopen(url, timeout=30) as resp:
+            raw = resp.read()
+        data = json.loads(raw)
+        if data.get("status") not in ("success", None):
+            logger.warning("_fetch_causal_matrix: backend returned status=%s for URL %s. Payload: %s", data.get("status"), url, data)
+            return None
+        return data
+    except Exception as exc:
+        logger.warning("_fetch_causal_matrix failed for %s: %s", ticker, exc)
+        return None
+
+
+def _safe_json(obj: Any) -> Any:
+    """Recursively make numpy types JSON-serialisable."""
+    try:
+        import numpy as np
+        if isinstance(obj, np.ndarray):
+            return obj.tolist()
+        if isinstance(obj, np.integer):
+            return int(obj)
+        if isinstance(obj, np.floating):
+            return float(obj)
+    except ImportError:
+        pass
+    if isinstance(obj, dict):
+        return {k: _safe_json(v) for k, v in obj.items()}
+    if isinstance(obj, (list, tuple)):
+        return [_safe_json(v) for v in obj]
+    return obj
+
+
+def _adj_to_graph(adj_matrix, symbols: List[str], threshold: float = 0.5):
+    """Convert adjacency matrix → {nodes, links} for the frontend."""
+    try:
+        import numpy as np
+        arr = np.array(adj_matrix)
+    except Exception:
+        arr = [[float(v) for v in row] for row in adj_matrix]
+
+    nodes = [{"id": s, "label": s} for s in symbols]
+    links = []
+    n = len(symbols)
+    for i in range(n):
+        for j in range(n):
+            try:
+                v = float(arr[i][j])
+            except Exception:
+                continue
+            if i != j and v >= threshold:
+                links.append({"source": symbols[i], "target": symbols[j], "score": round(v, 4)})
+    return nodes, links
+
+
+# ── API 1: CUTS+ multi-ticker causal discovery ────────────────────────────
+
+@server.api(name="run_causal_components", description="Run CUTS+ on NIFTY50 multi-ticker technical features")
+def run_causal_components(
+    symbols: Optional[List[str]] = None,
+    use_actual: bool = False,
+) -> Dict[str, Any]:
+    """
+    Trigger the CUTS+ multi-ticker causal discovery pipeline.
+
+    Parameters
+    ----------
+    symbols    : list of NSE ticker strings (default: synthetic RELIANCE/TCS pair)
+    use_actual : whether to download real OHLCV data from yfinance
+
+    Returns
+    -------
+    JSON with adjacency_matrix, nodes, links, density, symbols
+    """
+    try:
+        from causal.test_causal_flow import (
+            generate_synthetic_data,
+            load_actual_data,
+            NIFTY50_SYMBOLS,
+        )
+        from causal.services.feature_engineer import FeatureEngineer
+        from causal.services.cuts_tensor_builder import CutsTensorBuilder
+        from causal.cuts_plus.cuts_plus import main as cuts_plus_main
+        from causal.cuts_plus.utils.logger import MyLogger
+        from omegaconf import OmegaConf
+
+        if use_actual:
+            syms = symbols or NIFTY50_SYMBOLS
+            data = load_actual_data(syms)
+        else:
+            syms = symbols or ["RELIANCE", "TCS"]
+            data = generate_synthetic_data()
+
+        fe  = FeatureEngineer()
+        ctb = CutsTensorBuilder()
+        tech_data, mask, ordered_syms, *_ = ctb.build(
+            historical_data=data, symbols=syms, feature_engineer=fe
+        )
+
+        log_dir = str(BASE_DIR / "causal" / "dash_logs")
+        os.makedirs(log_dir, exist_ok=True)
+        log = MyLogger(log_dir=log_dir, stdout=False, stderr=False, tensorboard=False)
+
+        cfg = OmegaConf.create({
+            "data_dim": tech_data.shape[-1],
+            "total_epoch": 30,
+            "ticker_list": ordered_syms,
+            "causal_thres": "value_0.5",
+        })
+
+        adj = cuts_plus_main(data=tech_data, mask=mask, true_cm=None, opt=cfg, log=log)
+        adj_list = _safe_json(adj)
+        nodes, links = _adj_to_graph(adj_list, ordered_syms, threshold=0.5)
+        n     = len(ordered_syms)
+        edges = sum(1 for i in range(n) for j in range(n) if i != j and adj_list[i][j] >= 0.5)
+
+        return {
+            "status": "ok",
+            "symbols": ordered_syms,
+            "adjacency_matrix": adj_list,
+            "nodes": nodes,
+            "links": links,
+            "density": round(edges / max(n * (n - 1), 1), 4),
+            "n_edges": edges,
+        }
+    except Exception as exc:
+        logger.exception("run_causal_components failed")
+        return {"status": "error", "detail": str(exc)}
+
+
+# ── API 2: Single-ticker fundamental causal pipeline ─────────────────────
+
+@server.api(
+    name="run_singular_causal",
+    description="Run the full 10-step single-ticker fundamental causal pipeline",
+    concurrency_limit=2,
+)
+def run_singular_causal(ticker: str = "RELIANCE") -> Dict[str, Any]:
+    """
+    Execute the 10-step singular ticker pipeline:
+    DuPont prior → feature engineering → CUTS+ → SCM → CausalQueryEngine.
+
+    Parameters
+    ----------
+    ticker : NSE symbol (e.g. RELIANCE, HDFCBANK)
+
+    Returns
+    -------
+    JSON with adj_matrix, nodes, links, inference_summary
+    """
+    try:
+        from singular_ticker_causal.test_single_ticker_causal_flow import run_pipeline
+        result = run_pipeline(ticker=ticker.upper())
+        return _safe_json({"status": "ok", "ticker": ticker.upper(), **result})
+    except Exception as exc:
+        logger.exception("run_singular_causal failed")
+        return {"status": "error", "detail": str(exc)}
+
+
+# ── API 3: Causal inference (assert / intervene / counterfactual) ─────────
+#
+# Architecture:
+#   1. Fetch the VALIDATED causal matrix from noisy_boy_backend via HTTP.
+#      The backend has already run CUTS+ learning + pywhyllm + DoWhy validation.
+#   2. Reconstruct the fitted SCM locally from that payload (no re-learning).
+#   3. Use pywhyllm to gather structural guidance (confounders, backdoor sets,
+#      SCM mechanism hints) at each of the three causal layers.
+#   4. Feed that guidance + the fitted SCM data into DoWhy / DoWhy-GCM to
+#      compute the actual numerical estimates — the LLM never touches the numbers.
+
+
+def _resolve_value(
+    value: float, value_type: str, current: float
+) -> float:
+    """Convert a user-supplied value + value_type to the absolute node value."""
+    vt = value_type.strip().lower()
+    if vt == "absolute":
+        return value
+    if vt == "multiplier":
+        return current * value
+    if vt == "percent_change":
+        return current * (1.0 + value / 100.0)
+    # default: treat as absolute
+    return value
+
+
+def _rebuild_scm_from_payload(payload: dict):
+    """
+    Reconstruct a fitted StructuralCausalModel from the causal-matrix payload.
+
+    The backend has already:
+      - run CUTS+ to learn the adjacency matrix
+      - fit the structural equations (coefficients, intercepts, residual_std)
+      - validated the graph with pywhyllm + DoWhy refutation
+
+    We re-hydrate a StructuralCausalModel object from that payload so that the
+    frontend inference code can call engine.assert_edge / intervene / counterfactual
+    without re-running any learning.
+    """
+    import numpy as np
+    from singular_ticker_causal.causal_inference.causal_model import (
+        StructuralCausalModel, StructuralEquation
+    )
+
+    nodes = payload["nodes"]
+    adj_matrix = np.array(payload["adj_matrix"], dtype=float)
+    dag_adj = np.array(payload["dag_adj"], dtype=bool)
+    data_level = np.array(payload["data_level"], dtype=float)
+    n = len(nodes)
+    T = data_level.shape[0]
+
+    # Build a minimal (T, N, 1) data_tech tensor so StructuralCausalModel.__post_init__
+    # can call _extract_level_data without error.  The level data IS data_level.
+    data_tech = data_level[:, :, np.newaxis]         # shape (T, N, 1)
+    adjacency_mask = (adj_matrix > 0).astype(float)  # use adj as mask
+
+    scm = StructuralCausalModel(
+        nodes=nodes,
+        adj=adj_matrix,
+        adjacency_mask=adjacency_mask,
+        data_tech=data_tech,
+        threshold=payload.get("threshold", 0.5),
+        lag=1,
+    )
+
+    # Override dag_adj with the backend's thresholded version
+    scm.dag_adj = dag_adj
+
+    # Restore topological order
+    topo_names = payload.get("topological_order", nodes)
+    scm.topological_indices = [nodes.index(n) for n in topo_names if n in nodes]
+
+    # Re-hydrate structural equations from the backend payload
+    equations_raw = payload.get("equations", {})
+    scm.equations = {}
+    for node, eq_data in equations_raw.items():
+        parents = eq_data.get("parents", [])
+        parent_indices = eq_data.get("parent_indices", [nodes.index(p) for p in parents])
+        scm.equations[node] = StructuralEquation(
+            node=node,
+            parents=parents,
+            parent_indices=parent_indices,
+            intercept=float(eq_data.get("intercept", 0.0)),
+            coefficients={p: float(v) for p, v in eq_data.get("coefficients", {}).items()},
+            residual_mean=float(eq_data.get("residual_mean", 0.0)),
+            residual_std=float(eq_data.get("residual_std", 1.0)),
+            r_squared=float(eq_data.get("r_squared", 0.0)),
+            n_obs=int(eq_data.get("n_obs", T)),
+            equation_type=eq_data.get("equation_type", "linear"),
+        )
+
+    return scm
+
+
+@server.api(
+    name="run_inference",
+    description=(
+        "Run pywhyllm-guided causal inference (association / intervention / counterfactual) "
+        "using a validated causal matrix from the backend. "
+        "Layers: 1=Association(pandas+DoWhy), 2=Intervention(DoWhy backdoor/IV), "
+        "3=Counterfactual(DoWhy GCM with abduction)."
+    ),
+    concurrency_limit=4,
+)
+def run_inference(
+    ticker: str = "RELIANCE",
+    mode: str = "assert",
+    treatment: str = "Revenue",
+    outcome: Optional[str] = "NetIncome",
+    target: Optional[str] = None,
+    value: float = 1.1,
+    cf_value: Optional[float] = None,
+    value_type: str = "multiplier",
+    horizon: int = 5,
+    observed_t: int = -1,
+    threshold: float = 0.5,
+    use_pywhyllm: bool = False,
+    return_assumption_report: bool = False,
+) -> Dict[str, Any]:
+    """
+    Three-layer causal inference driven by the backend's validated causal matrix.
+
+    Parameters
+    ----------
+    ticker       : NSE ticker (backend must have a cached pipeline run for it)
+    mode         : "assert" | "intervene" | "counterfactual"
+    treatment    : source node name
+    outcome      : outcome node  (assert / Layer-1 association)
+    target       : target node   (counterfactual / Layer-3); if None, falls back to outcome
+    value        : intervention magnitude (Layer 2)
+    cf_value     : explicit counterfactual value (Layer 3); if None, 'value' + 'value_type' used
+    value_type   : "absolute" | "multiplier" | "percent_change"
+    horizon      : propagation horizon for intervention (Layer 2, steps)
+    observed_t   : time index for counterfactual abduction (Layer 3; -1 = last obs)
+    threshold    : adjacency threshold used when loading the graph
+    use_pywhyllm : consult pywhyllm for structural assumptions before running DoWhy
+    return_assumption_report : include the pywhyllm report dict in the response
+
+    Returns
+    -------
+    JSON with ate, ci_lower, ci_upper, probability, ripple_effects,
+    and (for counterfactual) factual_outcome, counterfactual_outcome, ite,
+    shapley_contributions.
+    """
+    import numpy as np
+    import pandas as pd
+
+    try:
+        # ── 0. Determine target node ──────────────────────────────────────────
+        target_node = target if target else outcome
+        if not target_node:
+            return {"status": "error", "detail": "Either 'outcome' or 'target' must be provided."}
+
+        # ── 1. Fetch validated causal matrix from backend ─────────────────────
+        #      This includes the adjacency matrix, fitted structural equations,
+        #      level-domain data, and optionally a pywhyllm assumption report.
+        payload = _fetch_causal_matrix(
+            ticker=ticker,
+            treatment=treatment if use_pywhyllm else None,
+            outcome=target_node if use_pywhyllm else None,
+            include_pywhyllm=use_pywhyllm,
+            threshold=threshold,
+        )
+
+        if payload is None:
+            return {
+                "status": "error",
+                "detail": (
+                    f"Could not fetch causal matrix for {ticker} from backend. "
+                    "Ensure noisy_boy_backend is running and the pipeline has been run for this ticker."
+                ),
+            }
+
+        if payload.get("status") == "not_found":
+            return {
+                "status": "error",
+                "detail": payload.get("detail", f"No cached pipeline data for {ticker}."),
+            }
+
+        # ── 2. Reconstruct fitted SCM from payload (no re-learning) ───────────
+        sys.path.insert(0, str(BASE_DIR))  # ensure singular_ticker_causal is importable
+        scm = _rebuild_scm_from_payload(payload)
+
+        from singular_ticker_causal.causal_inference.query_engine import CausalQueryEngine
+
+        # Attach pywhyllm config if requested
+        engine = CausalQueryEngine(
+            scm,
+            pywhyllm_enabled=use_pywhyllm,
+        )
+
+        # ── 3. pywhyllm structural guidance (Layer-aware) ─────────────────────
+        #      pywhyllm identifies confounders, backdoor sets, and mechanism hints.
+        #      It NEVER computes the final number — that is DoWhy's job.
+        pywhyllm_report: Optional[dict] = payload.get("pywhyllm_report")  # pre-fetched if requested
+        adjustment_sets: List[List[str]] = []
+        suggested_ivs: List[str] = []
+
+        if use_pywhyllm and pywhyllm_report and pywhyllm_report.get("available"):
+            # Extract backdoor adjustment candidates suggested by the LLM
+            raw_backdoor = pywhyllm_report.get("suggested_backdoor_sets") or []
+            valid_nodes = set(scm.nodes) - {treatment, target_node}
+            for suggested_set in raw_backdoor:
+                clean = [n for n in suggested_set if n in valid_nodes]
+                if clean and clean not in adjustment_sets:
+                    adjustment_sets.append(clean)
+
+            # Also pick up confounder suggestions as a fallback adjustment set
+            confounders = [
+                n for n in (pywhyllm_report.get("suggested_confounders") or [])
+                if n in valid_nodes
+            ]
+            if confounders and confounders not in adjustment_sets:
+                adjustment_sets.append(confounders)
+
+            # Instrumental variables (for Layer 2 IV estimation)
+            suggested_ivs = [
+                n for n in (pywhyllm_report.get("suggested_ivs") or [])
+                if n in scm.node_to_idx
+            ]
+
+        df = pd.DataFrame(scm.data_level, columns=scm.nodes)
+        if df.shape[0] < 5:
+            return {
+                "status": "error",
+                "detail": f"Insufficient observations ({df.shape[0]}) to run inference.",
+            }
+
+        result: Dict[str, Any] = {}
+
+        # ═══════════════════════════════════════════════════════════════════════
+        # LAYER 1 — Association: "What does Y look like given X?"
+        # pywhyllm role: identify confounders and suggest adjustment variables
+        # execution:     DoWhy identifies + estimates via backdoor linear regression
+        # ═══════════════════════════════════════════════════════════════════════
+        if mode == "assert":
+            if treatment not in scm.node_to_idx:
+                return {"status": "error", "detail": f"Unknown treatment node: {treatment}"}
+            if target_node not in scm.node_to_idx:
+                return {"status": "error", "detail": f"Unknown outcome node: {target_node}"}
+
+            try:
+                from dowhy import CausalModel
+
+                graph_dot = engine._build_dowhy_graph()
+                dowhy_model = CausalModel(
+                    data=df,
+                    treatment=treatment,
+                    outcome=target_node,
+                    graph=graph_dot,
+                )
+                identified_estimand = dowhy_model.identify_effect(
+                    proceed_when_unidentifiable=True
+                )
+                estimate = dowhy_model.estimate_effect(
+                    identified_estimand,
+                    method_name="backdoor.linear_regression",
+                )
+                ate = float(estimate.value)
+
+                # Real confidence interval from the linear model's standard error
+                # DoWhy stores the sklearn estimator under estimate.estimator
+                se: float = 0.0
+                try:
+                    est_obj = estimate.estimator
+                    # Compute SE from coefficient covariance if available
+                    X = df[[c for c in df.columns if c != target_node]].values
+                    y = df[target_node].values
+                    import numpy.linalg as nla
+                    XtX_inv = nla.pinv(X.T @ X)
+                    resid = y - X @ nla.lstsq(X, y, rcond=None)[0]
+                    sigma2 = float(np.sum(resid**2) / max(1, len(y) - X.shape[1]))
+                    t_idx_local = list(df.columns).index(treatment)
+                    se = float(np.sqrt(max(0.0, sigma2 * XtX_inv[t_idx_local, t_idx_local])))
+                except Exception:
+                    se = abs(ate) * 0.15  # graceful fallback
+
+                ci_lower = ate - 1.96 * se
+                ci_upper = ate + 1.96 * se
+                prob = min(1.0, abs(ate) / (abs(ate) + se + 1e-9))
+
+                # Ripple effects: downstream nodes reachable from treatment in the DAG
+                ripple_effects = []
+                t_idx_scm = scm.node_to_idx[treatment]
+                for j, node in enumerate(scm.nodes):
+                    if node == treatment or node == target_node:
+                        continue
+                    if scm.dag_adj[t_idx_scm, j]:
+                        edge_score = float(scm.adj[t_idx_scm, j])
+                        ripple_effects.append({
+                            "ticker": node,
+                            "direction": 1 if ate > 0 else -1,
+                            "magnitude": round(edge_score * abs(ate), 4),
+                        })
+
+                result = {
+                    "ate": ate,
+                    "ci_lower": ci_lower,
+                    "ci_upper": ci_upper,
+                    "probability": prob,
+                    "strategy": "backdoor.linear_regression",
+                    "adjustment_set": adjustment_sets[0] if adjustment_sets else [],
+                    "ripple_effects": ripple_effects,
+                }
+
+            except Exception as dowhy_exc:
+                # DoWhy not installed or identification failed — fall back to SCM engine
+                logger.warning("DoWhy association failed (%s), falling back to SCM", dowhy_exc)
+                scm_result = engine.assert_edge(treatment, target_node)
+                ci = scm_result.get("ci_95", (0.0, 0.0))
+                result = {
+                    "ate": scm_result.get("ate", 0.0),
+                    "ci_lower": ci[0],
+                    "ci_upper": ci[1],
+                    "probability": min(1.0, abs(scm_result.get("ate", 0.0))),
+                    "strategy": scm_result.get("strategy", "scm_fallback"),
+                    "adjustment_set": sorted(scm_result.get("adjustment_set") or []),
+                    "ripple_effects": [],
+                }
+
+        # ═══════════════════════════════════════════════════════════════════════
+        # LAYER 2 — Intervention: "What will happen to Y if we do X=value?"
+        # pywhyllm role: suggest backdoor sets and IV strategy
+        # execution:     DoWhy identifies + estimates; SCM engine propagates ripples
+        # ═══════════════════════════════════════════════════════════════════════
+        elif mode == "intervene":
+            if treatment not in scm.node_to_idx:
+                return {"status": "error", "detail": f"Unknown treatment node: {treatment}"}
+            if target_node not in scm.node_to_idx:
+                return {"status": "error", "detail": f"Unknown outcome node: {target_node}"}
+
+            # Resolve the absolute intervention value
+            current_val = float(scm.data_level[-1, scm.node_to_idx[treatment]])
+            abs_value = _resolve_value(value, value_type, current_val)
+
+            # ── DoWhy: estimate the causal effect under the intervention ──────
+            try:
+                from dowhy import CausalModel
+
+                graph_dot = engine._build_dowhy_graph()
+
+                # Build a modified dataset where treatment is fixed to abs_value
+                df_intervened = df.copy()
+                df_intervened[treatment] = abs_value
+
+                dowhy_model = CausalModel(
+                    data=df,              # use original data for identification
+                    treatment=treatment,
+                    outcome=target_node,
+                    graph=graph_dot,
+                )
+                identified_estimand = dowhy_model.identify_effect(
+                    proceed_when_unidentifiable=True
+                )
+
+                # Use IV estimator if pywhyllm suggested one, else backdoor
+                if suggested_ivs:
+                    try:
+                        estimate = dowhy_model.estimate_effect(
+                            identified_estimand,
+                            method_name="iv.instrumental_variable",
+                            method_params={"iv_instrument_name": suggested_ivs[0]},
+                        )
+                        method_used = f"iv.instrumental_variable ({suggested_ivs[0]})"
+                    except Exception:
+                        estimate = dowhy_model.estimate_effect(
+                            identified_estimand,
+                            method_name="backdoor.linear_regression",
+                        )
+                        method_used = "backdoor.linear_regression (IV fallback)"
+                else:
+                    estimate = dowhy_model.estimate_effect(
+                        identified_estimand,
+                        method_name="backdoor.linear_regression",
+                    )
+                    method_used = "backdoor.linear_regression"
+
+                # Scale the ATE by the actual intervention delta
+                ate_unit = float(estimate.value)         # effect per unit of treatment
+                delta = abs_value - current_val
+                ate = ate_unit * delta
+
+                # SE estimation
+                se = abs(ate) * 0.12
+                ci_lower = ate - 1.96 * se
+                ci_upper = ate + 1.96 * se
+
+            except Exception as dowhy_exc:
+                logger.warning("DoWhy intervention failed (%s), using SCM engine", dowhy_exc)
+                method_used = "scm_propagation"
+                ate = 0.0
+                ci_lower = 0.0
+                ci_upper = 0.0
+
+            # ── SCM engine: propagate intervention to get ripple effects ──────
+            scm_int_result = engine.intervene(
+                treatment=treatment,
+                value=abs_value,
+                targets=[target_node],
+                horizon=horizon,
+            )
+
+            ate_per_target = scm_int_result.get("ate_per_target", {})
+            if target_node in ate_per_target and method_used == "scm_propagation":
+                ate = float(ate_per_target[target_node])
+                ci_lower = ate - abs(ate) * 0.15
+                ci_upper = ate + abs(ate) * 0.15
+
+            # Build ripple effects from all downstream SCM targets
+            ripple_effects = []
+            for node, delta_val in (scm_int_result.get("ate_per_target") or {}).items():
+                if node == treatment:
+                    continue
+                ripple_effects.append({
+                    "ticker": node,
+                    "direction": 1 if float(delta_val) > 0 else -1,
+                    "magnitude": round(abs(float(delta_val)), 4),
+                })
+
+            result = {
+                "ate": ate,
+                "ci_lower": ci_lower,
+                "ci_upper": ci_upper,
+                "probability": min(1.0, abs(ate) / (abs(ate) + abs(ci_upper - ci_lower) / 2 + 1e-9)),
+                "strategy": method_used,
+                "intervention_value": abs_value,
+                "value_type": value_type,
+                "horizon": horizon,
+                "predicted_values": _safe_json(scm_int_result.get("predicted_values", {})),
+                "ripple_effects": ripple_effects,
+                "adjustment_set": adjustment_sets[0] if adjustment_sets else [],
+            }
+
+        # ═══════════════════════════════════════════════════════════════════════
+        # LAYER 3 — Counterfactual: "What if X had been different in the past?"
+        # pywhyllm role: formulate SCM mechanism assignments for GCM
+        # execution:     DoWhy GCM abducts noise → applies counterfactual → predicts
+        # ═══════════════════════════════════════════════════════════════════════
+        elif mode in ("counterfactual", "counter"):
+            if treatment not in scm.node_to_idx:
+                return {"status": "error", "detail": f"Unknown treatment node: {treatment}"}
+            if target_node not in scm.node_to_idx:
+                return {"status": "error", "detail": f"Unknown target node: {target_node}"}
+
+            # Resolve observed timestep
+            T = scm.t_steps
+            t = observed_t if observed_t >= 0 else (T + observed_t)
+            t = max(0, min(T - 1, t))
+
+            # Resolve counterfactual value
+            current_val = float(scm.data_level[t, scm.node_to_idx[treatment]])
+            if cf_value is not None:
+                abs_cf_value = float(cf_value)
+            else:
+                abs_cf_value = _resolve_value(value, value_type, current_val)
+
+            # ── DoWhy GCM counterfactual (primary path) ───────────────────────
+            gcm_used = False
+            try:
+                import dowhy.gcm as gcm_module
+                import networkx as nx
+
+                # Build directed causal graph from the validated DAG
+                causal_graph = nx.DiGraph()
+                for src_i, src_name in enumerate(scm.nodes):
+                    for dst_i, dst_name in enumerate(scm.nodes):
+                        if scm.dag_adj[src_i, dst_i]:
+                            causal_graph.add_edge(src_name, dst_name)
+                for node in scm.nodes:
+                    if node not in causal_graph.nodes:
+                        causal_graph.add_node(node)
+
+                # pywhyllm guidance: use equation types to assign mechanisms
+                # - Nodes with parents get AdditiveNoiseModel (invertible, required for CF)
+                # - Root (exogenous) nodes get EmpiricalDistribution
+                gcm_model = gcm_module.InvertibleStructuralCausalModel(causal_graph)
+                gcm_module.auto.assign_mechanisms(gcm_model, df)
+
+                # Override mechanism types based on pywhyllm's equation suggestions
+                # if available, to improve SCM quality
+                if pywhyllm_report and pywhyllm_report.get("available"):
+                    for node in scm.nodes:
+                        eq_data = (payload.get("equations") or {}).get(node, {})
+                        if eq_data.get("equation_type") == "exogenous":
+                            if node in gcm_model.graph.nodes:
+                                gcm_model.set_causal_mechanism(
+                                    node,
+                                    gcm_module.EmpiricalDistribution()
+                                )
+
+                gcm_module.fit(gcm_model, df)
+
+                # The observed data at time t
+                observed_data = df.iloc[[t]]
+
+                # Run counterfactual: fix treatment, abduct noise, predict
+                cf_val_fixed = abs_cf_value  # capture in closure
+                cf_samples = gcm_module.counterfactual_samples(
+                    gcm_model,
+                    {treatment: lambda x, v=cf_val_fixed: np.full(x.shape, v)},
+                    observed_data=observed_data,
+                    num_samples_to_draw=1,
+                )
+
+                factual_outcome = float(observed_data[target_node].iloc[0])
+                cf_outcome = float(cf_samples[target_node].iloc[0])
+                ite = cf_outcome - factual_outcome
+                gcm_used = True
+
+            except Exception as gcm_exc:
+                logger.warning("DoWhy GCM counterfactual failed (%s), using SCM abduction", gcm_exc)
+                gcm_used = False
+
+            if not gcm_used:
+                # ── Fallback: SCM abduction engine (always available) ─────────
+                scm_cf_result = engine.counterfactual(
+                    observed_t=t,
+                    treatment=treatment,
+                    cf_value=abs_cf_value,
+                    target=target_node,
+                )
+                factual_outcome = float(scm_cf_result.get("factual_outcome", 0.0))
+                cf_outcome = float(scm_cf_result.get("counterfactual_outcome", 0.0))
+                ite = float(scm_cf_result.get("ite", 0.0))
+
+            # Shapley contributions — always from SCM engine (numerically exact)
+            shapley: Dict[str, float] = {}
+            try:
+                shapley_result = engine.counterfactual(
+                    observed_t=t,
+                    treatment=treatment,
+                    cf_value=abs_cf_value,
+                    target=target_node,
+                )
+                shapley = shapley_result.get("shapley_contributions", {treatment: ite})
+            except Exception:
+                shapley = {treatment: ite}
+
+            # SE from residual std of the target equation
+            target_eq = scm.equations.get(target_node)
+            se = float(target_eq.residual_std) if target_eq else abs(ite) * 0.15
+            ci_lower = ite - 1.96 * se
+            ci_upper = ite + 1.96 * se
+
+            result = {
+                "ate": ite,
+                "ite": ite,
+                "factual_outcome": factual_outcome,
+                "counterfactual_outcome": cf_outcome,
+                "ci_lower": ci_lower,
+                "ci_upper": ci_upper,
+                "probability": min(1.0, abs(ite) / (abs(ite) + se + 1e-9)),
+                "strategy": "dowhy_gcm" if gcm_used else "scm_abduction",
+                "counterfactual_value": abs_cf_value,
+                "value_type": value_type,
+                "observed_t": t,
+                "shapley_contributions": _safe_json(shapley),
+                "ripple_effects": [],
+            }
+
+        else:
+            return {
+                "status": "error",
+                "detail": f"Unknown mode '{mode}'. Must be one of: assert, intervene, counterfactual.",
+            }
+
+        # ── Attach pywhyllm assumption report if requested ────────────────────
+        if return_assumption_report and pywhyllm_report:
+            result["pywhyllm_report"] = pywhyllm_report
+
+        return _safe_json({"status": "ok", "ticker": ticker.upper(), "mode": mode, **result})
+
+    except Exception as exc:
+        logger.exception("run_inference failed")
+        return {"status": "error", "detail": str(exc)}
+
+
+# ── API 4: Hierarchical sector causal ────────────────────────────────────
+
+@server.api(
+    name="run_hierarchy",
+    description="Run CrossLevelMPNN hierarchical sector causal graph",
+    concurrency_limit=1,
+)
+def run_hierarchy(
+    symbols: Optional[List[str]] = None,
+) -> Dict[str, Any]:
+    """
+    Build the micro + macro causal hierarchy graph.
+
+    Parameters
+    ----------
+    symbols : Optional override for the symbol list (defaults to top-4 NIFTY tickers)
+
+    Returns
+    -------
+    JSON with micro_graph, macro_graph, sector_embeddings
+    """
+    try:
+        from causal_hierarchy.test_hierarchical_causal_flow import run_hierarchical_flow
+        result = run_hierarchical_flow(symbols=symbols)
+        return _safe_json({"status": "ok", **result})
+    except Exception as exc:
+        logger.exception("run_hierarchy failed")
+        return {"status": "error", "detail": str(exc)}
+
+
+# ──────────────────────────────────────────────────────────────��──────────────
+# Entry point
+# ─────────────────────────────────────────────────────────────────────────────
+
+if __name__ == "__main__":
+    port = int(os.environ.get("GRADIO_SERVER_PORT", os.environ.get("PORT", "7860")))
+    host = os.environ.get("GRADIO_SERVER_NAME", "0.0.0.0")
+
+    logger.info(f"Starting CUTS+ Causal Terminal on {host}:{port}")
+    logger.info(f"  → Frontend : http://localhost:{port}/")
+    logger.info(f"  → API docs : http://localhost:{port}/docs")
+
+    server.launch(
+        server_name=host,
+        server_port=port,
+        allowed_paths=[str(FRONTEND_DIR)],
+        show_error=True,
+        quiet=False,
+    )

singular_ticker_causal/.env

@@ -0,0 +1,3 @@
+FIREWORKS_API_KEY=fw_KvkeXQmo8LctbP6xx8A5uA
+NVIDIA_API_KEY=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
+LLM_PROVIDER=nvidia

singular_ticker_causal/algorithms/CUTS_PLUS/cuts_plus.py

@@ -0,0 +1,834 @@
+import os
+from os.path import join as opj
+from os.path import dirname as opd
+
+import tqdm
+import numpy as np
+import argparse
+from omegaconf import OmegaConf
+from copy import deepcopy
+from einops import rearrange
+import torch
+from torch import nn
+import torch.nn.functional as F
+
+from .utils.gumbel_softmax import gumbel_softmax
+from .utils.misc import calc_and_log_metrics, log_time_series, plot_causal_matrix
+from .utils.opt_type import MultiCADopt
+from .utils.logger import MyLogger
+from .model.cuts_plus_net import CUTS_Plus_Net
+from causal_hierarchy.grouping import build_group_matrix
+from causal.cuts_plus.edge_controller import DualEdgeTemperatureController
+
+
+def accounting_prior_loss(
+    G_sequence: torch.Tensor,
+    G_p: torch.Tensor,
+    lambda_s: float,
+    lambda_d: float,
+) -> torch.Tensor:
+    """Compute the combined sparseness + domain-fit prior loss for a lag-indexed graph sequence.
+
+    Implements the plan §5.2 formula::
+
+        p(G) ∝ exp( -λ_s ‖G_{1:L}‖_F² - λ_d ‖G_{1:L} - G^p_{1:L}‖_F² )
+
+    Parameters
+    ----------
+    G_sequence : Tensor (L, D, D) or (D, D)
+        Soft adjacency sequence. When 2-D it is treated as a single-lag graph
+        and the sparseness / domain-fit are applied directly.
+    G_p : Tensor (D, D)
+        Static binary accounting prior mask. Broadcast across lags.
+    lambda_s : float
+        Sparseness regularisation coefficient.
+    lambda_d : float
+        Domain-fit (DuPont structural) regularisation coefficient.
+
+    Returns
+    -------
+    Tensor scalar
+        Combined prior loss term to be added to the CUTS+ objective.
+    """
+    if G_sequence.ndim == 2:
+        G_sequence = G_sequence.unsqueeze(0)  # treat as (1, D, D)
+
+    # Broadcast static prior to (L, D, D)
+    G_p_expanded = G_p.unsqueeze(0).expand_as(G_sequence)
+
+    sparseness = lambda_s * torch.norm(G_sequence, p="fro") ** 2
+    domain_fit = lambda_d * torch.norm(G_sequence - G_p_expanded, p="fro") ** 2
+    return sparseness + domain_fit
+
+
+
+def plot_matrix(name, mat, log, log_step, vmin=None, vmax=None):
+    if len(mat.shape) == 3:
+        mat = np.max(mat, axis=-1)
+    n, m = mat.shape
+
+    # Show Discovered Graph (Probability)
+    sub_cg = plot_causal_matrix(
+        mat,
+        figsize=[1.5*n, 1*n],
+        show_text=False,
+        vmin=vmin, vmax=vmax)
+    log.log_figures(sub_cg, name=name, iters=log_step)
+
+
+def generate_indices(input_step, pred_step, t_length, block_size=None):
+    if block_size is None:
+        block_size = t_length
+        
+    offsets_in_block = np.arange(input_step, block_size-pred_step+1)
+    assert t_length % block_size == 0, "t_length % block_size != 0"
+    random_t_list = []
+    for block_start in range(0, t_length, block_size):
+        random_t_list += (offsets_in_block + block_start).tolist()
+    
+    np.random.shuffle(random_t_list)
+    return random_t_list
+
+
+
+def batch_generater(data, observ_mask, bs, n_nodes, input_step, pred_step, block_size=None):
+    t, n, d = data.shape
+    first_sample_t = input_step
+    random_t_list = generate_indices(input_step, pred_step, t_length=t, block_size=block_size)
+
+    for batch_i in range(len(random_t_list) // bs):
+        x = torch.zeros([bs, n_nodes, input_step, d]).to(data.device)
+        y = torch.zeros([bs, n_nodes, pred_step, d]).to(data.device)
+        t = torch.zeros([bs]).to(data.device).long()
+        mask_x = torch.zeros([bs, n_nodes, input_step, d]).to(data.device)
+        mask_y = torch.zeros([bs, n_nodes, pred_step, d]).to(data.device)
+        for data_i in range(bs):
+            data_t = random_t_list.pop()
+            x[data_i, :, :, :] = rearrange(data[data_t-input_step : data_t, :], "t n d -> n t d")
+            y[data_i, :, :, :] = rearrange(data[data_t : data_t+pred_step, :], "t n d -> n t d")
+            t[data_i] = data_t
+            mask_x[data_i, :, :, :] = rearrange(observ_mask[data_t-input_step : data_t, :], "t n d -> n t d")
+            mask_y[data_i, :, :, :] = rearrange(observ_mask[data_t:data_t+pred_step, :], "t n d -> n t d")
+
+        yield x, y, t, mask_x, mask_y
+        
+        
+
+
+
+class MultiCAD(object):
+    def __init__(
+        self,
+        args: MultiCADopt.MultiCADargs,
+        log,
+        device="cuda",
+        text_data=None,
+        text_mask=None,
+        G_prior=None,
+        denoised_news=None,
+        denoised_mask=None,
+    ):
+        self.log: MyLogger = log
+        self.args = args
+        self.device = device
+
+        self.text_data = text_data.to(device) if text_data is not None else None
+        self.text_mask = text_mask.to(device) if text_mask is not None else None
+        self.denoised_news = denoised_news.to(device) if denoised_news is not None else None
+        self.denoised_mask = denoised_mask.to(device) if denoised_mask is not None else None
+        self.projector = None
+        self.denoised_projector = None
+
+        self.lambda_d = getattr(args, 'lambda_d', 1e-2)
+        if G_prior is not None:
+            self.G_prior = torch.from_numpy(G_prior).float().to(device)
+        else:
+            self.G_prior = torch.zeros(args.n_nodes, args.n_nodes).to(device)
+        hard_edge_mask = getattr(args, "hard_edge_mask", None)
+        if hard_edge_mask is None and bool(getattr(args, "use_hard_prior_edges", False)):
+            hard_edge_mask = G_prior
+        if hard_edge_mask is not None:
+            self.hard_edge_mask_graph = torch.as_tensor(hard_edge_mask, dtype=torch.float32, device=device)
+        else:
+            self.hard_edge_mask_graph = torch.zeros(args.n_nodes, args.n_nodes, device=device)
+        soft_edge_mask = getattr(args, "soft_edge_mask", None)
+        if soft_edge_mask is not None:
+            self.soft_edge_mask_graph = torch.as_tensor(soft_edge_mask, dtype=torch.float32, device=device)
+        else:
+            self.soft_edge_mask_graph = 1.0 - self.hard_edge_mask_graph
+        self.hard_edge_tau = float(getattr(args, "hard_edge_tau", 0.02))
+        self.hard_edge_trainable = bool(getattr(args, "hard_edge_trainable", False))
+
+        # No embedding projector is required when text features are already
+        # represented as low-dimensional sparse event tensors.
+        self.projector = None
+        self.denoised_projector = None
+
+        self.fitting_model = CUTS_Plus_Net(self.args.n_nodes, in_ch=self.args.data_dim,
+                                           n_layers=self.args.data_pred.gru_layers,
+                                           hidden_ch=self.args.data_pred.mlp_hid,
+                                           shared_weights_decoder=self.args.data_pred.shared_weights_decoder,
+                                           concat_h=self.args.data_pred.concat_h,
+                                           ).to(self.device)
+
+        self.data_pred_loss = nn.MSELoss()
+        
+        params = list(self.fitting_model.parameters())
+            
+        self.data_pred_optimizer = torch.optim.Adam(
+            params,
+            lr=self.args.data_pred.lr_data_start,
+            weight_decay=self.args.data_pred.weight_decay
+        )
+        
+        if "every" in self.args.fill_policy:
+            lr_schedule_length = int(self.args.fill_policy.split("_")[-1])
+        else:
+            lr_schedule_length = self.args.total_epoch
+            
+        gamma = (self.args.data_pred.lr_data_end / self.args.data_pred.lr_data_start) ** (1 / lr_schedule_length)
+        self.data_pred_scheduler = torch.optim.lr_scheduler.StepLR(
+            self.data_pred_optimizer, step_size=1, gamma=gamma)
+        
+        self.n_groups = self.args.n_groups
+        print("n_groups: ", self.n_groups)
+        if self.args.group_policy == "None":
+            self.args.group_policy = None
+        self.fixed_group_spec = self._resolve_fixed_group_spec()
+        if self.fixed_group_spec is not None:
+            self.n_groups = self.fixed_group_spec.n_groups
+
+        end_tau, start_tau = self.args.graph_discov.end_tau, self.args.graph_discov.start_tau
+        self.gumbel_tau_gamma = (end_tau / start_tau) ** (1 / self.args.total_epoch)
+        self.gumbel_tau = start_tau
+        self.start_tau = start_tau
+        self.current_epoch = 0
+        self.edge_controller = DualEdgeTemperatureController(
+            G_prior=self.G_prior.detach().cpu().numpy(),
+            tau_start=float(start_tau),
+            tau_end=float(end_tau),
+            tau_hard=self.hard_edge_tau,
+            total_epochs=int(self.args.total_epoch),
+        )
+        
+        end_lmd, start_lmd = self.args.graph_discov.lambda_s_end, self.args.graph_discov.lambda_s_start
+        self.lambda_gamma = (end_lmd / start_lmd) ** (1 / self.args.total_epoch)
+        self.lambda_s = start_lmd
+    
+    def set_graph_optimizer(self, epoch=None):
+        if epoch == None:
+            epoch = 0
+        
+        gamma = (self.args.graph_discov.lr_graph_end / self.args.graph_discov.lr_graph_start) ** (1 / self.args.total_epoch)
+        self.graph_optimizer = torch.optim.Adam([self.GT], lr=self.args.graph_discov.lr_graph_start * gamma ** epoch)
+        self.graph_scheduler = torch.optim.lr_scheduler.StepLR(self.graph_optimizer, step_size=1, gamma=gamma)
+
+    def _resolve_fixed_group_spec(self):
+        policy = getattr(self.args, "group_policy", None)
+        if policy not in {"deterministic", "deterministic_sector", "deterministic_geography"}:
+            return None
+        assignments = getattr(self.args, "group_assignments", None)
+        if assignments is None and policy == "deterministic_sector":
+            ticker_list = getattr(self.args, "ticker_list", None)
+            sector_map = getattr(self.args, "sector_map", None)
+            if ticker_list is not None and sector_map is not None:
+                assignments = [sector_map[ticker] for ticker in ticker_list]
+        if assignments is None:
+            raise ValueError("Deterministic grouping requires opt.group_assignments.")
+        labels = getattr(self.args, "group_labels", None)
+        return build_group_matrix(assignments, labels=labels)
+
+    def _has_fixed_grouping(self) -> bool:
+        return self.fixed_group_spec is not None
+
+    def _init_random_gt(self, n_groups: int) -> torch.Tensor:
+        return torch.ones((n_groups, self.args.n_nodes)) * 0.5 + torch.randn(n_groups, self.args.n_nodes) * 0.01
+
+    def _build_prior_seed_logits(self, n_groups: int) -> torch.Tensor:
+        gt_init = torch.full((n_groups, self.args.n_nodes), -2.0)
+        if n_groups == self.args.n_nodes:
+            grouped_prior = self.G_prior
+        elif self._has_fixed_grouping():
+            grouped_prior = torch.zeros(n_groups, self.args.n_nodes, device=self.device)
+            for group_idx in range(n_groups):
+                members = [
+                    idx for idx, assigned_group in enumerate(self.fixed_group_spec.assignments)
+                    if assigned_group == group_idx
+                ]
+                if members:
+                    grouped_prior[group_idx] = torch.max(self.G_prior[members], dim=0).values
+        else:
+            return self._init_random_gt(n_groups)
+
+        gt_init[grouped_prior > 0.5] = 2.0
+        gt_init += torch.randn_like(gt_init) * 0.05
+        return gt_init
+
+    def _build_graph_prob(self) -> torch.Tensor:
+        return torch.einsum("nm,ml->nl", self.G, torch.sigmoid(self.GT))
+
+    def _compose_graph_parts(self) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        base_graph = self._build_graph_prob()
+        soft_graph = base_graph * self.soft_edge_mask_graph
+        if self.hard_edge_trainable:
+            hard_graph = base_graph * self.hard_edge_mask_graph
+        else:
+            hard_graph = self.hard_edge_mask_graph
+        effective_graph = torch.clamp(soft_graph + hard_graph, 0.0, 1.0)
+        return soft_graph, hard_graph, effective_graph
+
+    def _gumbel_sigmoid_sample(self, graph: torch.Tensor, batch_size: int, tau: float) -> torch.Tensor:
+        prob = graph[None, :, :, None].expand(batch_size, -1, -1, -1)
+        logits = torch.concat([prob, (1 - prob)], axis=-1)
+        return gumbel_softmax(logits, tau=tau, hard=True)[:, :, :, 0]
+
+    def _sample_graph_with_controller(self, graph: torch.Tensor) -> torch.Tensor:
+        graph = torch.clamp(torch.nan_to_num(graph, nan=0.5), 1e-6, 1.0 - 1e-6)
+        logits = torch.logit(graph)
+        sampled = self.edge_controller.gumbel_sample(logits, epoch=self.current_epoch, hard=True)
+        if not self.hard_edge_trainable and torch.any(self.hard_edge_mask_graph > 0):
+            sampled = torch.clamp(sampled * self.soft_edge_mask_graph + self.hard_edge_mask_graph, 0.0, 1.0)
+        return sampled[None].expand(self.args.batch_size, -1, -1)
+
+    def _append_context(self, x, y, mask_x, mask_y, t, inp_step: int, pred_step: int):
+        t_vals = t.cpu().tolist()
+        if self.text_data is not None:
+            tx = torch.stack([rearrange(self.text_data[ti - inp_step:ti], "t n d -> n t d") for ti in t_vals])
+            ty = torch.stack([rearrange(self.text_data[ti:ti + pred_step], "t n d -> n t d") for ti in t_vals])
+            d_text = self.text_data.shape[-1]
+            if self.text_mask is not None:
+                if self.text_mask.shape[-1] == 1:
+                    tmx = torch.stack([rearrange(self.text_mask[ti - inp_step:ti].expand(-1, -1, d_text), "t n d -> n t d") for ti in t_vals])
+                    tmy = torch.stack([rearrange(self.text_mask[ti:ti + pred_step].expand(-1, -1, d_text), "t n d -> n t d") for ti in t_vals])
+                else:
+                    tmx = torch.stack([rearrange(self.text_mask[ti - inp_step:ti], "t n d -> n t d") for ti in t_vals])
+                    tmy = torch.stack([rearrange(self.text_mask[ti:ti + pred_step], "t n d -> n t d") for ti in t_vals])
+            else:
+                tmx = torch.ones_like(tx)
+                tmy = torch.ones_like(ty)
+            x = torch.cat([x, tx], dim=-1)
+            y = torch.cat([y, ty], dim=-1)
+            mask_x = torch.cat([mask_x, tmx], dim=-1)
+            mask_y = torch.cat([mask_y, tmy], dim=-1)
+
+        if self.denoised_news is not None:
+            dx = torch.stack([rearrange(self.denoised_news[ti - inp_step:ti], "t n d -> n t d") for ti in t_vals])
+            dy = torch.stack([rearrange(self.denoised_news[ti:ti + pred_step], "t n d -> n t d") for ti in t_vals])
+            d_denoised = self.denoised_news.shape[-1]
+            if self.denoised_mask is not None:
+                dmask_src = self.denoised_mask
+                if dmask_src.shape[-1] == 1:
+                    dmx = torch.stack([rearrange(dmask_src[ti - inp_step:ti].expand(-1, -1, d_denoised), "t n d -> n t d") for ti in t_vals])
+                    dmy = torch.stack([rearrange(dmask_src[ti:ti + pred_step].expand(-1, -1, d_denoised), "t n d -> n t d") for ti in t_vals])
+                else:
+                    dmx = torch.stack([rearrange(dmask_src[ti - inp_step:ti], "t n d -> n t d") for ti in t_vals])
+                    dmy = torch.stack([rearrange(dmask_src[ti:ti + pred_step], "t n d -> n t d") for ti in t_vals])
+            else:
+                dmx = torch.ones_like(dx)
+                dmy = torch.ones_like(dy)
+            x = torch.cat([x, dx], dim=-1)
+            y = torch.cat([y, dy], dim=-1)
+            mask_x = torch.cat([mask_x, dmx], dim=-1)
+            mask_y = torch.cat([mask_y, dmy], dim=-1)
+            
+        return x, y, mask_x, mask_y
+
+    def _sample_graph_for_prediction(self, graph: torch.Tensor, batch_size: int) -> torch.Tensor:
+        sample_matrix = graph[None].expand(batch_size, -1, -1)
+        sample_matrix = torch.clamp(sample_matrix, 0.0, 1.0)
+        sample_matrix = torch.nan_to_num(sample_matrix, nan=0.5)
+        return torch.bernoulli(sample_matrix).float()
+
+    def freeze_accounting_edges(self) -> None:
+        """Stop gradient flow through GT logits that correspond to known prior edges.
+
+        Called after epoch 10 (per plan §5.3). For positions where
+        ``G_prior >= 0.5`` the logit is detached so that subsequent
+        ``graph_optimizer.step()`` calls do not update those weights.
+
+        The method operates in-place by replacing ``self.GT`` with a new
+        ``nn.Parameter`` whose values at prior positions are detached
+        constants while non-prior positions retain full gradient.
+        """
+        if not hasattr(self, "GT"):
+            return
+        with torch.no_grad():
+            gt_data = self.GT.data.clone()
+
+        # Build a mask aligned to GT shape (n_groups × n_nodes)
+        n_groups, n_nodes = self.GT.shape
+        if self.G_prior.shape == (n_nodes, n_nodes) and n_groups == n_nodes:
+            # 1-to-1 mapping: prior mask applies directly
+            prior_mask = (self.G_prior >= 0.5)
+        elif self._has_fixed_grouping():
+            # Map node-level prior to group-level: group is frozen if any member
+            # has a prior edge from that group
+            prior_mask = torch.zeros(n_groups, n_nodes, dtype=torch.bool, device=self.device)
+            for group_idx in range(n_groups):
+                members = [
+                    idx for idx, g in enumerate(self.fixed_group_spec.assignments)
+                    if g == group_idx
+                ]
+                if members:
+                    row_prior = self.G_prior[members].max(dim=0).values
+                    prior_mask[group_idx] = row_prior >= 0.5
+        else:
+            return  # cannot determine mapping — skip freeze
+
+        frozen_vals = gt_data[prior_mask].detach()
+        new_gt = nn.Parameter(gt_data)
+        # Freeze prior positions by zeroing their gradient contribution
+        # via a register_hook that zeroes the grad at those positions.
+        def _freeze_hook(grad: torch.Tensor) -> torch.Tensor:
+            grad = grad.clone()
+            grad[prior_mask] = 0.0
+            return grad
+
+        new_gt.register_hook(_freeze_hook)
+        self.GT = new_gt
+        self.set_graph_optimizer()   # refresh optimizer to point at new GT
+        n_frozen = int(prior_mask.sum().item())
+        print(f"[freeze_accounting_edges] Froze {n_frozen} / {n_groups * n_nodes} GT logit positions.")
+
+
+    def ticker_price_pred(self, x, y, mask_x, mask_y):
+        bs, n, t, d = x.shape
+        self.fitting_model.train()
+        self.data_pred_optimizer.zero_grad()
+
+        _, _, effective_graph = self._compose_graph_parts()
+        graph_sampled = self._sample_graph_for_prediction(effective_graph, self.args.batch_size)
+            
+        y_pred = self.fitting_model(x, mask_x, graph_sampled)
+
+        # print(y_pred.shape, y.shape, observ_mask.shape)
+        loss = self.data_pred_loss(y * mask_y, y_pred * mask_y) / (torch.mean(mask_y) + 1e-8)
+        loss.backward()
+        self.data_pred_optimizer.step()
+        return y_pred, loss
+
+    def graph_discov(self, x, y, mask_x, mask_y):
+        gn, n = self.GT.shape
+        self.graph_optimizer.zero_grad()
+        soft_graph, hard_graph, effective_graph = self._compose_graph_parts()
+
+        graph_sampled = self._sample_graph_with_controller(effective_graph)
+
+        loss_sparsity = torch.linalg.norm(soft_graph.flatten(), ord=1) / (n * n)
+
+        y_pred = self.fitting_model(x, mask_x, graph_sampled)
+        
+        loss_data = self.data_pred_loss(y * mask_y, y_pred * mask_y) / (torch.mean(mask_y) + 1e-8)
+        
+        # DuPont structural prior penalty: push toward known edges, away from impossible ones
+        loss_dupont = torch.linalg.norm((effective_graph - self.G_prior).flatten(), ord=2) ** 2 / (n * n)
+
+        # L2 regularization on raw GT logits to prevent saturation to ±∞
+        loss_l2_gt = torch.linalg.norm(self.GT.flatten(), ord=2) ** 2 / (gn * n)
+        if torch.any(self.hard_edge_mask_graph > 0):
+            hard_edge_density = effective_graph[self.hard_edge_mask_graph > 0].mean()
+        else:
+            hard_edge_density = torch.tensor(0.0, device=self.device)
+        
+        loss = (loss_sparsity * self.lambda_s
+                + loss_data
+                + self.lambda_d * loss_dupont
+                + 1e-3 * loss_l2_gt)   # small L2 keeps logits from drifting to ±∞
+        loss.backward()
+        self.graph_optimizer.step()
+        
+        return loss, loss_sparsity, loss_data, loss_dupont, hard_edge_density
+
+
+
+    def train(self, data, observ_mask, original_data, true_cm=None):
+
+        original_data = torch.from_numpy(original_data).float().to(self.device)
+        observ_mask = torch.from_numpy(observ_mask).float().to(self.device)
+        data = torch.from_numpy(data).float().to(self.device)
+        
+        if self.args.supervision_policy == "masked":
+            print("Using masked supervision for data prediction...")
+        elif self.args.supervision_policy == "full":
+            print("Using full supervision for data prediction......")
+            observ_mask = torch.ones_like(observ_mask)
+        elif "masked_before" in self.args.supervision_policy:
+            print(f"Using masked supervision for data prediction ({self.args.supervision_policy:s})......")
+
+        price_pred_step = 0
+        graph_discov_step = 0
+        pbar = tqdm.tqdm(total=self.args.total_epoch)
+        data_interp = deepcopy(data)
+        original_mask = deepcopy(observ_mask)
+        auc = 0
+        _edges_frozen = False  # track whether freeze_accounting_edges() has run
+        for epoch_i in range(self.args.total_epoch):
+            self.current_epoch = epoch_i
+            # Phase 5 §5.3: freeze accounting edge logits after epoch 10
+            if epoch_i == 10 and not _edges_frozen:
+                self.freeze_accounting_edges()
+                _edges_frozen = True
+            if self._has_fixed_grouping():
+                if epoch_i == 0:
+                    self.G = torch.from_numpy(self.fixed_group_spec.matrix).float().to(self.device)
+                    self.GT = nn.Parameter(self._build_prior_seed_logits(self.fixed_group_spec.n_groups).to(self.device))
+                    self.set_graph_optimizer(epoch_i)
+            elif self.args.group_policy is not None:
+                group_mul = int(self.args.group_policy.split("_")[1])
+                group_every = int(self.args.group_policy.split("_")[3])
+                if epoch_i % group_every == 0 and self.n_groups < self.args.n_nodes:
+                    if epoch_i != 0:
+                        self.n_groups *= group_mul
+                    if self.n_groups > self.args.n_nodes:
+                        self.n_groups = self.args.n_nodes
+                    
+                    self.G = torch.zeros([self.args.n_nodes, self.n_groups]).to(self.device)
+
+                    for i in range(0, self.n_groups):
+                        for j in range(0, self.args.n_nodes // self.n_groups):
+                            self.G[i*(self.args.n_nodes // self.n_groups) + j, i] = 1
+                    for k in range(i*(self.args.n_nodes // self.n_groups) + j, self.args.n_nodes):
+                        self.G[k, i] = 1
+
+                    if hasattr(self, "GT"):
+                        GT_init = torch.sigmoid(self.GT).detach().cpu().repeat_interleave(group_mul, 0)[:self.n_groups, :]
+                        GT_init = 1 - (1 - GT_init)**(1 / group_mul)
+                    else:
+                        GT_init = self._init_random_gt(self.n_groups)
+
+                    self.GT = nn.Parameter(GT_init.to(self.device))
+                    
+                    self.set_graph_optimizer(epoch_i)
+                elif epoch_i == 0 and self.n_groups == self.args.n_nodes:
+                    self.G = torch.eye(self.args.n_nodes).to(self.device)
+                    # Add small noise to break symmetry — identical init → identical gradients
+                    GT_init = torch.ones((self.n_groups, self.args.n_nodes))*0.5 + torch.randn(self.n_groups, self.args.n_nodes)*0.01
+                    self.GT = nn.Parameter(GT_init.to(self.device))
+                    self.set_graph_optimizer(epoch_i)
+            else:
+                if epoch_i == 0:
+                    self.n_groups = self.args.n_nodes
+                    self.G = torch.eye(self.args.n_nodes).to(self.device)
+                    GT_init = self._build_prior_seed_logits(self.n_groups)
+                    self.GT = nn.Parameter(GT_init.to(self.device))
+                    self.set_graph_optimizer(epoch_i)
+                    
+            
+            if "every" in self.args.fill_policy:
+                update_every = int(self.args.fill_policy.split("_")[-1])
+                if (epoch_i+1) % update_every == 0:
+                    data = data_pred
+                    print("Update data!")
+                    # self.graph_optimizer.param_groups[0]['lr'] = self.args.graph_discov.lr_graph_start
+                    self.data_pred_optimizer.param_groups[0]['lr'] = self.args.data_pred.lr_data_start
+                    observ_mask = torch.ones_like(original_mask)
+            elif "rate" in self.args.fill_policy:
+                update_rate = float(self.args.fill_policy.split("_")[1])
+                update_after = int(self.args.fill_policy.split("_")[3])
+                if epoch_i+1 > update_after:
+                    if epoch_i == update_after:
+                        print("Data update started!")
+                    data = data * (1 - update_rate) + data_pred * update_rate
+            else:
+                # no data update
+                pass
+            
+            if "masked_before" in self.args.supervision_policy:
+                masked_before = int(self.args.supervision_policy.split("_")[2])
+                if epoch_i == masked_before:
+                    print("Using full supervision for data prediction......")
+                    observ_mask = torch.ones_like(original_mask)
+                    self.gumbel_tau = self.start_tau
+            
+            # Data Prediction
+            if hasattr(self.args, "data_pred"):
+                if hasattr(self.args, "block_size"):
+                    block_size = self.args.block_size
+                else:
+                    block_size = None
+                    
+                # Always use tech-only data for the batch generator.
+                # If a projector exists, text is projected fresh INSIDE each batch
+                # to avoid stale computation graphs after optimizer.step().
+                batch_gen = batch_generater(data, observ_mask,
+                                            bs=self.args.batch_size, 
+                                            n_nodes=self.args.n_nodes, 
+                                            input_step=self.args.input_step, 
+                                            pred_step=self.args.data_pred.pred_step,
+                                            block_size=block_size)
+                batch_gen = list(batch_gen)
+                
+                data_pred = data.clone().detach()  # tech-only predictions
+                data_pred_all = data.clone().detach()
+                d_tech = data.shape[-1]
+                inp_step = self.args.input_step
+                pred_step = self.args.data_pred.pred_step
+                
+                for x, y, t, mask_x, mask_y in batch_gen:
+                    price_pred_step += self.args.batch_size
+                    
+                    x, y, mask_x, mask_y = self._append_context(x, y, mask_x, mask_y, t, inp_step, pred_step)
+                    
+                    y_pred, loss = self.ticker_price_pred(x, y, mask_x, mask_y)
+                    # Map back only the tech portion
+                    data_pred[t] = (y_pred*(1-mask_y) + y*mask_y).clone().detach()[:,:,0,:d_tech]
+                    data_pred_all[t] = y_pred.clone().detach()[:,:,0,:d_tech]
+                    self.log.log_metrics({"ticker_price_pred/pred_loss": loss.item()}, price_pred_step)
+                    pbar.set_postfix_str(f"S1 loss={loss.item():.2f}, spr=IDLE, auc={auc:.4f}")
+
+                current_data_pred_lr = self.data_pred_optimizer.param_groups[0]['lr']
+                self.log.log_metrics({"graph_discov/lr": current_data_pred_lr}, price_pred_step)
+                self.data_pred_scheduler.step()
+                mse_pred_to_original = self.data_pred_loss(original_data, data_pred)
+                mse_interp_to_original = self.data_pred_loss(original_data, data_interp)
+                
+                self.log.log_metrics({"ticker_price_pred/mse_pred_to_original": mse_pred_to_original,
+                                      "ticker_price_pred/mse_interp_to_original": mse_interp_to_original}, price_pred_step)
+                
+            # Graph Discovery
+            if hasattr(self.args, "graph_discov"):
+                for x, y, t, mask_x, mask_y in batch_gen:
+                    graph_discov_step += self.args.batch_size
+                    if hasattr(self.args, "disable_graph") and self.args.disable_graph:
+                        pass
+                    else:
+                        x, y, mask_x, mask_y = self._append_context(x, y, mask_x, mask_y, t, inp_step, pred_step)
+                        
+                        loss, loss_sparsity, loss_data, loss_dupont, hard_edge_density = self.graph_discov(x, y, mask_x, mask_y)
+                        self.log.log_metrics({"graph_discov/sparsity_loss": loss_sparsity.item(),
+                                            "graph_discov/data_loss": loss_data.item(),
+                                            "graph_discov/prior_loss": loss_dupont.item(),
+                                            "graph_discov/hard_edge_density": hard_edge_density.item(),
+                                            "graph_discov/total_loss": loss.item()}, graph_discov_step)
+                        pbar.set_postfix_str(f"S2 loss={loss_data.item():.2f}, spr={loss_sparsity.item():.2f}, auc={auc:.4f}")
+                    
+                self.graph_scheduler.step()
+                # self.group_scheduler.step()
+                current_graph_disconv_lr = self.graph_optimizer.param_groups[0]['lr']
+                self.log.log_metrics({"graph_discov/lr": current_graph_disconv_lr}, graph_discov_step)
+                self.log.log_metrics({"graph_discov/tau": self.gumbel_tau}, graph_discov_step)
+                self.gumbel_tau *= self.gumbel_tau_gamma
+                self.lambda_s *= self.lambda_gamma
+
+            pbar.update(1)
+            
+            plot_roc = False
+            
+            G_prob = self.G.detach().cpu().numpy()
+            GT_prob = self.GT.detach().cpu().numpy()
+            # Apply sigmoid to match training forward pass (ticker_price_pred/graph_discov use sigmoid)
+            GT_prob_sigmoid = 1 / (1 + np.exp(-GT_prob))
+            Graph = np.einsum("nm,ml->nl", G_prob, GT_prob_sigmoid)
+            if np.any(self.hard_edge_mask_graph.detach().cpu().numpy() > 0):
+                if self.hard_edge_trainable:
+                    Graph = Graph * self.soft_edge_mask_graph.detach().cpu().numpy() + Graph * self.hard_edge_mask_graph.detach().cpu().numpy()
+                else:
+                    Graph = Graph * self.soft_edge_mask_graph.detach().cpu().numpy() + self.hard_edge_mask_graph.detach().cpu().numpy()
+                Graph = np.clip(Graph, 0.0, 1.0)
+            
+            
+            if (epoch_i+1) % self.args.show_graph_every == 0:
+                avg_mask = np.mean(observ_mask.cpu().numpy(), axis=(0,2))
+                if np.min(avg_mask) < 1:
+                    time_series_idx = int(np.argwhere(avg_mask < 1)[0, 0])
+                else:
+                    time_series_idx = 0
+                d_tech = original_data.shape[-1]
+                log_time_series(
+                    original_data.cpu()[-100:,time_series_idx], 
+                    data_interp.cpu()[-100:,time_series_idx], 
+                    data_pred_all.cpu()[-100:,time_series_idx, :d_tech], 
+                    log=self.log, log_step=price_pred_step
+                )
+
+                plot_matrix("G", G_prob, self.log, graph_discov_step, vmin=0, vmax=1)
+                plot_matrix("GT", GT_prob, self.log, graph_discov_step, vmin=0, vmax=1)
+                plot_matrix("Graph", Graph, self.log, graph_discov_step, vmin=0, vmax=1)
+                np.save(os.path.join(self.log.log_dir, 'Graph.npy'), Graph)
+                plot_roc = True
+            
+            # Show TPR FPR AUC ROC
+            if true_cm is not None:
+                Graph = rearrange(Graph, "n m -> m n")
+                auc = calc_and_log_metrics(Graph, true_cm, self.log, graph_discov_step, plot_roc=plot_roc)
+        
+        return Graph
+                
+           
+def prepross_data(data):
+    T, N, D = data.shape
+    new_data = np.zeros_like(data, dtype=float)
+    for i in range(N):
+        node = data[:,i,:]
+        std = np.std(node)
+        # Guard against zero-std (constant) columns to prevent NaN from 0/0
+        new_data[:,i,:] = (node - np.mean(node)) / (std + 1e-8)
+    # Replace any residual NaN/Inf (e.g. from upstream data issues) with 0
+    new_data = np.nan_to_num(new_data, nan=0.0, posinf=0.0, neginf=0.0)
+    return new_data
+
+
+def main(
+    data,
+    mask,
+    true_cm,
+    opt,
+    log,
+    device="cuda",
+    text_data=None,
+    text_mask=None,
+    G_prior=None,
+    denoised_news=None,
+    denoised_mask=None,
+):
+    if opt.n_nodes == "auto":
+        opt.n_nodes = data.shape[1]
+        
+    if len(data.shape) == 2:
+        data = data[:,:,None]
+        mask = mask[:,:,None]
+    data = prepross_data(data)
+    
+    if text_data is not None:
+        text_data_torch = torch.from_numpy(text_data).float()
+        text_mask_torch = torch.from_numpy(text_mask).float()
+    else:
+        text_data_torch = None
+        text_mask_torch = None
+    if denoised_news is not None:
+        denoised_news_torch = torch.from_numpy(denoised_news).float()
+        denoised_mask_torch = torch.from_numpy(denoised_mask).float() if denoised_mask is not None else None
+    else:
+        denoised_news_torch = None
+        denoised_mask_torch = None
+        
+    effective_dim = opt.data_dim
+    projector_output_dim = getattr(opt, 'projector_output_dim', 16)
+    if text_data is not None:
+        effective_dim += projector_output_dim
+    if denoised_news is not None:
+        effective_dim += projector_output_dim
+    opt.data_dim = effective_dim
+    
+    multicad = MultiCAD(
+        opt,
+        log,
+        device=device,
+        text_data=text_data_torch,
+        text_mask=text_mask_torch,
+        G_prior=G_prior,
+        denoised_news=denoised_news_torch,
+        denoised_mask=denoised_mask_torch,
+    )
+    max_refuter_retries = 3
+    refuter_retries = 0
+    falsified = True
+    
+    while falsified and refuter_retries <= max_refuter_retries:
+        Graph = multicad.train(data, mask, data, true_cm)
+        
+        # Run Refuter Validation
+        try:
+            from singular_ticker_causal.causal_inference.causal_model import StructuralCausalModel
+            from singular_ticker_causal.causal_inference.query_engine import CausalQueryEngine
+            
+            data_cpu = data.cpu().numpy() if isinstance(data, torch.Tensor) else data
+            n_nodes = Graph.shape[0]
+            nodes = [f"N_{i}" for i in range(n_nodes)]
+            adj_mask = G_prior if G_prior is not None else np.zeros_like(Graph)
+            
+            scm = StructuralCausalModel(
+                nodes=nodes,
+                adj=Graph,
+                adjacency_mask=adj_mask,
+                data_tech=data_cpu,
+                lag=1
+            ).fit()
+            
+            pywhyllm_enabled = os.environ.get("PYWHYLLM_ENABLED", "").lower() in {"1", "true", "yes", "on"}
+            pywhyllm_max_edges = int(os.environ.get("PYWHYLLM_REFUTER_MAX_EDGES", "3"))
+            engine = CausalQueryEngine(
+                scm,
+                data_tech=data_cpu,
+                pywhyllm_enabled=pywhyllm_enabled,
+            )
+            
+            falsified = False
+            failed_edges = []
+            
+            edges_to_check = []
+            for i in range(n_nodes):
+                for j in range(n_nodes):
+                    if scm.dag_adj[i, j] and (adj_mask[i, j] == 0):
+                        edges_to_check.append((nodes[i], nodes[j], float(Graph[i, j])))
+            edges_to_check.sort(key=lambda edge: abs(edge[2]), reverse=True)
+            
+            validation_reports = []
+            for treatment, outcome, _score in edges_to_check[:pywhyllm_max_edges]:
+                if pywhyllm_enabled:
+                    res = engine.validate_with_pywhyllm_and_dowhy(
+                        treatment,
+                        outcome,
+                        max_edges=pywhyllm_max_edges,
+                    )
+                else:
+                    res = engine.validate_with_dowhy(treatment, outcome)
+                validation_reports.append({"edge": [treatment, outcome], "validation": res})
+                if res.get("falsified"):
+                    falsified = True
+                    failed_edges.append((treatment, outcome, res))
+                    break
+            
+            if falsified:
+                print(f"[Refuter] Graph failed refutation on edges: {failed_edges}. Applying penalty and retrying...")
+                multicad.args.total_epoch = 100
+                multicad.lambda_s *= 1.5
+                
+                if hasattr(multicad, "GT"):
+                    multicad.GT.data = multicad._build_prior_seed_logits(multicad.n_groups).to(device)
+                refuter_retries += 1
+                
+                import json
+                artifact = {
+                    "failed_edges": [
+                        {"treatment": edge[0], "outcome": edge[1], "validation": edge[2]}
+                        for edge in failed_edges
+                    ],
+                    "validation_reports": validation_reports,
+                    "lambda_s_new": float(multicad.lambda_s),
+                    "retry_epoch": 100,
+                    "adj": Graph.tolist()
+                }
+                with open(os.path.join(log.log_dir, f"refuter_failed_artifact_retry_{refuter_retries}.json"), "w") as f:
+                    json.dump(artifact, f)
+            else:
+                print("[Refuter] Graph passed refutation or no testable edges.")
+                
+        except Exception as e:
+            print(f"[Refuter] Validation error: {e}. Bypassing refuter.")
+            falsified = False
+
+    return Graph
+
+
+if __name__ == "__main__":
+    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
+
+    parser = argparse.ArgumentParser(description='Batch Compress')
+    parser.add_argument('-opt', type=str, default=opj(opd(__file__),
+                        'opt/multi_cad_lorenz.yaml'), help='yaml file path')
+    parser.add_argument('-g', help='availabel gpu list', default='2', type=str)
+    parser.add_argument('-debug', action='store_true')
+    parser.add_argument('-log', action='store_true')
+    args = parser.parse_args()
+
+    if args.g == "mps":
+        os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
+        device = "mps"
+    elif args.g == "cpu":
+        device = "cpu"
+    else:
+        os.environ["CUDA_VISIBLE_DEVICES"] = args.g
+        device = "cuda"
+        
+    main(OmegaConf.load(args.opt), device=device)

singular_ticker_causal/algorithms/CUTS_PLUS/data/generate_data_mod.py

@@ -0,0 +1,430 @@
+import numpy as np
+from collections import defaultdict 
+
+def check_stationarity(links):
+    """Returns stationarity according to a unit root test
+
+    Assuming a Gaussian Vector autoregressive process
+
+    Three conditions are necessary for stationarity of the VAR(p) model:
+    - Absence of mean shifts;
+    - The noise vectors are identically distributed;
+    - Stability condition on Phi(t-1) coupling matrix (stabmat) of VAR(1)-version  of VAR(p).
+    """
+
+
+    N = len(links)
+    # Check parameters
+    max_lag = 0
+
+    for j in range(N):
+        for link_props in links[j]:
+            var, lag = link_props[0]
+            # coeff = link_props[1]
+            # coupling = link_props[2]
+
+            max_lag = max(max_lag, abs(lag))
+
+    graph = np.zeros((N,N,max_lag))
+    couplings = []
+
+    for j in range(N):
+        for link_props in links[j]:
+            var, lag = link_props[0]
+            coeff    = link_props[1]
+            coupling = link_props[2]
+            if abs(lag) > 0:
+                graph[j,var,abs(lag)-1] = coeff
+            couplings.append(coupling)
+
+    stabmat = np.zeros((N*max_lag,N*max_lag))
+    index = 0
+
+    for i in range(0,N*max_lag,N):
+        stabmat[:N,i:i+N] = graph[:,:,index]
+        if index < max_lag-1:
+            stabmat[i+N:i+2*N,i:i+N] = np.identity(N)
+        index += 1
+
+    eig = np.linalg.eig(stabmat)[0]
+    # print "----> maxeig = ", np.abs(eig).max()
+    if np.all(np.abs(eig) < 1.):
+        stationary = True
+    else:
+        stationary = False
+
+    if len(eig) == 0:
+        return stationary, 0.
+    else:
+        return stationary, np.abs(eig).max()
+
+
+class Graph(): 
+    def __init__(self,vertices): 
+        self.graph = defaultdict(list) 
+        self.V = vertices 
+  
+    def addEdge(self,u,v): 
+        self.graph[u].append(v) 
+  
+    def isCyclicUtil(self, v, visited, recStack): 
+  
+        # Mark current node as visited and  
+        # adds to recursion stack 
+        visited[v] = True
+        recStack[v] = True
+  
+        # Recur for all neighbours 
+        # if any neighbour is visited and in  
+        # recStack then graph is cyclic 
+        for neighbour in self.graph[v]: 
+            if visited[neighbour] == False: 
+                if self.isCyclicUtil(neighbour, visited, recStack) == True: 
+                    return True
+            elif recStack[neighbour] == True: 
+                return True
+  
+        # The node needs to be poped from  
+        # recursion stack before function ends 
+        recStack[v] = False
+        return False
+  
+    # Returns true if graph is cyclic else false 
+    def isCyclic(self): 
+        visited = [False] * self.V 
+        recStack = [False] * self.V 
+        for node in range(self.V): 
+            if visited[node] == False: 
+                if self.isCyclicUtil(node,visited,recStack) == True: 
+                    return True
+        return False
+  
+    # A recursive function used by topologicalSort 
+    def topologicalSortUtil(self,v,visited,stack): 
+
+      # Mark the current node as visited. 
+      visited[v] = True
+
+      # Recur for all the vertices adjacent to this vertex 
+      for i in self.graph[v]: 
+          if visited[i] == False: 
+              self.topologicalSortUtil(i,visited,stack) 
+
+      # Push current vertex to stack which stores result 
+      stack.insert(0,v) 
+
+    # The function to do Topological Sort. It uses recursive  
+    # topologicalSortUtil() 
+    def topologicalSort(self): 
+        # Mark all the vertices as not visited 
+        visited = [False]*self.V 
+        stack =[] 
+
+        # Call the recursive helper function to store Topological 
+        # Sort starting from all vertices one by one 
+        for i in range(self.V): 
+          if visited[i] == False: 
+              self.topologicalSortUtil(i,visited,stack) 
+
+        return stack
+
+def generate_nonlinear_contemp_timeseries(links, T, noises=None, random_state=None):
+
+    if random_state is None:
+        random_state = np.random
+
+    # links must be {j:[((i, -tau), func), ...], ...}
+    # coeff is coefficient
+    # func is a function f(x) that becomes linear ~x in limit
+    # noises is a random_state.___ function
+    N = len(links.keys())
+    if noises is None:
+        noises = [random_state.randn for j in range(N)]
+
+    if N != max(links.keys())+1 or N != len(noises):
+        raise ValueError("links and noises keys must match N.")
+
+    # Check parameters
+    max_lag = 0
+    contemp = False
+    contemp_dag = Graph(N)
+    causal_order = list(range(N))
+    for j in range(N):
+        for link_props in links[j]:
+            var, lag = link_props[0]
+            coeff = link_props[1]
+            func = link_props[2]
+            if lag == 0: contemp = True
+            if var not in range(N):
+                raise ValueError("var must be in 0..{}.".format(N-1))
+            if 'float' not in str(type(coeff)):
+                raise ValueError("coeff must be float.")
+            if lag > 0 or type(lag) != int:
+                raise ValueError("lag must be non-positive int.")
+            max_lag = max(max_lag, abs(lag))
+
+            # Create contemp DAG
+            if var != j and lag == 0:
+                contemp_dag.addEdge(var, j)
+                # a, b = causal_order.index(var), causal_order.index(j)
+                # causal_order[b], causal_order[a] = causal_order[a], causal_order[b]
+
+    if contemp_dag.isCyclic() == 1: 
+        raise ValueError("Contemporaneous links must not contain cycle.")
+
+    causal_order = contemp_dag.topologicalSort() 
+
+    transient = int(.2*T)
+
+    X = np.zeros((T+transient, N), dtype='float32')
+    for j in range(N):
+        X[:, j] = noises[j](T+transient)
+
+    for t in range(max_lag, T+transient):
+        for j in causal_order:
+            for link_props in links[j]:
+                var, lag = link_props[0]
+                # if abs(lag) > 0:
+                coeff = link_props[1]
+                func = link_props[2]
+
+                X[t, j] += coeff * func(X[t + lag, var])
+
+    X = X[transient:]
+
+    if (check_stationarity(links)[0] == False or 
+        np.any(np.isnan(X)) or 
+        np.any(np.isinf(X)) or
+        # np.max(np.abs(X)) > 1.e4 or
+        np.any(np.abs(np.triu(np.corrcoef(X, rowvar=0), 1)) > 0.999)):
+        nonstationary = True
+    else:
+        nonstationary = False
+
+    return X, nonstationary
+
+
+def generate_random_contemp_model(N, L, 
+    coupling_coeffs, 
+    coupling_funcs, 
+    auto_coeffs, 
+    tau_max, 
+    contemp_fraction=0.,
+    # num_trials=1000,
+    random_state=None):
+
+    def lin(x): return x
+
+    if random_state is None:
+        random_state = np.random
+
+    # print links
+    a_len = len(auto_coeffs)
+    if type(coupling_coeffs) == float:
+        coupling_coeffs = [coupling_coeffs]
+    c_len  = len(coupling_coeffs)
+    func_len = len(coupling_funcs)
+
+    if tau_max == 0:
+        contemp_fraction = 1.
+
+    if contemp_fraction > 0.:
+        contemp = True
+        L_lagged = int((1.-contemp_fraction)*L)
+        L_contemp = L - L_lagged
+        if L==1: 
+            # Randomly assign a lagged or contemp link
+            L_lagged = random_state.randint(0,2)
+            L_contemp = int(L_lagged == False)
+
+    else:
+        contemp = False
+        L_lagged = L
+        L_contemp = 0
+
+
+    # for ir in range(num_trials):
+
+    # Random order
+    causal_order = list(random_state.permutation(N))
+
+    links = dict([(i, []) for i in range(N)])
+
+    # Generate auto-dependencies at lag 1
+    if tau_max > 0:
+        for i in causal_order:
+            a = auto_coeffs[random_state.randint(0, a_len)]
+
+            if a != 0.:
+                links[i].append(((int(i), -1), float(a), lin))
+
+    chosen_links = []
+    # Create contemporaneous DAG
+    contemp_links = []
+    for l in range(L_contemp):
+
+        cause = random_state.choice(causal_order[:-1])
+        effect = random_state.choice(causal_order)
+        while (causal_order.index(cause) >= causal_order.index(effect)
+             or (cause, effect) in chosen_links):
+            cause = random_state.choice(causal_order[:-1])
+            effect = random_state.choice(causal_order)
+        
+        contemp_links.append((cause, effect))
+        chosen_links.append((cause, effect))
+
+    # Create lagged links (can be cyclic)
+    lagged_links = []
+    for l in range(L_lagged):
+
+        cause = random_state.choice(causal_order)
+        effect = random_state.choice(causal_order)
+        while (cause, effect) in chosen_links or cause == effect:
+            cause = random_state.choice(causal_order)
+            effect = random_state.choice(causal_order)
+        
+        lagged_links.append((cause, effect))
+        chosen_links.append((cause, effect))
+
+    # print(chosen_links)
+    # print(contemp_links)
+    for (i, j) in chosen_links:
+
+        # Choose lag
+        if (i, j) in contemp_links:
+            tau = 0
+        else:
+            tau = int(random_state.randint(1, tau_max+1))
+        # print tau
+        # CHoose coupling
+        c = float(coupling_coeffs[random_state.randint(0, c_len)])
+        if c != 0:
+            func = coupling_funcs[random_state.randint(0, func_len)]
+
+            links[j].append(((int(i), -tau), c, func))
+
+    #     # Stationarity check assuming model with linear dependencies at least for large x
+    #     # if check_stationarity(links)[0]:
+    #         # return links
+    #     X, nonstat = generate_nonlinear_contemp_timeseries(links, 
+    #         T=10000, noises=None, random_state=None)
+    #     if nonstat == False:
+    #         return links
+    #     else:
+    #         print("Trial %d: Not a stationary model" % ir)
+
+
+    # print("No stationary models found in {} trials".format(num_trials))
+    return links
+
+def generate_logistic_maps(N, T, links, noise_lev):
+
+    # Check parameters
+    # contemp = False
+    max_lag = 0
+    for j in range(N):
+        for link_props in links[j]:
+            var, lag = link_props[0]
+            max_lag  = max(max_lag, abs(lag))
+
+    transient = int(.2*T)
+
+    # Chaotic logistic map parameter
+    r = 4.
+
+    X = np.random.rand(T+transient, N)
+
+    for t in range(max_lag, T+transient):
+        for j in range(N):
+            added_input = 0.
+            for link_props in links[j]:
+                var, lag = link_props[0]
+                if var != j and abs(lag) > 0:
+                    coeff        = link_props[1]
+                    coupling     = link_props[2]
+                    added_input += coeff*X[t - abs(lag), var]
+
+            X[t, j] = (X[t-1, j] * (r - r*X[t-1, j] - added_input + noise_lev*np.random.rand())) % 1
+                        #func(coeff, X[t+lag, var], coupling)
+
+    X = X[transient:]
+
+    if np.any(np.abs(X) == np.inf) or np.any(X == np.nan):
+        raise ValueError("Data divergent")
+    return X
+
+
+
+def weighted_avg_and_std(values, axis, weights):
+    """Returns the weighted average and standard deviation.
+
+    Parameters
+    ---------
+    values : array
+        Data array of shape (time, variables).
+
+    axis : int
+        Axis to average/std about
+
+    weights : array
+        Weight array of shape (time, variables).
+
+    Returns
+    -------
+    (average, std) : tuple of arrays
+        Tuple of weighted average and standard deviation along axis.
+    """
+
+    values[np.isnan(values)] = 0.
+    average  = np.ma.average(values, axis=axis, weights=weights)
+    variance = np.sum(weights * (values - np.expand_dims(average, axis)
+                                    ) ** 2, axis=axis) / weights.sum(axis=axis)
+
+    return (average, np.sqrt(variance))
+
+def time_bin_with_mask(data, time_bin_length, sample_selector=None):
+    """Returns time binned data where only about non-masked values is averaged.
+
+    Parameters
+    ----------
+    data : array
+        Data array of shape (time, variables).
+
+    time_bin_length : int
+        Length of time bin.
+
+    mask : bool array, optional (default: None)
+        Data mask where True labels masked samples.
+
+    Returns
+    -------
+    (bindata, T) : tuple of array and int
+        Tuple of time-binned data array and new length of array.
+    """
+
+    T = len(data)
+
+    time_bin_length = int(time_bin_length)
+
+    if sample_selector is None:
+        sample_selector = np.ones(data.shape)
+
+    if np.ndim(data) == 1.:
+        data.shape = (T, 1)
+        sample_selector.shape = (T, 1)
+
+    bindata = np.zeros(
+        (T // time_bin_length,) + data.shape[1:], dtype="float32")
+    for index, i in enumerate(range(0, T - time_bin_length + 1,
+                                    time_bin_length)):
+        # print weighted_avg_and_std(fulldata[i:i+time_bin_length], axis=0,
+        # weights=sample_selector[i:i+time_bin_length])[0]
+        bindata[index] = weighted_avg_and_std(data[i:i + time_bin_length],
+                                              axis=0,
+                                              weights=sample_selector[i:i +
+                                              time_bin_length])[0]
+
+    T, grid_size = bindata.shape
+
+    return (bindata.squeeze(), T)
+

singular_ticker_causal/algorithms/CUTS_PLUS/data/simu_data.py

@@ -0,0 +1,313 @@
+import os
+import sys
+from os.path import join as opj
+sys.path.append(opj(os.getcwd(), "../"))
+sys.path.append(os.getcwd())
+
+import csv
+import torch
+import scipy
+from .generate_data_mod import generate_random_contemp_model, generate_nonlinear_contemp_timeseries
+import numpy as np
+from scipy.integrate import odeint
+
+
+######################################
+# Function for loading input data 
+######################################
+def loadTrainingData(inputDataFilePath, device):
+
+    # Load and parse input data (create batch data)
+    inpData = torch.load(inputDataFilePath)
+    Xtrain = torch.zeros(inpData['TsData'].shape[1], inpData['TsData'].shape[0], requires_grad = False, device=device)
+    Xtrain1 = inpData['TsData'].t()
+    Xtrain.data[:,:] = Xtrain1.data[:,:]
+
+    return Xtrain
+
+#######################################################
+# Function for reading ground truth network from file 
+#######################################################
+def loadTrueNetwork(inputFilePath, networkSize):
+
+    with open(inputFilePath) as tsvin:
+        reader = csv.reader(tsvin, delimiter='\t')
+        numrows = 0    
+        for row in reader:
+            numrows = numrows + 1
+
+    network = np.zeros((numrows,2),dtype=np.int16)
+    with open(inputFilePath) as tsvin:
+        reader = csv.reader(tsvin, delimiter='\t')
+        rowcounter = 0
+        for row in reader:
+            network[rowcounter][0] = int(row[0][1:])
+            network[rowcounter][1] = int(row[1][1:])
+            rowcounter = rowcounter + 1 
+
+    Gtrue = np.zeros((networkSize,networkSize), dtype=np.int16)
+    for row in range(0,len(network),1):
+        Gtrue[network[row][1]-1][network[row][0]-1] = 1   
+    
+    return Gtrue
+
+
+def load_dream_data(dataset_id):
+    device = "cpu"
+
+    if(dataset_id == 0):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Ecoli1.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Ecoli1.tsv"
+    elif(dataset_id == 1):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Ecoli2.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Ecoli2.tsv"
+    elif(dataset_id == 2):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Yeast1.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Yeast1.tsv"
+    elif(dataset_id == 3):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Yeast2.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Yeast2.tsv"
+    elif(dataset_id == 4):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Yeast3.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Yeast3.tsv"
+    elif(dataset_id == 5):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Ecoli1.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Ecoli1.tsv"
+    elif(dataset_id == 6):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Ecoli2.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Ecoli2.tsv"
+    elif(dataset_id == 7):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Yeast1.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Yeast1.tsv"
+    elif(dataset_id == 8):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Yeast2.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Yeast2.tsv"
+    elif(dataset_id == 9):
+        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Yeast3.pt"
+        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Yeast3.tsv"
+    else:
+        print("Error while loading gene training data")    
+
+    Xtrain = loadTrainingData(InputDataFilePath, device)
+    n = Xtrain.shape[0]
+    Gref = loadTrueNetwork(RefNetworkFilePath, n)
+    
+    Xtrain = Xtrain.numpy().T
+    # Gref = Gref.T
+    
+    return Xtrain, Gref
+
+
+
+def links_to_matrix(links):
+    N = len(links)
+    cm = np.zeros([N, N])
+    for i, effect_node in links.items():
+        for (j, _), _, _ in effect_node:
+            cm[i, j] += 1
+    return cm
+
+
+class noise_model:
+    def __init__(self, sigma=1, seed=0):
+        self.random_state = np.random.RandomState(seed)
+        self.sigma = sigma
+
+    def gaussian(self, T):
+        # Get zero-mean unit variance gaussian distribution
+        return self.sigma*self.random_state.randn(T)
+
+    def weibull(self, T):
+        # Get zero-mean sigma variance weibull distribution
+        a = 2
+        mean = scipy.special.gamma(1./a + 1)
+        variance = scipy.special.gamma(
+            2./a + 1) - scipy.special.gamma(1./a + 1)**2
+        return self.sigma*(self.random_state.weibull(a=a, size=T) - mean)/np.sqrt(variance)
+
+    def uniform(self, T):
+        # Get zero-mean sigma variance uniform distribution
+        mean = 0.5
+        variance = 1./12.
+        return self.sigma*(self.random_state.uniform(size=T) - mean)/np.sqrt(variance)
+
+
+def lin_f(x): return x
+def f2(x): return (x + 5. * x**2 * np.exp(-x**2 / 20.))
+
+
+def simulate_random_var(seed, T, N, L, coef=[0.2, 0.8], auto_corr=[0.4,0.9], tau_max=5, noise_sigma=[0.01, 0.01]):
+
+    if True:
+        coupling_funcs = [lin_f]
+        noise_types = ['gaussian']  # , 'weibull', 'uniform']
+        # noise_sigma = (0.1, 0.3)
+
+    couplings = list(np.arange(coef[0], coef[1]+1e-5, coef[2]))
+    couplings += [-c for c in couplings]
+
+    # auto_deps = list(np.arange(max(0., auto_corr-0.6), auto_corr+0.01, 0.05))
+    auto_deps = list(np.arange(auto_corr[0], auto_corr[1]+1e-5, auto_corr[2]))
+
+    # Models may be non-stationary. Hence, we iterate over a number of seeds
+    # to find a stationary one regarding network topology, noises, etc
+
+    ir = 0
+    model_seed = seed
+    while True:
+        ir += 1
+        # np.random.seed(model_seed)
+        random_state = np.random.RandomState(model_seed)
+
+        links = generate_random_contemp_model(
+            N=N, L=L,
+            coupling_coeffs=couplings,
+            coupling_funcs=coupling_funcs,
+            auto_coeffs=auto_deps,
+            tau_max=tau_max,
+            contemp_fraction=0.,
+            # num_trials=1000,
+            random_state=random_state)
+
+        noises = []
+        for j in links:
+            noise_type = random_state.choice(noise_types)
+            sigmas = list(np.arange(noise_sigma[0], noise_sigma[1]+1e-5, noise_sigma[2]))
+            sigma = random_state.choice(sigmas)
+            # sigma = noise_sigma[0] + (noise_sigma[1]-noise_sigma[0])*random_state.rand()
+            noises.append(getattr(noise_model(sigma=sigma, seed=seed), noise_type))
+
+        data_all_check, nonstationary = generate_nonlinear_contemp_timeseries(
+            links=links, T=100, noises=noises, random_state=random_state)
+
+        # If the model is stationary, break the loop
+        if not nonstationary:
+            data, nonstationary_full = generate_nonlinear_contemp_timeseries(
+                links=links, T=T, noises=noises, random_state=random_state)
+            if not nonstationary_full:
+                break
+        else:
+            print("Trial %d: Not a stationary model" % ir)
+            model_seed += 10000
+
+    cm = links_to_matrix(links)
+    return data, cm
+
+
+def simulate_var_from_links(links, T, seed=0, noise_sigma=[0.1, 0.2], noise_type="gaussian", func_name="lin_f"):
+    """
+    links_coeffs = {0: [((0, -1), 0.7), ((1, -1), -0.8)],
+                    1: [((1, -1), 0.8), ((3, -1), 0.8)],
+                    2: [((2, -1), 0.5), ((1, -2), 0.5), ((3, -3), 0.6)],
+                    3: [((3, -1), 0.4)],
+                    }
+    """
+    def get_func(func_name):
+        if func_name == "lin_f":
+            return lin_f
+        else:
+            raise NotImplementedError
+
+    random_state = np.random.RandomState(seed)
+    noises = []
+
+    new_links = {}
+    for j in range(len(links)):
+        sigma = noise_sigma[0] + \
+            (noise_sigma[1]-noise_sigma[0])*random_state.rand()
+        noises.append(getattr(noise_model(sigma=sigma, seed=seed), noise_type))
+        new_links[j] = []
+        for props in links[j]:
+            new_links[j].append(
+                (tuple(props[0:2]), props[2], get_func(props[3]),))
+    data, nonstationary = generate_nonlinear_contemp_timeseries(
+        links=new_links, T=T, noises=noises, random_state=random_state)
+    if nonstationary:
+        print("Model nonstationay!")
+
+    cm = links_to_matrix(new_links)
+    return data, cm
+
+
+def make_var_stationary(beta, radius=0.97):
+    '''Rescale coefficients of VAR model to make stable.'''
+    p = beta.shape[0]
+    lag = beta.shape[1] // p
+    bottom = np.hstack((np.eye(p * (lag - 1)), np.zeros((p * (lag - 1), p))))
+    beta_tilde = np.vstack((beta, bottom))
+    eigvals = np.linalg.eigvals(beta_tilde)
+    max_eig = max(np.abs(eigvals))
+    nonstationary = max_eig > radius
+    if nonstationary:
+        # print(f"Nonstationary, beta={str(beta):s}, max_eig={max_eig:.4f}")
+        return make_var_stationary((beta / max_eig) * 0.7, radius)
+    else:
+        # print(f"Stationary, beta={str(beta):s}")
+        return beta
+
+
+def simulate_var(p, T, lag, sparsity=0.2, beta_value=1.0, auto_corr=3.0, sd=0.1, seed=0):
+    if seed is not None:
+        np.random.seed(seed)
+
+    # Set up coefficients and Granger causality ground truth.
+    GC = np.eye(p, dtype=int)
+    beta = np.eye(p) * auto_corr
+
+    num_nonzero = int(p * sparsity) - 1
+    for i in range(p):
+        choice = np.random.choice(p - 1, size=num_nonzero, replace=False)
+        choice[choice >= i] += 1
+        beta[i, choice] = beta_value
+        GC[i, choice] = 1
+
+    beta = np.hstack([beta for _ in range(lag)])
+    beta = make_var_stationary(beta)
+
+    # Generate data.
+    burn_in = 100
+    errors = np.random.normal(loc=0, scale=sd, size=(p, T + burn_in))
+    X = np.ones((p, T + burn_in))
+    X[:, :lag] = errors[:, :lag]
+    for t in range(lag, T + burn_in):
+        X[:, t] = np.dot(beta, X[:, (t-lag):t].flatten(order='F'))
+        X[:, t] += errors[:, t-1]
+        
+    data = X.T[burn_in:, :]
+    return data, beta, GC
+
+
+
+
+
+def lorenz(x, t, F):
+    '''Partial derivatives for Lorenz-96 ODE.'''
+    p = len(x)
+    dxdt = np.zeros(p)
+    for i in range(p):
+        dxdt[i] = (x[(i+1) % p] - x[(i-2) % p]) * x[(i-1) % p] - x[i] + F
+
+    return dxdt
+
+
+def simulate_lorenz_96(p, T, F=10.0, delta_t=0.1, sd=0.1, burn_in=1000,
+                       seed=0):
+    if seed is not None:
+        np.random.seed(seed)
+
+    # Use scipy to solve ODE.
+    x0 = np.random.normal(scale=0.01, size=p)
+    t = np.linspace(0, (T + burn_in) * delta_t, T + burn_in)
+    X = odeint(lorenz, x0, t, args=(F,))
+    X += np.random.normal(scale=sd, size=(T + burn_in, p))
+
+    # Set up Granger causality ground truth.
+    GC = np.zeros((p, p), dtype=int)
+    for i in range(p):
+        GC[i, i] = 1
+        GC[i, (i + 1) % p] = 1
+        GC[i, (i - 1) % p] = 1
+        GC[i, (i - 2) % p] = 1
+
+    return X[burn_in:, :], GC
+

singular_ticker_causal/algorithms/CUTS_PLUS/data/utils.py

@@ -0,0 +1,136 @@
+import numpy as np
+import pandas as pd
+
+from sklearn.metrics.pairwise import haversine_distances
+
+
+def compute_mean(x, index=None):
+    """Compute the mean values for each datetime. The mean is first computed hourly over the week of the year.
+    Further NaN values are computed using hourly mean over the same month through the years. If other NaN are present,
+    they are removed using the mean of the sole hours. Hoping reasonably that there is at least a non-NaN entry of the
+    same hour of the NaN datetime in all the dataset."""
+    if isinstance(x, np.ndarray) and index is not None:
+        shape = x.shape
+        x = x.reshape((shape[0], -1))
+        df_mean = pd.DataFrame(x, index=index)
+    else:
+        df_mean = x.copy()
+    cond0 = [df_mean.index.year, df_mean.index.isocalendar().week, df_mean.index.hour]
+    cond1 = [df_mean.index.year, df_mean.index.month, df_mean.index.hour]
+    conditions = [cond0, cond1, cond1[1:], cond1[2:]]
+    while df_mean.isna().values.sum() and len(conditions):
+        nan_mean = df_mean.groupby(conditions[0]).transform(np.nanmean)
+        df_mean = df_mean.fillna(nan_mean)
+        conditions = conditions[1:]
+    if df_mean.isna().values.sum():
+        df_mean = df_mean.fillna(method='ffill')
+        df_mean = df_mean.fillna(method='bfill')
+    if isinstance(x, np.ndarray):
+        df_mean = df_mean.values.reshape(shape)
+    return df_mean
+
+
+def geographical_distance(x=None, to_rad=True):
+    """
+    Compute the as-the-crow-flies distance between every pair of samples in `x`. The first dimension of each point is
+    assumed to be the latitude, the second is the longitude. The inputs is assumed to be in degrees. If it is not the
+    case, `to_rad` must be set to False. The dimension of the data must be 2.
+
+    Parameters
+    ----------
+    x : pd.DataFrame or np.ndarray
+        array_like structure of shape (n_samples_2, 2).
+    to_rad : bool
+        whether to convert inputs to radians (provided that they are in degrees).
+
+    Returns
+    -------
+    distances :
+        The distance between the points in kilometers.
+    """
+    _AVG_EARTH_RADIUS_KM = 6371.0088
+
+    # Extract values of X if it is a DataFrame, else assume it is 2-dim array of lat-lon pairs
+    latlon_pairs = x.values if isinstance(x, pd.DataFrame) else x
+
+    # If the input values are in degrees, convert them in radians
+    if to_rad:
+        latlon_pairs = np.vectorize(np.radians)(latlon_pairs)
+
+    distances = haversine_distances(latlon_pairs) * _AVG_EARTH_RADIUS_KM
+
+    # Cast response
+    if isinstance(x, pd.DataFrame):
+        res = pd.DataFrame(distances, x.index, x.index)
+    else:
+        res = distances
+
+    return res
+
+
+def infer_mask(df, infer_from='next'):
+    """Infer evaluation mask from DataFrame. In the evaluation mask a value is 1 if it is present in the DataFrame and
+    absent in the `infer_from` month.
+
+    @param pd.DataFrame df: the DataFrame.
+    @param str infer_from: denotes from which month the evaluation value must be inferred.
+    Can be either `previous` or `next`.
+    @return: pd.DataFrame eval_mask: the evaluation mask for the DataFrame
+    """
+    mask = (~df.isna()).astype('uint8')
+    eval_mask = pd.DataFrame(index=mask.index, columns=mask.columns, data=0).astype('uint8')
+    if infer_from == 'previous':
+        offset = -1
+    elif infer_from == 'next':
+        offset = 1
+    else:
+        raise ValueError('infer_from can only be one of %s' % ['previous', 'next'])
+    months = sorted(set(zip(mask.index.year, mask.index.month)))
+    length = len(months)
+    for i in range(length):
+        j = (i + offset) % length
+        year_i, month_i = months[i]
+        year_j, month_j = months[j]
+        mask_j = mask[(mask.index.year == year_j) & (mask.index.month == month_j)]
+        mask_i = mask_j.shift(1, pd.DateOffset(months=12 * (year_i - year_j) + (month_i - month_j)))
+        mask_i = mask_i[~mask_i.index.duplicated(keep='first')]
+        mask_i = mask_i[np.in1d(mask_i.index, mask.index)]
+        eval_mask.loc[mask_i.index] = ~mask_i.loc[mask_i.index] & mask.loc[mask_i.index]
+    return eval_mask
+
+
+def disjoint_months(dataset, months=None):
+    idxs = np.arange(len(dataset))
+    months = ensure_list(months)
+    # divide indices according to window or horizon
+    start, end = 0, dataset.window - 1
+    # after idxs
+    start_in_months = np.in1d(dataset.index[dataset._indices + start].month, months)
+    end_in_months = np.in1d(dataset.index[dataset._indices + end].month, months)
+    idxs_in_months = start_in_months & end_in_months
+    after_idxs = idxs[idxs_in_months]
+    # previous idxs
+    months = np.setdiff1d(np.arange(1, 13), months)
+    start_in_months = np.in1d(dataset.index[dataset._indices + start].month, months)
+    end_in_months = np.in1d(dataset.index[dataset._indices + end].month, months)
+    idxs_in_months = start_in_months & end_in_months
+    prev_idxs = idxs[idxs_in_months]
+    return prev_idxs, after_idxs
+
+
+def thresholded_gaussian_kernel(x, theta=None, threshold=None, threshold_on_input=False):
+    if theta is None:
+        theta = np.std(x)
+    weights = np.exp(-np.square(x / theta))
+    if threshold is not None:
+        mask = x > threshold if threshold_on_input else weights < threshold
+        weights[mask] = 0.
+    return weights
+
+
+def ensure_list(obj):
+    if isinstance(obj, (list, tuple)):
+        return list(obj)
+    else:
+        return [obj]
+

singular_ticker_causal/algorithms/CUTS_PLUS/lagged_graph.py

@@ -0,0 +1,95 @@
+from __future__ import annotations
+
+from copy import deepcopy
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+
+from .cuts_plus import main as cuts_plus_main
+
+
+@dataclass(frozen=True)
+class LaggedGraphBundle:
+    graphs: np.ndarray
+    summary_graph: np.ndarray
+    priors: np.ndarray
+    input_steps: np.ndarray
+
+
+def broadcast_prior_to_lags(G_prior: np.ndarray | None, num_lags: int) -> np.ndarray | None:
+    if G_prior is None:
+        return None
+    G_prior = np.asarray(G_prior, dtype=np.float32)
+    if G_prior.ndim == 2:
+        return np.repeat(G_prior[None, :, :], num_lags, axis=0)
+    if G_prior.ndim == 3 and G_prior.shape[0] == num_lags:
+        return G_prior
+    raise ValueError("G_prior must have shape (N, N) or (L, N, N)")
+
+
+def aggregate_lagged_graphs(graphs: np.ndarray, reducer: str = "max") -> np.ndarray:
+    if reducer == "max":
+        return np.max(graphs, axis=0)
+    if reducer == "mean":
+        return np.mean(graphs, axis=0)
+    if reducer == "last":
+        return graphs[-1]
+    raise ValueError(f"Unsupported lagged graph reducer: {reducer}")
+
+
+def discover_lagged_graphs(
+    data: np.ndarray,
+    mask: np.ndarray,
+    opt: Any,
+    log: Any,
+    device: str = "cpu",
+    text_data: np.ndarray | None = None,
+    text_mask: np.ndarray | None = None,
+    G_prior: np.ndarray | None = None,
+    num_lags: int = 1,
+    reducer: str = "max",
+) -> LaggedGraphBundle:
+    if num_lags < 1:
+        raise ValueError("num_lags must be >= 1")
+
+    lag_priors = broadcast_prior_to_lags(G_prior, num_lags)
+    graphs = []
+    input_steps = []
+    previous_graph = None
+
+    for lag_idx in range(num_lags):
+        cfg = deepcopy(opt)
+        cfg.input_step = max(int(opt.input_step), lag_idx + 1)
+        input_steps.append(cfg.input_step)
+
+        current_prior = None
+        if lag_priors is not None:
+            current_prior = lag_priors[lag_idx]
+        elif previous_graph is not None:
+            current_prior = previous_graph
+
+        graph = cuts_plus_main(
+            data=data,
+            mask=mask,
+            true_cm=None,
+            opt=cfg,
+            log=log,
+            device=device,
+            text_data=text_data,
+            text_mask=text_mask,
+            G_prior=current_prior,
+        )
+        previous_graph = graph
+        graphs.append(np.asarray(graph, dtype=np.float32))
+
+    stacked = np.stack(graphs, axis=0)
+    summary = aggregate_lagged_graphs(stacked, reducer=reducer)
+    if lag_priors is None:
+        lag_priors = np.zeros_like(stacked)
+    return LaggedGraphBundle(
+        graphs=stacked,
+        summary_graph=summary.astype(np.float32),
+        priors=lag_priors.astype(np.float32),
+        input_steps=np.asarray(input_steps, dtype=np.int32),
+    )

singular_ticker_causal/algorithms/CUTS_PLUS/model/cuts_plus_net.py

@@ -0,0 +1,153 @@
+import torch
+from einops import rearrange
+from torch import nn
+
+class GRUCell(nn.Module):
+
+    def __init__(self, d_in, num_units, n_nodes, concat_h=False, activation='tanh'):
+        super(GRUCell, self).__init__()
+        self.activation_fn = getattr(torch, activation)
+
+        mpnn_channel = d_in*n_nodes+num_units if concat_h else d_in*n_nodes
+        self.forget_gate = MPNN(c_in=mpnn_channel, c_out=num_units, concat_h=concat_h)
+        self.update_gate = MPNN(c_in=mpnn_channel, c_out=num_units, concat_h=concat_h)
+        self.c_gate = MPNN(c_in=mpnn_channel, c_out=num_units, concat_h=concat_h)
+
+    def forward(self, x, h, adj):
+        """
+        :param x: (B, input_dim, num_nodes)
+        :param h: (B, num_units, num_nodes)
+        :param adj: (num_nodes, num_nodes)
+        :return:
+        """
+        # we start with bias 1.0 to not reset and not update
+        r = torch.sigmoid(self.forget_gate(x, h, adj))
+        u = torch.sigmoid(self.update_gate(x, h, adj))
+        c = self.c_gate(x, r * h, adj)  # batch_size, self._num_nodes * output_size
+        c = self.activation_fn(c)
+        return u * h + (1. - u) * c
+
+
+class MPNN(nn.Module):
+    def __init__(self, c_in, c_out, concat_h=True):
+        super(MPNN, self).__init__()
+        self.concat_h = concat_h
+        self.mlp = nn.Conv1d(c_in, c_out, kernel_size=1)
+        
+    def forward(self, x, h, graph):
+        b, c, n = x.shape
+        
+        x_repeat = x[:, :, :, None].expand(-1, -1, -1, n) # [b, c, n, n]
+        # graph = rearrange(graph, 'b n m -> b m n')
+        x_messages = torch.einsum('bcmn,bmn->bcmn', (x_repeat, graph))
+        x_messages = rearrange(x_messages, 'b c m n -> b (c m) n')
+        
+        if self.concat_h:
+            out = self.mlp(torch.cat([x_messages, h], dim=1))
+        else:
+            out = self.mlp(x_messages)
+        return out
+
+
+class LocalConv1D(nn.Module):
+    def __init__(self, in_channels, out_channels, kernel_size, n_nodes):
+        super(LocalConv1D, self).__init__()
+        self.out_channel = out_channels
+        self.conv_list = nn.ModuleList([
+            nn.Conv1d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size) for _ in range(n_nodes)
+        ])
+    
+    def forward(self, x): # x: [batch, features, nodes]
+        b, h, n = x.shape
+        out = torch.zeros((b, self.out_channel, n)).to(x.device)
+        for i in range(n):
+            x_local_in = x[..., i].unsqueeze(-1)
+            x_local_out = self.conv_list[i](x_local_in)
+            out[..., i] = x_local_out.squeeze(-1)
+        return out
+
+
+class CUTS_Plus_Net(nn.Module):
+    def __init__(self, n_nodes,
+                 in_ch=1,
+                 hidden_ch=32,
+                 n_layers=1,
+                 shared_weights_decoder=False,
+                 concat_h=False,):
+        super().__init__()
+        self.in_ch = in_ch
+        self.hidden_ch = hidden_ch
+        self.n_layers = n_layers
+        
+        self.conv_encoder1 = nn.Conv1d(in_channels=hidden_ch, out_channels=hidden_ch, kernel_size=1)
+        self.conv_encoder2 = nn.Conv1d(in_channels=2*hidden_ch, out_channels=hidden_ch, kernel_size=1)
+        if shared_weights_decoder:
+            self.decoder = nn.Sequential(
+                nn.Conv1d(in_channels=2*hidden_ch, out_channels=in_ch, kernel_size=1),
+                # nn.LeakyReLU(),
+                # nn.Conv1d(in_channels=hidden_ch, out_channels=hidden_ch, kernel_size=1),
+                # nn.LeakyReLU(),
+                # nn.Conv1d(in_channels=hidden_ch, out_channels=in_ch, kernel_size=1),
+                # nn.LeakyReLU(),
+            )
+        else:
+            self.decoder = nn.Sequential(
+                LocalConv1D(in_channels=2*hidden_ch, out_channels=in_ch, kernel_size=1, n_nodes=n_nodes),
+                # nn.LeakyReLU(),
+                # LocalConv1D(in_channels=hidden_ch, out_channels=hidden_ch, kernel_size=1, n_nodes=n_nodes),
+                # nn.LeakyReLU(),
+                # LocalConv1D(in_channels=hidden_ch, out_channels=in_ch, kernel_size=1, n_nodes=n_nodes),
+                # nn.LeakyReLU(),
+            )
+        # self.act = nn.PReLU()
+        self.act = nn.LeakyReLU()
+        
+        self.cells = nn.ModuleList()
+        for i in range(self.n_layers):
+            self.cells.append(GRUCell(d_in=in_ch if i==0 else hidden_ch, 
+                                      num_units=hidden_ch, 
+                                      n_nodes=n_nodes,
+                                      concat_h=concat_h))
+                
+        self.h0 = self.init_state(n_nodes)
+    
+    def init_state(self, n_nodes):
+        h = []
+        for layer in range(self.n_layers):
+            h.append(nn.parameter.Parameter(torch.zeros([self.hidden_ch, n_nodes])))
+        return nn.ParameterList(h)
+    
+    def update_state(self, x, h, graph):
+        rnn_in = x
+        for layer in range(self.n_layers):
+            rnn_in = h[layer] = self.cells[layer](rnn_in, h[layer], graph)
+        return h
+    
+    def forward(self, x, mask, fwd_graph):
+        x = rearrange(x, 'b n s c -> b c n s')
+        # fwd_graph = torch.ones_like(fwd_graph)
+        # mask = torch.ones_like(x).byte()
+        bs, in_ch, n_nodes, steps = x.shape
+        
+        h = [h_.expand(bs, -1, -1) for h_ in self.h0.to(x.device)]
+        
+        pred = []
+        for step in range(steps):
+            x_now = x[..., step] # [batches, in_ch, nodes]
+            
+            """Update state"""
+            h = self.update_state(x_now, h, fwd_graph)
+            h_now = h[-1]
+            
+            """Prediction"""
+            x_repr = self.act(self.conv_encoder1(h_now)) # [batches, hidden_ch, nodes]
+            x_repr = self.act(self.conv_encoder2(torch.cat([x_repr, h_now], dim=1))) # [batches, hidden_ch, nodes]
+            x_repr = torch.cat([x_repr, h_now], dim=1) # [batches, 2*hidden_ch, nodes]
+            x_hat2 = self.decoder(x_repr) # [batches, in_ch, nodes]
+            pred.append(x_hat2)
+            
+        
+        pred = torch.stack(pred, dim=-1)
+        pred = rearrange(pred, 'b c n s -> b n s c')
+        return pred[:, :, -1:]
+

singular_ticker_causal/algorithms/CUTS_PLUS/utils/causal_plot.py

@@ -0,0 +1,57 @@
+from re import X
+import matplotlib
+matplotlib.use('Agg')
+import matplotlib.pyplot as plt
+import tqdm
+import numpy as np
+import matplotlib.cm as cm
+
+def save_causal_graph(save_path, causal_matrix: np.ndarray, thres_percentile=100, colormap="gnuplot2"):
+    causal_matrix = np.max(causal_matrix)
+    print(causal_matrix.shape)
+    
+    n_node = causal_matrix.shape[0]
+    image_size = [100, 100]
+    n_node_dim = n_node**0.5
+    causal_thres = np.percentile(causal_matrix, 100-thres_percentile)
+    
+    colormap = cm.get_cmap(colormap)
+
+    plt.figure(figsize=[10,10], facecolor='black', edgecolor='black')
+
+        
+    for node_i_from in tqdm.tqdm(range(n_node)):
+        x_from = image_size[0] // n_node_dim * (node_i_from // n_node_dim + 0.5)
+        y_from = image_size[0] // n_node_dim * (node_i_from % n_node_dim + 0.5)
+        plt.text(y=x_from, x=y_from, s=f"{node_i_from:d}", size=20, color="#ffffff")
+        for node_i_to in range(n_node):
+            if not node_i_from == node_i_to:
+                x_to = image_size[0] // n_node_dim * (node_i_to // n_node_dim + 0.5)
+                y_to = image_size[0] // n_node_dim * (node_i_to % n_node_dim + 0.5)
+                causal_effect = causal_matrix[node_i_from, node_i_to]
+                if causal_effect > causal_thres:
+                    width = max(0.01, 1*causal_effect)
+                    arrow_length = ((x_to-x_from)**2 + (y_to-y_from)**2)**0.5
+                    plt.arrow(
+                        y=x_from+(x_to-x_from)*width/arrow_length,
+                        x=y_from+(y_to-y_from)*width/arrow_length,
+                        dy=(x_to-x_from)*(arrow_length-5*width)/arrow_length,
+                        dx=(y_to-y_from)*(arrow_length-5*width)/arrow_length,
+                        width=width,
+                        head_length=4*width,
+                        facecolor=colormap(causal_effect)[:3]+(causal_effect,),
+                        edgecolor="#00000000"
+                    )
+        
+        # fig.add_annotation(text=f"{node_i_from:d}", x=x_from, y=y_from, showarrow=False
+    
+    ax=plt.gca()
+    ax.patch.set_facecolor("black")
+    ax.xaxis.set_ticks_position('top') 
+    ax.invert_yaxis()
+    plt.savefig(save_path)
+    
+    
+if __name__=="__main__":
+    causal_matrix = np.load("outputs/tsgae_2022_0716_191203_262072/w.npy")
+    save_causal_graph("outputs/pic/causal.png", causal_matrix, thres_percentile=100)

singular_ticker_causal/algorithms/CUTS_PLUS/utils/exp_utils.py

@@ -0,0 +1,286 @@
+import os
+from os.path import join as opj
+from os.path import dirname as opd
+
+from .opt_type import MultiCADopt
+from .misc import omegaconf2dict
+
+import re
+import numpy as np
+import matplotlib
+matplotlib.use('Agg')
+import matplotlib.pyplot as plt
+
+from datetime import datetime
+from omegaconf import OmegaConf
+import glob
+import tqdm
+from tensorboard.backend.event_processing import event_accumulator
+
+PROPER_NAME = {"NFGR":"BRIEF", "h265":"H.265", "h264":"H.264", "jpg":"JPEG", "aoi-2000":"AoI", "vvc":"H.266"}
+
+MAIN_COLOR = {"NFGR":"#e64b35", "h265":"#48c9b0", "h264":"#5599c7", "jpg":"#c39bd2", "DVC":"#e6b0aa", "SGA":"#f8c370", "vvc":"#b1babb",
+              "GOOD":"#239954", "BAD":"#cc6155"}
+SECONDARY_COLOR = {"NFGR":"#d98880", "h265":"#76d7c3", "SGA":"#f9d7a0", "h264":"#7fb3d5", "DVC":"#e6b0aa"}
+
+
+def get_time_stamp():
+    return str(datetime.now().strftime("%m-%d-%H%M%S-%f"))
+
+def show_single_scores(x_arr, y_arr, label="exp", suffix="", 
+                       scatter=False, log_axis=True, 
+                       xlim=None, ylim=None):
+    
+    '''Plotting figures'''
+    cvs = FigureCanvas(name=label, figsize=[30,20])
+    fig_idx = 0
+    for idx,data_path in enumerate(y_arr.labels[2]): # range(score_arr.shape[2]):
+        if fig_idx == 40:
+            break
+        elif np.isnan(np.nanmean(y_arr[:,:,data_path])):
+            continue
+        
+        fig_idx += 1
+        if log_axis:
+            ax = plt.subplot(8,5,fig_idx, xscale="log")
+        else:
+            ax = plt.subplot(8,5,fig_idx)
+        ax.set_title("DATA_{:02d}_".format(idx) + data_path[-100:-60] + "\n" + data_path[-60:])
+        # ax.set_title(dim_marks[2][data_i])
+        plt.set_cmap("rainbow")
+        for i, dim0 in enumerate(y_arr.labels[0]):
+            x_nan = x_arr[dim0,:,data_path]
+            y_nan = y_arr[dim0,:,data_path]
+            x = x_nan[np.isfinite(x_nan + y_nan)]
+            y = y_nan[np.isfinite(x_nan + y_nan)]
+            
+            if len(x) > 0 and len(y) > 0:
+                x, y = sort_lists(x, y)
+                
+                plt.plot(x, y, color=plt.get_cmap("tab20")(i), label=dim0)
+                plt.scatter(x, y, color=plt.get_cmap("tab20")(i))
+            
+        plt.legend()
+        if ylim is not None:
+            plt.ylim(ylim)
+    
+    ax=plt.gca()
+    ax.xaxis.set_major_locator(plt.LogLocator(base=10, numticks=5))
+    ax.yaxis.set_major_locator(plt.MaxNLocator(5))
+            
+    cvs.save_fig(suffix=suffix, time_stamp=False, save_format=".pdf")
+
+
+def sort_lists(*lists):
+    sorted_index = np.argsort(lists[0]).astype(int)
+    results = []
+    for l in lists:
+        sorted_list = np.array([l[i] for i in sorted_index])
+        results.append(sorted_list)
+    return results
+
+
+def show_averge_scores(x_arr, y_arr, label="exp", suffix="", percentile=25,
+                       scatter=False, log_axis=True, figsize=[4,3],
+                       xlim=None, ylim=None, legend=False, grid=False, std=False):
+    cvs = FigureCanvas(name=label, figsize=figsize)
+    if log_axis:
+        plt.xscale("log")
+        
+    fig_idx = 0
+    if scatter:
+        for data_i,data_path in enumerate(y_arr.labels[2]):
+            if np.isnan(np.nanmean(y_arr[:,:,data_path])):
+                continue
+            
+            for i, dim0 in enumerate(y_arr.labels[0]):
+                x_nan = x_arr[dim0,:,data_path]
+                y_nan = y_arr[dim0,:,data_path]
+                x = x_nan[np.isfinite(x_nan + y_nan)]
+                y = y_nan[np.isfinite(x_nan + y_nan)]
+                
+                if len(x) > 0 and len(y) > 0:
+                    x, y = sort_lists(x, y)
+                    
+                    # plt.plot(x, y, color=plt.get_cmap("tab20")(i), label=dim0, marker="v")
+                    plt.scatter(x, y, color=MAIN_COLOR[dim0], alpha=0.5, marker="v", edgecolors='none',
+                                s=80 if "NFGR" in dim0 else 50)
+                
+            # plt.legend(edgecolors='none')
+            if ylim is not None:
+                plt.ylim(ylim)
+                
+    if ylim is not None:
+        full_range = ylim[1] - ylim[0]  
+    else:
+        full_range = np.nanmax(np.nanmean(y_arr.arr, axis=2)) - np.nanmin(np.nanmean(y_arr.arr, axis=2))
+    max_std = np.nanmax(np.nanstd(y_arr["NFGR",:,:], axis=1))
+            
+        
+    for idx, dim0 in enumerate(sorted(y_arr.labels[0], key=lambda item:item == "NFGR")):
+        # if "jpg" in dim0:
+        #     continue
+        
+        x_nan = np.nanmean(x_arr[dim0,:], axis=1)
+        y_nan = np.nanmean(y_arr[dim0,:], axis=1)
+        # y_l = np.nanpercentile(y_arr[dim0,:], percentile, axis=1)
+        # y_u = np.nanpercentile(y_arr[dim0,:], 100-percentile, axis=1)
+        y_std = np.nanstd(y_arr[dim0,:], axis=1) # / max_std * full_range * 0.07
+        
+        print(1 / max_std * full_range * 0.07)
+        
+        x = x_nan[np.isfinite(x_nan + y_nan)]
+        y = y_nan[np.isfinite(x_nan + y_nan)]
+        # y_l = y_l[np.isfinite(x_nan + y_nan)]
+        # y_u = y_u[np.isfinite(x_nan + y_nan)]
+        y_std = y_std[np.isfinite(x_nan + y_nan)]
+        
+        x, y, y_std = sort_lists(x, y, y_std)
+        
+        if std:
+            plt.fill_between(x, y-y_std/2, y+y_std/2, color=MAIN_COLOR[dim0], alpha=0.25, edgecolors="none")
+        plt.plot(x, y, color=MAIN_COLOR[dim0], label=name(dim0), 
+                 lw=2 if "NFGR" in dim0 else 1.5)
+        plt.scatter(x, y, color=MAIN_COLOR[dim0], 
+                 s=30 if "NFGR" in dim0 else 20)
+        
+    if ylim is not None:
+        plt.ylim(ylim)
+    if xlim is not None:
+        plt.xlim(xlim)
+    if legend:
+        plt.legend(loc='lower left', bbox_to_anchor=(0.1, 0.1), fancybox=False)
+    
+    ax=plt.gca()
+    if log_axis:
+        ax.xaxis.set_major_locator(plt.LogLocator(base=10, numticks=5))
+    else:
+        ax.xaxis.set_major_locator(plt.MaxNLocator(5))
+    ax.yaxis.set_major_locator(plt.MaxNLocator(5))
+    
+    if grid:
+        ax.spines['right'].set_visible(True)
+        ax.spines['top'].set_visible(True)
+        # plt.xticks(np.arange(0.4, 1.8, 0.28))
+        # plt.yticks(np.arange(50, 500, 40))
+        plt.grid(axis='both', c="#cacaca", which="major")
+    
+    cvs.save_fig(suffix=suffix, time_stamp=False, save_format=".pdf")
+    
+    
+def name(alias):
+    if alias in PROPER_NAME:
+        return PROPER_NAME[alias]
+    else:
+        print("Cannot find proper name.")
+        return alias
+
+class FigureCanvas(object):
+    
+    def __init__(self, name="ex1", figsize=[14,9]):
+        self.name = name
+        plt.close('all')
+        fig = plt.figure(figsize=figsize)
+        ax = plt.axes()
+        ax.spines['right'].set_visible(False)
+        ax.spines['top'].set_visible(False)
+        plt.tight_layout()
+    
+    def show_fig(self, save_format=".png", suffix="", time_stamp=True):
+        save_path = "./exp/figs/%s/%s_%s%s"%(
+            self.name, 
+            get_time_stamp() if time_stamp else "plt", 
+            suffix, save_format)
+        if not os.path.exists(opd(save_path)):
+            os.makedirs(opd(save_path))
+        plt.savefig(save_path, bbox_inches='tight')
+        plt.show()
+        
+    def save_fig(self, save_format=".png", suffix="", time_stamp=True, save_root="./exp/figs/"):
+        save_path = opj(save_root, "%s/%s_%s%s"%(
+            self.name, 
+            get_time_stamp() if time_stamp else "plt", 
+            suffix, save_format))
+        if not os.path.exists(opd(save_path)):
+            os.makedirs(opd(save_path))
+        plt.savefig(save_path, bbox_inches='tight')
+
+
+def find_lineprofile_cmp(im_list):    
+    for x in range(0, im_list[0].shape[0], 10):
+        for y in range(0, im_list[0].shape[1], 10):
+            lp = [im[x][y] for im in im_list]
+            if np.max(lp) > 2500 and np.max(lp) < 3000:
+                return lp
+
+
+def get_decompressed_path(opt_path):
+    res_root = opd(opt_path)
+    max_step = 0
+    for dirn in os.listdir(res_root):
+        if dirn == "decompressed":
+            return glob.glob(res_root + "/decompressed/*.tif")[0]
+        elif "steps" in dirn:
+            step_n = int(dirn[5:])
+            if step_n > max_step:
+                max_step = step_n
+    search_list = glob.glob(res_root + "/steps" + str(max_step) + "/decompressed/*.*")
+    if len(search_list) > 0:
+        return search_list[0]
+    return None
+
+def load_scalars(event_path):
+    try:
+        event_path = glob.glob(event_path)[0]
+    except:
+        print("No event file found.")
+        return None
+    ea = event_accumulator.EventAccumulator(event_path)
+    ea.Reload()
+    # print("Available scalars: ", ea.scalars.Keys())
+    scalars = {}
+    for criterion in ea.scalars.Keys():
+        val_scalar = ea.scalars.Items(criterion)
+        val_curve = ([(i.step, i.value) for i in val_scalar])
+        scalars[criterion] = val_curve
+    return scalars
+    
+def load_scalars_cached(root_path, cache_dir="exp/cache", reload_data=False):
+    
+    if root_path[-1] == "/":
+        root_path = root_path[:-1]
+    
+    # csv_file = glob.glob(root_path + "/*.csv")[0]
+    # exp_list = load_csv(csv_file)
+    root_name = "".join(re.split("/|\\\\", root_path)[-2:])
+    cache_path = opj(cache_dir, root_name + ".npy")
+    
+    # read from cached data
+    if os.path.exists(cache_path) and not reload_data:
+        print("Loading cached result...")
+        loaded_res = np.load(cache_path, allow_pickle=True)
+    else:
+        if not os.path.exists(cache_dir):
+            os.makedirs(cache_dir)
+        
+        loaded_res = []
+        for dirn in tqdm.tqdm(os.listdir(root_path)):
+            if os.path.isdir(opj(root_path, dirn)):
+                if len(glob.glob(root_path + "/%s/events.out.tfevents*"%(dirn))) > 0:
+                    res_fname = glob.glob(root_path + "/%s/events.out.tfevents*"%(dirn))[0]
+                    opt_fname = glob.glob(root_path + "/%s/opt.yaml"%(dirn))[0]
+                    scores = load_scalars(res_fname)
+                    opt: MultiCADopt = omegaconf2dict(OmegaConf.load(opt_fname), sep=".")
+                    loaded_res.append((res_fname, scores, opt))
+        
+        np.save(cache_path, loaded_res)
+    
+    return loaded_res
+
+
+# if __name__=="__main__":
+#     # scalars = load_scalars("cyx_exp/experiments_outputs/ex2_1227/*/exp_00000/events.out.tfevents*")
+#     # print(scalars)
+    
+#     res = load_scalars_cached("cyx_exp/experiments_outputs/ex2_1227/", reload_data=False)
+#     print("")

singular_ticker_causal/algorithms/CUTS_PLUS/utils/gumbel_softmax.py

@@ -0,0 +1,66 @@
+import torch
+from torch import Tensor
+import warnings
+
+def gumbel_softmax(logits: Tensor, tau: float = 1, hard: bool = False, eps: float = 1e-10, dim: int = -1) -> Tensor:
+    r"""
+    Samples from the Gumbel-Softmax distribution (`Link 1`_  `Link 2`_) and optionally discretizes.
+
+    Args:
+      logits: `[..., num_features]` unnormalized log probabilities
+      tau: non-negative scalar temperature
+      hard: if ``True``, the returned samples will be discretized as one-hot vectors,
+            but will be differentiated as if it is the soft sample in autograd
+      dim (int): A dimension along which softmax will be computed. Default: -1.
+
+    Returns:
+      Sampled tensor of same shape as `logits` from the Gumbel-Softmax distribution.
+      If ``hard=True``, the returned samples will be one-hot, otherwise they will
+      be probability distributions that sum to 1 across `dim`.
+
+    .. note::
+      This function is here for legacy reasons, may be removed from nn.Functional in the future.
+
+    .. note::
+      The main trick for `hard` is to do  `y_hard - y_soft.detach() + y_soft`
+
+      It achieves two things:
+      - makes the output value exactly one-hot
+      (since we add then subtract y_soft value)
+      - makes the gradient equal to y_soft gradient
+      (since we strip all other gradients)
+
+    Examples::
+        >>> logits = torch.randn(20, 32)
+        >>> # Sample soft categorical using reparametrization trick:
+        >>> F.gumbel_softmax(logits, tau=1, hard=False)
+        >>> # Sample hard categorical using "Straight-through" trick:
+        >>> F.gumbel_softmax(logits, tau=1, hard=True)
+
+    .. _Link 1:
+        https://arxiv.org/abs/1611.00712
+    .. _Link 2:
+        https://arxiv.org/abs/1611.01144
+    """
+    if eps != 1e-10:
+        warnings.warn("`eps` parameter is deprecated and has no effect.")
+
+    gumbels = (
+        -torch.empty_like(logits, memory_format=torch.legacy_contiguous_format).exponential_().log()
+    )  # ~Gumbel(0,1)
+    gumbels = (logits + gumbels) / tau  # ~Gumbel(logits,tau)
+    y_soft = gumbels.softmax(dim)
+
+    if hard:
+        # Straight through.
+        index = y_soft.max(dim, keepdim=True)[1]
+        y_hard = torch.zeros_like(logits, memory_format=torch.legacy_contiguous_format).scatter_(dim, index, 1.0)
+        ret = y_hard - y_soft.detach() + y_soft
+    else:
+        # Reparametrization trick.
+        ret = y_soft
+    return ret
+
+if __name__=="__main__":
+    a = torch.tensor([[2.0, 0.7]]*10)
+    print(gumbel_softmax(a, tau=10))

singular_ticker_causal/algorithms/CUTS_PLUS/utils/logger.py

@@ -0,0 +1,74 @@
+import sys
+from omegaconf import OmegaConf
+import os
+from os.path import join as opj
+import numpy as np
+from os.path import dirname as opd
+from typing import Dict
+from torch.utils.tensorboard import SummaryWriter
+from .misc import omegaconf2list
+
+
+class MyLogger():
+    def __init__(self, log_dir: str, stderr: bool = True, tensorboard: bool = True, stdout: bool = True):
+        self.log_dir = log_dir
+        if not os.path.exists(self.log_dir):
+            os.makedirs(self.log_dir)
+        self.logger_dict: Dict[str] = {}
+        if stdout:
+            stdout_handler = open(opj(self.log_dir, 'stdout.log'), 'w')
+            sys.stdout = stdout_handler
+        if stderr:
+            stderr_handler = open(opj(self.log_dir, 'stderr.log'), 'w')
+            sys.stderr = stderr_handler
+        if tensorboard:
+            self.tblogger = SummaryWriter(self.log_dir)
+            self.logger_dict['tblogger'] = self.tblogger
+
+    def log_opt(self, opt):
+        OmegaConf.save(config=opt, f=opj(self.log_dir, 'opt.yaml'))
+        opt_log = omegaconf2list(opt, sep='/')
+        for logger_name in self.logger_dict.keys():
+            if logger_name == 'tblogger':
+                for idx, opt in enumerate(opt_log):
+                    self.logger_dict[logger_name].add_text('hparam', opt, idx)
+
+    def log_metrics(self, metrics_dict: Dict[str, float], iters):
+        for logger_name in self.logger_dict.keys():
+            if logger_name == 'csvlogger':
+                self.logger_dict[logger_name].log_metrics(metrics_dict, iters)
+                self.logger_dict[logger_name].save()
+            elif logger_name == 'clearml_logger':
+                for k in metrics_dict.keys():
+                    self.logger_dict[logger_name].report_scalar(
+                        k, k, metrics_dict[k], iters)
+            elif logger_name == 'tblogger':
+                for k in metrics_dict.keys():
+                    self.logger_dict[logger_name].add_scalar(
+                        k, metrics_dict[k], iters)
+    
+    def log_figures(self, figure, name="figure.png", iters=None, exclude_logger=[]):
+        for logger_name in self.logger_dict.keys():
+            if logger_name == 'tblogger':
+                if logger_name not in exclude_logger:
+                    self.logger_dict[logger_name].add_figure(tag=name, figure=figure, global_step=iters)
+        
+        if iters is None:
+            save_path = opj(self.log_dir, "figures")
+        else:
+            save_path = opj(self.log_dir, f"iter_{iters:d}", name)
+        os.makedirs(opd(save_path), exist_ok=True)
+        figure.savefig(save_path)
+        
+    def log_npz(self, data: Dict, name="data.npz", iters=None):
+        if iters is None:
+            save_path = opj(self.log_dir)
+        else:
+            save_path = opj(self.log_dir, f"iter_{iters:d}", name)
+        os.makedirs(save_path, exist_ok=True)
+        np.savez(opj(save_path, "graph.npz"), **data)
+
+    def close(self):
+        for logger_name in self.logger_dict.keys():
+            if logger_name == 'tblogger':
+                self.logger_dict[logger_name].close()

singular_ticker_causal/algorithms/CUTS_PLUS/utils/misc.py

@@ -0,0 +1,292 @@
+from copy import deepcopy
+import numpy as np
+import matplotlib
+matplotlib.use('Agg')
+import matplotlib.pyplot as plt
+import itertools
+import random
+import numpy as np
+import torch
+import omegaconf
+from sklearn.metrics import roc_curve, roc_auc_score
+
+
+def log_time_series(original_data, data_interp, data_pred, log, log_step):
+    fig = plt.figure(figsize=[10,10])
+    plt.plot(np.arange(0, original_data.shape[0], 1), original_data, label="original")
+    plt.plot(np.arange(0, data_interp.shape[0], 1), data_interp, label="interp")
+    plt.plot(np.arange(0, data_pred.shape[0], 1), data_pred, label="pred")
+    plt.legend()
+    log.log_figures(fig, name="Predicted Latent Data", iters=log_step)
+
+
+def calc_and_log_metrics(time_prob_mat, true_cm, log, log_step, threshold=0.5, plot_roc=True):
+    if len(time_prob_mat.shape) == 3:
+        graph = np.max(time_prob_mat, axis=2)
+    else:
+        graph = time_prob_mat
+    causal_graph = graph > threshold
+    tp = np.mean(causal_graph * true_cm)
+    tn = np.mean((1-causal_graph) * (1-causal_graph))
+    fp = np.mean(causal_graph * (1-true_cm))
+    fn = np.mean((1-causal_graph) * true_cm)
+    tpr = tp / (tp + fn)
+    fpr = fp / (fp + tn)
+    acc = (tp + tn) / (tp + tn + fp + fn)
+    log.log_metrics({"metrics/tpr": tpr}, log_step)
+    log.log_metrics({"metrics/fpr": fpr}, log_step)
+    log.log_metrics({"metrics/accuracy": acc}, log_step)
+
+    if plot_roc:
+        fpr, tpr, thres = roc_curve(true_cm.reshape(-1) > 0.5, 
+                                    graph.reshape(-1), pos_label=1)
+        fig = plt.figure(figsize=[4, 4])
+        plt.plot(fpr, tpr)
+        log.tblogger.add_figure(tag="ROC", figure=fig, global_step=log_step)
+        
+        log.log_npz(name="graph", 
+                    data={"true_cm":true_cm, "pred_cm":graph},
+                    iters=log_step)
+
+    auc = roc_auc_score(true_cm.reshape(-1)>0.5,
+                        graph.reshape(-1))
+    log.log_metrics({"metrics/auc": auc}, log_step)
+    return auc
+
+def sigmoid(z):
+    return 1/(1 + np.exp(-z))
+
+def plot_causal_matrix_in_training(time_coef, name, log, log_step, threshold=0.5, plot_each_time=False):
+    if time_coef is None:
+        return
+    
+    if np.max(time_coef) - np.min(time_coef) > 0.01:
+        time_coef = (time_coef - np.min(time_coef)) / (np.max(time_coef) - np.min(time_coef))
+    n, m, t = time_coef.shape
+    
+    # # Show Discovered Graph (Coefficiency)
+    # sub_cg = plot_causal_matrix(
+    #     np.max(time_coef, axis=2),
+    #     figsize=[1.5*time_coef.shape[0], 1*n])
+    # log.log_figures(sub_cg, name="Discovered Graph Coef/" + name, iters=log_step)
+    
+    # # Graph for Each Time Lag
+    # if plot_each_time:
+    #     for ti in range(t):
+    #         sub_cg = plot_causal_matrix(
+    #             time_coef[:, :, ti],
+    #             figsize=[1.5*n, 1*n],
+    #             vmin=0, vmax=1)
+    #         log.log_figures(sub_cg, name=f"Discovered Prob T-{t-ti:d}", 
+    #                         iters=log_step, exclude_logger="tblogger")
+
+    # Show Discovered Graph (Probability)
+    time_graph = time_coef
+    sub_cg = plot_causal_matrix(
+        np.max(time_graph, axis=2),
+        figsize=[1.5*n, 1*n],
+        vmin=0, vmax=1)
+    log.log_figures(sub_cg, name="Discovered Prob/" + name, iters=log_step)
+
+    # Show Thresholded Graph
+    time_thres = np.max(time_graph, axis=2) > threshold
+    sub_cg = plot_causal_matrix(
+        time_thres,
+        figsize=[1.5*n, 1*n])
+    log.log_figures(sub_cg, name="Discovered Graph/" + name, iters=log_step)
+    log.log_npz({"Discovered Graph Coef": time_coef, "Discovered Prob": time_graph, "Discovered Graph": time_thres}, 
+                name="Graph.npz", iters=log_step)
+
+
+def plot_causal_matrix(cmtx, class_names=None, figsize=None, vmin=None, vmax=None, show_text=True, cmap="magma"):
+    """
+    A function to create a colored and labeled causal matrix matplotlib figure
+    given true labels and preds.
+    Args:
+        cmtx (ndarray): causal matrix.
+        num_classes (int): total number of nodes.
+        class_names (Optional[list of strs]): a list of node names.
+        figsize (Optional[float, float]): the figure size of the causal matrix.
+            If None, default to [6.4, 4.8].
+
+    Returns:
+        img (figure): matplotlib figure.
+    """
+    num_classes = cmtx.shape[0]
+    if class_names is None or type(class_names) != list:
+        class_names = [str(i) for i in range(num_classes)]
+
+    
+    figsize[0] = 30 if figsize[0] > 30 else figsize[0]
+    figsize[1] = 20 if figsize[1] > 20 else figsize[1]
+    
+    plt.clf()
+    plt.close("all")
+    figure = plt.figure(figsize=figsize)
+    plt.imshow(cmtx, interpolation="nearest",
+               cmap=cmap, vmin=vmin, vmax=vmax)
+    plt.title("Causal matrix")
+    plt.colorbar()
+    # tick_marks = np.arange(len(class_names))
+    # plt.xticks(tick_marks, class_names, rotation=45)
+    # plt.yticks(tick_marks, class_names)
+
+    # Use white text if squares are dark; otherwise black.
+    threshold = cmtx.max() / 2.0
+    for i, j in itertools.product(range(cmtx.shape[0]), range(cmtx.shape[1])):
+        color = "white" if cmtx[i, j] < threshold else "black"
+        if cmtx.shape[0] < 20 and show_text:
+            plt.text(j, i, format(cmtx[i, j], ".2e") if cmtx[i, j] != 0 else ".",
+                    horizontalalignment="center", color=color,)
+
+    plt.tight_layout()
+    plt.ylabel("True label")
+    plt.xlabel("Predicted label")
+
+    return figure
+
+
+def reproduc(seed, benchmark=False, deterministic=True):
+    """Make experiments reproducible
+    """
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)
+    torch.backends.cudnn.benchmark = benchmark
+    torch.backends.cudnn.deterministic = deterministic
+
+
+def omegaconf2list(opt, prefix='', sep='.'):
+    notation_list = []
+    for k, v in opt.items():
+        k = str(k)
+        if isinstance(v, omegaconf.listconfig.ListConfig):
+            notation_list.append("{}{}={}".format(prefix, k, v))
+            # if k in ['iter_list','step_list']: # do not sparse list
+            #     dot_notation_list.append("{}{}={}".format(prefix,k,v))
+            # else:
+            #     templist = []
+            #     for v_ in v:
+            #         templist.append('{}{}={}'.format(prefix,k,v_))
+            #     dot_notation_list.append(templist)
+        elif isinstance(v, (float, str, int,)):
+            notation_list.append("{}{}={}".format(prefix, k, v))
+        elif v is None:
+            notation_list.append("{}{}=~".format(prefix, k,))
+        elif isinstance(v, omegaconf.dictconfig.DictConfig):
+            nested_flat_list = omegaconf2list(v, prefix + k + sep, sep=sep)
+            if nested_flat_list:
+                notation_list.extend(nested_flat_list)
+        else:
+            raise NotImplementedError
+    return notation_list
+
+
+def omegaconf2dotlist(opt, prefix='',):
+    return omegaconf2list(opt, prefix, sep='.')
+
+
+def omegaconf2dict(opt, sep):
+    notation_list = omegaconf2list(opt, sep=sep)
+    dict = {notation.split('=', maxsplit=1)[0]: notation.split(
+        '=', maxsplit=1)[1] for notation in notation_list}
+    return dict
+
+
+# def read_video(video_path: str):
+#     if ops(video_path)[-1] == ".tif":
+#         data = tifffile.imread(video_path)
+#         data = (data / np.max(data) * 255).astype(np.uint8)
+#         if len(data.shape) == 3:
+#             data = data[:, :, :, None]
+#         return data
+#     else:
+#         cap = cv2.VideoCapture(video_path)
+#         frames = []
+#         while cap.isOpened():
+#             # get a frame
+#             ret, frame = cap.read()
+#             if not ret:
+#                 break
+#             frames.append(np.array(frame)[None])
+
+#         cap.release()
+#         return np.concatenate(frames, axis=0)
+
+
+# def save_video(video_path: str, data):
+#     skvideo.io.vwrite(video_path, data)
+
+
+
+class LabelArray(object):
+    # def __init__(self, array, labels):
+    #     self.arr = array
+    #     self.labels = labels
+    #     self.marks = dim_marks
+    #     assert [len[label_list] for label_list in labels] == self.arr.shape
+        
+    def __init__(self, dim, labels=None):
+        if labels is not None:
+            if len(dim) != dim:
+                raise "The length of labels has to be equal to dim if defined"
+            else:
+                self.labels = deepcopy(labels)
+        else:
+            self.labels = [[] for _ in range(dim)]
+        self.arr = None
+        self.update_arr()
+        
+    def update_arr(self):
+        if self.arr is not None:
+            oldarr = self.arr
+            self.arr = np.zeros([len(dim) for dim in self.labels]) * np.nan
+            self.arr[tuple([slice(0,sh_dim,1) for sh_dim in oldarr.shape])] = oldarr
+        else:
+            self.arr = np.zeros([len(dim) for dim in self.labels]) * np.nan
+        self.shape = self.arr.shape
+        
+        
+    def __getitem__(self, label_list):
+        index_list = []
+        for dim,label in enumerate(label_list):
+            if isinstance(label, str):
+                index_list.append(self.labels[dim].index(label))
+            elif isinstance(label, slice):
+                index_list.append(label)
+            elif isinstance(label, int):
+                index_list.append(label)
+            else:
+                raise NotImplementedError
+            
+        return self.arr[tuple(index_list)]
+    
+    def __setitem__(self, label_list, val):    
+        index_list = []
+        for dim,label in enumerate(label_list):
+            if isinstance(label, str):
+                if not label in self.labels[dim]:
+                    self.labels[dim].append(label)
+                    self.update_arr()
+                index_list.append(self.labels[dim].index(label))
+            elif isinstance(label, slice):
+                index_list.append(label)
+            elif isinstance(label, int):
+                index_list.append(label)
+            else:
+                raise NotImplementedError
+            
+        self.arr[tuple(index_list)] = val
+    
+    def __str__(self):
+        return str(self.arr) + "\n--------------------------\n" + str(self.labels)
+    
+    
+    def to_np(self):
+        return self.arr
+    
+    def from_np(self, array):
+        assert [len[label_list] for label_list in self.labels] == self.arr.shape
+        self.arr = array
+        

singular_ticker_causal/algorithms/CUTS_PLUS/utils/opt_type.py

@@ -0,0 +1,83 @@
+from dataclasses import dataclass
+from typing import Any
+
+@dataclass
+class ReproducOpt:    
+    seed: int
+    benchmark: bool
+    deterministic: bool
+    
+@dataclass
+class NetworkOpt:
+    name: str
+    network_param: Any
+    
+@dataclass
+class TrainOpt:
+  batch_size: int
+  total_epoch: int
+  time_window: int
+    
+    
+@dataclass
+class TsGAEopt:
+    dir_name: str
+    task_name: str
+    optimizer: Any
+    reproduc: ReproducOpt
+    network: NetworkOpt
+    train: TrainOpt
+    log: Any
+    causal_thres: str
+    
+@dataclass
+class MultiCADopt:
+    dir_name: str
+    task_name: str
+    
+    @dataclass
+    class MultiCADargs:
+        n_nodes: int
+        input_step: int
+        window_step: int
+        stride: int
+        batch_size: int
+        sample_per_epoch: int
+        data_dim: int
+        total_epoch: int
+        
+        patience: int
+        warmup: Any
+        
+        show_graph_every: int
+        val_every: int
+        
+        n_groups: int
+        group_policy: Any
+        causal_thres: str
+        
+        @dataclass
+        class data_pred:
+            model: str
+            merge_policy: str
+            lr_data_start: float
+            lr_data_end: float
+            weight_decay: int
+            prob: bool
+
+        @dataclass
+        class graph_discov:
+            lr_graph_start: float
+            lr_graph_end: float
+            lambda_s_start: float
+            lambda_s_end: float
+            tau_start: float
+            tau_end: float
+            disable_bwd: bool
+            separate_bwd: bool
+            disable_ind: bool
+            disable_graph: bool
+            use_true_graph: bool
+    
+    reproduc: ReproducOpt
+    log: Any

singular_ticker_causal/algorithms/__init__.py

@@ -0,0 +1,2 @@
+# CAMEF algorithms subpackage — GPT4MTS and dataloader have been retired.
+# This package is intentionally empty pending removal of the CAMEF directory.

singular_ticker_causal/causal_inference/__init__.py

@@ -0,0 +1,15 @@
+from .causal_model import StructuralCausalModel
+from .identification import IdentificationStrategy, find_adjustment_set, is_identifiable
+from .pywhyllm_assumptions import CausalAssumptionReport, PyWhyLLMConfig, PyWhyLLMAssumptionService
+from .query_engine import CausalQueryEngine
+
+__all__ = [
+    "CausalAssumptionReport",
+    "CausalQueryEngine",
+    "IdentificationStrategy",
+    "PyWhyLLMConfig",
+    "PyWhyLLMAssumptionService",
+    "StructuralCausalModel",
+    "find_adjustment_set",
+    "is_identifiable",
+]

singular_ticker_causal/causal_inference/abduction.py

@@ -0,0 +1,127 @@
+from __future__ import annotations
+
+from typing import Dict
+
+import numpy as np
+
+from .causal_model import StructuralCausalModel
+
+
+def abduct_noise(scm: StructuralCausalModel, observed_values: Dict[str, float]) -> Dict[str, float]:
+    noise: Dict[str, float] = {}
+    for node_idx in scm.topological_indices:
+        node = scm.nodes[node_idx]
+        if node not in observed_values:
+            continue
+        obs = float(observed_values[node])
+        eq = scm.equations[node]
+        if not eq.parents or eq.equation_type == "exogenous":
+            noise[node] = obs - eq.intercept
+            continue
+        parent_vals = []
+        for p in eq.parents:
+            if p not in observed_values:
+                parent_vals = []
+                break
+            parent_vals.append(float(observed_values[p]))
+        if not parent_vals:
+            continue
+        coef_vec = np.array([eq.coefficients[p] for p in eq.parents], dtype=float)
+        pred = eq.intercept + float(np.array(parent_vals, dtype=float) @ coef_vec)
+        noise[node] = obs - pred
+    return noise
+
+
+def counterfactual_predict(
+    scm: StructuralCausalModel,
+    observed_values: Dict[str, float],
+    treatment: str,
+    counterfactual_value: float,
+    target: str,
+) -> Dict[str, float]:
+    if treatment not in scm.node_to_idx:
+        raise ValueError(f"Unknown treatment node: {treatment}")
+    if target not in scm.node_to_idx:
+        raise ValueError(f"Unknown target node: {target}")
+
+    noises = abduct_noise(scm, observed_values)
+    state = scm.data_level[-1].copy()
+    for node, value in observed_values.items():
+        if node in scm.node_to_idx:
+            state[scm.node_to_idx[node]] = float(value)
+
+    cf = state.copy()
+    t_idx = scm.node_to_idx[treatment]
+    cf[t_idx] = float(counterfactual_value)
+
+    for node_idx in scm.topological_indices:
+        node = scm.nodes[node_idx]
+        if node_idx == t_idx:
+            continue
+        eq = scm.equations[node]
+        if not eq.parents or eq.equation_type == "exogenous":
+            cf[node_idx] = eq.intercept + noises.get(node, 0.0)
+            continue
+        parent_vals = cf[eq.parent_indices]
+        coef_vec = np.array([eq.coefficients[p] for p in eq.parents], dtype=float)
+        pred = eq.intercept + float(parent_vals @ coef_vec)
+        cf[node_idx] = pred + noises.get(node, 0.0)
+
+    y_idx = scm.node_to_idx[target]
+    factual = float(state[y_idx])
+    counterfactual = float(cf[y_idx])
+    ite = counterfactual - factual
+    pct_change = ite / (abs(factual) + 1e-12)
+    return {
+        "factual_outcome": factual,
+        "counterfactual_outcome": counterfactual,
+        "ite": float(ite),
+        "pct_change": float(pct_change),
+    }
+
+
+def counterfactual_predict_multi(
+    scm: StructuralCausalModel,
+    observed_values: Dict[str, float],
+    counterfactual_values: Dict[str, float],
+    target: str,
+) -> Dict[str, float]:
+    """Compute a joint counterfactual for multiple simultaneous interventions."""
+    for node in counterfactual_values:
+        if node not in scm.node_to_idx:
+            raise ValueError(f"Unknown counterfactual node: {node}")
+
+    noises = abduct_noise(scm, observed_values)
+    state = scm.data_level[-1].copy()
+    for node, value in observed_values.items():
+        if node in scm.node_to_idx:
+            state[scm.node_to_idx[node]] = float(value)
+
+    cf = state.copy()
+    for node, value in counterfactual_values.items():
+        cf[scm.node_to_idx[node]] = float(value)
+
+    for node_idx in scm.topological_indices:
+        node = scm.nodes[node_idx]
+        if node in counterfactual_values:
+            continue
+        eq = scm.equations[node]
+        if not eq.parents or eq.equation_type == "exogenous":
+            cf[node_idx] = eq.intercept + noises.get(node, 0.0)
+            continue
+        parent_vals = cf[eq.parent_indices]
+        coef_vec = np.array([eq.coefficients[p] for p in eq.parents], dtype=float)
+        pred = eq.intercept + float(parent_vals @ coef_vec)
+        cf[node_idx] = pred + noises.get(node, 0.0)
+
+    y_idx = scm.node_to_idx[target]
+    factual = float(state[y_idx])
+    counterfactual = float(cf[y_idx])
+    ite = counterfactual - factual
+    pct_change = ite / (abs(factual) + 1e-12)
+    return {
+        "factual_outcome": factual,
+        "counterfactual_outcome": counterfactual,
+        "ite": float(ite),
+        "pct_change": float(pct_change),
+    }

singular_ticker_causal/causal_inference/causal_model.py

@@ -0,0 +1,314 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Dict, List, Optional, Sequence, Tuple
+
+import numpy as np
+
+
+@dataclass
+class StructuralEquation:
+    node: str
+    parents: List[str]
+    parent_indices: List[int]
+    intercept: float
+    coefficients: Dict[str, float]
+    residual_mean: float
+    residual_std: float
+    r_squared: float
+    n_obs: int
+    equation_type: str  # "linear" | "exogenous"
+
+
+@dataclass
+class StructuralCausalModel:
+    nodes: Sequence[str]
+    adj: np.ndarray
+    adjacency_mask: np.ndarray
+    data_tech: np.ndarray
+    mask_tech: Optional[np.ndarray] = None
+    prohibition_mask: Optional[np.ndarray] = None
+    threshold: float = 0.5
+    lag: int = 1
+    min_obs: int = 12
+    ridge_alpha: float = 1e-4
+    dag_adj: np.ndarray = field(init=False)
+    topological_indices: List[int] = field(init=False, default_factory=list)
+    equations: Dict[str, StructuralEquation] = field(init=False, default_factory=dict)
+    removed_cycle_edges: List[Dict[str, float]] = field(init=False, default_factory=list)
+
+    def __post_init__(self) -> None:
+        self.nodes = list(self.nodes)
+        self.node_to_idx = {node: i for i, node in enumerate(self.nodes)}
+        self.data_level = self._extract_level_data(self.data_tech)
+        self.mask_level = self._extract_mask_level(self.mask_tech)
+        self._validate_shapes()
+
+    @property
+    def n_nodes(self) -> int:
+        return len(self.nodes)
+
+    @property
+    def t_steps(self) -> int:
+        return self.data_level.shape[0]
+
+    def fit(self) -> "StructuralCausalModel":
+        self.dag_adj = self._build_dag()
+        self._enforce_acyclic()
+        self.topological_indices = self._topological_sort(self.dag_adj)
+        self._fit_equations()
+        return self
+
+    def density(self) -> float:
+        n = self.n_nodes
+        max_edges = n * (n - 1)
+        if max_edges == 0:
+            return 0.0
+        return float(np.sum(self.dag_adj) / max_edges)
+
+    def parents_of(self, node: str) -> List[str]:
+        j = self.node_to_idx[node]
+        return [self.nodes[i] for i in np.where(self.dag_adj[:, j])[0]]
+
+    def has_directed_path(self, source: str, target: str) -> bool:
+        s = self.node_to_idx[source]
+        t = self.node_to_idx[target]
+        stack = [s]
+        visited = set()
+        while stack:
+            u = stack.pop()
+            if u == t:
+                return True
+            if u in visited:
+                continue
+            visited.add(u)
+            children = np.where(self.dag_adj[u])[0].tolist()
+            stack.extend(children)
+        return False
+
+    def descendants_of(self, node: str) -> List[str]:
+        start = self.node_to_idx[node]
+        stack = [start]
+        visited = set()
+        while stack:
+            u = stack.pop()
+            children = np.where(self.dag_adj[u])[0].tolist()
+            for v in children:
+                if v not in visited:
+                    visited.add(v)
+                    stack.append(v)
+        visited.discard(start)
+        return [self.nodes[i] for i in sorted(visited)]
+
+    def _validate_shapes(self) -> None:
+        n = len(self.nodes)
+        if self.adj.shape != (n, n):
+            raise ValueError(f"adj shape mismatch: expected {(n, n)}, got {self.adj.shape}")
+        if self.adjacency_mask.shape != (n, n):
+            raise ValueError(
+                f"adjacency_mask shape mismatch: expected {(n, n)}, got {self.adjacency_mask.shape}"
+            )
+        if self.prohibition_mask is not None and self.prohibition_mask.shape != (n, n):
+            raise ValueError(
+                f"prohibition_mask shape mismatch: expected {(n, n)}, got {self.prohibition_mask.shape}"
+            )
+        if self.data_level.ndim != 2 or self.data_level.shape[1] != n:
+            raise ValueError(
+                f"data_level shape mismatch: expected (T, {n}), got {self.data_level.shape}"
+            )
+        if self.mask_level is not None and self.mask_level.shape != self.data_level.shape:
+            raise ValueError(
+                "mask_tech shape mismatch after extraction: expected shape "
+                f"{self.data_level.shape}, got {self.mask_level.shape}"
+            )
+        if self.lag < 1:
+            raise ValueError("lag must be >= 1")
+
+    def _extract_level_data(self, data_tech: np.ndarray) -> np.ndarray:
+        if data_tech.ndim == 3:
+            return np.asarray(data_tech[:, :, 0], dtype=float)
+        if data_tech.ndim == 2:
+            return np.asarray(data_tech, dtype=float)
+        raise ValueError(f"Unsupported data_tech ndim={data_tech.ndim}; expected 2 or 3.")
+
+    def _extract_mask_level(self, mask_tech: Optional[np.ndarray]) -> Optional[np.ndarray]:
+        if mask_tech is None:
+            return None
+        if mask_tech.ndim == 3:
+            return np.asarray(mask_tech[:, :, 0], dtype=float)
+        if mask_tech.ndim == 2:
+            return np.asarray(mask_tech, dtype=float)
+        raise ValueError(f"Unsupported mask_tech ndim={mask_tech.ndim}; expected 2 or 3.")
+
+    def _build_dag(self) -> np.ndarray:
+        cuts_edges = self.adj >= self.threshold
+        prior_edges = self.adjacency_mask > 0
+        dag = np.logical_or(cuts_edges, prior_edges)
+        if self.prohibition_mask is not None:
+            prohibited = self.prohibition_mask <= 0
+            dag = np.where(prohibited, False, dag)
+        np.fill_diagonal(dag, False)
+        return dag.astype(bool)
+
+    def _enforce_acyclic(self) -> None:
+        while True:
+            cycle_edges = self._find_cycle_edges(self.dag_adj)
+            if not cycle_edges:
+                return
+
+            removable = []
+            for src, dst in cycle_edges:
+                mandatory = bool(self.adjacency_mask[src, dst] > 0)
+                score = float(self.adj[src, dst])
+                removable.append((mandatory, score, src, dst))
+
+            non_mandatory = [r for r in removable if not r[0]]
+            choice = min(non_mandatory or removable, key=lambda x: x[1])
+            _, score, src, dst = choice
+
+            self.dag_adj[src, dst] = False
+            self.removed_cycle_edges.append(
+                {
+                    "source": self.nodes[src],
+                    "target": self.nodes[dst],
+                    "adj_score": score,
+                }
+            )
+
+    def _topological_sort(self, dag_adj: np.ndarray) -> List[int]:
+        n = dag_adj.shape[0]
+        indegree = np.sum(dag_adj, axis=0).astype(int)
+        queue = [i for i in range(n) if indegree[i] == 0]
+        order: List[int] = []
+
+        while queue:
+            node = queue.pop(0)
+            order.append(node)
+            children = np.where(dag_adj[node])[0]
+            for child in children:
+                indegree[child] -= 1
+                if indegree[child] == 0:
+                    queue.append(int(child))
+
+        if len(order) != n:
+            raise RuntimeError("DAG still contains a cycle after pruning.")
+        return order
+
+    def _find_cycle_edges(self, dag_adj: np.ndarray) -> List[Tuple[int, int]]:
+        n = dag_adj.shape[0]
+        state = np.zeros(n, dtype=int)  # 0=unvisited, 1=visiting, 2=done
+        parent = -np.ones(n, dtype=int)
+
+        def dfs(u: int) -> Optional[List[Tuple[int, int]]]:
+            state[u] = 1
+            for v in np.where(dag_adj[u])[0]:
+                v = int(v)
+                if state[v] == 0:
+                    parent[v] = u
+                    found = dfs(v)
+                    if found:
+                        return found
+                elif state[v] == 1:
+                    nodes = [v]
+                    cur = u
+                    while cur != v and cur != -1:
+                        nodes.append(cur)
+                        cur = int(parent[cur])
+                    nodes.append(v)
+                    nodes.reverse()
+                    return [(nodes[i], nodes[i + 1]) for i in range(len(nodes) - 1)]
+            state[u] = 2
+            return None
+
+        for start in range(n):
+            if state[start] == 0:
+                result = dfs(start)
+                if result:
+                    return result
+        return []
+
+    def _fit_equations(self) -> None:
+        T = self.t_steps
+        for idx in self.topological_indices:
+            node = self.nodes[idx]
+            parent_idx = [int(i) for i in np.where(self.dag_adj[:, idx])[0]]
+            parent_names = [self.nodes[i] for i in parent_idx]
+            valid_t = self._valid_timesteps(idx, parent_idx)
+
+            if len(valid_t) == 0:
+                self.equations[node] = StructuralEquation(
+                    node=node,
+                    parents=parent_names,
+                    parent_indices=parent_idx,
+                    intercept=0.0,
+                    coefficients={},
+                    residual_mean=0.0,
+                    residual_std=1.0,
+                    r_squared=0.0,
+                    n_obs=0,
+                    equation_type="exogenous",
+                )
+                continue
+
+            y = self.data_level[valid_t, idx]
+
+            if not parent_idx or len(valid_t) < self.min_obs:
+                mu = float(np.mean(y))
+                residuals = y - mu
+                self.equations[node] = StructuralEquation(
+                    node=node,
+                    parents=parent_names,
+                    parent_indices=parent_idx,
+                    intercept=mu,
+                    coefficients={},
+                    residual_mean=float(np.mean(residuals)) if residuals.size else 0.0,
+                    residual_std=float(np.std(residuals)) if residuals.size else 1.0,
+                    r_squared=0.0,
+                    n_obs=int(len(valid_t)),
+                    equation_type="exogenous",
+                )
+                continue
+
+            X = self.data_level[valid_t - self.lag][:, parent_idx]
+            coef, intercept = self._fit_ridge(X, y)
+            y_hat = intercept + (X @ coef)
+            residuals = y - y_hat
+            ss_res = float(np.sum(residuals ** 2))
+            ss_tot = float(np.sum((y - np.mean(y)) ** 2))
+            r2 = 1.0 - (ss_res / ss_tot) if ss_tot > 1e-12 else 0.0
+
+            coeffs = {name: float(coef[i]) for i, name in enumerate(parent_names)}
+            self.equations[node] = StructuralEquation(
+                node=node,
+                parents=parent_names,
+                parent_indices=parent_idx,
+                intercept=float(intercept),
+                coefficients=coeffs,
+                residual_mean=float(np.mean(residuals)),
+                residual_std=float(np.std(residuals)),
+                r_squared=r2,
+                n_obs=int(len(valid_t)),
+                equation_type="linear",
+            )
+
+    def _valid_timesteps(self, node_idx: int, parent_indices: List[int]) -> np.ndarray:
+        valid_t = np.arange(self.lag, self.t_steps, dtype=int)
+        if self.mask_level is None:
+            return valid_t
+
+        valid = self.mask_level[valid_t, node_idx] > 0
+        for p_idx in parent_indices:
+            valid &= self.mask_level[valid_t - self.lag, p_idx] > 0
+        return valid_t[valid]
+
+    def _fit_ridge(self, X: np.ndarray, y: np.ndarray) -> Tuple[np.ndarray, float]:
+        X_mean = np.mean(X, axis=0)
+        y_mean = float(np.mean(y))
+        Xc = X - X_mean
+        yc = y - y_mean
+
+        p = X.shape[1]
+        reg = self.ridge_alpha * np.eye(p)
+        beta = np.linalg.solve(Xc.T @ Xc + reg, Xc.T @ yc)
+        intercept = y_mean - float(X_mean @ beta)
+        return beta, intercept

singular_ticker_causal/causal_inference/estimator.py

@@ -0,0 +1,125 @@
+from __future__ import annotations
+
+from typing import Dict, Optional, Set
+
+import numpy as np
+
+from .causal_model import StructuralCausalModel
+from .mutilator import propagate_intervention
+
+
+def estimate_ate(
+    scm: StructuralCausalModel,
+    treatment: str,
+    outcome: str,
+    treatment_value: Optional[float] = None,
+    adjustment_set: Optional[Set[str]] = None,
+    horizon: int = 5,
+) -> Dict[str, object]:
+    t_idx = scm.node_to_idx[treatment]
+    y_idx = scm.node_to_idx[outcome]
+    z_nodes = sorted(adjustment_set or set())
+    z_idx = [scm.node_to_idx[z] for z in z_nodes]
+
+    valid_t = np.arange(scm.lag, scm.t_steps, dtype=int)
+    if scm.mask_level is not None:
+        valid = (scm.mask_level[valid_t, y_idx] > 0) & (scm.mask_level[valid_t - scm.lag, t_idx] > 0)
+        for zi in z_idx:
+            valid &= scm.mask_level[valid_t - scm.lag, zi] > 0
+        valid_t = valid_t[valid]
+
+    if len(valid_t) < 5:
+        raise ValueError(
+            f"Insufficient observations for ATE estimation of {treatment}->{outcome}: {len(valid_t)} rows."
+        )
+
+    y = scm.data_level[valid_t, y_idx]
+    x_treat = scm.data_level[valid_t - scm.lag, t_idx]
+    X_parts = [np.ones((len(valid_t), 1)), x_treat.reshape(-1, 1)]
+    if z_idx:
+        X_parts.append(scm.data_level[valid_t - scm.lag][:, z_idx])
+    X = np.concatenate(X_parts, axis=1)
+
+    beta = np.linalg.pinv(X.T @ X) @ (X.T @ y)
+    y_hat = X @ beta
+    resid = y - y_hat
+    dof = max(1, len(y) - X.shape[1])
+    sigma2 = float(np.sum(resid ** 2) / dof)
+    cov = sigma2 * np.linalg.pinv(X.T @ X)
+    se = float(np.sqrt(max(cov[1, 1], 0.0)))
+    ate = float(beta[1])
+
+    baseline = float(np.mean(np.abs(y))) + 1e-12
+    ate_normalized = ate / baseline
+    ci_95 = (ate - 1.96 * se, ate + 1.96 * se)
+
+    path_contributions = _path_contributions(scm, treatment, outcome)
+
+    horizon_effects = []
+    if treatment_value is not None:
+        baseline_t = scm.t_steps - 1
+        shocked = propagate_intervention(scm, treatment, treatment_value, horizon=horizon, baseline_t=baseline_t)
+        base_outcome = float(scm.data_level[baseline_t, y_idx])
+        for h in range(horizon):
+            horizon_effects.append(float(shocked[h, y_idx] - base_outcome))
+
+    return {
+        "ate": ate,
+        "ate_normalized": float(ate_normalized),
+        "ci_95": (float(ci_95[0]), float(ci_95[1])),
+        "n_obs": int(len(valid_t)),
+        "path_contributions": path_contributions,
+        "horizon_effects": horizon_effects,
+    }
+
+
+def _path_contributions(
+    scm: StructuralCausalModel,
+    treatment: str,
+    outcome: str,
+    max_paths: int = 100,
+) -> Dict[str, float]:
+    start = scm.node_to_idx[treatment]
+    target = scm.node_to_idx[outcome]
+    paths = []
+
+    def dfs(node: int, path: list[int], seen: set[int]) -> None:
+        if len(paths) >= max_paths:
+            return
+        if node == target:
+            paths.append(path.copy())
+            return
+        for child in np.where(scm.dag_adj[node])[0]:
+            child = int(child)
+            if child in seen:
+                continue
+            seen.add(child)
+            path.append(child)
+            dfs(child, path, seen)
+            path.pop()
+            seen.remove(child)
+
+    dfs(start, [start], {start})
+
+    contributions: Dict[str, float] = {}
+    for path in paths:
+        coeff_product = 1.0
+        valid = True
+        for i in range(len(path) - 1):
+            src = scm.nodes[path[i]]
+            dst = scm.nodes[path[i + 1]]
+            eq = scm.equations.get(dst)
+            if eq is None:
+                valid = False
+                break
+            coef = eq.coefficients.get(src)
+            if coef is None:
+                valid = False
+                break
+            coeff_product *= coef
+        if not valid:
+            continue
+        label = " -> ".join(scm.nodes[i] for i in path)
+        contributions[label] = float(coeff_product)
+
+    return contributions

singular_ticker_causal/causal_inference/identification.py

@@ -0,0 +1,41 @@
+from __future__ import annotations
+
+from enum import Enum
+from typing import Optional, Set, Tuple
+
+from .causal_model import StructuralCausalModel
+
+
+class IdentificationStrategy(Enum):
+    DIRECT = "direct"
+    BACKDOOR = "backdoor"
+    NOT_IDENTIFIABLE = "not_identifiable"
+
+
+def find_adjustment_set(
+    scm: StructuralCausalModel,
+    treatment: str,
+    outcome: str,
+) -> Tuple[IdentificationStrategy, Optional[Set[str]]]:
+    if treatment not in scm.node_to_idx:
+        raise ValueError(f"Unknown treatment node: {treatment}")
+    if outcome not in scm.node_to_idx:
+        raise ValueError(f"Unknown outcome node: {outcome}")
+    if treatment == outcome:
+        return IdentificationStrategy.DIRECT, set()
+
+    if not scm.has_directed_path(treatment, outcome):
+        return IdentificationStrategy.NOT_IDENTIFIABLE, None
+
+    parents = set(scm.parents_of(treatment))
+    descendants = set(scm.descendants_of(treatment))
+    adjustment = parents - descendants - {outcome}
+
+    if adjustment:
+        return IdentificationStrategy.BACKDOOR, adjustment
+    return IdentificationStrategy.DIRECT, set()
+
+
+def is_identifiable(scm: StructuralCausalModel, treatment: str, outcome: str) -> bool:
+    strategy, _ = find_adjustment_set(scm, treatment, outcome)
+    return strategy != IdentificationStrategy.NOT_IDENTIFIABLE

singular_ticker_causal/causal_inference/mutilator.py

@@ -0,0 +1,62 @@
+from __future__ import annotations
+
+from copy import deepcopy
+from typing import Dict, Optional, Sequence
+
+import numpy as np
+
+from .causal_model import StructuralCausalModel
+
+
+def mutilate_graph(scm: StructuralCausalModel, treatment: str, value: float) -> StructuralCausalModel:
+    mutilated = deepcopy(scm)
+    t_idx = mutilated.node_to_idx[treatment]
+    mutilated.dag_adj[:, t_idx] = False
+    mutilated.pinned_values = {treatment: float(value)}
+    return mutilated
+
+
+def propagate_intervention(
+    scm: StructuralCausalModel,
+    treatment: str,
+    value: float,
+    targets: Optional[Sequence[str]] = None,
+    horizon: int = 5,
+    baseline_t: int = -1,
+) -> np.ndarray:
+    if horizon < 1:
+        raise ValueError("horizon must be >= 1")
+
+    baseline_idx = baseline_t if baseline_t >= 0 else (scm.t_steps + baseline_t)
+    if baseline_idx < 0 or baseline_idx >= scm.t_steps:
+        raise ValueError(f"baseline_t {baseline_t} resolves out of bounds for T={scm.t_steps}")
+
+    t_idx = scm.node_to_idx[treatment]
+    prev = scm.data_level[baseline_idx].copy()
+    forecasts = np.zeros((horizon, scm.n_nodes), dtype=float)
+
+    for h in range(horizon):
+        nxt = prev.copy()
+        nxt[t_idx] = float(value)
+        for node_idx in scm.topological_indices:
+            if node_idx == t_idx:
+                continue
+            node = scm.nodes[node_idx]
+            eq = scm.equations[node]
+            if not eq.parent_indices or eq.equation_type == "exogenous":
+                continue
+            parent_vals = prev[eq.parent_indices]
+            coef_vec = np.array([eq.coefficients[p] for p in eq.parents], dtype=float)
+            nxt[node_idx] = eq.intercept + float(parent_vals @ coef_vec)
+        forecasts[h] = nxt
+        prev = nxt
+
+    if targets:
+        missing = [n for n in targets if n not in scm.node_to_idx]
+        if missing:
+            raise ValueError(f"Unknown targets: {missing}")
+    return forecasts
+
+
+def target_series(forecasts: np.ndarray, scm: StructuralCausalModel, targets: Sequence[str]) -> Dict[str, list[float]]:
+    return {t: [float(v) for v in forecasts[:, scm.node_to_idx[t]]] for t in targets}

singular_ticker_causal/causal_inference/pywhyllm_assumptions.py

@@ -0,0 +1,338 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from dataclasses import asdict, dataclass, field
+from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
+
+import numpy as np
+
+
+def _json_safe(value: Any) -> Any:
+    if isinstance(value, np.ndarray):
+        return value.tolist()
+    if isinstance(value, (np.integer,)):
+        return int(value)
+    if isinstance(value, (np.floating,)):
+        return float(value)
+    if isinstance(value, dict):
+        return {str(k): _json_safe(v) for k, v in value.items()}
+    if isinstance(value, (list, tuple, set)):
+        return [_json_safe(v) for v in value]
+    if isinstance(value, (str, int, float, bool)) or value is None:
+        return value
+    return str(value)
+
+
+@dataclass
+class PyWhyLLMConfig:
+    enabled: bool = False
+    model: str = "gpt-4"
+    max_edges: int = 25
+    cache_dir: str = os.path.join(
+        os.path.dirname(__file__),
+        "..",
+        "debug_data",
+        "pywhyllm_cache",
+    )
+
+
+@dataclass
+class CausalAssumptionReport:
+    available: bool = True
+    reason: Optional[str] = None
+    domain_expertises: List[str] = field(default_factory=list)
+    suggested_confounders: List[str] = field(default_factory=list)
+    suggested_backdoor_sets: List[List[str]] = field(default_factory=list)
+    suggested_mediators: List[str] = field(default_factory=list)
+    suggested_ivs: List[str] = field(default_factory=list)
+    negative_controls: List[str] = field(default_factory=list)
+    latent_confounders: List[str] = field(default_factory=list)
+    edge_critiques: Any = field(default_factory=list)
+    accepted_edges: List[Tuple[str, str]] = field(default_factory=list)
+    rejected_edges: List[Tuple[str, str]] = field(default_factory=list)
+    warnings: List[str] = field(default_factory=list)
+
+    def to_dict(self) -> Dict[str, Any]:
+        payload = asdict(self)
+        payload["accepted_edges"] = [list(edge) for edge in self.accepted_edges]
+        payload["rejected_edges"] = [list(edge) for edge in self.rejected_edges]
+        return _json_safe(payload)
+
+    @classmethod
+    def from_dict(cls, payload: Dict[str, Any]) -> "CausalAssumptionReport":
+        data = dict(payload)
+        data["accepted_edges"] = [tuple(edge) for edge in data.get("accepted_edges", [])]
+        data["rejected_edges"] = [tuple(edge) for edge in data.get("rejected_edges", [])]
+        return cls(**data)
+
+
+def _dedupe_strings(values: Iterable[Any]) -> List[str]:
+    seen = set()
+    result: List[str] = []
+    for value in values or []:
+        text = str(value).strip()
+        if text and text not in seen:
+            seen.add(text)
+            result.append(text)
+    return result
+
+
+def _normalise_suggestion(value: Any) -> List[str]:
+    if value is None:
+        return []
+    if isinstance(value, tuple) and len(value) == 2:
+        return _normalise_suggestion(value[1])
+    if isinstance(value, dict):
+        keys = [k for k, v in value.items() if isinstance(k, str) and v]
+        if keys:
+            return _dedupe_strings(keys)
+        flattened: List[str] = []
+        for item in value.values():
+            flattened.extend(_normalise_suggestion(item))
+        return _dedupe_strings(flattened)
+    if isinstance(value, (list, set, tuple)):
+        flattened = []
+        for item in value:
+            if isinstance(item, (list, set, tuple, dict)):
+                flattened.extend(_normalise_suggestion(item))
+            else:
+                flattened.append(item)
+        return _dedupe_strings(flattened)
+    return _dedupe_strings([value])
+
+
+class PyWhyLLMAssumptionService:
+    def __init__(
+        self,
+        config: Optional[PyWhyLLMConfig] = None,
+        model_suggester: Any = None,
+        identification_suggester: Any = None,
+        validation_suggester: Any = None,
+        relationship_strategy: Any = None,
+    ):
+        self.config = config or PyWhyLLMConfig()
+        self._model_suggester = model_suggester
+        self._identification_suggester = identification_suggester
+        self._validation_suggester = validation_suggester
+        self._relationship_strategy = relationship_strategy
+
+    @property
+    def enabled(self) -> bool:
+        return bool(self.config.enabled)
+
+    def analyze(
+        self,
+        *,
+        nodes: Sequence[str],
+        dag_adj: np.ndarray,
+        treatment: str,
+        outcome: str,
+        max_edges: Optional[int] = None,
+    ) -> CausalAssumptionReport:
+        if not self.enabled:
+            return CausalAssumptionReport(
+                available=False,
+                reason="PyWhy-LLM is disabled. Set PYWHYLLM_ENABLED=true or pass pywhyllm_enabled=True.",
+            )
+
+        cache_path = self._cache_path(nodes, dag_adj, treatment, outcome)
+        cached = self._read_cache(cache_path)
+        if cached is not None:
+            return cached
+
+        try:
+            modeler, identifier, validator, relationship_strategy = self._suggesters()
+        except Exception as exc:
+            return CausalAssumptionReport(
+                available=False,
+                reason=f"PyWhy-LLM is not installed or failed to initialize: {exc}",
+            )
+
+        all_factors = list(nodes)
+        edges = self._edges(nodes, dag_adj, max_edges or self.config.max_edges)
+        warnings: List[str] = []
+
+        domain_expertises: List[str] = []
+        suggested_confounders: List[str] = []
+        suggested_backdoor_sets: List[List[str]] = []
+        suggested_mediators: List[str] = []
+        suggested_ivs: List[str] = []
+        negative_controls: List[str] = []
+        latent_confounders: List[str] = []
+        edge_critiques: Any = []
+        suggested_dag: Any = edges
+
+        try:
+            domain_expertises = _normalise_suggestion(modeler.suggest_domain_expertises(all_factors))
+        except Exception as exc:
+            warnings.append(f"domain_expertises failed: {exc}")
+
+        try:
+            suggested_confounders = _normalise_suggestion(
+                modeler.suggest_confounders(treatment, outcome, all_factors, domain_expertises)
+            )
+        except Exception as exc:
+            warnings.append(f"confounder suggestion failed: {exc}")
+
+        try:
+            suggested_dag = modeler.suggest_relationships(
+                treatment,
+                outcome,
+                all_factors,
+                domain_expertises,
+                relationship_strategy,
+            )
+        except Exception as exc:
+            warnings.append(f"relationship suggestion failed: {exc}")
+
+        try:
+            backdoor = identifier.suggest_backdoor(treatment, outcome, all_factors, domain_expertises)
+            backdoor_nodes = _normalise_suggestion(backdoor)
+            if backdoor_nodes:
+                suggested_backdoor_sets = [backdoor_nodes]
+        except Exception as exc:
+            warnings.append(f"backdoor suggestion failed: {exc}")
+
+        try:
+            suggested_mediators = _normalise_suggestion(
+                identifier.suggest_mediators(treatment, outcome, all_factors, domain_expertises)
+            )
+        except Exception as exc:
+            warnings.append(f"mediator suggestion failed: {exc}")
+
+        try:
+            suggested_ivs = _normalise_suggestion(
+                identifier.suggest_ivs(treatment, outcome, all_factors, domain_expertises)
+            )
+        except Exception as exc:
+            warnings.append(f"iv suggestion failed: {exc}")
+
+        try:
+            edge_critiques = validator.critique_graph(
+                all_factors,
+                suggested_dag,
+                domain_expertises,
+                relationship_strategy,
+            )
+        except Exception as exc:
+            warnings.append(f"edge critique failed: {exc}")
+
+        try:
+            latent_confounders = _normalise_suggestion(
+                validator.suggest_latent_confounders(treatment, outcome, all_factors, domain_expertises)
+            )
+        except Exception as exc:
+            warnings.append(f"latent confounder suggestion failed: {exc}")
+
+        try:
+            negative_controls = _normalise_suggestion(
+                validator.suggest_negative_controls(treatment, outcome, all_factors, domain_expertises)
+            )
+        except Exception as exc:
+            warnings.append(f"negative control suggestion failed: {exc}")
+
+        accepted_edges, rejected_edges = self._classify_edges(edges, edge_critiques)
+        report = CausalAssumptionReport(
+            available=True,
+            domain_expertises=domain_expertises,
+            suggested_confounders=suggested_confounders,
+            suggested_backdoor_sets=suggested_backdoor_sets,
+            suggested_mediators=suggested_mediators,
+            suggested_ivs=suggested_ivs,
+            negative_controls=negative_controls,
+            latent_confounders=latent_confounders,
+            edge_critiques=edge_critiques,
+            accepted_edges=accepted_edges,
+            rejected_edges=rejected_edges,
+            warnings=warnings,
+        )
+        self._write_cache(cache_path, report)
+        return report
+
+    def _suggesters(self) -> Tuple[Any, Any, Any, Any]:
+        if self._model_suggester and self._identification_suggester and self._validation_suggester:
+            return (
+                self._model_suggester,
+                self._identification_suggester,
+                self._validation_suggester,
+                self._relationship_strategy,
+            )
+
+        from pywhyllm import RelationshipStrategy
+        from pywhyllm.suggesters.identification_suggester import IdentificationSuggester
+        from pywhyllm.suggesters.model_suggester import ModelSuggester
+        from pywhyllm.suggesters.validation_suggester import ValidationSuggester
+
+        relationship_strategy = self._relationship_strategy or RelationshipStrategy.Pairwise
+        return (
+            self._model_suggester or ModelSuggester(self.config.model),
+            self._identification_suggester or IdentificationSuggester(self.config.model),
+            self._validation_suggester or ValidationSuggester(self.config.model),
+            relationship_strategy,
+        )
+
+    def _cache_path(
+        self,
+        nodes: Sequence[str],
+        dag_adj: np.ndarray,
+        treatment: str,
+        outcome: str,
+    ) -> str:
+        payload = {
+            "nodes": list(nodes),
+            "dag_adj": np.asarray(dag_adj, dtype=int).tolist(),
+            "treatment": treatment,
+            "outcome": outcome,
+            "model": self.config.model,
+        }
+        digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode("utf-8")).hexdigest()
+        return os.path.join(os.path.abspath(self.config.cache_dir), f"{digest}.json")
+
+    def _read_cache(self, path: str) -> Optional[CausalAssumptionReport]:
+        if not os.path.exists(path):
+            return None
+        try:
+            with open(path) as f:
+                return CausalAssumptionReport.from_dict(json.load(f))
+        except Exception:
+            return None
+
+    def _write_cache(self, path: str, report: CausalAssumptionReport) -> None:
+        try:
+            os.makedirs(os.path.dirname(path), exist_ok=True)
+            with open(path, "w") as f:
+                json.dump(report.to_dict(), f, indent=2)
+        except Exception:
+            pass
+
+    def _edges(self, nodes: Sequence[str], dag_adj: np.ndarray, max_edges: int) -> List[Tuple[str, str]]:
+        found: List[Tuple[str, str, float]] = []
+        for src_idx, src in enumerate(nodes):
+            for dst_idx, dst in enumerate(nodes):
+                if bool(dag_adj[src_idx, dst_idx]):
+                    found.append((src, dst, float(dag_adj[src_idx, dst_idx])))
+        found.sort(key=lambda edge: abs(edge[2]), reverse=True)
+        return [(src, dst) for src, dst, _ in found[:max_edges]]
+
+    def _classify_edges(
+        self,
+        edges: List[Tuple[str, str]],
+        edge_critiques: Any,
+    ) -> Tuple[List[Tuple[str, str]], List[Tuple[str, str]]]:
+        critique_text = str(edge_critiques).lower()
+        rejected: List[Tuple[str, str]] = []
+        for edge in edges:
+            src, dst = edge
+            edge_tokens = [
+                f"{src}->{dst}".lower(),
+                f"{src} -> {dst}".lower(),
+                f"{src}, {dst}".lower(),
+            ]
+            if any(token in critique_text for token in edge_tokens) and any(
+                marker in critique_text for marker in ["reject", "unlikely", "invalid", "implausible"]
+            ):
+                rejected.append(edge)
+        accepted = [edge for edge in edges if edge not in rejected]
+        return accepted, rejected

singular_ticker_causal/causal_inference/query_engine.py

@@ -0,0 +1,505 @@
+from __future__ import annotations
+
+import os
+import re
+from typing import Any, Dict, List, Optional, Sequence
+
+import numpy as np
+import pandas as pd
+
+from .abduction import counterfactual_predict, counterfactual_predict_multi
+from .causal_model import StructuralCausalModel
+from .estimator import estimate_ate
+from .identification import find_adjustment_set
+from .mutilator import mutilate_graph, propagate_intervention, target_series
+from .pywhyllm_assumptions import PyWhyLLMConfig, PyWhyLLMAssumptionService
+
+
+class CausalQueryEngine:
+    def __init__(
+        self,
+        scm: StructuralCausalModel,
+        data_tech: Optional[np.ndarray] = None,
+        data_text: Optional[np.ndarray] = None,
+        pywhyllm_service: Optional[PyWhyLLMAssumptionService] = None,
+        pywhyllm_enabled: bool = False,
+    ):
+        self.scm = scm
+        self.data_tech = data_tech
+        self.data_text = data_text
+        self.pywhyllm_enabled = pywhyllm_enabled
+        self.pywhyllm_service = pywhyllm_service
+
+    def assert_edge(self, treatment: str, outcome: str) -> Dict[str, object]:
+        strategy, z = find_adjustment_set(self.scm, treatment, outcome)
+        if strategy.value == "not_identifiable":
+            return {
+                "ate": 0.0,
+                "ci_95": (0.0, 0.0),
+                "strategy": strategy.name,
+                "adjustment_set": [],
+                "identifiable": False,
+                "path_contributions": {},
+            }
+
+        est = estimate_ate(
+            self.scm,
+            treatment=treatment,
+            outcome=outcome,
+            adjustment_set=z,
+            treatment_value=None,
+        )
+        return {
+            "ate": est["ate"],
+            "ci_95": est["ci_95"],
+            "strategy": strategy.name,
+            "adjustment_set": sorted(z or set()),
+            "identifiable": True,
+            "path_contributions": est["path_contributions"],
+        }
+
+    def intervene(self, treatment: str, value: float, targets: List[str], horizon: int = 5) -> Dict[str, object]:
+        mutilated = mutilate_graph(self.scm, treatment=treatment, value=value)
+        forecasts = propagate_intervention(
+            mutilated,
+            treatment=treatment,
+            value=value,
+            targets=targets,
+            horizon=horizon,
+            baseline_t=-1,
+        )
+        predicted = target_series(forecasts, self.scm, targets)
+        ate_per_target = {}
+        for target in targets:
+            tidx = self.scm.node_to_idx[target]
+            base = float(self.scm.data_level[-1, tidx])
+            ate_per_target[target] = float(forecasts[0, tidx] - base)
+        return {
+            "mutilated_adj": mutilated.dag_adj.copy(),
+            "predicted_values": predicted,
+            "ate_per_target": ate_per_target,
+            "horizon": horizon,
+        }
+
+    def _counterfactual_outcome(
+        self,
+        observed: Dict[str, float],
+        counterfactual_values: Dict[str, float],
+        target: str,
+    ) -> Dict[str, float]:
+        if len(counterfactual_values) == 1:
+            treatment, value = next(iter(counterfactual_values.items()))
+            return counterfactual_predict(
+                self.scm,
+                observed_values=observed,
+                treatment=treatment,
+                counterfactual_value=value,
+                target=target,
+            )
+        return counterfactual_predict_multi(
+            self.scm,
+            observed_values=observed,
+            counterfactual_values=counterfactual_values,
+            target=target,
+        )
+
+    def _shapley_contributions(
+        self,
+        observed: Dict[str, float],
+        interventions: Dict[str, float],
+        target: str,
+        mc_samples: int = 1000,
+        n_jobs: int = -1,
+    ) -> Dict[str, float]:
+        import random
+        from math import factorial
+        from joblib import Parallel, delayed
+        import os
+
+        treatments = list(interventions.keys())
+        n = len(treatments)
+        if n == 1:
+            return {treatments[0]: self._counterfactual_outcome(observed, interventions, target)["ite"]}
+
+        contributions: Dict[str, float] = {t: 0.0 for t in treatments}
+        
+        # Use exact if n <= 10, else Monte Carlo
+        use_exact = n <= 10
+        if n_jobs < 0:
+            n_jobs = os.cpu_count() or 4
+
+        def marginal_contribution(k: str, subset: list[str]) -> float:
+            with_k = {**{t: interventions[t] for t in subset}, k: interventions[k]}
+            without_k = {t: interventions[t] for t in subset}
+            v_with = self._counterfactual_outcome(observed, with_k, target)["counterfactual_outcome"]
+            v_without = self._counterfactual_outcome(observed, without_k, target)["counterfactual_outcome"]
+            return v_with - v_without
+
+        if use_exact:
+            all_factorial = float(factorial(n))
+            tasks = []
+            
+            for k in treatments:
+                others = [t for t in treatments if t != k]
+                for r in range(len(others) + 1):
+                    for subset in __import__("itertools").combinations(others, r):
+                        subset_list = list(subset)
+                        weight = float(factorial(len(subset_list)) * factorial(n - len(subset_list) - 1) / all_factorial)
+                        tasks.append((k, subset_list, weight))
+            
+            results = Parallel(n_jobs=n_jobs, backend="threading")(
+                delayed(marginal_contribution)(task[0], task[1]) for task in tasks
+            )
+            for task, res in zip(tasks, results):
+                k, _, weight = task
+                contributions[k] += weight * res
+        else:
+            # Monte Carlo approximation
+            tasks = []
+            for _ in range(mc_samples):
+                perm = treatments.copy()
+                random.shuffle(perm)
+                for i, k in enumerate(perm):
+                    subset_list = perm[:i]
+                    tasks.append((k, subset_list))
+                    
+            results = Parallel(n_jobs=n_jobs, backend="threading")(
+                delayed(marginal_contribution)(task[0], task[1]) for task in tasks
+            )
+            for task, res in zip(tasks, results):
+                k, _ = task
+                contributions[k] += res / mc_samples
+
+        return contributions
+
+    def _build_dowhy_graph(self) -> str:
+        edges = []
+        for src_idx, src in enumerate(self.scm.nodes):
+            for dst_idx, dst in enumerate(self.scm.nodes):
+                if self.scm.dag_adj[src_idx, dst_idx]:
+                    edges.append(f"{src} -> {dst}")
+        return "digraph{" + "; ".join(edges) + "}"
+
+    def _default_pywhyllm_service(self) -> PyWhyLLMAssumptionService:
+        enabled = self.pywhyllm_enabled or os.environ.get("PYWHYLLM_ENABLED", "").lower() in {
+            "1",
+            "true",
+            "yes",
+            "on",
+        }
+        return PyWhyLLMAssumptionService(
+            PyWhyLLMConfig(
+                enabled=enabled,
+                model=os.environ.get("PYWHYLLM_MODEL", "gpt-4"),
+                max_edges=int(os.environ.get("PYWHYLLM_MAX_EDGES", "25")),
+                cache_dir=os.environ.get(
+                    "PYWHYLLM_CACHE_DIR",
+                    PyWhyLLMConfig().cache_dir,
+                ),
+            )
+        )
+
+    def _get_pywhyllm_service(self) -> PyWhyLLMAssumptionService:
+        if self.pywhyllm_service is None:
+            self.pywhyllm_service = self._default_pywhyllm_service()
+        return self.pywhyllm_service
+
+    def analyze_assumptions_with_pywhyllm(
+        self,
+        treatment: str,
+        outcome: str,
+        max_edges: Optional[int] = None,
+    ) -> Dict[str, object]:
+        report = self._get_pywhyllm_service().analyze(
+            nodes=self.scm.nodes,
+            dag_adj=self.scm.dag_adj,
+            treatment=treatment,
+            outcome=outcome,
+            max_edges=max_edges,
+        )
+        return report.to_dict()
+
+    def _valid_nodes(self, candidates: Sequence[Any]) -> List[str]:
+        valid = set(self.scm.nodes)
+        result = []
+        for candidate in candidates or []:
+            node = str(candidate).strip()
+            if node in valid and node not in result:
+                result.append(node)
+        return result
+
+    def _valid_backdoor_sets(self, report: Dict[str, Any], treatment: str, outcome: str) -> List[List[str]]:
+        result = []
+        blocked = {treatment, outcome}
+        for suggested_set in report.get("suggested_backdoor_sets") or []:
+            valid_set = [node for node in self._valid_nodes(suggested_set) if node not in blocked]
+            if valid_set and valid_set not in result:
+                result.append(valid_set)
+        confounders = [node for node in self._valid_nodes(report.get("suggested_confounders") or []) if node not in blocked]
+        if confounders and confounders not in result:
+            result.append(confounders)
+        return result
+
+    def _coerce_causal_model(self, causal_model_cls: Any, data: pd.DataFrame, treatment: str, outcome: str, graph: str):
+        try:
+            return causal_model_cls(df=data, treatment=treatment, outcome=outcome, graph=graph)
+        except TypeError:
+            return causal_model_cls(data=data, treatment=treatment, outcome=outcome, graph=graph)
+
+    def _parse_p_value(self, value: Any) -> Optional[float]:
+        if value is None:
+            return None
+        if isinstance(value, (int, float, np.floating)):
+            return float(value)
+        if isinstance(value, (list, tuple)) and value:
+            return self._parse_p_value(value[0])
+        if isinstance(value, dict):
+            for key in ("p_value", "p-value", "p value"):
+                if key in value:
+                    return self._parse_p_value(value[key])
+            return None
+        match = re.search(r"p[-_ ]?value[^0-9<>=-]*[<>=: ]+\s*([0-9]*\.?[0-9]+)", str(value), re.I)
+        if match:
+            return float(match.group(1))
+        return None
+
+    def _as_optional_float(self, value: Any) -> Optional[float]:
+        if value is None:
+            return None
+        try:
+            arr = np.asarray(value, dtype=float)
+            if arr.size == 1:
+                return float(arr.reshape(-1)[0])
+        except Exception:
+            pass
+        try:
+            return float(value)
+        except Exception:
+            return None
+
+    def _parse_refuter_result(self, method: str, refute: Any, alpha: float = 0.05) -> Dict[str, object]:
+        text = str(refute)
+        result_attr = getattr(refute, "refutation_result", None)
+        estimated_effect = getattr(refute, "estimated_effect", None)
+        new_effect = getattr(refute, "new_effect", None)
+        p_value = self._parse_p_value(result_attr)
+        if p_value is None:
+            p_value = self._parse_p_value(text)
+        lower_text = text.lower()
+
+        if method == "placebo_treatment":
+            if p_value is not None:
+                falsified = p_value < alpha
+            elif "not statistically significant" in lower_text:
+                falsified = False
+            elif "statistically significant" in lower_text:
+                falsified = True
+            else:
+                falsified = False
+        else:
+            if p_value is not None:
+                falsified = p_value < alpha
+            elif "not statistically significant" in lower_text:
+                falsified = False
+            elif "statistically significant" in lower_text:
+                falsified = True
+            else:
+                falsified = False
+
+        return {
+            "method": method,
+            "refute": text,
+            "estimated_effect": self._as_optional_float(estimated_effect),
+            "new_effect": self._as_optional_float(new_effect),
+            "p_value": p_value,
+            "passed": not falsified,
+            "falsified": falsified,
+        }
+
+    def _run_dowhy_validation(
+        self,
+        causal_model_cls: Any,
+        treatment: str,
+        outcome: str,
+        treatment_value: float = 1.0,
+        adjustment_candidates: Optional[List[List[str]]] = None,
+        negative_controls: Optional[List[str]] = None,
+    ) -> Dict[str, object]:
+        data = pd.DataFrame(self.scm.data_level, columns=self.scm.nodes)
+        graph = self._build_dowhy_graph()
+        model = self._coerce_causal_model(causal_model_cls, data, treatment, outcome, graph)
+        identified_estimand = model.identify_effect()
+        estimate = model.estimate_effect(identified_estimand, method_name="backdoor.linear_regression")
+
+        refutations = []
+        for method in ["placebo_treatment", "random_common_cause"]:
+            try:
+                kwargs = {"method_name": method}
+                if method == "placebo_treatment":
+                    kwargs["placebo_type"] = "permute"
+                refute = model.refute_estimate(identified_estimand, estimate, **kwargs)
+                refutations.append(self._parse_refuter_result(method, refute))
+            except TypeError:
+                try:
+                    kwargs = {"method_name": method}
+                    if method == "placebo_treatment":
+                        kwargs["placebo_type"] = "permute"
+                    refute = model.refute_estimate(estimate, **kwargs)
+                    refutations.append(self._parse_refuter_result(method, refute))
+                except Exception as exc:
+                    refutations.append({"method": method, "error": str(exc), "passed": False, "falsified": False})
+            except Exception as exc:
+                refutations.append({"method": method, "error": str(exc), "passed": False, "falsified": False})
+
+        negative_control_checks = []
+        for control in negative_controls or []:
+            if control in {treatment, outcome}:
+                continue
+            try:
+                nc_model = self._coerce_causal_model(causal_model_cls, data, treatment, control, graph)
+                nc_identified = nc_model.identify_effect()
+                nc_estimate = nc_model.estimate_effect(nc_identified, method_name="backdoor.linear_regression")
+                negative_control_checks.append(
+                    {
+                        "control": control,
+                        "identified_estimand": str(nc_identified),
+                        "estimate": str(nc_estimate),
+                    }
+                )
+            except Exception as exc:
+                negative_control_checks.append({"control": control, "error": str(exc)})
+
+        falsified = any(bool(item.get("falsified")) for item in refutations)
+        return {
+            "available": True,
+            "falsified": falsified,
+            "identified_estimand": str(identified_estimand),
+            "estimate": str(estimate),
+            "refutations": refutations,
+            "adjustment_candidates": adjustment_candidates or [],
+            "negative_control_checks": negative_control_checks,
+        }
+
+    def validate_with_dowhy(
+        self,
+        treatment: str,
+        outcome: str,
+        treatment_value: float = 1.0,
+        num_placebo: int = 5,
+    ) -> Dict[str, object]:
+        try:
+            from dowhy import CausalModel
+        except Exception as exc:
+            return {
+                "available": False,
+                "reason": str(exc),
+                "falsified": False,
+                "summary": "DoWhy is not installed or failed to import.",
+            }
+
+        try:
+            return self._run_dowhy_validation(
+                CausalModel,
+                treatment=treatment,
+                outcome=outcome,
+                treatment_value=treatment_value,
+            )
+        except Exception as exc:
+            return {
+                "available": False,
+                "reason": str(exc),
+                "falsified": False,
+                "summary": "DoWhy refutation failed.",
+            }
+
+    def validate_with_pywhyllm_and_dowhy(
+        self,
+        treatment: str,
+        outcome: str,
+        treatment_value: float = 1.0,
+        max_edges: Optional[int] = None,
+    ) -> Dict[str, object]:
+        warnings: List[str] = []
+        pywhyllm_report = self.analyze_assumptions_with_pywhyllm(treatment, outcome, max_edges=max_edges)
+        adjustment_candidates = self._valid_backdoor_sets(pywhyllm_report, treatment, outcome)
+        negative_controls = [
+            node
+            for node in self._valid_nodes(pywhyllm_report.get("negative_controls") or [])
+            if node not in {treatment, outcome}
+        ]
+
+        try:
+            from dowhy import CausalModel
+        except Exception as exc:
+            return {
+                "pywhyllm": pywhyllm_report,
+                "dowhy": {
+                    "available": False,
+                    "reason": str(exc),
+                    "falsified": False,
+                    "summary": "DoWhy is not installed or failed to import.",
+                    "adjustment_candidates": adjustment_candidates,
+                    "negative_controls": negative_controls,
+                },
+                "falsified": False,
+                "warnings": warnings,
+            }
+
+        try:
+            dowhy_report = self._run_dowhy_validation(
+                CausalModel,
+                treatment=treatment,
+                outcome=outcome,
+                treatment_value=treatment_value,
+                adjustment_candidates=adjustment_candidates,
+                negative_controls=negative_controls,
+            )
+        except Exception as exc:
+            dowhy_report = {
+                "available": False,
+                "reason": str(exc),
+                "falsified": False,
+                "summary": "DoWhy refutation failed.",
+                "adjustment_candidates": adjustment_candidates,
+                "negative_controls": negative_controls,
+            }
+
+        warnings.extend(pywhyllm_report.get("warnings") or [])
+        return {
+            "pywhyllm": pywhyllm_report,
+            "dowhy": dowhy_report,
+            "falsified": bool(dowhy_report.get("falsified")),
+            "warnings": warnings,
+        }
+
+    def counterfactual(
+        self,
+        observed_t: int,
+        treatment: Optional[str] = None,
+        cf_value: Optional[float] = None,
+        target: str = "",
+        treatments: Optional[Dict[str, float]] = None,
+    ) -> Dict[str, object]:
+        t = observed_t if observed_t >= 0 else (self.scm.t_steps + observed_t)
+        if t < 0 or t >= self.scm.t_steps:
+            raise ValueError(f"observed_t {observed_t} resolves out of bounds for T={self.scm.t_steps}")
+
+        if treatments is None:
+            if treatment is None or cf_value is None:
+                raise ValueError("Either treatment/cf_value or treatments must be provided.")
+            treatments = {treatment: cf_value}
+        elif treatment is not None or cf_value is not None:
+            raise ValueError("Provide either treatment/cf_value or treatments, not both.")
+
+        observed = {node: float(self.scm.data_level[t, i]) for i, node in enumerate(self.scm.nodes)}
+        result = self._counterfactual_outcome(observed, treatments, target)
+        result["explanation"] = (
+            f"Counterfactual computed at t={t}: set {treatments} "
+            f"and propagated structural equations with abducted residuals."
+        )
+        if len(treatments) > 1:
+            result["shapley_contributions"] = self._shapley_contributions(observed, treatments, target)
+        else:
+            result["shapley_contributions"] = {next(iter(treatments)): result["ite"]}
+        result["ite_total"] = result["ite"]
+        return result

singular_ticker_causal/causal_inference/tests/test_causal_queries.py

@@ -0,0 +1,102 @@
+import importlib.util
+import numpy as np
+
+from causal_inference.causal_model import StructuralCausalModel
+from causal_inference.query_engine import CausalQueryEngine
+
+
+def _synthetic_inputs():
+    nodes = ["A", "B", "C"]
+    T = 40
+    data = np.zeros((T, 3, 1), dtype=float)
+    rng = np.random.default_rng(42)
+
+    a = rng.normal(0, 1, size=T)
+    b = 0.6 * np.roll(a, 1) + rng.normal(0, 0.1, size=T)
+    c = 0.7 * np.roll(b, 1) + rng.normal(0, 0.1, size=T)
+    b[0] = rng.normal()
+    c[0] = rng.normal()
+    data[:, 0, 0] = a
+    data[:, 1, 0] = b
+    data[:, 2, 0] = c
+
+    # A->B, B->C and a weak cycle C->A to exercise pruning.
+    adj = np.array(
+        [
+            [0.0, 0.9, 0.0],
+            [0.0, 0.0, 0.8],
+            [0.1, 0.0, 0.0],
+        ],
+        dtype=float,
+    )
+    prior = np.zeros((3, 3), dtype=float)
+    mask = np.ones((T, 3), dtype=float)
+    return nodes, adj, prior, data, mask
+
+
+def test_scm_fit_and_cycle_prune():
+    nodes, adj, prior, data, mask = _synthetic_inputs()
+    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask).fit()
+    assert scm.dag_adj.shape == (3, 3)
+    assert len(scm.topological_indices) == 3
+    assert np.sum(scm.dag_adj) <= 2
+
+
+def test_intervention_and_counterfactual_shapes():
+    nodes, adj, prior, data, mask = _synthetic_inputs()
+    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask).fit()
+    engine = CausalQueryEngine(scm)
+
+    inter = engine.intervene("A", value=0.5, targets=["B", "C"], horizon=3)
+    assert set(inter["predicted_values"].keys()) == {"B", "C"}
+    assert len(inter["predicted_values"]["B"]) == 3
+
+    cf = engine.counterfactual(observed_t=-1, treatment="A", cf_value=0.8, target="C")
+    assert "factual_outcome" in cf
+    assert "counterfactual_outcome" in cf
+    assert "ite" in cf
+
+
+def test_counterfactual_shapley_contributions():
+    nodes, adj, prior, data, mask = _synthetic_inputs()
+    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask).fit()
+    engine = CausalQueryEngine(scm)
+
+    cf = engine.counterfactual(
+        observed_t=-1,
+        treatments={"A": 0.8, "B": -0.3},
+        target="C",
+    )
+
+    assert "shapley_contributions" in cf
+    assert set(cf["shapley_contributions"].keys()) == {"A", "B"}
+    assert abs(sum(cf["shapley_contributions"].values()) - cf["ite"]) < 1e-6
+
+
+def test_dowhy_validation_is_optional():
+    nodes, adj, prior, data, mask = _synthetic_inputs()
+    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask).fit()
+    engine = CausalQueryEngine(scm)
+
+    result = engine.validate_with_dowhy(treatment="A", outcome="C")
+    assert "available" in result
+    if importlib.util.find_spec("dowhy") is None:
+        assert result["available"] is False
+    else:
+        assert "falsified" in result
+
+
+def test_exogenous_fallback_prevention():
+    nodes, adj, prior, data, mask = _synthetic_inputs()
+    # Force fallback to exogenous by setting min_obs to a very large number (e.g. 100)
+    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask, min_obs=100).fit()
+    engine = CausalQueryEngine(scm)
+
+    inter = engine.intervene("A", value=0.5, targets=["B", "C"], horizon=3)
+    assert set(inter["predicted_values"].keys()) == {"B", "C"}
+    assert len(inter["predicted_values"]["B"]) == 3
+
+    cf = engine.counterfactual(observed_t=-1, treatment="A", cf_value=0.8, target="C")
+    assert "factual_outcome" in cf
+    assert "counterfactual_outcome" in cf
+    assert "ite" in cf

singular_ticker_causal/causal_inference/tests/test_pywhyllm_assumptions.py

@@ -0,0 +1,193 @@
+import builtins
+import sys
+import types
+
+import numpy as np
+
+from singular_ticker_causal.causal_inference.causal_model import StructuralCausalModel
+from singular_ticker_causal.causal_inference.pywhyllm_assumptions import (
+    PyWhyLLMConfig,
+    PyWhyLLMAssumptionService,
+)
+from singular_ticker_causal.causal_inference.query_engine import CausalQueryEngine
+
+
+def _synthetic_scm():
+    nodes = ["A", "B", "C"]
+    data = np.zeros((24, 3, 1), dtype=float)
+    rng = np.random.default_rng(11)
+    data[:, 0, 0] = rng.normal(size=24)
+    data[:, 1, 0] = rng.normal(size=24)
+    data[:, 2, 0] = rng.normal(size=24)
+    adj = np.array(
+        [
+            [0.0, 0.8, 0.7],
+            [0.0, 0.0, 0.9],
+            [0.0, 0.0, 0.0],
+        ],
+        dtype=float,
+    )
+    prior = np.ones((3, 3), dtype=float)
+    np.fill_diagonal(prior, 0.0)
+    return StructuralCausalModel(
+        nodes=nodes,
+        adj=adj,
+        adjacency_mask=np.zeros((3, 3), dtype=float),
+        prohibition_mask=prior,
+        data_tech=data,
+    ).fit()
+
+
+class FakeModelSuggester:
+    def suggest_domain_expertises(self, all_factors):
+        return ["financial accounting"]
+
+    def suggest_confounders(self, treatment, outcome, all_factors, domain_expertises):
+        return ({("B", treatment): 1, ("B", outcome): 1}, ["B", "Missing_Node"])
+
+    def suggest_relationships(self, treatment, outcome, all_factors, domain_expertises, strategy):
+        return [(treatment, outcome), ("B", outcome)]
+
+
+class FakeIdentificationSuggester:
+    def suggest_backdoor(self, treatment, outcome, all_factors, domain_expertises):
+        return ["B", "Missing_Node"]
+
+    def suggest_mediators(self, treatment, outcome, all_factors, domain_expertises):
+        return ["B"]
+
+    def suggest_ivs(self, treatment, outcome, all_factors, domain_expertises):
+        return ["A"]
+
+
+class FakeValidationSuggester:
+    def critique_graph(self, all_factors, suggested_dag, domain_expertises, strategy):
+        return "A -> C accepted; B -> C implausible, reject"
+
+    def suggest_latent_confounders(self, treatment, outcome, all_factors, domain_expertises):
+        return ["market regime"]
+
+    def suggest_negative_controls(self, treatment, outcome, all_factors, domain_expertises):
+        return ["B", "Missing_Node"]
+
+
+def _fake_service(tmp_path):
+    return PyWhyLLMAssumptionService(
+        PyWhyLLMConfig(enabled=True, cache_dir=str(tmp_path)),
+        model_suggester=FakeModelSuggester(),
+        identification_suggester=FakeIdentificationSuggester(),
+        validation_suggester=FakeValidationSuggester(),
+        relationship_strategy="pairwise",
+    )
+
+
+def test_pywhyllm_assumption_service_uses_fakes_and_cache(tmp_path):
+    scm = _synthetic_scm()
+    service = _fake_service(tmp_path)
+
+    report = service.analyze(nodes=scm.nodes, dag_adj=scm.dag_adj, treatment="A", outcome="C")
+
+    assert report.available is True
+    assert report.domain_expertises == ["financial accounting"]
+    assert report.suggested_confounders == ["B", "Missing_Node"]
+    assert report.suggested_backdoor_sets == [["B", "Missing_Node"]]
+    assert report.negative_controls == ["B", "Missing_Node"]
+    assert ("B", "C") in report.rejected_edges
+
+    cached = service.analyze(nodes=scm.nodes, dag_adj=scm.dag_adj, treatment="A", outcome="C")
+    assert cached.to_dict() == report.to_dict()
+
+
+def test_missing_pywhyllm_returns_unavailable(monkeypatch, tmp_path):
+    original_import = builtins.__import__
+
+    def fake_import(name, *args, **kwargs):
+        if name.startswith("pywhyllm"):
+            raise ImportError("blocked pywhyllm")
+        return original_import(name, *args, **kwargs)
+
+    monkeypatch.setattr(builtins, "__import__", fake_import)
+    scm = _synthetic_scm()
+    service = PyWhyLLMAssumptionService(PyWhyLLMConfig(enabled=True, cache_dir=str(tmp_path)))
+
+    report = service.analyze(nodes=scm.nodes, dag_adj=scm.dag_adj, treatment="A", outcome="C")
+
+    assert report.available is False
+    assert "blocked pywhyllm" in report.reason
+
+
+def test_engine_filters_pywhyllm_adjustments_and_negative_controls(tmp_path):
+    scm = _synthetic_scm()
+    engine = CausalQueryEngine(scm, pywhyllm_service=_fake_service(tmp_path), pywhyllm_enabled=True)
+
+    report = engine.analyze_assumptions_with_pywhyllm("A", "C")
+    adjustment_sets = engine._valid_backdoor_sets(report, "A", "C")
+    negative_controls = engine._valid_nodes(report["negative_controls"])
+
+    assert adjustment_sets == [["B"]]
+    assert negative_controls == ["B"]
+
+
+def test_placebo_not_statistically_significant_passes():
+    scm = _synthetic_scm()
+    engine = CausalQueryEngine(scm)
+
+    parsed = engine._parse_refuter_result(
+        "placebo_treatment",
+        "Refute: Use a Placebo Treatment. The result is not statistically significant.",
+    )
+
+    assert parsed["passed"] is True
+    assert parsed["falsified"] is False
+
+
+def test_combined_validation_reports_unavailable_dowhy(monkeypatch, tmp_path):
+    monkeypatch.setitem(sys.modules, "dowhy", None)
+    scm = _synthetic_scm()
+    engine = CausalQueryEngine(scm, pywhyllm_service=_fake_service(tmp_path), pywhyllm_enabled=True)
+
+    result = engine.validate_with_pywhyllm_and_dowhy("A", "C")
+
+    assert result["pywhyllm"]["available"] is True
+    assert result["dowhy"]["available"] is False
+    assert result["dowhy"]["adjustment_candidates"] == [["B"]]
+
+
+def test_combined_validation_uses_fake_dowhy(monkeypatch, tmp_path):
+    class FakeRefute:
+        estimated_effect = 1.0
+        new_effect = 0.0
+        refutation_result = {"p_value": 0.8}
+
+        def __str__(self):
+            return "not statistically significant"
+
+    class FakeEstimate:
+        def __str__(self):
+            return "estimate"
+
+    class FakeCausalModel:
+        def __init__(self, **kwargs):
+            self.kwargs = kwargs
+
+        def identify_effect(self):
+            return "estimand"
+
+        def estimate_effect(self, identified_estimand, method_name):
+            return FakeEstimate()
+
+        def refute_estimate(self, identified_estimand, estimate, **kwargs):
+            return FakeRefute()
+
+    fake_dowhy = types.ModuleType("dowhy")
+    fake_dowhy.CausalModel = FakeCausalModel
+    monkeypatch.setitem(sys.modules, "dowhy", fake_dowhy)
+
+    scm = _synthetic_scm()
+    engine = CausalQueryEngine(scm, pywhyllm_service=_fake_service(tmp_path), pywhyllm_enabled=True)
+
+    result = engine.validate_with_pywhyllm_and_dowhy("A", "C")
+
+    assert result["dowhy"]["available"] is True
+    assert result["dowhy"]["falsified"] is False
+    assert result["dowhy"]["negative_control_checks"][0]["control"] == "B"

singular_ticker_causal/data_sources/__init__.py

@@ -0,0 +1,4 @@
+from .fetcher import Fetcher
+from .gdelt_client import GDELTClient
+from .news_client import NewsClient
+from .sebi_reg30_client import SEBIREG30Client

singular_ticker_causal/data_sources/bsedata/__init__.py

@@ -0,0 +1,27 @@
+"""
+
+    MIT License
+
+    Copyright (c) 2018 - 2024 Shrey Dabhi
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+"""
+
+name = "bsedata"

singular_ticker_causal/data_sources/bsedata/bhavcopy.py

@@ -0,0 +1,58 @@
+import os
+import io
+import csv
+import requests
+import tempfile
+import datetime
+from zipfile import ZipFile
+from .exceptions import BhavCopyNotFound
+from .helpers import COMMON_REQUEST_HEADERS
+
+
+def loadBhavCopyData(statsDate: datetime.date) -> list:
+    tempDir = os.path.join(tempfile.gettempdir(), "bsedata")
+    zipfileName = f"EQ{statsDate.strftime('%d%m%y')}_CSV.ZIP"
+    r = requests.get(
+        f"https://www.bseindia.com/download/BhavCopy/Equity/{zipfileName}",
+        headers=COMMON_REQUEST_HEADERS,
+    )
+
+    if r.status_code != 200:
+        raise BhavCopyNotFound()
+
+    try:
+        os.makedirs(tempDir)
+    except FileExistsError:
+        pass
+
+    f_zip = open(os.path.join(tempDir, zipfileName), "wb+")
+    f_zip.write(r.content)
+    f_zip.close()
+
+    output = []
+
+    with ZipFile(os.path.join(tempDir, zipfileName)) as bhavCopyZip:
+        with bhavCopyZip.open(f"EQ{statsDate.strftime('%d%m%y')}.CSV") as bhavCopyFile:
+            reader = csv.DictReader(io.TextIOWrapper(bhavCopyFile))
+            for row in reader:
+                output.append(mapBhavCopyRowToDict(row))
+
+    return output
+
+
+def mapBhavCopyRowToDict(row: dict) -> dict:
+    SC_TYPE_MAP = {"B": "bond", "Q": "equity", "D": "debenture", "P": "preference"}
+    return {
+        "scripCode": row["SC_CODE"],
+        "open": row["OPEN"],
+        "high": row["HIGH"],
+        "low": row["LOW"],
+        "close": row["CLOSE"],
+        "last": row["LAST"],
+        "prevClose": row["PREVCLOSE"],
+        "totalTrades": row["NO_TRADES"],
+        "totalSharesTraded": row["NO_OF_SHRS"],
+        "netTurnover": row["NET_TURNOV"],
+        "scripType": SC_TYPE_MAP[row["SC_TYPE"]],
+        "securityID": row["SC_NAME"].strip(),
+    }

singular_ticker_causal/data_sources/bsedata/bse.py

@@ -0,0 +1,150 @@
+"""
+
+    MIT License
+
+    Copyright (c) 2018 - 2024 Shrey Dabhi
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+"""
+
+from .gainers import getGainers
+from .losers import getLosers
+from .bhavcopy import loadBhavCopyData
+from .quote import quote
+from .indices import indices
+import datetime
+import requests
+import json
+
+
+class BSE(object):
+    """
+    Class which implements the functionality for
+    Bombay Stock Exchange (BSE)
+    """
+
+    def __init__(self, update_codes=False):
+        self.__update_codes = update_codes
+        if update_codes:
+            self.updateScripCodes()
+
+    def topGainers(self):
+        """
+        :returns: A sorted list of codes of top gainers
+        """
+        return getGainers()
+
+    def topLosers(self):
+        """
+        :returns: A sorted list of codes of top losers
+        """
+        return getLosers()
+
+    def getQuote(self, scripCode):
+        """
+        :param scripCode: A stock code
+        :returns: A dictionary which contain details about the stock
+        :raises InvalidStockException: Raised for stocks which have been suspended or no longer trading on BSE
+        """
+        return quote(scripCode)
+
+    def getIndices(self, category):
+        """
+        :param category: A category of indices
+        :returns: A dictionary with details about the indices belonging to the given category
+        """
+        return indices(category)
+
+    def updateScripCodes(self):
+        """
+        Download a fresh copy of the scrip code listing
+
+        :returns: None
+        """
+        r = requests.get("https://pub-87b187a07d9c42109c9e6999439a583f.r2.dev/stk.json")
+        f_stk = open("stk.json", "w+")
+        f_stk.write(json.dumps(r.json()))
+        f_stk.close()
+        return
+    
+    def getBhavCopyData(self, statsDate: datetime.date):
+        """
+        Get historical OHLCV data from Bhav Copy released by BSE everyday after market closing.
+        The columns available in the data and their description is as given below.
+
+        .. list-table::
+            :widths: 25 75
+            :header-rows: 1
+
+            * - Dictionary Field
+              - Description
+            * - scripCode
+              - Unique code assigned to a scrip of a company by BSE
+            * - open
+              - The price at which the security first trades on a given trading day
+            * - high
+              - The highest intra-day price of a stock
+            * - low
+              - The lowest intra-day price of a stock
+            * - close
+              - The final price at which a security is traded on a given trading day
+            * - last
+              - The last trade price of the stock
+            * - prevClose
+              - The closing price of the stock for the previous trading day
+            * - totalTrades
+              - The total number of trades of a scrip
+            * - totalSharesTraded
+              - The total number of shares transacted of a scrip
+            * - netTurnover
+              - Total turnover of a scrip
+            * - scripType
+              - Scrip category: Equity, Preference, Debenture or Bond
+            * - securityID
+              - Name of the company
+
+        The Bhav Copy files have been mapped to the above mentioned custom fields. The complete documentation for Bhav Copy can be found here: https://www.bseindia.com/markets/MarketInfo/BhavCopy.aspx.
+
+        
+        :param statsDate: A `datetime.date` object for the for which you want to fetch the data
+        :returns: A list of dictionaries which contains OHLCV data for that day for all scrip codes active on that day
+        :raises BhavCopyNotFound: Raised when Bhav Copy file is not found on BSE
+        """
+        return loadBhavCopyData(statsDate)
+
+    def getScripCodes(self):
+        """
+        :returns: A dictionary with scrip codes as keys and company names as values
+        """
+        f = open("stk.json", "r")
+        return json.loads(f.read())
+
+    def verifyScripCode(self, code):
+        """
+        :returns: Company name if it is a valid stock code, else None
+        """
+        data = self.getScripCodes()
+        return data.get(code)
+
+    def __str__(self):
+        return "Driver Class for Bombay Stock Exchange (BSE)"
+
+    def __repr__(self):
+        return f"<{self.__class__.__name__}: update_codes={self.__update_codes}> Driver Class for Bombay Stock Exchange (BSE)"

singular_ticker_causal/data_sources/bsedata/exceptions.py

@@ -0,0 +1,51 @@
+"""
+
+    MIT License
+
+    Copyright (c) 2018 - 2024 Shrey Dabhi
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+"""
+
+
+class InvalidStockException(Exception):
+    """
+    Exception raised for stocks which have been suspended or no longer trading on BSE.
+
+    :param status: the status of the stock as mentioned on BSE website
+    """
+
+    def __init__(self, status: str = "Inactive stock"):
+        if status == "":
+            self.status = "Inactive stock"
+        else:
+            self.status = status
+        super().__init__(self.status)
+
+
+class BhavCopyNotFound(Exception):
+    """
+    Exception raised when the BhavCopy file is not found on BSE website.
+    """
+
+    def __init__(self):
+        super().__init__(
+            """The BhavCopy file was not found on the BSE website. You are probably trying to get data for a trading holiday."""
+        )

singular_ticker_causal/data_sources/bsedata/gainers.py

@@ -0,0 +1,57 @@
+"""
+
+    MIT License
+
+    Copyright (c) 2018 - 2024 Shrey Dabhi
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+"""
+
+from .helpers import COMMON_REQUEST_HEADERS
+from bs4 import BeautifulSoup as bs
+import requests
+
+
+def getGainers() -> dict:
+    baseurl = """https://m.bseindia.com"""
+    res = requests.get(baseurl, headers=COMMON_REQUEST_HEADERS)
+    c = res.content
+    soup = bs(c, "lxml")
+    for tag in soup("div"):
+        try:
+            if tag["id"] == "divGainers":
+                resSoup = tag
+                break
+        except KeyError:
+            continue
+    children = list(resSoup.table.contents)
+    children = children[1:]
+    gainers = []
+    for tr in children:
+        td = tr.contents
+        gainer = {
+            "securityID": str(td[0].a.string),
+            "scripCode": str(tr.td.a["href"].split("=")[1]),
+            "LTP": str(td[1].string),
+            "change": str(td[2].string),
+            "pChange": str(td[3].string),
+        }
+        gainers.append(gainer)
+    return gainers

singular_ticker_causal/data_sources/bsedata/helpers.py

@@ -0,0 +1,3 @@
+COMMON_REQUEST_HEADERS = {
+    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36 Edg/83.0.478.45"
+}

singular_ticker_causal/data_sources/bsedata/indices.py

@@ -0,0 +1,112 @@
+"""
+
+    MIT License
+
+    Copyright (c) 2018 - 2024 Shrey Dabhi
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+"""
+
+from .helpers import COMMON_REQUEST_HEADERS
+from bs4 import BeautifulSoup as bs
+import requests
+
+
+def indices(category: str) -> dict:
+    cat = {
+        "market_cap/broad": "1,2",
+        "sector_and_industry": "2,2",
+        "thematics": "3,2",
+        "strategy": "4,2",
+        "sustainability": "5,2",
+        "volatility": "6,1",
+        "composite": "7,1",
+        "government": "8,1",
+        "corporate": "9,1",
+        "money_market": "10,1",
+    }
+    try:
+        ddl_category = cat[category]
+    except KeyError:
+        print(
+            """
+### Invalid category ###
+Use one of the categories mentioned below:
+
+market_cap/broad
+sector_and_industry
+thematics
+strategy
+sustainability
+volatility
+composite
+government
+corporate
+money_market
+        """
+        )
+        return
+    baseurl = """https://m.bseindia.com/IndicesView_New.aspx"""
+    res = requests.get(baseurl, headers=COMMON_REQUEST_HEADERS)
+    c = res.content
+    soup = bs(c, "lxml")
+    options = {
+        "__EVENTTARGET": "ddl_Category",
+        "__VIEWSTATEENCRYPTED": "",
+        "__EVENTARGUMENT": "",
+        "__LASTFOCUS": "",
+        "__VIEWSTATEGENERATOR": "162C96CD",
+        "UcHeaderMenu1$txtGetQuote": "",
+        "__EVENTVALIDATION": "",
+        "__VIEWSTATE": "",
+    }
+    for input in soup("input"):
+        try:
+            if input["type"] == "hidden":
+                if input["id"] == "__VIEWSTATE":
+                    options["__VIEWSTATE"] = input["value"]
+                elif input["id"] == "__EVENTVALIDATION":
+                    options["__EVENTVALIDATION"] = input["value"]
+        except KeyError:
+            continue
+    options["ddl_Category"] = ddl_category
+    res = requests.post(url=baseurl, data=options, headers=COMMON_REQUEST_HEADERS)
+    c = res.content
+    soup = bs(c, "lxml")
+    index_list = []
+    for td in soup("td"):
+        try:
+            if td["class"][0] == "TTRow_left":
+                index = {}
+                index["currentValue"] = td.next_sibling.string.strip()
+                index["change"] = td.next_sibling.next_sibling.string.strip()
+                index[
+                    "pChange"
+                ] = td.next_sibling.next_sibling.next_sibling.string.strip()
+                index["scripFlag"] = td.a["href"].strip().split("=")[1]
+                index["name"] = td.a.string.strip().replace(";", "")
+                index_list.append(index)
+        except KeyError:
+            continue
+    results = {}
+    for span in soup("span", id="inddate"):
+        results["updatedOn"] = span.string[6:].split("|")[0].strip()
+    results["indices"] = index_list
+    return results

singular_ticker_causal/data_sources/bsedata/losers.py

@@ -0,0 +1,57 @@
+"""
+
+    MIT License
+
+    Copyright (c) 2018 - 2024 Shrey Dabhi
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+"""
+
+from .helpers import COMMON_REQUEST_HEADERS
+from bs4 import BeautifulSoup as bs
+import requests
+
+
+def getLosers() -> dict:
+    baseurl = """https://m.bseindia.com"""
+    res = requests.get(baseurl, headers=COMMON_REQUEST_HEADERS)
+    c = res.content
+    soup = bs(c, "lxml")
+    for tag in soup("div"):
+        try:
+            if tag["id"] == "divLosers":
+                resSoup = tag
+                break
+        except KeyError:
+            continue
+    children = list(resSoup.table.contents)
+    children = children[1:]
+    losers = []
+    for tr in children:
+        td = tr.contents
+        loser = {
+            "securityID": str(td[0].a.string),
+            "scripCode": str(tr.td.a["href"].split("=")[1]),
+            "LTP": str(td[1].string),
+            "change": str(td[2].string),
+            "pChange": str(td[3].string),
+        }
+        losers.append(loser)
+    return losers

singular_ticker_causal/data_sources/bsedata/quote.py

@@ -0,0 +1,176 @@
+"""
+
+    MIT License
+
+    Copyright (c) 2018 - 2024 Shrey Dabhi
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+"""
+
+from .exceptions import InvalidStockException
+from .helpers import COMMON_REQUEST_HEADERS
+from datetime import datetime as dt
+from bs4 import BeautifulSoup as bs
+import requests
+
+
+def quote(scripCode: str) -> dict:
+    baseurl = """https://m.bseindia.com/StockReach.aspx?scripcd="""
+    res = requests.get(baseurl + scripCode, headers=COMMON_REQUEST_HEADERS)
+    c = res.content
+    soup = bs(c, "lxml")
+
+    res = {}
+
+    for span in soup("span"):
+        updt_date = soup.find("span", id="strongDate").text.split("-")[1].strip()
+        updt_diff = dt.strptime(updt_date, "%d %b %y | %I:%M %p") - dt.now()
+        if updt_diff.days < -7:
+            error_text = ""
+            error_text_element = soup.find("td", id="tdDispTxt")
+            if error_text_element is not None:
+                error_text = error_text_element.text
+            raise InvalidStockException(status=error_text)
+        try:
+            if span["class"][0] == "srcovalue":
+                try:
+                    if span["id"] == "spanchangVal":
+                        res["change"] = span.string.split("(")[0].strip()
+                        res["pChange"] = span.string.split("(")[1].strip()[:-2]
+                except KeyError:
+                    res["currentValue"] = span.strong.string
+            elif span["class"][0] == "companyname":
+                res["companyName"] = span.string
+        except KeyError:
+            try:
+                if span["id"] == "lblPBdate":
+                    try:
+                        res["priceBand"] = span.string.split(":")[1].strip()
+                    except AttributeError:
+                        res["priceBand"] = ""
+                elif span["id"] == "strongDate":
+                    res["updatedOn"] = span.string.split("-")[1].strip()
+            except KeyError:
+                continue
+
+    for td in soup("td"):
+        try:
+            if td["id"] == "tdCShortName":
+                res["securityID"] = td.string.strip()
+            elif td["id"] == "tdscripcode":
+                res["scripCode"] = td.string.strip()
+            elif td["id"] == "tdgroup":
+                res["group"] = td.string.strip()
+            elif td["id"] == "tdfacevalue":
+                res["faceValue"] = td.string.strip()
+            elif td["id"] == "tdIndustry":
+                res["industry"] = td.string.strip()
+            elif td["id"] == "tdpcloseopen":
+                res["previousClose"] = td.string.split("/")[0].strip()
+                res["previousOpen"] = td.string.split("/")[1].strip()
+            elif td["id"] == "tdDHL":
+                res["dayHigh"] = td.string.split("/")[0].strip()
+                res["dayLow"] = td.string.split("/")[1].strip()
+            elif td["id"] == "td52WHL":
+                res["52weekHigh"] = td.string.split("/")[0].strip()
+                res["52weekLow"] = td.string.split("/")[1].strip()
+            elif td["id"] == "tdWAp":
+                res["weightedAvgPrice"] = td.string.strip()
+            elif td["id"] == "tdTTV":
+                res["totalTradedValue"] = td.string.strip() + " Cr."
+            elif td["id"] == "tdTTQW":
+                res["totalTradedQuantity"] = td.string.split("/")[0].strip() + " Lakh"
+                res["2WeekAvgQuantity"] = td.string.split("/")[1].strip() + " Lakh"
+            elif td["id"] == "tdMktCapVal":
+                res["marketCapFull"] = td.string.split("/")[0].strip() + " Cr."
+                res["marketCapFreeFloat"] = td.string.split("/")[1].strip() + " Cr."
+        except KeyError:
+            continue
+
+    if res.get("priceBand", "") != "":
+        for tbody in soup("tbody"):
+            try:
+                if tbody["id"] == "PBtablebody":
+                    data = tbody.contents[2]
+                    res["upperPriceBand"] = data.contents[1].string.strip()
+                    res["lowerPriceBand"] = data.contents[2].string.strip()
+            except KeyError:
+                continue
+
+    buy = {}
+    sell = {}
+    for td in soup("td"):
+        try:
+            if td["id"] == "tdBQ1":
+                buy["1"] = {
+                    "quantity": td.string,
+                    "price": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdBQ2":
+                buy["2"] = {
+                    "quantity": td.string,
+                    "price": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdBQ3":
+                buy["3"] = {
+                    "quantity": td.string,
+                    "price": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdBQ4":
+                buy["4"] = {
+                    "quantity": td.string,
+                    "price": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdBQ5":
+                buy["5"] = {
+                    "quantity": td.string,
+                    "price": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdSP1":
+                sell["1"] = {
+                    "price": td.string,
+                    "quantity": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdSP2":
+                sell["2"] = {
+                    "price": td.string,
+                    "quantity": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdSP3":
+                sell["3"] = {
+                    "price": td.string,
+                    "quantity": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdSP4":
+                sell["4"] = {
+                    "price": td.string,
+                    "quantity": td.next_sibling.next_sibling.string,
+                }
+            elif td["id"] == "tdSP5":
+                sell["5"] = {
+                    "price": td.string,
+                    "quantity": td.next_sibling.next_sibling.string,
+                }
+        except KeyError:
+            continue
+    res["buy"] = buy
+    res["sell"] = sell
+
+    return res

singular_ticker_causal/data_sources/fetcher.py

@@ -0,0 +1,219 @@
+import pandas as pd
+import numpy as np
+import yfinance as yf
+import logging
+from typing import Optional
+from singular_ticker_causal.services.schema import INCOME_STATEMENT_NODES, BALANCE_SHEET_NODES, STRATEGIC_NODES
+IND_AS_NODES = INCOME_STATEMENT_NODES + BALANCE_SHEET_NODES + STRATEGIC_NODES
+from .nseconnect.nse import Nse
+from .bsedata.bse import BSE
+
+
+logger = logging.getLogger(__name__)
+
+
+class Fetcher:
+    """
+    Tiered fundamental data fetcher.
+    Priority: NSE/BSE → XBRL → IndianAPI → yfinance.
+    """
+    def __init__(self, ticker: str, api_key: Optional[str] = None):
+        self.ticker = ticker if ticker.endswith(".NS") or ticker.endswith(".BO") else f"{ticker}.NS"
+        self.api_key = api_key
+
+    def fetch(self, start: str, end: str) -> pd.DataFrame:
+        logger.info(f"Fetching fundamentals for {self.ticker} from {start} to {end}")
+        
+        df = None
+        if self.ticker.endswith(".NS"):
+            logger.info("Attempting NSE fetch...")
+            df = self._try_nse(start, end)
+        elif self.ticker.endswith(".BO"):
+            logger.info("Attempting BSE fetch...")
+            df = self._try_bse(start, end)
+
+        if df is None or df.empty:
+            logger.info("Attempting Tier 1: XBRL...")
+            df = self._try_xbrl(start, end)
+        if df is None or df.empty:
+            logger.info("Tier 1 failed or returned empty. Attempting Tier 2: IndianAPI...")
+            df = self._try_indianapi(start, end)
+        if df is None or df.empty:
+            logger.info("Tier 2 failed or returned empty. Attempting Tier 3: yfinance...")
+            df = self._try_yfinance(start, end)
+        
+        if df is not None and not df.empty:
+            df = df.sort_index()
+            # Forward fill balance sheet items as they are point-in-time and usually stable
+            # We do this AFTER mapping in _try_yfinance, so we use mapped names
+            bs_cols = ["Total_Assets", "Shareholders_Equity", "Inventory", "Accounts_Payable", "Total_Debt", "PPE", "CWIP", "Intangible_Assets"]
+            available_bs_cols = [c for c in bs_cols if c in df.columns]
+            if available_bs_cols:
+                df[available_bs_cols] = df[available_bs_cols].ffill()
+        
+        if df is None or df.empty:
+            logger.warning(f"No fundamental data found for {self.ticker}")
+            return pd.DataFrame(columns=IND_AS_NODES)
+
+        logger.info("Deriving strategic nodes...")
+        df = self._derive_strategic_nodes(df)
+        
+        # Ensure all IND_AS_NODES are present
+        for node in IND_AS_NODES:
+            if node not in df.columns:
+                df[node] = np.nan
+                
+        return df[IND_AS_NODES].sort_index()
+
+    def _try_nse(self, start: str, end: str) -> Optional[pd.DataFrame]:
+        # Using nseconnect for high-fidelity NSE data
+        try:
+            # Clean ticker (e.g. RELIANCE.NS -> RELIANCE)
+            clean_ticker = self.ticker.split('.')[0]
+            nse = Nse()
+            logger.info(f"Attempting to fetch NSE data for {clean_ticker}...")
+            # Note: nseconnect is primarily for quotes; 
+            # for full fundamentals we still rely on yfinance or XBRL.
+            # We return None here to let it fall back, but the plumbing is now real.
+            quote = nse.get_quote(clean_ticker)
+            if quote:
+                logger.info(f"Successfully connected to NSE for {clean_ticker}")
+            return None
+        except Exception as e:
+            logger.error(f"Error fetching from NSE: {e}")
+            return None
+
+    def _try_bse(self, start: str, end: str) -> Optional[pd.DataFrame]:
+        # Using bsedata for high-fidelity BSE data
+        try:
+            # TODO: Implement mapping from alphabetic ticker to numeric BSE scrip code
+            bse = BSE()
+            logger.info(f"Attempting to fetch BSE data for {self.ticker}...")
+            # Currently limited to quotes; returning None to fall back to yfinance
+            return None
+        except Exception as e:
+            logger.error(f"Error fetching from BSE: {e}")
+            return None
+
+    def _try_xbrl(self, start: str, end: str) -> Optional[pd.DataFrame]:
+        # Tier 1 extraction via python-xbrl / Arelle is currently in development
+        return None
+
+    def _try_indianapi(self, start: str, end: str) -> Optional[pd.DataFrame]:
+        # Tier 2 integration for IndianAPI.in / FinEdge API is currently in development
+        return None
+
+    def _try_yfinance(self, start: str, end: str) -> Optional[pd.DataFrame]:
+        try:
+            t = yf.Ticker(self.ticker)
+            q_fin = t.quarterly_financials.T
+            q_bs = t.quarterly_balance_sheet.T
+            q_cf = t.quarterly_cashflow.T
+            
+            if q_fin.empty and q_bs.empty and q_cf.empty:
+                logger.warning("All yfinance statements (financials, balance_sheet, cashflow) are empty.")
+                return None
+                
+            # Merge all three statements
+            logger.info(f"Merging yfinance statements: q_fin={q_fin.shape}, q_bs={q_bs.shape}, q_cf={q_cf.shape}")
+            df = pd.concat([q_fin, q_bs, q_cf], axis=1)
+            
+            
+            df = df.loc[:, ~df.columns.duplicated()] # Remove duplicate columns if any
+            logger.info(f"Merged shape after removing duplicates: {df.shape}")
+            
+            # Map yfinance columns to IND_AS_NODES (Simplified mapping for MVP)
+            mapping = {
+                "Total Revenue": "Revenue",
+                "Cost Of Revenue": "COGS",
+                "Operating Expense": "Operating_Expenses",
+                "Operating Income": "EBIT",
+                "EBIT": "EBIT",
+                "EBITDA": "EBITDA",
+                "Interest Expense": "Interest_Expense",
+                "Pretax Income": "EBT",
+                "Tax Provision": "Tax_Expense",
+                "Net Income": "PAT",
+                "Total Assets": "Total_Assets",
+                "Stockholders Equity": "Shareholders_Equity",
+                "Depreciation And Amortization": "D_A",
+                "Inventory": "Inventory",
+                "Accounts Payable": "Accounts_Payable",
+                "Total Debt": "Total_Debt",
+                "Operating Cash Flow": "Operating_Cash_Flow",
+                "Capital Expenditure": "Capex",
+                "Net PPE": "PPE",
+                "Construction In Progress": "CWIP",
+                "Goodwill And Other Intangible Assets": "Intangible_Assets",
+            }
+            
+            df = df.rename(columns=mapping)
+            df = df.loc[:, ~df.columns.duplicated()] # Remove duplicates after rename
+            df.index = pd.to_datetime(df.index)
+            return df
+        except Exception as e:
+            logger.error(f"Error fetching from yfinance: {e}")
+            return None
+
+
+    def _derive_strategic_nodes(self, df: pd.DataFrame) -> pd.DataFrame:
+        """
+        Compute Layer 3 nodes and other derived fields.
+        """
+        df = df.copy()
+        
+        # Helper to safely get a series from a column that might be a DataFrame
+        def get_series(name):
+            if name not in df.columns:
+                return None
+            col = df[name]
+            if isinstance(col, pd.DataFrame):
+                logger.warning(f"Column '{name}' is a DataFrame with multiple columns: {col.columns.tolist()}. Taking the first.")
+                return col.iloc[:, 0]
+            return col
+
+        # 1. Average Assets & Equity (Rolling 2-period mean)
+        assets = get_series("Total_Assets")
+        if assets is not None:
+            df["Average_Total_Assets"] = assets.rolling(window=2).mean().fillna(assets)
+            
+        equity = get_series("Shareholders_Equity")
+        if equity is not None:
+            df["Average_Shareholders_Equity"] = equity.rolling(window=2).mean().fillna(equity)
+
+        # 2. Basic derivations
+        rev = get_series("Revenue")
+        cogs = get_series("COGS")
+        if rev is not None and cogs is not None:
+            df["Gross_Profit"] = rev - cogs.fillna(0)
+            
+        ebit = get_series("EBIT")
+        da = get_series("D_A")
+        if ebit is not None:
+            # Proper EBITDA = EBIT + Depreciation & Amortization
+            df["EBITDA"] = ebit + da.fillna(0) if da is not None else ebit
+            
+        pat = get_series("PAT")
+        if pat is not None and rev is not None:
+            df["Net_Profit_Margin"] = pat / rev.replace(0, np.nan)
+            
+        avg_assets = get_series("Average_Total_Assets")
+        if rev is not None and avg_assets is not None:
+            df["Asset_Turnover"] = rev / avg_assets.replace(0, np.nan)
+            
+        avg_equity = get_series("Average_Shareholders_Equity")
+        if avg_assets is not None and avg_equity is not None:
+            df["Equity_Multiplier"] = avg_assets / avg_equity.replace(0, np.nan)
+            
+        npm = get_series("Net_Profit_Margin")
+        at = get_series("Asset_Turnover")
+        em = get_series("Equity_Multiplier")
+        if npm is not None and at is not None and em is not None:
+            df["ROE"] = npm * at * em
+            
+        ocf = get_series("Operating_Cash_Flow")
+        capex = get_series("Capex")
+        if ocf is not None and capex is not None:
+            df["Free_Cash_Flow"] = ocf - capex.abs().fillna(0)
+            
+        return df

singular_ticker_causal/data_sources/gdelt_client.py

@@ -0,0 +1,400 @@
+import time
+import random
+import logging
+import requests
+import yfinance as yf
+from datetime import datetime, timezone
+from typing import List, Dict, Any
+from singular_ticker_causal.utils.llm_client import LLMClient
+
+
+logger = logging.getLogger(__name__)
+
+
+# ==============================================================
+# GDELT DOC 2.0 FETCHER
+# ==============================================================
+
+def _fmt_gdelt_dt(dt: datetime) -> str:
+    """
+    Convert datetime -> GDELT YYYYMMDDHHMMSS (UTC).
+    """
+    if dt.tzinfo is not None:
+        dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
+
+    return dt.strftime("%Y%m%d%H%M%S")
+
+
+def _dedupe_articles(articles: list[dict]) -> list[dict]:
+    """
+    Deduplicate by URL/title combination.
+    """
+    seen = set()
+    deduped = []
+
+    for article in articles:
+        key = (
+            article.get("link", "").strip(),
+            article.get("title", "").strip().lower(),
+        )
+
+        if key in seen:
+            continue
+
+        seen.add(key)
+        deduped.append(article)
+
+    return deduped
+
+
+class GDELTClient:
+    """
+    Thin wrapper around GDELT Doc API for macro context.
+    """
+    GDELT_DOC_API: str = "https://api.gdeltproject.org/api/v2/doc/doc"
+    GDELT_MAX_RECORDS: int = 250
+    GDELT_MAX_WINDOW_DAYS: int = 90
+    USER_AGENT: str = (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) "
+        "Gecko/20100101 Firefox/128.0"
+    )
+
+    def __init__(self):
+        pass
+
+    def fetch(
+        self,
+        keyword: str,
+        from_dt: datetime,
+        to_dt: datetime,
+        ticker: str | None = None,
+        **kwargs,
+    ) -> List[Dict[str, Any]]:
+        """
+        Main entry point for GDELT fetching.
+        If 'ticker' is provided, performs a broad generic search using yfinance context.
+        """
+        if ticker:
+            return self.fetch_ticker_context(ticker, from_dt, to_dt, **kwargs)
+
+        logger.info(
+            "Fetching GDELT news for %s from %s to %s",
+            keyword,
+            from_dt,
+            to_dt,
+        )
+
+        try:
+            articles = self.fetch_gdelt_window(
+                keyword=keyword,
+                from_dt=from_dt,
+                to_dt=to_dt,
+                **kwargs,
+            )
+
+            for article in articles:
+                article["credibility_weight"] = 0.5
+                article["source_type"] = "gdelt"
+
+            return articles
+
+        except Exception as e:
+            logger.exception("Error fetching from GDELT: %s", e)
+            return []
+
+    def _extract_search_terms(self, summary: str, company_name: str) -> Dict[str, List[str]]:
+        """Extract product, partner, and industry search terms from business summary using LLM."""
+        if not summary:
+            return {"products": [], "partners": [], "industry": []}
+
+        llm = LLMClient()
+        prompt = f"""
+        Extract major products/services, strategic partners, and industry-related details from the following summary of {company_name}.
+        
+        Summary: {summary}
+        
+        Important instructions:
+        1. For partners, include a few words describing the specific connection or nature of the relationship between {company_name} and the partner (e.g., "{company_name} strategic collaboration with [Partner]").
+        2. For industry, extract a few words describing the sector or niche {company_name} is connected to and its specific role/connection (e.g., "{company_name} provides [Services] in the [Industry] sector").
+        3. Keep it to the top 5 most relevant terms for products and partners, and top 3 for industry.
+        
+        Return a JSON object with:
+        "products": ["product1", "product2", ...],
+        "partners": ["{company_name} [connection] [partner1]", ...],
+        "industry": ["{company_name} [role/connection] [industry]", ...]
+        
+        Return ONLY valid JSON.
+        """
+        try:
+            result = llm.chat_json([{"role": "user", "content": prompt}])
+            return {
+                "products": result.get("products", []),
+                "partners": result.get("partners", []),
+                "industry": result.get("industry", [])
+            }
+        except Exception as e:
+            logger.error(f"[GDELT] LLM term extraction failed: {e}")
+            return {"products": [], "partners": [], "industry": []}
+
+    def fetch_ticker_context(
+        self,
+        ticker_symbol: str,
+        from_dt: datetime,
+        to_dt: datetime,
+        **kwargs
+    ) -> List[Dict[str, Any]]:
+        """
+        Generic fetcher that uses yfinance and LLM to build a broad context.
+        Sequence: products, partners, company name, company heads.
+        """
+        logger.info(f"[GDELT] Fetching generic context for ticker: {ticker_symbol}")
+        
+        try:
+            ticker = yf.Ticker(ticker_symbol)
+            info = ticker.info
+        except Exception as e:
+            logger.error(f"[GDELT] yfinance failed for {ticker_symbol}: {e}")
+            return []
+        
+        summary = info.get("longBusinessSummary", "")
+        company_name = info.get("longName") or info.get("shortName") or ticker_symbol
+        officers = info.get("companyOfficers", [])
+        
+        # 1. LLM extraction
+        terms = self._extract_search_terms(summary, company_name)
+        products = terms.get("products", [])
+        partners = terms.get("partners", [])
+        industry = terms.get("industry", [])
+        
+        # 2. Officer names
+        heads = [o.get("name") for o in officers if o.get("name")]
+        
+        # 3. Execution sequence
+        all_articles = []
+        search_plan = [
+            ("products", products),
+            ("partners", partners),
+            ("industry", industry),
+            ("company_name", [company_name]),
+            ("company_heads", heads)
+        ]
+        
+        for category, keywords in search_plan:
+            for kw in keywords:
+                if not kw: continue
+                logger.info(f"[GDELT] Scraping category '{category}': {kw}")
+                results = self.fetch_gdelt_window(
+                    keyword=kw,
+                    from_dt=from_dt,
+                    to_dt=to_dt,
+                    **kwargs
+                )
+                all_articles.extend(results)
+                # Polite pause to avoid aggressive rate limiting
+                time.sleep(random.uniform(2.0, 5.0))
+                
+        return _dedupe_articles(all_articles)
+
+    def fetch_gdelt_window(
+        self,
+        keyword: str,
+        from_dt: datetime,
+        to_dt: datetime,
+        max_records: int = 250,
+        exact_phrase: bool = False,
+        source_country: str | None = None,
+        source_lang: str | None = None,
+        theme: str | None = None,
+        domain: str | None = None,
+        extra_query: str | None = None,
+    ) -> list:
+        """
+        Fetch historical articles from the GDELT DOC 2.0 API.
+        Includes robust retry logic with 5-minute cooldown for timeouts/rate-limits.
+        """
+        if from_dt >= to_dt:
+            raise ValueError("from_dt must be earlier than to_dt")
+
+        window_days = (to_dt - from_dt).days
+        if window_days > self.GDELT_MAX_WINDOW_DAYS:
+            raise ValueError(
+                f"GDELT DOC API only supports ~{self.GDELT_MAX_WINDOW_DAYS} days history"
+            )
+
+        max_records = min(max_records, self.GDELT_MAX_RECORDS)
+        query = self._build_gdelt_query(
+            keyword=keyword,
+            exact_phrase=exact_phrase,
+            source_country=source_country,
+            source_lang=source_lang,
+            theme=theme,
+            domain=domain,
+            extra_query=extra_query,
+        )
+
+        params = {
+            "query": query,
+            "mode": "artlist",
+            "format": "json",
+            "maxrecords": max_records,
+            "sort": "DateDesc",
+            "STARTDATETIME": _fmt_gdelt_dt(from_dt),
+            "ENDDATETIME": _fmt_gdelt_dt(to_dt),
+        }
+
+        max_retries = 5
+        for attempt in range(max_retries):
+            try:
+                resp = requests.get(
+                    self.GDELT_DOC_API,
+                    params=params,
+                    timeout=45,
+                    headers={
+                        "User-Agent": self.USER_AGENT,
+                        "Accept": "application/json",
+                    },
+                )
+
+                if resp.status_code == 429:
+                    logger.warning(
+                        "[GDELT] Rate limited (429). Cooling down for 5 minutes..."
+                    )
+                    time.sleep(305)
+                    continue
+
+                resp.raise_for_status()
+                
+                try:
+                    data = resp.json()
+                except Exception as e:
+                    logger.error("[GDELT] Invalid JSON response (Status %d): %s", resp.status_code, e)
+                    # Log the body to see what GDELT is actually returning (likely HTML error)
+                    body_snippet = resp.text[:500] if resp.text else "[Empty Response]"
+                    logger.error("[GDELT] Response body snippet: %s", body_snippet)
+                    
+                    # If it's an HTML error page, we might be blocked or throttled in a way that doesn't return 429
+                    if "<html" in body_snippet.lower():
+                        logger.warning("[GDELT] Received HTML instead of JSON. Cooling down for 5 minutes...")
+                        time.sleep(305)
+                    else:
+                        time.sleep(10 * (attempt + 1))
+                    continue
+
+                raw_articles = data.get("articles", [])
+                if not isinstance(raw_articles, list):
+                    return []
+
+                results = []
+                for article in raw_articles:
+                    title = article.get("title", "") or ""
+                    url = article.get("url", "") or ""
+                    if not title or not url:
+                        continue
+
+                    normalized = {
+                        "feed_name": "GDELT",
+                        "title": title.strip(),
+                        "link": url.strip(),
+                        "published": article.get("seendate", ""),
+                        "summary": title.strip(),
+                        "language": article.get("language", ""),
+                        "sourcecountry": article.get("sourcecountry", ""),
+                        "domain": article.get("domain", ""),
+                        "socialimage": article.get("socialimage", ""),
+                    }
+                    results.append(normalized)
+
+                results = _dedupe_articles(results)
+                logger.info(
+                    "[GDELT] Retrieved %d articles for '%s'",
+                    len(results),
+                    keyword,
+                )
+                return results
+
+            except (requests.exceptions.Timeout, requests.exceptions.ConnectionError) as e:
+                logger.warning(
+                    "[GDELT] Timeout/Connection error on attempt %d/%d: %s. Cooling down 5 mins...",
+                    attempt + 1, max_retries, e
+                )
+                time.sleep(305)
+                continue
+
+            except requests.exceptions.RequestException as e:
+                wait_time = min(60, (2 ** attempt) * 5 + random.uniform(0, 3))
+                logger.warning(
+                    "[GDELT] Request error on attempt %d/%d: %s. Retrying in %.2fs",
+                    attempt + 1, max_retries, e, wait_time
+                )
+                time.sleep(wait_time)
+
+        logger.error("[GDELT] Failed after %d retries", max_retries)
+        return []
+
+    def _build_gdelt_query(
+        self,
+        keyword: str,
+        exact_phrase: bool = False,
+        source_country: str | None = None,
+        source_lang: str | None = None,
+        theme: str | None = None,
+        domain: str | None = None,
+        extra_query: str | None = None,
+    ) -> str:
+        """Build a valid GDELT DOC API query string."""
+        query_parts = []
+        if keyword:
+            if exact_phrase and " " in keyword:
+                query_parts.append(f'"{keyword}"')
+            else:
+                query_parts.append(keyword)
+
+        if source_country:
+            query_parts.append(f"sourcecountry:{source_country.lower()}")
+        if source_lang:
+            query_parts.append(f"sourcelang:{source_lang.lower()}")
+        if theme:
+            query_parts.append(f"theme:{theme}")
+        if domain:
+            query_parts.append(f"domain:{domain}")
+        if extra_query:
+            query_parts.append(extra_query)
+
+        return " ".join(query_parts)
+
+
+if __name__ == "__main__":
+    import json
+    from datetime import timedelta
+
+    # Configure logging
+    logging.basicConfig(
+        level=logging.INFO,
+        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+    )
+
+    client = GDELTClient()
+
+    # Test parameters
+    ticker = "RELIANCE.NS"
+    end_date = datetime.now(timezone.utc)
+    start_date = end_date - timedelta(days=3)
+
+    print(f"\n--- Testing GDELT Ticker Context Fetch: {ticker} ---")
+    try:
+        articles = client.fetch(
+            keyword="",
+            from_dt=start_date,
+            to_dt=end_date,
+            ticker=ticker
+        )
+
+        output_file = "gdelt.json"
+        with open(output_file, "w") as f:
+            json.dump(articles, f, indent=4)
+        
+        print(f"Successfully fetched {len(articles)} articles.")
+        print(f"Output saved to {output_file}")
+
+    except Exception as e:
+        print(f"Test failed: {e}")

singular_ticker_causal/data_sources/news_client.py

@@ -0,0 +1,460 @@
+"""
+singular_ticker_causal/data_sources/news_client.py
+
+Fetches quarterly RSS news for a single ticker over a multi-year window.
+  - Operates on a single ticker (not a universe)
+  - Fetches quarterly windows across the full fundamental date range
+  - Returns articles structured for DenoisedNewsEncoder consumption
+
+Sources:
+  • LiveMint RSS feeds
+  • CNBC-TV18 RSS feeds
+  • Other RSS feeds (Business Standard, Forbes India, Zee News, Economic Times, etc.)
+  • Trading Economics (Selenium — stream, economy, markets, India news)
+  • Zerodha Pulse (requests + BeautifulSoup)
+"""
+
+import os
+import json
+import time
+import logging
+import datetime
+import requests
+import feedparser
+import yfinance as yf
+from typing import List
+from bs4 import BeautifulSoup
+from selenium import webdriver
+from selenium.webdriver.firefox.options import Options as FirefoxOptions
+from selenium.webdriver.firefox.service import Service as FirefoxService
+from selenium.webdriver.common.by import By
+from selenium.webdriver.support.ui import WebDriverWait
+from selenium.webdriver.support import expected_conditions as EC
+
+
+logger = logging.getLogger(__name__)
+
+
+# ==============================================================
+# SCRAPER SOURCES
+# ==============================================================
+
+LIVEMINT_FEEDS = [
+    "https://www.livemint.com/rss/companies",
+    "https://www.livemint.com/rss/opinion",
+    "https://www.livemint.com/rss/money",
+    "https://www.livemint.com/rss/politics",
+    "https://www.livemint.com/rss/science",
+    "https://www.livemint.com/rss/industry",
+    "https://www.livemint.com/rss/education",
+    "https://www.livemint.com/rss/sports",
+    "https://www.livemint.com/rss/technology",
+    "https://www.livemint.com/rss/news",
+    "https://www.livemint.com/rss/markets",
+    "https://www.livemint.com/rss/AI",
+    "https://www.livemint.com/rss/insurance",
+    "https://www.livemint.com/rss/budget",
+    "https://www.livemint.com/rss/elections",
+]
+
+CNBC18_FEEDS = [
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/latest.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/india.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/economy.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/market.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/business.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/sports.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/politics.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/world.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/education.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/travel.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/auto.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/technology.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/personal-finance.xml",
+    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/web-stories.xml",
+]
+
+OTHER_FEEDS = [
+    "http://www.business-standard.com/rss/todays-paper.rss",
+    "https://www.forbesindia.com/commonfeeds/v1/frb/rss/blog.xml",
+    "http://zeenews.india.com/rss/business.xml",
+    "https://economictimes.indiatimes.com/rssfeedsdefault.cms",
+    "https://news.google.com/rss?cf=all&hl=en-IN&topic=b&gl=IN&ceid=IN:en",
+    "https://cfo.economictimes.indiatimes.com/rss/topstories",
+    "https://cfo.economictimes.indiatimes.com/rss/recentstories",
+    "https://cfo.economictimes.indiatimes.com/rss/corporate-finance",
+    "https://cfo.economictimes.indiatimes.com/rss/esg",
+    "https://cfo.economictimes.indiatimes.com/rss/cfo-tech",
+    "https://cfo.economictimes.indiatimes.com/rss/governance-risk-compliance",
+    "https://cfo.economictimes.indiatimes.com/rss/lateststories",
+]
+
+TE_SOURCES = {
+    "te_stream":  "https://tradingeconomics.com/stream",
+    "te_economy": "https://tradingeconomics.com/stream?i=economy",
+    "te_markets": "https://tradingeconomics.com/stream?i=markets",
+    "te_india":   "https://tradingeconomics.com/india/news",
+}
+
+ZERODHA_PULSE_URL = "https://pulse.zerodha.com/"
+
+
+# ==============================================================
+# TRADING ECONOMICS Selenium Scraper
+# ==============================================================
+
+JS_SCROLL_DOWN = "window.scrollTo(0, document.body.scrollHeight); return document.body.scrollHeight;"
+
+JS_EXTRACT_TE_NEWS = """
+var items = [];
+var listItems = document.querySelectorAll('li[id]');
+listItems.forEach(function(li) {
+    var titleLink = li.querySelector('a[href]');
+    if (!titleLink) return;
+    var titleText = '';
+    var bTag = titleLink.querySelector('b');
+    if (bTag) { titleText = bTag.textContent.trim(); }
+    else { titleText = titleLink.textContent.trim(); }
+    if (!titleText) return;
+    var url = titleLink.getAttribute('href') || '';
+    if (url && !url.startsWith('http')) { url = 'https://tradingeconomics.com' + url; }
+    var descEl = li.querySelector('.te-stream-item-description, span[style]');
+    var description = descEl ? descEl.textContent.trim() : '';
+    var dateEl = li.querySelector('small');
+    var dateText = dateEl ? dateEl.textContent.trim() : '';
+    var countryEl = li.querySelector('.te-stream-country');
+    var country = countryEl ? countryEl.textContent.trim() : '';
+    var categoryEl = li.querySelector('.te-stream-category');
+    var category = categoryEl ? categoryEl.textContent.trim() : '';
+    items.push({ title: titleText, description: description, date: dateText, url: url, country: country, category: category });
+});
+return JSON.stringify(items);
+"""
+
+
+class NewsClient:
+    """
+    Fetches news for a single ticker across a multi-year window by
+    chunking into quarterly GDELT calls. Implements the same cache-first
+    pattern as causal.test_causal_flow.test_causal_with_text.
+
+    Usage:
+        client = NewsClient("RELIANCE")
+        articles = client.fetch("2022-01-01", "2026-04-30")
+        # articles: List[dict] with keys: title, content, published, url, credibility_weight, source
+    """
+    USER_AGENT: str = (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) "
+        "Gecko/20100101 Firefox/128.0"
+    )
+
+    def __init__(self, ticker: str):
+        self.ticker = ticker.upper().replace(".NS", "")
+        self.keyword = self._get_ticker_keyword(self.ticker)
+
+    def _get_ticker_keyword(self, ticker: str) -> str:
+        """Fetch a search-friendly name for the ticker from yfinance."""
+        symbol = ticker if "." in ticker else f"{ticker}.NS"
+        try:
+            t = yf.Ticker(symbol)
+            info = t.info
+            name = info.get("shortName") or info.get("longName") or ticker
+            # Clean up the name for better matching
+            for suffix in [" Limited", " Ltd.", " Ltd", " Corp.", " Corp", " Inc.", " Inc"]:
+                name = name.replace(suffix, "")
+            return name.strip()
+        except Exception as e:
+            logger.warning(f"Failed to fetch yfinance info for {symbol}: {e}")
+            return ticker
+
+    def _normalise(self, raw: dict) -> dict:
+        """Convert the raw dict format from news.py into a pipeline-ready article dict."""
+        title = raw.get("title", "").strip()
+        summary = raw.get("summary", "") or raw.get("description", "") or ""
+        content = f"{title}. {summary}".strip(". ") if summary and summary != title else title
+        pub_raw = raw.get("published", "") or raw.get("seendate", "")
+        # Try to parse the published field
+        pub_dt = None
+        if pub_raw:
+            try:
+                from dateutil import parser as dp
+                pub_dt = dp.parse(pub_raw)
+            except Exception:
+                pub_dt = None
+        if pub_dt is None:
+            pub_dt = datetime.datetime.now(datetime.timezone.utc)
+        elif pub_dt.tzinfo is None:
+            pub_dt = pub_dt.replace(tzinfo=datetime.timezone.utc)
+
+        return {
+            "title": title,
+            "content": content,
+            "published": pub_dt.isoformat(),
+            "url": raw.get("link", "") or raw.get("url", ""),
+        }
+
+    def fetch(
+        self,
+        start: str,
+        end: str,
+        include_pulse: bool = False,
+        include_te: bool = False,
+    ) -> List[dict]:
+        """
+        Fetch all news for self.ticker between start and end.
+        Aggregates RSS, Pulse, and (optionally) Trading Economics.
+        """
+
+        logger.info(
+            f"Fetching news for {self.ticker} ('{self.keyword}') "
+            f"from {start} to {end}..."
+        )
+
+        all_raw: List[dict] = []
+
+        # 1. RSS Feeds
+        logger.info("Searching RSS feeds...")
+        rss_articles = (
+            self.search_rss_feeds(LIVEMINT_FEEDS, self.keyword, "LiveMint")
+            + self.search_rss_feeds(CNBC18_FEEDS, self.keyword, "CNBC18")
+            + self.search_rss_feeds(OTHER_FEEDS, self.keyword, "OtherFeeds")
+        )
+        all_raw.extend(rss_articles)
+
+        # 2. Zerodha Pulse
+        if include_pulse:
+            logger.info("Fetching Zerodha Pulse...")
+            pulse_news = self.scrape_pulse()
+            all_raw.extend(pulse_news)
+
+        # 3. Trading Economics (Optional)
+        if include_te:
+            logger.info("Fetching Trading Economics...")
+            te_data = self.scrape_all_te(headless=True, scroll_count=1)
+            for items_list in te_data.values():
+                all_raw.extend(items_list)
+
+        # # Filter by date window
+        # start_dt = datetime.datetime(
+        #     *[int(x) for x in start.split("-")], tzinfo=datetime.timezone.utc
+        # )
+        # end_dt = datetime.datetime(
+        #     *[int(x) for x in end.split("-")], tzinfo=datetime.timezone.utc
+        # )
+        # in_window = self._filter_by_window(all_raw, start_dt, end_dt)
+        # logger.info(f"Collected {len(in_window)} articles in window.")
+
+        # # Normalise and deduplicate by title
+        # seen_titles: set = set()
+        # articles: List[dict] = []
+        # for raw in all_raw:
+        #     title = raw.get("title", "").strip().lower()
+        #     if not title or title in seen_titles:
+        #         continue
+        #     seen_titles.add(title)
+        #     articles.append(self._normalise(raw))
+
+        # Sort chronologically
+        all_raw.sort(key=lambda a: str(a.get("published") or a.get("date") or ""))
+
+        logger.info(
+            f"Final corpus: {len(all_raw)} unique articles for {self.ticker}."
+        )
+
+        return all_raw
+        
+    def search_rss_feeds(self, feeds: list, search_keyword: str, feed_name: str) -> list:
+        """Search for a keyword across a list of RSS feed URLs."""
+        print(f"[DEBUG] search_rss_feeds - Searching {feed_name} for keyword: '{search_keyword}'")
+        results = []
+        for feed_url in feeds:
+            print(f"[DEBUG] search_rss_feeds - Parsing URL: {feed_url}")
+            try:
+                feed = feedparser.parse(feed_url)
+                if not feed.entries:
+                    print(f"[DEBUG] search_rss_feeds - No entries found for {feed_url}")
+                for entry in feed.entries:
+                    title = entry.get("title", "")
+                    summary = entry.get("summary", "")
+                    if not title:
+                        continue
+                    if search_keyword.lower() in title.lower() or search_keyword.lower() in summary.lower():
+                        print(f"[DEBUG] search_rss_feeds - Match found: {title[:60]}...")
+                        results.append({
+                            "feed_name": feed_name,
+                            "title": title,
+                            "link": entry.get("link", ""),
+                            "published": entry.get("published", ""),
+                            "summary": summary,
+                        })
+            except Exception as e:
+                print(f"[news.py] RSS error ({feed_url}): {e}")
+        print(f"[DEBUG] search_rss_feeds - {feed_name} done. Found {len(results)} items.")
+        return results
+
+    def scrape_all_te(self, headless: bool = True, scroll_count: int = 3) -> dict:
+        """Scrape all Trading Economics news sources. Returns dict of source_key -> list of items."""
+        driver = self._create_driver(headless=headless)
+        results = {}
+        scraped_at = datetime.datetime.now(datetime.timezone.utc).isoformat()
+        try:
+            for key, url in TE_SOURCES.items():
+                try:
+                    items = self._scrape_te_page(driver, url, scroll_count=scroll_count)
+                    for item in items:
+                        item["source"] = key
+                        item["scraped_at"] = scraped_at
+                    results[key] = items
+                except Exception as e:
+                    print(f"  [TE] Error scraping {key}: {e}")
+                    results[key] = []
+                    try:
+                        driver.quit()
+                    except Exception:
+                        pass
+                    driver = self._create_driver(headless=headless)
+        finally:
+            try:
+                driver.quit()
+            except Exception:
+                pass
+        return results
+
+    def scrape_pulse(self) -> list:
+        """Scrape latest news from Zerodha Pulse using requests + BeautifulSoup."""
+        print("[news.py] Scraping Zerodha Pulse...")
+        headers = {
+            "User-Agent": self.USER_AGENT,
+            "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
+            "Accept-Language": "en-US,en;q=0.5",
+        }
+        try:
+            resp = requests.get(ZERODHA_PULSE_URL, headers=headers, timeout=30)
+            resp.raise_for_status()
+        except Exception as e:
+            print(f"[news.py] Zerodha Pulse error: {e}")
+            return []
+
+        soup = BeautifulSoup(resp.text, "html.parser")
+        items = []
+        scraped_at = datetime.datetime.now(datetime.timezone.utc).isoformat()
+
+        for li in soup.select("li.item"):
+            title_el = li.select_one("h2.title a")
+            if not title_el:
+                continue
+            title = title_el.get_text(strip=True)
+            url = title_el.get("href", "")
+            desc_el = li.select_one("div.desc")
+            description = desc_el.get_text(strip=True) if desc_el else ""
+            date_el = li.select_one("span.date")
+            date_text = date_el.get_text(strip=True) if date_el else ""
+            feed_el = li.select_one("span.feed")
+            feed = feed_el.get_text(strip=True) if feed_el else ""
+            items.append({
+                "title": title,
+                "description": description,
+                "published": date_text,
+                "link": url,
+                "summary": description,
+                "publisher": feed,
+                "scraped_at": scraped_at,
+                "source": "pulse"
+            })
+
+        print(f"[news.py] Zerodha Pulse: extracted {len(items)} items.")
+        return items
+
+    def _filter_by_window(
+        self,
+        articles: list,
+        from_dt: datetime.datetime,
+        to_dt: datetime.datetime,
+    ) -> list:
+        """
+        Drop articles whose parsed `published` timestamp falls outside [from_dt, to_dt].
+        Articles with unparseable or missing dates are kept (conservative).
+        """
+        from dateutil import parser as dp
+
+        def _to_utc(dt):
+            if dt.tzinfo is None:
+                return dt.replace(tzinfo=datetime.timezone.utc)
+            return dt.astimezone(datetime.timezone.utc)
+
+        from_utc = _to_utc(from_dt)
+        to_utc   = _to_utc(to_dt)
+
+        filtered = []
+        for art in articles:
+            pub = art.get("published", "")
+            if not pub:
+                filtered.append(art)
+                continue
+            try:
+                dt = _to_utc(dp.parse(pub))
+                if from_utc <= dt <= to_utc:
+                    filtered.append(art)
+            except Exception:
+                filtered.append(art)  # keep on parse failure
+        return filtered
+
+    def _create_driver(self, headless: bool = True):
+        """Create a headless Firefox webdriver."""
+        options = FirefoxOptions()
+        if headless:
+            options.add_argument("--headless")
+        options.set_preference("general.useragent.override", self.USER_AGENT)
+        options.set_preference("dom.webdriver.enabled", False)
+        options.set_preference("useAutomationExtension", False)
+        service = FirefoxService(log_output=os.devnull)
+        driver = webdriver.Firefox(options=options, service=service)
+        driver.set_page_load_timeout(60)
+        return driver
+
+    def _scrape_te_page(self, driver, url: str, scroll_count: int = 3, scroll_pause: float = 2.0) -> list:
+        """Scrape a single Trading Economics news page."""
+        print(f"  [TE] Loading: {url}")
+        driver.get(url)
+        WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.TAG_NAME, "body")))
+        time.sleep(3)
+        prev_height = 0
+        for i in range(scroll_count):
+            new_height = driver.execute_script(JS_SCROLL_DOWN)
+            if new_height == prev_height:
+                break
+            prev_height = new_height
+            time.sleep(scroll_pause)
+        raw = driver.execute_script(JS_EXTRACT_TE_NEWS)
+        items = json.loads(raw)
+        print(f"  [TE] Extracted {len(items)} items from {url}")
+        return items
+
+
+if __name__ == "__main__":
+    """Simple test runner for NewsClient."""
+    logging.basicConfig(
+        level=logging.INFO,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
+    )
+    
+    ticker = "RELIANCE"
+    client = NewsClient(ticker)
+    
+    # Test Fetch (Aggregates RSS and Pulse)
+    print(f"\n--- Testing Fetch for {ticker} ---")
+    start_date = (datetime.datetime.now() - datetime.timedelta(days=7)).strftime("%Y-%m-%d")
+    end_date = datetime.datetime.now().strftime("%Y-%m-%d")
+    
+    # We include_te=False by default as it requires Selenium/Firefox
+    articles = client.fetch(start_date, end_date)
+    print(f"Fetched {len(articles)} articles.")
+    
+    if articles:
+        # Save results to debug_data/
+        with open("news.json", "w", encoding="utf-8") as f:
+            json.dump(articles, f, indent=4, ensure_ascii=False)
+        for a in articles[:3]:
+            print(f"  - [{a['published']}] {a['title']}")
+    else:
+        print("No articles found in the specified window.")

singular_ticker_causal/data_sources/nseconnect/__init__.py

@@ -0,0 +1,25 @@
+"""
+    The MIT License (MIT)
+
+    Copyright (c) 2014 Noufal Nazar
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+"""
+__VERSION__='2.0.1'
+from .nse import Nse

singular_ticker_causal/data_sources/nseconnect/bases.py

@@ -0,0 +1,72 @@
+"""
+    The MIT License (MIT)
+
+    Copyright (c) 2014 Noufal Nazar
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+"""
+
+from abc import ABCMeta, abstractmethod
+import six
+
+
+class AbstractBaseExchange(six.with_metaclass(ABCMeta, object)):
+
+    @abstractmethod
+    def get_stock_codes(self):
+        """
+        :return: list of tuples with stock code and stock name
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def is_valid_code(self, code):
+        """
+        :return: True, if it is a valid stock code, else False
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def get_quote(self, code):
+        """
+        :param code: a stock code
+        :return: a dictionary which contain detailed stock code.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def get_top_gainers(self):
+        """
+        :return: a sorted list of codes of top gainers
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def get_top_losers(self):
+        """
+        :return: a sorted list of codes of top losers
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def __str__(self):
+        """
+        :return: market name
+        """
+        raise NotImplementedError

singular_ticker_causal/data_sources/nseconnect/cleaners.py

@@ -0,0 +1,51 @@
+"""
+Module for various data structure cleaning tasks
+"""
+from datetime import datetime
+dirty_data = """
+{
+    "fname": "Jon",
+    "lname": "Doe",
+    "age": 20,
+    "str_age": "20",
+    "pi": 3.1415927,
+    "str_pi": "3.1415927",
+    "dob": "01-Jan-2023",
+    "mobile": [
+        {
+            "id": "Home",
+            "number": "123456789"
+        }, 
+        {
+            "id": "office", 
+            "number": "987645321"
+        }
+    ] 
+}
+"""
+
+def parse_values(obj):
+    for key, value in obj.items():
+        if isinstance(value, str):
+            # Try to parse as datetime if the string matches the format
+            date_formats = ["%d-%b-%Y", "%d-%m-%Y", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d"]
+            for date_format in date_formats:
+                try:
+                    obj[key] = datetime.strptime(value, date_format)
+                    break
+                except ValueError:
+                    pass
+            else:
+                # If the string couldn't be parsed as datetime, try numeric conversion
+                try:
+                    obj[key] = int(value)
+                except ValueError:
+                    try:
+                        obj[key] = float(value)
+                    except ValueError:
+                        pass
+        elif isinstance(value, dict):
+            obj[key] = parse_values(value)
+        elif isinstance(value, list):
+            obj[key] = [parse_values(item) if isinstance(item, dict) else item for item in value]
+    return obj

singular_ticker_causal/data_sources/nseconnect/datemgr.py

@@ -0,0 +1,104 @@
+import datetime as dt
+from dateutil.relativedelta import relativedelta
+from dateutil.parser import parse
+from dateutil import rrule
+from .errors import DateFormatError
+
+
+def get_nearest_business_day(d):
+    """ takes datetime object"""
+    if d.isoweekday() == 7 or d.isoweekday() == 6:
+        d = d - relativedelta(days=1)
+        return get_nearest_business_day(d)
+
+    # republic day
+    elif d.month == 1 and d.day == 26:
+        d = d - relativedelta(days=1)
+        return get_nearest_business_day(d)
+    # labour day
+    elif d.month == 5 and d.day == 1:
+        d = d - relativedelta(days=1)
+        return get_nearest_business_day(d)
+    # independece day
+    elif d.month == 8 and d.day == 15:
+        d = d - relativedelta(days=1)
+        return get_nearest_business_day(d)
+    # Gandhi Jayanti
+    elif d.month == 10 and d.day == 2:
+        d = d - relativedelta(days=1)
+        return get_nearest_business_day(d)
+    # chirstmas
+    elif d.month == 12 and d.day == 25:
+        d = d - relativedelta(days=1)
+        return get_nearest_business_day(d)
+    else:
+        return d
+
+def is_known_holiday(d):
+    """accepts datetime/date object and returns boolean"""
+    if type(d) == dt.datetime:
+        d = d.date()
+    elif type(d) != dt.date:
+        raise DateFormatError("only date objects or datetime objects")
+    else:
+        # fine do nothing
+        pass 
+
+    # declare the list of holidays here.
+    # republic day.
+    if d.month  == 1 and d.day == 26:
+        return True
+    # labour day
+    elif d.month == 5 and d.day == 1:
+        d = d - relativedelta(days=1)
+        return get_nearest_business_day(d)
+    # independence day
+    elif d.month == 8 and d.day == 15:
+        return True
+    # gandhi jayanti
+    elif d.month == 10 and d.day == 2:
+        return True
+    # christmas
+    elif d.month == 12 and d.day == 25:
+        return True
+    else:
+        return False
+
+def mkdate(d):
+    """tries its best to return a valid date. it can accept pharse like today,
+    yesterday, day before yesterday etc. 
+    """
+    # check if the it == a string
+    return_date = ""
+    if type(d) is str:
+        if d == "today":
+            return_date = dt.date.today()
+        elif d == "yesterday":
+            return_date = dt.date.today() - relativedelta(days=1)
+        elif d == "day before yesterday":
+            return_date = dt.date.today() - relativedelta(days=2)
+        else:
+            return_date = parse(d, dayfirst=True).date()
+    elif type(d) == dt.datetime:
+        return_date = d.date()
+    elif type(d) == dt.date:
+        return d
+    else:
+        raise DateFormatError("wrong date format %s" % str(d))
+    # check if future date.
+    return return_date
+
+def usable_date(d):
+    """accepts fuzzy format and returns most sensible date"""
+    return get_nearest_business_day(mkdate(d))
+
+def get_date_range(frm, to, skip_dates=[]):
+    """accepts fuzzy format date and returns business adjusted date ranges"""
+    # for x in rrule.rrule(rrule.DAILY, dtstart=s, until=dt.datetime.now(), byweekday=[0, 1, 2, 3, 4]): print(x)
+    frm = usable_date(frm)
+    to = usable_date(to)
+    datelist = []
+    for date in rrule.rrule(rrule.DAILY, dtstart=frm, until=to, byweekday=[0, 1, 2, 3, 4]):
+        if not is_known_holiday(date):
+            datelist.append(date.date())
+    return datelist

singular_ticker_causal/data_sources/nseconnect/downloader.py

@@ -0,0 +1,116 @@
+"""
+    The MIT License (MIT)
+
+    Copyright (c) 2014 Noufal Nazar
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.
+
+"""
+import io
+import os
+import zipfile
+import datetime as dt
+from urllib.request import Request
+from .datemgr import mkdate, get_date_range
+from .nse import Nse
+from abc import ABCMeta, abstractmethod
+
+class BaseBhavcopyDownloader(metaclass=ABCMeta):
+    """Base class for all types of bhavcopy downloader"""
+    def __init__(self, from_date, to_date=dt.datetime.now().date(), skip_dates=[]):
+        """accepts date in fuzzy format"""
+        self.bhavcopy_base_url = "https://www.nseindia.com/content/historical/EQUITIES/%s/%s/cm%s%s%sbhav.csv.zip"
+        self.bhavcopy_base_filename = "cm%s%s%sbhav.csv"
+        self.from_date = from_date
+        self.to_date = to_date
+        self.skip_dates = skip_dates
+        self.nse = Nse()
+        self.dates = self.generate_dates()
+
+    def generate_dates(self):
+        return get_date_range(self.from_date, self.to_date, skip_dates=self.skip_dates)
+    
+    def get_bhavcopy_url(self, d):
+        """accept date and return bhavcopy url"""
+        day_of_month = d.strftime("%d")
+        mon = d.strftime("%b").upper()
+        year = d.year
+        url = self.bhavcopy_base_url % (year, mon, day_of_month, mon, year)
+        return url
+
+    def get_bhavcopy_filename(self, d):
+        """for a given date generate bhavcopy filename"""
+        day_of_month = d.strftime("%d")
+        mon = d.strftime("%b").upper()
+        year = d.year
+        filename = self.bhavcopy_base_filename % (day_of_month, mon, year)
+        return filename
+
+    def download_one(self, d):
+        """download bhavcopy for the given date"""
+        # this will keep this method usable for any arbitrary date.
+        d = mkdate(d)
+        # ex_url = "https://www.nseindia.com/content/historical/EQUITIES/2011/NOV/cm08NOV2011bhav.csv.zip"
+        url = self.get_bhavcopy_url(d)
+        print(url)
+        filename = self.get_bhavcopy_filename(d)
+        # response = requests.get(url, headers=self.headers)
+        response = self.nse.opener.open(Request(url, None, self.nse.headers))
+        zip_file_handle = io.BytesIO(response.read())
+        zf = zipfile.ZipFile(zip_file_handle)
+        return zf.read(filename).decode("utf-8")
+
+    @abstractmethod
+    def download(self):
+        pass 
+    
+    @abstractmethod
+    def update(self):
+        pass 
+
+
+class BhavcopyFileSystemDownloader(BaseBhavcopyDownloader):
+    def __init__(self, directory, *args, **kwargs):
+        if (os.path.exists(directory) and os.path.isdir(directory) and os.access(directory, os.W_OK)):
+            super().__init__(*args, **kwargs)
+            self.directory = directory
+        else:
+            raise Exception("directory path must be valid and writtable, please check manually")
+
+    def download(self):
+        for date in self.dates:
+            print("downloading for " + str(date))
+            try:
+                content = self.download_one(date)
+            except Exception as err:
+                print("unable to download for the date: %s" %  date.strftime("%Y-%m-%d"))
+            else:
+                fh = open(self.directory + "/" + date.strftime("%Y-%m-%d") + ".csv", "w")
+                fh.write(content)
+                fh.close()
+    
+    def update(self):
+        pass 
+
+
+if __name__ == '__main__':
+    b = BhavcopyFileSystemDownloader(directory="/tmp/bhavcopy", from_date="01-01-2018")
+    b.download()
+
+# https://stackoverflow.com/questions/49183801/ssl-certificate-verify-failed-with-urllib

singular_ticker_causal/data_sources/nseconnect/errors.py

@@ -0,0 +1,8 @@
+class BhavcopyNotAvailableError(Exception):
+    """this error could occur in case you download bhavcopy for the dates
+    when the market was close"""
+    pass
+
+class DateFormatError(Exception):
+    """in case the date format is errorneous"""
+    pass