Removed causal learning and validation

README.md

@@ -14,648 +14,3 @@ pinned: false
 **Description:** End-to-end financial intelligence pipeline for Net-of-Tax Alpha decisions using causal chain analysis, web scraping, and data aggregation.
 
 ---
-
-## Table of Contents
-
-1. [Overview](#overview)
-2. [Authentication](#authentication)
-3. [API Endpoints](#api-endpoints)
-   - [Server Endpoints](#server-endpoints)
-   - [V2 API Endpoints](#v2-api-endpoints)
-4. [Core Functions](#core-functions)
-5. [Supported Tickers](#supported-tickers)
-6. [Error Handling](#error-handling)
-7. [Examples](#examples)
-
----
-
-## Overview
-
-The Noisy Boy API is a FastAPI-based financial intelligence system that:
-
-- **Fetches multi-source financial data** from various Indian market sources (BSE, NSE, RBI, SEBI, etc.)
-- **Analyzes causal chains** between market events and price movements using BERT and embedding models
-- **Generates aggregated text** from diverse data sources for each ticker
-- **Builds knowledge graphs** with semantic and evidence layers
-- **Streams real-time SSE responses** for long-running analysis tasks
-
-### Technology Stack
-
-- **Framework:** FastAPI with async/await support
-- **Database:** SQLite (local) + Supabase (cloud)
-- **Models:**
-  - BERT-CAUSE-EFFECT for causal relationship extraction
-  - Fireworks AI embeddings for similarity scoring
-  - OpenAI LLM for graph summarization
-  - TinyFish AI for web automation
-- **Data Sources:** 50+ fetchers covering macro, regulatory, and company-specific data
-
----
-
-## Authentication
-
-### API Key Authentication
-
-All endpoints (except `/api/source-refs`) require an `X-API-Key` header:
-
-```bash
-curl -H "X-API-Key: your-secret-token" https://api.example.com/api/endpoint
-curl -H "X-API-Key: your-api-key" http://localhost:8000/v2/api/endpoint
-```
-
-**Environment Variable**: `API_KEY` (default: `secret-token`)
-
----
-
-## Base URL
-
-```
-http://localhost:8000/v2
-```
-
----
-
-## API Endpoints
-
-### Scraper API
-
-#### 1. GET `/api/source-refs`
-Returns all fetcher-to-source-URL mappings and reference links.
-
-**Authentication**: None required
-
-**Response**:
-```json
-{
-  "fetcher_refs": {
-    "repo_rate": "https://rbi.org.in/...",
-    "bse": "https://bseindia.com/...",
-    ...
-  },
-  "cat_refs": {
-    "macro": {...},
-    "corporate": {...}
-  }
-}
-```
-
----
-
-#### 2. POST `/api/run`
-Streams real-time scraping results via Server-Sent Events (SSE) from TinyFish API with local caching.
-
-**Authentication**: Required (`X-API-Key`)
-
-**Request Body**:
-```json
-{
-  "url": "https://example.com/page",
-  "goal": "Extract financial data",
-  "ticker": "HDFCBANK",
-  "stealth": false
-}
-```
-
-**Query Parameters**:
-- `url` (required): Target URL to scrape
-- `goal` (required): Scraping objective description
-- `ticker` (optional): Stock ticker for caching purposes
-- `stealth` (optional, bool): Enable stealth mode
-
-**Response** (SSE):
-```
-data: {"type": "STARTED", "run_id": "run_123"}
-data: {"type": "PROGRESS", "purpose": "Fetching data..."}
-data: {"type": "COMPLETE", "status": "COMPLETED", "result_json": {...}}
-```
-
-**Caching**: Results are cached per ticker per day via SQLite.
-
----
-
-#### 3. GET `/api/ticker-data/{ticker}`
-Fetch all cached data for a ticker across all sources (Supabase).
-
-**Authentication**: Required (`X-API-Key`)
-
-**Path Parameters**:
-- `ticker`: Stock ticker symbol (e.g., `HDFCBANK`)
-
-**Response**:
-```json
-{
-  "ticker": "HDFCBANK",
-  "status": "success",
-  "data": {
-    "repo_rate": {
-      "data": {...},
-      "fetched_at": "2024-01-15T10:30:00Z"
-    },
-    "bse": {
-      "data": [...],
-      "fetched_at": "2024-01-15T10:30:00Z"
-    },
-    "aggregated_text": {
-      "aggregated_text": "...",
-      "fetched_at": "2024-01-15T10:30:00Z"
-    },
-    "causal_chain": {
-      "nodes": [...],
-      "links": [...],
-      "all_chains": [...],
-      "biggest_chain": [...]
-    }
-  }
-}
-```
-
----
-
-### Causal Chain API
-
-#### 4. GET `/api/generate-causal-chain-stream`
-Streams causal chain generation via SSE (two phases: nodes, then edges).
-
-**Authentication**: Required (`X-API-Key`)
-
-**Query Parameters**:
-- `ticker` (required): Stock ticker symbol
-
-**Response** (SSE - Streaming):
-```
-data: {"type": "status", "message": "Generating text for ticker..."}
-data: {"type": "nodes", "nodes": [{"id": "RBI Rate Hike", "label": "RBI Rate Hike"}, ...]}
-data: {"type": "edges", "links": [{"source": "Rate Hike", "target": "Inflation", "score": 0.89}, ...]}
-data: {"type": "done", "all_chains": [[...], [...]], "biggest_chain": [...]}
-```
-
-**Phases**:
-1. **Status**: Initial status message
-2. **Nodes**: Unique causal entities extracted from BERT model
-3. **Edges**: Connections between nodes from embedding similarity
-4. **Done**: Final chains and biggest causal path
-
----
-
-### Ontology API
-
-#### 5. POST `/api/generate-ontology`
-Generate financial ontology from input text.
-
-**Authentication**: Required (via implicit call)
-
-**Request Body**:
-```json
-{
-  "ticker": "RELIANCE",
-  "text": "Recent crude oil rally drives RELIANCE revenue..."
-}
-```
-
-**Response**:
-```json
-{
-  "status": "success",
-  "ontology": {
-    "entities": ["crude", "revenue", "RELIANCE"],
-    "relations": [{"type": "affects", "from": "crude", "to": "RELIANCE"}]
-  }
-}
-```
-
----
-
-#### 6. POST `/api/extract-entities`
-Extract named entities using the provided ontology.
-
-**Authentication**: Required (via implicit call)
-
-**Request Body**:
-```json
-{
-  "ticker": "ITC",
-  "text": "Coal prices surge amid monsoon fears...",
-  "ontology": {
-    "entities": ["coal", "monsoon", "price"],
-    "relations": [...]
-  }
-}
-```
-
-**Response**:
-```json
-{
-  "status": "success",
-  "entities": [
-    {"entity": "coal", "type": "commodity", "confidence": 0.92},
-    {"entity": "monsoon", "type": "weather_event", "confidence": 0.88}
-  ]
-}
-```
-
----
-
-#### 7. POST `/api/build-knowledge-graph`
-Build complete knowledge graph with nodes, edges, and causal chains.
-
-**Authentication**: Required (via implicit call)
-
-**Request Body**:
-```json
-{
-  "graph_id": "graph_001",
-  "ticker": "BHEL",
-  "text": "Full financial text for analysis...",
-  "financial_results": {...},
-  "forensic_results": {...},
-  "tech_results": {...},
-  "cached_data": {...}
-}
-```
-
-**Response**:
-```json
-{
-  "meta": {
-    "ticker": "BHEL",
-    "exchange": "NSE",
-    "generated_at": "2024-01-15T12:45:00Z",
-    "status": "success",
-    "chain_count": 5,
-    "node_count": 23,
-    "edge_count": 45
-  },
-  "summary": {
-    "narrative": "Coal shortage → Power demand surge → BHEL tariff opportunity",
-    "net_sentiment_for_ticker": "bullish",
-    "ticker_relevance_score": 0.87,
-    "macro_regimes_active": ["coal-supply-shock", "power-demand-rally"],
-    "top_causal_nodes": ["Coal shortage", "Power demand", "BHEL capacity"],
-    "ria_alert": {
-      "level": "medium",
-      "reason": "Monitor coal supply for sustained impact on tariff structure."
-    }
-  },
-  "biggest_chain": ["Coal shortage", "Power demand", "BHEL contract wins", "Revenue growth"],
-  "all_chains": [[...], [...], ...],
-  "nodes": [...],
-  "edges": [...]
-}
-```
-
----
-
-### Server API (Streaming)
-
-#### 8. GET `/api/generate-causal-chain-stream` (Alternative)
-Same as endpoint #4 but with optional caching from server-side TinyFish integration.
-
----
-
-## Data Fetchers
-
-The system supports multiple data fetchers for different tickers. Each fetcher aggregates specific financial signals:
-
-### Supported Fetchers
-
-| Fetcher | Description | Tickers |
-|---------|-------------|---------|
-| `repo_rate` | RBI Repo Rate | HDFCBANK, TCS, PAYTM, TMPV |
-| `bse` | BSE Announcements | All supported tickers |
-| `fii_dii` | FII/DII Flows | HDFCBANK |
-| `brent` | Brent Crude Oil Price | HDFCBANK, RELIANCE, TMPV |
-| `news` | Financial News Articles | All supported tickers |
-| `npp` | National Power Portal Data | IEX, BHEL |
-| `imd_monsoon` | IMD Monsoon Status | RELIANCE, IEX, BHEL, ETERNAL, ULTRACEMCO |
-| `coal` | Coal Price Index | IEX, BHEL, ITC, ULTRACEMCO |
-| `bhel_tenders` | BHEL Active Tenders | BHEL |
-| `agmarknet` | Agricultural Prices | ETERNAL, ITC |
-| `weather` | Weather in Key Cities | ETERNAL |
-| `h1b` | H1B Visa Filings | TCS |
-| `us_fed` | US Fed Interest Rate | TCS |
-| `us_pmi` | US Services PMI | TCS |
-| `trai_reports` | TRAI Telecom Reports | RELIANCE |
-| `npci` | NPCI UPI Statistics | ETERNAL, IRCTC, PAYTM, MAPMYINDIA |
-| `dgca_traffic` | DGCA Air Traffic Data | IRCTC |
-| `tourist_arrivals` | Foreign Tourist Arrivals | IRCTC |
-| `india_cpi` | India CPI Inflation | ITC |
-| `food_cpi` | Food CPI Inflation | ETERNAL |
-| `labour` | Labour Ministry Releases | ETERNAL |
-| `sebi_orders` | SEBI Orders | PAYTM |
-| `mca_filings` | MCA Corporate Filings | PAYTM |
-| `cma_capacity` | Cement Capacity | ULTRACEMCO |
-| `datareportal` | Digital India Stats | MAPMYINDIA |
-| `pib_highways` | PIB Highway Announcements | ULTRACEMCO, MAPMYINDIA |
-| `pib_vb` | PIB Vande Bharat Updates | BHEL, IRCTC |
-| `nse_bulk_deals` | NSE Bulk Deals | RELIANCE |
-| `cci_orders` | CCI Antitrust Orders | RELIANCE |
-| `erc_orders` | ERC Tariff Orders | IEX |
-| `saubhagya` | Saubhagya Electrification | IEX |
-| `ppac` | India Basket Crude | RELIANCE, ULTRACEMCO |
-
-### Fetcher Response Format
-
-Each fetcher returns formatted text with relevant financial signals:
-
-```
-Brent crude oil price is $85.50 showing a (upward trend, with 5.2% change 30-day).
-RBI Repo Rate is 6.5%.
-FII/DII flows for 2024-01-15: FII net is 450 Cr with net (buy action, holding a positive view).
-```
-
----
-
-## Core Functions
-
-### CausalChain Class (`app/services/causal_chains.py`)
-
-#### `__init__(chunks, fireworks_api_key, fireworks_model)`
-Initialize causal chain processor.
-
-**Parameters**:
-- `chunks` (list): Text chunks to analyze
-- `fireworks_api_key` (str): Fireworks AI API key
-- `fireworks_model` (str): Embedding model name (default: `qwen3-embedding-8b`)
-
----
-
-#### `create_effects(batch_size=16)`
-Extract triggers and effects using BERT-CAUSE-EFFECT model.
-
-**Process**:
-1. Sends text chunks to causal model API
-2. Parses model responses for event triggers and descriptions
-3. Stores triggers and effects
-
-**Retries**: Up to 5 attempts with exponential backoff for failed requests
-
----
-
-#### `create_connections(batch_size=16, chain_threshold=0.85)`
-Build causal connections using embeddings with grounding.
-
-**Process**:
-1. Encodes triggers and effects via Fireworks embeddings
-2. Computes cosine similarity between effects and triggers
-3. Grounds connections by shared entities and keywords
-4. Filters spurious links using entity intersection
-
-**Parameters**:
-- `chain_threshold`: Similarity score threshold (default: 0.85)
-
----
-
-#### `get_all_chains(min_length=2, max_paths=500, time_budget=30)`
-Extract unique causal chains from connections.
-
-**Parameters**:
-- `min_length`: Minimum chain length to keep
-- `max_paths`: Stop after collecting N paths
-- `time_budget`: Wall-clock time limit (seconds)
-
-**Returns**: List of chains, each a list of nodes
-
----
-
-#### `find_biggest_chain(time_budget=20)`
-Find longest causal chain using iterative DFS.
-
-**Returns**: Longest causal path as list of nodes
-
----
-
-#### `to_dict()` / `from_dict()`
-Serialize/deserialize causal chain state.
-
----
-
-### Utility Functions (`util` class)
-
-#### `cos_sim(a, b)`
-Compute cosine similarity matrix between two embedding arrays.
-
-```python
-scores = util.cos_sim(effect_embeddings, trigger_embeddings)
-# scores[i,j] = cosine similarity between effect i and trigger j
-```
-
----
-
-#### `create_chunks(text_input, target_size=700, overlap_sentences=1)`
-Create semantic chunks with overlap for context preservation.
-
-**Parameters**:
-- `target_size`: Target chunk size (characters)
-- `overlap_sentences`: Number of sentences to overlap
-
-**Returns**: List of text chunks ≤ 1000 chars each
-
----
-
-### Fetcher Functions (`scrapper/tiny_fish.py`)
-
-#### `run_fetcher(fetcher_key, ticker_config, persist=False)`
-Execute a single data fetcher.
-
-**Parameters**:
-- `fetcher_key`: Name of fetcher (e.g., `repo_rate`, `bse`)
-- `ticker_config`: Config dict with BSE code, ticker symbol, fetchers list
-- `persist`: Whether to cache results to SQLite
-
-**Returns**: Formatted data dict
-
-**Caching**: Per-ticker, per-day SQLite caching
-
----
-
-### Text Formatting
-
-#### `format_fetcher_text(fetcher_key, data)`
-Format raw fetcher data into human-readable text.
-
-**Example**:
-```python
-text = format_fetcher_text("repo_rate", {"repo_rate_pct": 6.5})
-# Returns: "The current RBI Repo Rate is 6.5%.\n"
-```
-
----
-
-## Examples
-
-### Example 1: Fetch and Cache Aggregated Data
-
-```bash
-curl -X GET "http://localhost:8000/v2/api/ticker-data/RELIANCE" \
-  -H "X-API-Key: secret-token"
-```
-
----
-
-### Example 2: Stream Causal Chain Generation (SSE)
-
-```bash
-curl -X GET "http://localhost:8000/v2/api/generate-causal-chain-stream?ticker=HDFCBANK" \
-  -H "X-API-Key: secret-token" \
-  -N
-```
-
-**Output** (streaming):
-```
-data: {"type":"status","message":"Generating text for ticker..."}
-data: {"type":"nodes","nodes":[{"id":"RBI Rate Hike","label":"RBI Rate Hike"},{"id":"Inflation Risk","label":"Inflation Risk"}]}
-data: {"type":"edges","links":[{"source":"RBI Rate Hike","target":"Inflation Risk","score":0.89}]}
-data: {"type":"done","all_chains":[["RBI Rate Hike","Inflation Risk","Market Correction"]],"biggest_chain":["RBI Rate Hike","Inflation Risk","Market Correction"]}
-```
-
----
-
-### Example 3: Build Knowledge Graph
-
-```bash
-curl -X POST "http://localhost:8000/v2/api/build-knowledge-graph" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "graph_id": "g001",
-    "ticker": "TCS",
-    "text": "H1B visa approvals surge amid US tech expansion...",
-    "financial_results": null,
-    "forensic_results": null,
-    "tech_results": null,
-    "cached_data": null
-  }'
-```
-
----
-
-### Example 4: Ontology Generation
-
-```bash
-curl -X POST "http://localhost:8000/v2/api/generate-ontology" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "ticker": "PAYTM",
-    "text": "NPCI UPI transaction surge drives fintech growth..."
-  }'
-```
-
----
-
-### Example 5: Run Custom Web Scraper
-
-```bash
-curl -X POST "http://localhost:8000/v2/api/run" \
-  -H "Content-Type: application/json" \
-  -H "X-API-Key: secret-token" \
-  -d '{
-    "url": "https://example.com/financial-news",
-    "goal": "Extract Q4 earnings announcements",
-    "ticker": "RELIANCE",
-    "stealth": true
-  }' \
-  -N
-```
-
----
-
-## Error Handling
-
-### Standard Error Responses
-
-#### 400 Bad Request
-```json
-{
-  "detail": "url and goal are required"
-}
-```
-
-#### 401 Unauthorized
-```json
-{
-  "detail": "Invalid or missing API Key"
-}
-```
-
-#### 500 Internal Server Error
-```json
-{
-  "detail": "TINYFISH_API_KEY not found in .env"
-}
-```
-
----
-
-## Supported Tickers
-
-| Ticker | BSE Code | Use Case |
-|--------|----------|----------|
-| HDFCBANK | 500180 | Banking sector, RBI rates |
-| RELIANCE | 500325 | Energy, crude oil exposure |
-| IEX | 540716 | Power sector, capacity data |
-| TCS | 532540 | IT sector, US visa trends |
-| BHEL | 500103 | Power equipment, tender data |
-| ETERNAL | 543320 | Agri-related, food inflation |
-| IRCTC | 542830 | Transportation, tourist flows |
-| ITC | 500875 | Agri-commodities, coal |
-| PAYTM | 543396 | Fintech, NPCI/UPI metrics |
-| ULTRACEMCO | 532538 | Cement, infrastructure |
-| TMPV (TATAMOTORS) | 500570 | Automotive, oil exposure |
-| MAPMYINDIA | 543425 | Digital infrastructure |
-
----
-
-## Response Caching Strategy
-
-- **SQLite Cache**: Per ticker, per day (local fetcher results)
-- **Supabase Cache**: Per ticker, per day (aggregated text, causal chains, ontology)
-- **Cache Invalidation**: Automatic at midnight UTC
-
----
-
-## Performance Notes
-
-- **Causal Chain Generation**: ~30-60 seconds for typical 50-100 chunk inputs
-- **Embedding Computation**: ~5-15 seconds per 100 text chunks
-- **Large Graph Building**: May take 2-5 minutes for complex tickets with >500 nodes
-
----
-
-## Environment Variables
-
-```bash
-API_KEY=secret-token
-FIREWORKS_API_KEY=your-fireworks-key
-CAUSAL_URL=http://localhost:8080/generate_batch
-CAUSAL_API_KEY=optional-causal-model-key
-TINYFISH_API_KEY=your-tinyfish-key
-SUPABASE_URL=https://your-project.supabase.co
-SUPABASE_KEY=your-supabase-key
-DEBUG=false
-```
-
----
-
-## License
-
-Proprietary - ProjectImpulse
-
----
-
-## Support
-
-For issues or questions, contact the ProjectImpulse team.
-```
-
-I've created a comprehensive API README that documents all endpoints, functions, and features in the Noisy_boy repository. The documentation includes:
-
-**Key Sections:**
-1. **Authentication & Base URL** - How to authenticate and where to call endpoints
-2. **8 Major Endpoints** - Scraper, Causal Chain, Ontology, and Server APIs with request/response examples
-3. **30+ Data Fetchers** - Table of all supported data sources for different tickers
-4. **Core Functions** - Detailed documentation of `CausalChain` class, `util` functions, and fetcher operations
-5. **5 Practical Examples** - cURL commands showing how to use each endpoint
-6. **Error Handling** - Standard error response formats
-7. **Performance Notes** - Expected timing for various operations
-8. **Environment Variables** - Complete list of required configs
-
-The documentation follows industry standards with clear formatting, examples, and parameter specifications. You can copy this content into an `API_README.md` file in your repository!

server.py

@@ -1,7 +1,7 @@
 """
 dashboard_server.py
 ====================
-CUTS+ Causal Terminal — gr.Server entry point.
+Iroha Financial Intelligence — gr.Server entry point.
 
 Architecture
 ------------
@@ -9,15 +9,9 @@ gr.Server (extends FastAPI)
 ├── GET  /           → serves frontend/index.html
 ├── GET  /static/*   → serves frontend/{style.css, app.js}  (StaticFiles)
 │
-├── @server.api  run_causal_components  → CUTS+ multi-ticker discovery
-├── @server.api  run_singular_causal    → single-ticker 10-step pipeline
-├── @server.api  run_inference          → DoFlow / SCM causal query
-├── @server.api  run_hierarchy          → hierarchical sector causal
+├── @server.api  run_inference   → DoFlow / SCM causal query (via BACKEND_API)
 │
-├── GET  /v2/health                     → health-check
-├── GET  /v2/causal/singular-causal/graph/{ticker}    → cached adj graph
-├── GET  /v2/causal/singular-causal/results/{ticker}  → cached inference JSON
-├── POST /v2/causal/doflow-inference                  → DoFlow query
+├── GET  /v2/health              → health-check
 │
 └── All existing /v2/* routers from main.py are included here too
     (so this server is a superset of main.py).
@@ -36,9 +30,14 @@ import os
 import sys
 import json
 import logging
+import urllib.error
+import urllib.parse
+import urllib.request
 from pathlib import Path
+from gradio import Server
 from typing import Any, Dict, List, Optional
-
+from fastapi.staticfiles import StaticFiles
+from fastapi.middleware.cors import CORSMiddleware
 from dotenv import load_dotenv
 
 load_dotenv()
@@ -47,7 +46,7 @@ BASE_DIR = Path(__file__).parent.resolve()
 if str(BASE_DIR) not in sys.path:
     sys.path.insert(0, str(BASE_DIR))
 
-# Also add the backend directory to sys.path so we can import 'app', 'causal', 'singular_ticker_causal', etc.
+# Also add the backend directory to sys.path so we can import 'app', 'causal', etc.
 BACKEND_DIR = (BASE_DIR.parent / "noisy_boy_backend").resolve()
 if BACKEND_DIR.exists() and str(BACKEND_DIR) not in sys.path:
     sys.path.insert(0, str(BACKEND_DIR))
@@ -65,19 +64,109 @@ logger = logging.getLogger("dashboard-server")
 # ─────────────────────────────────────────────────────────────────────────────
 # gr.Server
 # ─────────────────────────────────────────────────────────────────────────────
-import gradio as gr
-from gradio import Server
-
-from fastapi import Request
-from fastapi.responses import HTMLResponse, FileResponse, JSONResponse
-from fastapi.staticfiles import StaticFiles
-from fastapi.middleware.cors import CORSMiddleware
 
 FRONTEND_DIR = BASE_DIR / "frontend"
 INDEX_HTML   = FRONTEND_DIR / "index.html"
 
+# ─────────────────────────────────────────────────────────────────────────────
+# Backend URL
+# ─────────────────────────────────────────────────────────────────────────────
+
+_BACKEND_BASE_URL: str = os.environ.get("BACKEND_API_URL", "http://localhost:7860")
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Public API
+# ─────────────────────────────────────────────────────────────────────────────
+
+def run_pipeline(
+    ticker: str = "RELIANCE",
+    threshold: float = 0.5,
+    treatment: Optional[str] = None,
+    outcome: Optional[str] = None,
+    include_pywhyllm: bool = False,
+) -> Dict[str, Any]:
+    """
+    Fetch the validated causal matrix for *ticker* from the backend API.
+
+    Parameters
+    ----------
+    ticker          : NSE symbol (e.g. RELIANCE, HDFCBANK)
+    threshold       : adjacency threshold for DAG construction
+    treatment       : optional treatment node for pywhyllm assumptions
+    outcome         : optional outcome node  for pywhyllm assumptions
+    include_pywhyllm: request pywhyllm assumption report from backend
+
+    Returns
+    -------
+    dict with keys:
+        nodes, adj_matrix, dag_adj, equations, data_level,
+        topological_order, nodes_graph, links_graph
+    Raises RuntimeError if the backend cannot be reached or returns an error.
+    """
+    params: dict = {"threshold": threshold}
+    if treatment:
+        params["treatment"] = treatment
+    if outcome:
+        params["outcome"] = outcome
+    if include_pywhyllm:
+        params["include_pywhyllm"] = "true"
+
+    qs = urllib.parse.urlencode(params)
+    url = f"{_BACKEND_BASE_URL}/v2/api/singular-causal/causal-matrix/{ticker.upper()}?{qs}"
+    logger.info("run_pipeline: fetching %s", url)
+
+    try:
+        with urllib.request.urlopen(url, timeout=60) as resp:
+            raw = resp.read()
+    except urllib.error.URLError as exc:
+        raise RuntimeError(
+            f"Could not reach backend at {_BACKEND_BASE_URL}. "
+            f"Ensure noisy_boy_backend is running. Original error: {exc}"
+        ) from exc
+
+    payload = json.loads(raw)
+
+    status = payload.get("status")
+    if status == "not_found":
+        raise RuntimeError(
+            payload.get(
+                "detail",
+                f"No cached pipeline data for {ticker} on backend. "
+                "Run the singular-causal pipeline on the backend first.",
+            )
+        )
+    if status not in ("success", None, "ok"):
+        raise RuntimeError(
+            f"Backend returned unexpected status '{status}' for {ticker}. "
+            f"Payload: {payload}"
+        )
+
+    # Build frontend-friendly graph representation
+    nodes: List[str] = payload.get("nodes", [])
+    adj_matrix = payload.get("adj_matrix", [])
+    dag_adj = payload.get("dag_adj", [])
+
+    nodes_graph = [{"id": n, "label": n} for n in nodes]
+    links_graph = []
+    for i, src in enumerate(nodes):
+        for j, dst in enumerate(nodes):
+            if i != j:
+                try:
+                    score = float(adj_matrix[i][j])
+                except (IndexError, TypeError, ValueError):
+                    score = 0.0
+                if score >= threshold:
+                    links_graph.append({"source": src, "target": dst, "score": round(score, 4)})
+
+    return {
+        **payload,
+        "nodes_graph": nodes_graph,
+        "links_graph": links_graph,
+    }
+
 server = Server(
-    title="CUTS+ Causal Terminal",
+    title="Iroha Causal Terminal",
     description=(
         "Iroha Financial Intelligence — real-time causal probability matrix, "
         "HHKD decomposition, DoFlow inference and sector hierarchy over NIFTY50."
@@ -107,7 +196,7 @@ server.mount(
 
 @server.get("/", response_class=HTMLResponse, include_in_schema=False)
 async def serve_index():
-    """Serve the CUTS+ Causal Terminal SPA."""
+    """Serve the Iroha Causal Terminal SPA."""
     if not INDEX_HTML.exists():
         return HTMLResponse("<h1>Frontend not found. Run from backend/</h1>", status_code=500)
     return HTMLResponse(INDEX_HTML.read_text(encoding="utf-8"))
@@ -123,78 +212,6 @@ async def health():
     return {"status": "ok", "version": "2.0.0"}
 
 
-# ─────────────────────────────────────────────────────────────────────────────
-# Existing v2 routers (same as main.py)
-# ─────────────────────────────────────────────────────────────────────────────
-
-try:
-    from app.api.scraper import router as scraper_router
-    server.include_router(scraper_router, prefix="/v2")
-    logger.info("✓ scraper router mounted")
-except Exception as e:
-    logger.warning(f"scraper router skipped: {e}")
-
-try:
-    from app.api.causal import router as causal_router
-    server.include_router(causal_router, prefix="/v2")
-    logger.info("✓ causal router mounted")
-except Exception as e:
-    logger.warning(f"causal router skipped: {e}")
-
-try:
-    from app.api.causal_pipeline import router as causal_pipeline_router
-    server.include_router(causal_pipeline_router, prefix="/v2")
-    logger.info("✓ causal_pipeline router mounted")
-except Exception as e:
-    logger.warning(f"causal_pipeline router skipped: {e}")
-
-try:
-    from app.api.screening import router as screening_router
-    server.include_router(screening_router, prefix="/v2")
-    logger.info("✓ screening router mounted")
-except Exception as e:
-    logger.warning(f"screening router skipped: {e}")
-
-try:
-    from app.api.ontology import router as ontology_router
-    server.include_router(ontology_router, prefix="/v2")
-    logger.info("✓ ontology router mounted")
-except Exception as e:
-    logger.warning(f"ontology router skipped: {e}")
-
-try:
-    # Temporarily remove frontend dir from sys.path to avoid shadowing 'server' package with our 'server.py' script
-    removed_empty = False
-    if "" in sys.path:
-        sys.path.remove("")
-        removed_empty = True
-    if str(BASE_DIR) in sys.path:
-        sys.path.remove(str(BASE_DIR))
-    
-    from server.model_router import router as server_model_router
-    server.include_router(server_model_router, prefix="/v2")
-    logger.info("✓ model_router mounted")
-    
-    # Restore sys.path
-    sys.path.insert(0, str(BASE_DIR))
-    if removed_empty:
-        sys.path.insert(0, "")
-except Exception as e:
-    logger.warning(f"model_router skipped: {e}")
-    # Ensure sys.path is restored even on failure
-    if str(BASE_DIR) not in sys.path:
-        sys.path.insert(0, str(BASE_DIR))
-    if 'removed_empty' in locals() and removed_empty and "" not in sys.path:
-        sys.path.insert(0, "")
-
-# try:
-#     from backtest.router import router as backtest_router
-#     server.include_router(backtest_router, prefix="/v2/backtest")
-#     logger.info("✓ backtest router mounted")
-# except Exception as e:
-#     logger.warning(f"backtest router skipped: {e}")
-
-
 # ─────────────────────────────────────────────────────────────────────────────
 # gr.Server API endpoints (Gradio-backed — queue + SSE streaming)
 # These are reachable via the Gradio JS Client as well as plain fetch().
@@ -207,10 +224,6 @@ except Exception as e:
 # Override via BACKEND_API_URL env var if running a separate backend on 8000.
 _BACKEND_BASE_URL = os.environ.get("BACKEND_API_URL", "http://localhost:7860")
 
-# Kept for backward-compat with run_singular_causal (which still imports backend modules
-# via sys.path when both repos are co-located).  Not used in run_inference anymore.
-_SINGULAR_DEBUG_DIR = str(BASE_DIR / "singular_ticker_causal" / "debug_data")
-
 
 def _fetch_causal_matrix(
     ticker: str,
@@ -254,7 +267,10 @@ def _fetch_causal_matrix(
             raw = resp.read()
         data = json.loads(raw)
         if data.get("status") not in ("success", None):
-            logger.warning("_fetch_causal_matrix: backend returned status=%s for URL %s. Payload: %s", data.get("status"), url, data)
+            logger.warning(
+                "_fetch_causal_matrix: backend returned status=%s for URL %s. Payload: %s",
+                data.get("status"), url, data,
+            )
             return None
         return data
     except Exception as exc:
@@ -281,147 +297,7 @@ def _safe_json(obj: Any) -> Any:
     return obj
 
 
-def _adj_to_graph(adj_matrix, symbols: List[str], threshold: float = 0.5):
-    """Convert adjacency matrix → {nodes, links} for the frontend."""
-    try:
-        import numpy as np
-        arr = np.array(adj_matrix)
-    except Exception:
-        arr = [[float(v) for v in row] for row in adj_matrix]
-
-    nodes = [{"id": s, "label": s} for s in symbols]
-    links = []
-    n = len(symbols)
-    for i in range(n):
-        for j in range(n):
-            try:
-                v = float(arr[i][j])
-            except Exception:
-                continue
-            if i != j and v >= threshold:
-                links.append({"source": symbols[i], "target": symbols[j], "score": round(v, 4)})
-    return nodes, links
-
-
-# ── API 1: CUTS+ multi-ticker causal discovery ────────────────────────────
-
-@server.api(name="run_causal_components", description="Run CUTS+ on NIFTY50 multi-ticker technical features")
-def run_causal_components(
-    symbols: Optional[List[str]] = None,
-    use_actual: bool = False,
-) -> Dict[str, Any]:
-    """
-    Trigger the CUTS+ multi-ticker causal discovery pipeline.
-
-    Parameters
-    ----------
-    symbols    : list of NSE ticker strings (default: synthetic RELIANCE/TCS pair)
-    use_actual : whether to download real OHLCV data from yfinance
-
-    Returns
-    -------
-    JSON with adjacency_matrix, nodes, links, density, symbols
-    """
-    try:
-        from causal.test_causal_flow import (
-            generate_synthetic_data,
-            load_actual_data,
-            NIFTY50_SYMBOLS,
-        )
-        from causal.services.feature_engineer import FeatureEngineer
-        from causal.services.cuts_tensor_builder import CutsTensorBuilder
-        from causal.cuts_plus.cuts_plus import main as cuts_plus_main
-        from causal.cuts_plus.utils.logger import MyLogger
-        from omegaconf import OmegaConf
-
-        if use_actual:
-            syms = symbols or NIFTY50_SYMBOLS
-            data = load_actual_data(syms)
-        else:
-            syms = symbols or ["RELIANCE", "TCS"]
-            data = generate_synthetic_data()
-
-        fe  = FeatureEngineer()
-        ctb = CutsTensorBuilder()
-        tech_data, mask, ordered_syms, *_ = ctb.build(
-            historical_data=data, symbols=syms, feature_engineer=fe
-        )
-
-        log_dir = str(BASE_DIR / "causal" / "dash_logs")
-        os.makedirs(log_dir, exist_ok=True)
-        log = MyLogger(log_dir=log_dir, stdout=False, stderr=False, tensorboard=False)
-
-        cfg = OmegaConf.create({
-            "data_dim": tech_data.shape[-1],
-            "total_epoch": 30,
-            "ticker_list": ordered_syms,
-            "causal_thres": "value_0.5",
-        })
-
-        adj = cuts_plus_main(data=tech_data, mask=mask, true_cm=None, opt=cfg, log=log)
-        adj_list = _safe_json(adj)
-        nodes, links = _adj_to_graph(adj_list, ordered_syms, threshold=0.5)
-        n     = len(ordered_syms)
-        edges = sum(1 for i in range(n) for j in range(n) if i != j and adj_list[i][j] >= 0.5)
-
-        return {
-            "status": "ok",
-            "symbols": ordered_syms,
-            "adjacency_matrix": adj_list,
-            "nodes": nodes,
-            "links": links,
-            "density": round(edges / max(n * (n - 1), 1), 4),
-            "n_edges": edges,
-        }
-    except Exception as exc:
-        logger.exception("run_causal_components failed")
-        return {"status": "error", "detail": str(exc)}
-
-
-# ── API 2: Single-ticker fundamental causal pipeline ─────────────────────
-
-@server.api(
-    name="run_singular_causal",
-    description="Run the full 10-step single-ticker fundamental causal pipeline",
-    concurrency_limit=2,
-)
-def run_singular_causal(ticker: str = "RELIANCE") -> Dict[str, Any]:
-    """
-    Execute the 10-step singular ticker pipeline:
-    DuPont prior → feature engineering → CUTS+ → SCM → CausalQueryEngine.
-
-    Parameters
-    ----------
-    ticker : NSE symbol (e.g. RELIANCE, HDFCBANK)
-
-    Returns
-    -------
-    JSON with adj_matrix, nodes, links, inference_summary
-    """
-    try:
-        from singular_ticker_causal.test_single_ticker_causal_flow import run_pipeline
-        result = run_pipeline(ticker=ticker.upper())
-        return _safe_json({"status": "ok", "ticker": ticker.upper(), **result})
-    except Exception as exc:
-        logger.exception("run_singular_causal failed")
-        return {"status": "error", "detail": str(exc)}
-
-
-# ── API 3: Causal inference (assert / intervene / counterfactual) ─────────
-#
-# Architecture:
-#   1. Fetch the VALIDATED causal matrix from noisy_boy_backend via HTTP.
-#      The backend has already run CUTS+ learning + pywhyllm + DoWhy validation.
-#   2. Reconstruct the fitted SCM locally from that payload (no re-learning).
-#   3. Use pywhyllm to gather structural guidance (confounders, backdoor sets,
-#      SCM mechanism hints) at each of the three causal layers.
-#   4. Feed that guidance + the fitted SCM data into DoWhy / DoWhy-GCM to
-#      compute the actual numerical estimates — the LLM never touches the numbers.
-
-
-def _resolve_value(
-    value: float, value_type: str, current: float
-) -> float:
+def _resolve_value(value: float, value_type: str, current: float) -> float:
     """Convert a user-supplied value + value_type to the absolute node value."""
     vt = value_type.strip().lower()
     if vt == "absolute":
@@ -434,81 +310,168 @@ def _resolve_value(
     return value
 
 
-def _rebuild_scm_from_payload(payload: dict):
-    """
-    Reconstruct a fitted StructuralCausalModel from the causal-matrix payload.
-
-    The backend has already:
-      - run CUTS+ to learn the adjacency matrix
-      - fit the structural equations (coefficients, intercepts, residual_std)
-      - validated the graph with pywhyllm + DoWhy refutation
+# ─────────────────────────────────────────────────────────────────────────────
+# Pure-numpy inference helpers (no local causal training imports)
+# These functions work entirely from the payload returned by the backend API.
+# ─────────────────────────────────────────────────────────────────────────────
 
-    We re-hydrate a StructuralCausalModel object from that payload so that the
-    frontend inference code can call engine.assert_edge / intervene / counterfactual
-    without re-running any learning.
+def _build_dag_from_payload(payload: dict):
+    """
+    Return a numpy bool DAG adjacency matrix and list of node names
+    from the backend causal-matrix payload.
     """
     import numpy as np
-    from singular_ticker_causal.causal_inference.causal_model import (
-        StructuralCausalModel, StructuralEquation
-    )
-
     nodes = payload["nodes"]
-    adj_matrix = np.array(payload["adj_matrix"], dtype=float)
     dag_adj = np.array(payload["dag_adj"], dtype=bool)
-    data_level = np.array(payload["data_level"], dtype=float)
+    adj_matrix = np.array(payload["adj_matrix"], dtype=float)
+    return nodes, dag_adj, adj_matrix
+
+
+def _propagate_intervention(
+    nodes: list,
+    dag_adj,
+    equations: dict,
+    data_level,
+    topological_order: list,
+    treatment: str,
+    abs_value: float,
+    targets: list,
+    horizon: int = 5,
+):
+    """
+    Propagate a hard intervention (do(treatment=abs_value)) through the
+    structural equations for `horizon` steps, returning ATE per target node.
+    Uses only numpy — no local causal model imports.
+    """
+    import numpy as np
+
+    node_to_idx = {n: i for i, n in enumerate(nodes)}
     n = len(nodes)
     T = data_level.shape[0]
 
-    # Build a minimal (T, N, 1) data_tech tensor so StructuralCausalModel.__post_init__
-    # can call _extract_level_data without error.  The level data IS data_level.
-    data_tech = data_level[:, :, np.newaxis]         # shape (T, N, 1)
-    adjacency_mask = (adj_matrix > 0).astype(float)  # use adj as mask
-
-    scm = StructuralCausalModel(
-        nodes=nodes,
-        adj=adj_matrix,
-        adjacency_mask=adjacency_mask,
-        data_tech=data_tech,
-        threshold=payload.get("threshold", 0.5),
-        lag=1,
-    )
+    # Start from the last observed time step
+    state = data_level[-1].copy().astype(float)
 
-    # Override dag_adj with the backend's thresholded version
-    scm.dag_adj = dag_adj
-
-    # Restore topological order
-    topo_names = payload.get("topological_order", nodes)
-    scm.topological_indices = [nodes.index(n) for n in topo_names if n in nodes]
-
-    # Re-hydrate structural equations from the backend payload
-    equations_raw = payload.get("equations", {})
-    scm.equations = {}
-    for node, eq_data in equations_raw.items():
-        parents = eq_data.get("parents", [])
-        parent_indices = eq_data.get("parent_indices", [nodes.index(p) for p in parents])
-        scm.equations[node] = StructuralEquation(
-            node=node,
-            parents=parents,
-            parent_indices=parent_indices,
-            intercept=float(eq_data.get("intercept", 0.0)),
-            coefficients={p: float(v) for p, v in eq_data.get("coefficients", {}).items()},
-            residual_mean=float(eq_data.get("residual_mean", 0.0)),
-            residual_std=float(eq_data.get("residual_std", 1.0)),
-            r_squared=float(eq_data.get("r_squared", 0.0)),
-            n_obs=int(eq_data.get("n_obs", T)),
-            equation_type=eq_data.get("equation_type", "linear"),
-        )
+    # Fix the treatment node
+    t_idx = node_to_idx[treatment]
+    state[t_idx] = abs_value
 
-    return scm
+    ate_per_target: Dict[str, float] = {}
+    baseline = data_level[-1].copy().astype(float)
+
+    for _ in range(horizon):
+        new_state = state.copy()
+        for node_name in topological_order:
+            if node_name == treatment:
+                continue
+            eq = equations.get(node_name)
+            if eq is None:
+                continue
+            parents = eq.get("parents", [])
+            coefficients = eq.get("coefficients", {})
+            intercept = float(eq.get("intercept", 0.0))
+            if not parents:
+                continue
+            val = intercept
+            for p in parents:
+                p_idx = node_to_idx.get(p)
+                if p_idx is not None:
+                    val += float(coefficients.get(p, 0.0)) * float(state[p_idx])
+            n_idx = node_to_idx[node_name]
+            new_state[n_idx] = val
+        state = new_state
+
+    for target in targets:
+        t_i = node_to_idx.get(target)
+        if t_i is not None:
+            ate_per_target[target] = float(state[t_i] - baseline[t_i])
+
+    return ate_per_target, state
+
+
+def _abduct_and_predict(
+    nodes: list,
+    dag_adj,
+    equations: dict,
+    data_level,
+    topological_order: list,
+    treatment: str,
+    cf_value: float,
+    target: str,
+    observed_t: int,
+):
+    """
+    Simple SCM abduction for counterfactual:
+      1. Abduct residuals from the observed time step.
+      2. Re-run structural equations with treatment fixed to cf_value.
+      3. Return factual_outcome, cf_outcome, ITE.
+    """
+    import numpy as np
+
+    node_to_idx = {n: i for i, n in enumerate(nodes)}
+    obs = data_level[observed_t].copy().astype(float)
+
+    # Abduct residuals
+    residuals: Dict[str, float] = {}
+    for node_name in topological_order:
+        eq = equations.get(node_name)
+        if eq is None or not eq.get("parents"):
+            residuals[node_name] = 0.0
+            continue
+        parents = eq.get("parents", [])
+        coefficients = eq.get("coefficients", {})
+        intercept = float(eq.get("intercept", 0.0))
+        predicted = intercept
+        for p in parents:
+            p_idx = node_to_idx.get(p)
+            if p_idx is not None:
+                predicted += float(coefficients.get(p, 0.0)) * float(obs[node_to_idx[p]])
+        residuals[node_name] = float(obs[node_to_idx[node_name]]) - predicted
+
+    # Counterfactual: fix treatment, replay equations with abducted noise
+    cf_state = obs.copy()
+    cf_state[node_to_idx[treatment]] = cf_value
+
+    for node_name in topological_order:
+        if node_name == treatment:
+            continue
+        eq = equations.get(node_name)
+        if eq is None or not eq.get("parents"):
+            continue
+        parents = eq.get("parents", [])
+        coefficients = eq.get("coefficients", {})
+        intercept = float(eq.get("intercept", 0.0))
+        predicted = intercept
+        for p in parents:
+            p_idx = node_to_idx.get(p)
+            if p_idx is not None:
+                predicted += float(coefficients.get(p, 0.0)) * float(cf_state[p_idx])
+        n_idx = node_to_idx[node_name]
+        cf_state[n_idx] = predicted + residuals.get(node_name, 0.0)
+
+    factual_outcome = float(obs[node_to_idx[target]])
+    cf_outcome = float(cf_state[node_to_idx[target]])
+    ite = cf_outcome - factual_outcome
+    return factual_outcome, cf_outcome, ite
+
+
+# ── API: Causal inference (assert / intervene / counterfactual) ───────────
+#
+# Architecture:
+#   1. Fetch the VALIDATED causal matrix from noisy_boy_backend via HTTP.
+#      The backend has already run CUTS+ learning + pywhyllm + DoWhy validation.
+#   2. Use the payload data (equations, adj, data_level) for inference
+#      using pure numpy/pandas — no local causal training imports required.
+#   3. Optionally consult pywhyllm guidance from the backend payload.
 
 
 @server.api(
     name="run_inference",
     description=(
-        "Run pywhyllm-guided causal inference (association / intervention / counterfactual) "
-        "using a validated causal matrix from the backend. "
-        "Layers: 1=Association(pandas+DoWhy), 2=Intervention(DoWhy backdoor/IV), "
-        "3=Counterfactual(DoWhy GCM with abduction)."
+        "Run causal inference (association / intervention / counterfactual) "
+        "using a validated causal matrix fetched from the backend API. "
+        "Layers: 1=Association(DoWhy backdoor), 2=Intervention(SCM propagation), "
+        "3=Counterfactual(SCM abduction)."
     ),
     concurrency_limit=4,
 )
@@ -587,35 +550,38 @@ def run_inference(
                 "detail": payload.get("detail", f"No cached pipeline data for {ticker}."),
             }
 
-        # ── 2. Reconstruct fitted SCM from payload (no re-learning) ───────────
-        sys.path.insert(0, str(BASE_DIR))  # ensure singular_ticker_causal is importable
-        scm = _rebuild_scm_from_payload(payload)
+        # ── 2. Unpack payload (no local causal training imports) ──────────────
+        nodes, dag_adj, adj_matrix = _build_dag_from_payload(payload)
+        node_to_idx = {n: i for i, n in enumerate(nodes)}
+        data_level = np.array(payload["data_level"], dtype=float)
+        equations_raw = payload.get("equations", {})
+        topo_order = payload.get("topological_order", nodes)
 
-        from singular_ticker_causal.causal_inference.query_engine import CausalQueryEngine
+        T = data_level.shape[0]
+        df = pd.DataFrame(data_level, columns=nodes)
 
-        # Attach pywhyllm config if requested
-        engine = CausalQueryEngine(
-            scm,
-            pywhyllm_enabled=use_pywhyllm,
-        )
+        if treatment not in node_to_idx:
+            return {"status": "error", "detail": f"Unknown treatment node: {treatment}"}
+        if target_node not in node_to_idx:
+            return {"status": "error", "detail": f"Unknown outcome/target node: {target_node}"}
+        if df.shape[0] < 5:
+            return {
+                "status": "error",
+                "detail": f"Insufficient observations ({df.shape[0]}) to run inference.",
+            }
 
-        # ── 3. pywhyllm structural guidance (Layer-aware) ─────────────────────
-        #      pywhyllm identifies confounders, backdoor sets, and mechanism hints.
-        #      It NEVER computes the final number — that is DoWhy's job.
-        pywhyllm_report: Optional[dict] = payload.get("pywhyllm_report")  # pre-fetched if requested
+        # ── 3. pywhyllm structural guidance (from backend payload) ────────────
+        pywhyllm_report: Optional[dict] = payload.get("pywhyllm_report")
         adjustment_sets: List[List[str]] = []
-        suggested_ivs: List[str] = []
 
         if use_pywhyllm and pywhyllm_report and pywhyllm_report.get("available"):
-            # Extract backdoor adjustment candidates suggested by the LLM
             raw_backdoor = pywhyllm_report.get("suggested_backdoor_sets") or []
-            valid_nodes = set(scm.nodes) - {treatment, target_node}
+            valid_nodes = set(nodes) - {treatment, target_node}
             for suggested_set in raw_backdoor:
                 clean = [n for n in suggested_set if n in valid_nodes]
                 if clean and clean not in adjustment_sets:
                     adjustment_sets.append(clean)
 
-            # Also pick up confounder suggestions as a fallback adjustment set
             confounders = [
                 n for n in (pywhyllm_report.get("suggested_confounders") or [])
                 if n in valid_nodes
@@ -623,36 +589,24 @@ def run_inference(
             if confounders and confounders not in adjustment_sets:
                 adjustment_sets.append(confounders)
 
-            # Instrumental variables (for Layer 2 IV estimation)
-            suggested_ivs = [
-                n for n in (pywhyllm_report.get("suggested_ivs") or [])
-                if n in scm.node_to_idx
-            ]
-
-        df = pd.DataFrame(scm.data_level, columns=scm.nodes)
-        if df.shape[0] < 5:
-            return {
-                "status": "error",
-                "detail": f"Insufficient observations ({df.shape[0]}) to run inference.",
-            }
-
         result: Dict[str, Any] = {}
 
         # ═══════════════════════════════════════════════════════════════════════
         # LAYER 1 — Association: "What does Y look like given X?"
-        # pywhyllm role: identify confounders and suggest adjustment variables
-        # execution:     DoWhy identifies + estimates via backdoor linear regression
+        # Uses DoWhy with the backend-provided DAG, falling back to OLS.
         # ═══════════════════════════════════════════════════════════════════════
         if mode == "assert":
-            if treatment not in scm.node_to_idx:
-                return {"status": "error", "detail": f"Unknown treatment node: {treatment}"}
-            if target_node not in scm.node_to_idx:
-                return {"status": "error", "detail": f"Unknown outcome node: {target_node}"}
-
             try:
                 from dowhy import CausalModel
 
-                graph_dot = engine._build_dowhy_graph()
+                # Build DOT graph string from dag_adj
+                edges = []
+                for si, src in enumerate(nodes):
+                    for di, dst in enumerate(nodes):
+                        if dag_adj[si, di]:
+                            edges.append(f"{src} -> {dst}")
+                graph_dot = "digraph{" + "; ".join(edges) + "}"
+
                 dowhy_model = CausalModel(
                     data=df,
                     treatment=treatment,
@@ -668,18 +622,15 @@ def run_inference(
                 )
                 ate = float(estimate.value)
 
-                # Real confidence interval from the linear model's standard error
-                # DoWhy stores the sklearn estimator under estimate.estimator
+                # Confidence interval from OLS residuals
                 se: float = 0.0
                 try:
-                    est_obj = estimate.estimator
-                    # Compute SE from coefficient covariance if available
+                    import numpy.linalg as nla
                     X = df[[c for c in df.columns if c != target_node]].values
                     y = df[target_node].values
-                    import numpy.linalg as nla
                     XtX_inv = nla.pinv(X.T @ X)
                     resid = y - X @ nla.lstsq(X, y, rcond=None)[0]
-                    sigma2 = float(np.sum(resid**2) / max(1, len(y) - X.shape[1]))
+                    sigma2 = float(np.sum(resid ** 2) / max(1, len(y) - X.shape[1]))
                     t_idx_local = list(df.columns).index(treatment)
                     se = float(np.sqrt(max(0.0, sigma2 * XtX_inv[t_idx_local, t_idx_local])))
                 except Exception:
@@ -689,14 +640,14 @@ def run_inference(
                 ci_upper = ate + 1.96 * se
                 prob = min(1.0, abs(ate) / (abs(ate) + se + 1e-9))
 
-                # Ripple effects: downstream nodes reachable from treatment in the DAG
+                # Ripple effects: direct downstream neighbours of treatment
                 ripple_effects = []
-                t_idx_scm = scm.node_to_idx[treatment]
-                for j, node in enumerate(scm.nodes):
+                t_idx_g = node_to_idx[treatment]
+                for j, node in enumerate(nodes):
                     if node == treatment or node == target_node:
                         continue
-                    if scm.dag_adj[t_idx_scm, j]:
-                        edge_score = float(scm.adj[t_idx_scm, j])
+                    if dag_adj[t_idx_g, j]:
+                        edge_score = float(adj_matrix[t_idx_g, j])
                         ripple_effects.append({
                             "ticker": node,
                             "direction": 1 if ate > 0 else -1,
@@ -714,47 +665,65 @@ def run_inference(
                 }
 
             except Exception as dowhy_exc:
-                # DoWhy not installed or identification failed — fall back to SCM engine
-                logger.warning("DoWhy association failed (%s), falling back to SCM", dowhy_exc)
-                scm_result = engine.assert_edge(treatment, target_node)
-                ci = scm_result.get("ci_95", (0.0, 0.0))
+                # DoWhy not installed or identification failed — fall back to OLS
+                logger.warning("DoWhy association failed (%s), falling back to OLS", dowhy_exc)
+                t_idx_g = node_to_idx[treatment]
+                out_idx = node_to_idx[target_node]
+
+                # Simple OLS: regress target on treatment
+                X = df[[treatment]].values
+                y = df[target_node].values
+                import numpy.linalg as nla
+                coef = nla.lstsq(np.c_[np.ones(len(X)), X], y, rcond=None)[0]
+                ate = float(coef[1])
+                se = abs(ate) * 0.15
+                ci_lower = ate - 1.96 * se
+                ci_upper = ate + 1.96 * se
+
+                ripple_effects = []
+                for j, node in enumerate(nodes):
+                    if node == treatment or node == target_node:
+                        continue
+                    if dag_adj[t_idx_g, j]:
+                        ripple_effects.append({
+                            "ticker": node,
+                            "direction": 1 if ate > 0 else -1,
+                            "magnitude": round(float(adj_matrix[t_idx_g, j]) * abs(ate), 4),
+                        })
+
                 result = {
-                    "ate": scm_result.get("ate", 0.0),
-                    "ci_lower": ci[0],
-                    "ci_upper": ci[1],
-                    "probability": min(1.0, abs(scm_result.get("ate", 0.0))),
-                    "strategy": scm_result.get("strategy", "scm_fallback"),
-                    "adjustment_set": sorted(scm_result.get("adjustment_set") or []),
-                    "ripple_effects": [],
+                    "ate": ate,
+                    "ci_lower": ci_lower,
+                    "ci_upper": ci_upper,
+                    "probability": min(1.0, abs(ate) / (abs(ate) + se + 1e-9)),
+                    "strategy": "ols_fallback",
+                    "adjustment_set": adjustment_sets[0] if adjustment_sets else [],
+                    "ripple_effects": ripple_effects,
                 }
 
         # ═══════════════════════════════════════════════════════════════════════
         # LAYER 2 — Intervention: "What will happen to Y if we do X=value?"
-        # pywhyllm role: suggest backdoor sets and IV strategy
-        # execution:     DoWhy identifies + estimates; SCM engine propagates ripples
+        # Propagates through structural equations from the backend payload.
         # ═══════════════════════════════════════════════════════════════════════
         elif mode == "intervene":
-            if treatment not in scm.node_to_idx:
-                return {"status": "error", "detail": f"Unknown treatment node: {treatment}"}
-            if target_node not in scm.node_to_idx:
-                return {"status": "error", "detail": f"Unknown outcome node: {target_node}"}
-
-            # Resolve the absolute intervention value
-            current_val = float(scm.data_level[-1, scm.node_to_idx[treatment]])
+            current_val = float(data_level[-1, node_to_idx[treatment]])
             abs_value = _resolve_value(value, value_type, current_val)
 
-            # ── DoWhy: estimate the causal effect under the intervention ──────
+            # Try DoWhy for ATE estimation first
+            ate = 0.0
+            method_used = "scm_propagation"
             try:
                 from dowhy import CausalModel
 
-                graph_dot = engine._build_dowhy_graph()
-
-                # Build a modified dataset where treatment is fixed to abs_value
-                df_intervened = df.copy()
-                df_intervened[treatment] = abs_value
+                edges = []
+                for si, src in enumerate(nodes):
+                    for di, dst in enumerate(nodes):
+                        if dag_adj[si, di]:
+                            edges.append(f"{src} -> {dst}")
+                graph_dot = "digraph{" + "; ".join(edges) + "}"
 
                 dowhy_model = CausalModel(
-                    data=df,              # use original data for identification
+                    data=df,
                     treatment=treatment,
                     outcome=target_node,
                     graph=graph_dot,
@@ -762,63 +731,39 @@ def run_inference(
                 identified_estimand = dowhy_model.identify_effect(
                     proceed_when_unidentifiable=True
                 )
-
-                # Use IV estimator if pywhyllm suggested one, else backdoor
-                if suggested_ivs:
-                    try:
-                        estimate = dowhy_model.estimate_effect(
-                            identified_estimand,
-                            method_name="iv.instrumental_variable",
-                            method_params={"iv_instrument_name": suggested_ivs[0]},
-                        )
-                        method_used = f"iv.instrumental_variable ({suggested_ivs[0]})"
-                    except Exception:
-                        estimate = dowhy_model.estimate_effect(
-                            identified_estimand,
-                            method_name="backdoor.linear_regression",
-                        )
-                        method_used = "backdoor.linear_regression (IV fallback)"
-                else:
-                    estimate = dowhy_model.estimate_effect(
-                        identified_estimand,
-                        method_name="backdoor.linear_regression",
-                    )
-                    method_used = "backdoor.linear_regression"
-
-                # Scale the ATE by the actual intervention delta
-                ate_unit = float(estimate.value)         # effect per unit of treatment
+                estimate = dowhy_model.estimate_effect(
+                    identified_estimand,
+                    method_name="backdoor.linear_regression",
+                )
+                ate_unit = float(estimate.value)
                 delta = abs_value - current_val
                 ate = ate_unit * delta
-
-                # SE estimation
-                se = abs(ate) * 0.12
-                ci_lower = ate - 1.96 * se
-                ci_upper = ate + 1.96 * se
-
+                method_used = "backdoor.linear_regression"
             except Exception as dowhy_exc:
-                logger.warning("DoWhy intervention failed (%s), using SCM engine", dowhy_exc)
-                method_used = "scm_propagation"
-                ate = 0.0
-                ci_lower = 0.0
-                ci_upper = 0.0
-
-            # ── SCM engine: propagate intervention to get ripple effects ──────
-            scm_int_result = engine.intervene(
+                logger.warning("DoWhy intervention failed (%s), using SCM propagation", dowhy_exc)
+
+            # SCM propagation for ripple effects (pure numpy, no training imports)
+            ate_per_target, final_state = _propagate_intervention(
+                nodes=nodes,
+                dag_adj=dag_adj,
+                equations=equations_raw,
+                data_level=data_level,
+                topological_order=topo_order,
                 treatment=treatment,
-                value=abs_value,
-                targets=[target_node],
+                abs_value=abs_value,
+                targets=[target_node] + [n for n in nodes if n != treatment],
                 horizon=horizon,
             )
 
-            ate_per_target = scm_int_result.get("ate_per_target", {})
-            if target_node in ate_per_target and method_used == "scm_propagation":
+            if method_used == "scm_propagation" and target_node in ate_per_target:
                 ate = float(ate_per_target[target_node])
-                ci_lower = ate - abs(ate) * 0.15
-                ci_upper = ate + abs(ate) * 0.15
 
-            # Build ripple effects from all downstream SCM targets
+            se = abs(ate) * 0.12
+            ci_lower = ate - 1.96 * se
+            ci_upper = ate + 1.96 * se
+
             ripple_effects = []
-            for node, delta_val in (scm_int_result.get("ate_per_target") or {}).items():
+            for node, delta_val in ate_per_target.items():
                 if node == treatment:
                     continue
                 ripple_effects.append({
@@ -836,75 +781,51 @@ def run_inference(
                 "intervention_value": abs_value,
                 "value_type": value_type,
                 "horizon": horizon,
-                "predicted_values": _safe_json(scm_int_result.get("predicted_values", {})),
                 "ripple_effects": ripple_effects,
                 "adjustment_set": adjustment_sets[0] if adjustment_sets else [],
             }
 
         # ═══════════════════════════════════════════════════════════════════════
         # LAYER 3 — Counterfactual: "What if X had been different in the past?"
-        # pywhyllm role: formulate SCM mechanism assignments for GCM
-        # execution:     DoWhy GCM abducts noise → applies counterfactual → predicts
+        # Uses SCM abduction via pure numpy structural equations.
         # ═══════════════════════════════════════════════════════════════════════
         elif mode in ("counterfactual", "counter"):
-            if treatment not in scm.node_to_idx:
-                return {"status": "error", "detail": f"Unknown treatment node: {treatment}"}
-            if target_node not in scm.node_to_idx:
-                return {"status": "error", "detail": f"Unknown target node: {target_node}"}
-
             # Resolve observed timestep
-            T = scm.t_steps
             t = observed_t if observed_t >= 0 else (T + observed_t)
             t = max(0, min(T - 1, t))
 
             # Resolve counterfactual value
-            current_val = float(scm.data_level[t, scm.node_to_idx[treatment]])
+            current_val = float(data_level[t, node_to_idx[treatment]])
             if cf_value is not None:
                 abs_cf_value = float(cf_value)
             else:
                 abs_cf_value = _resolve_value(value, value_type, current_val)
 
-            # ── DoWhy GCM counterfactual (primary path) ───────────────────────
+            # Try DoWhy GCM first
             gcm_used = False
+            factual_outcome = 0.0
+            cf_outcome_val = 0.0
+            ite = 0.0
+
             try:
                 import dowhy.gcm as gcm_module
                 import networkx as nx
 
-                # Build directed causal graph from the validated DAG
                 causal_graph = nx.DiGraph()
-                for src_i, src_name in enumerate(scm.nodes):
-                    for dst_i, dst_name in enumerate(scm.nodes):
-                        if scm.dag_adj[src_i, dst_i]:
-                            causal_graph.add_edge(src_name, dst_name)
-                for node in scm.nodes:
+                for si, src in enumerate(nodes):
+                    for di, dst in enumerate(nodes):
+                        if dag_adj[si, di]:
+                            causal_graph.add_edge(src, dst)
+                for node in nodes:
                     if node not in causal_graph.nodes:
                         causal_graph.add_node(node)
 
-                # pywhyllm guidance: use equation types to assign mechanisms
-                # - Nodes with parents get AdditiveNoiseModel (invertible, required for CF)
-                # - Root (exogenous) nodes get EmpiricalDistribution
                 gcm_model = gcm_module.InvertibleStructuralCausalModel(causal_graph)
                 gcm_module.auto.assign_mechanisms(gcm_model, df)
-
-                # Override mechanism types based on pywhyllm's equation suggestions
-                # if available, to improve SCM quality
-                if pywhyllm_report and pywhyllm_report.get("available"):
-                    for node in scm.nodes:
-                        eq_data = (payload.get("equations") or {}).get(node, {})
-                        if eq_data.get("equation_type") == "exogenous":
-                            if node in gcm_model.graph.nodes:
-                                gcm_model.set_causal_mechanism(
-                                    node,
-                                    gcm_module.EmpiricalDistribution()
-                                )
-
                 gcm_module.fit(gcm_model, df)
 
-                # The observed data at time t
                 observed_data = df.iloc[[t]]
-
-                # Run counterfactual: fix treatment, abduct noise, predict
-                cf_val_fixed = abs_cf_value  # capture in closure
+                cf_val_fixed = abs_cf_value
                 cf_samples = gcm_module.counterfactual_samples(
                     gcm_model,
                     {treatment: lambda x, v=cf_val_fixed: np.full(x.shape, v)},
@@ -913,42 +834,32 @@ def run_inference(
                 )
 
                 factual_outcome = float(observed_data[target_node].iloc[0])
-                cf_outcome = float(cf_samples[target_node].iloc[0])
-                ite = cf_outcome - factual_outcome
+                cf_outcome_val = float(cf_samples[target_node].iloc[0])
+                ite = cf_outcome_val - factual_outcome
                 gcm_used = True
 
             except Exception as gcm_exc:
                 logger.warning("DoWhy GCM counterfactual failed (%s), using SCM abduction", gcm_exc)
-                gcm_used = False
 
             if not gcm_used:
-                # ── Fallback: SCM abduction engine (always available) ─────────
-                scm_cf_result = engine.counterfactual(
-                    observed_t=t,
+                factual_outcome, cf_outcome_val, ite = _abduct_and_predict(
+                    nodes=nodes,
+                    dag_adj=dag_adj,
+                    equations=equations_raw,
+                    data_level=data_level,
+                    topological_order=topo_order,
                     treatment=treatment,
                     cf_value=abs_cf_value,
                     target=target_node,
-                )
-                factual_outcome = float(scm_cf_result.get("factual_outcome", 0.0))
-                cf_outcome = float(scm_cf_result.get("counterfactual_outcome", 0.0))
-                ite = float(scm_cf_result.get("ite", 0.0))
-
-            # Shapley contributions — always from SCM engine (numerically exact)
-            shapley: Dict[str, float] = {}
-            try:
-                shapley_result = engine.counterfactual(
                     observed_t=t,
-                    treatment=treatment,
-                    cf_value=abs_cf_value,
-                    target=target_node,
                 )
-                shapley = shapley_result.get("shapley_contributions", {treatment: ite})
-            except Exception:
-                shapley = {treatment: ite}
 
-            # SE from residual std of the target equation
-            target_eq = scm.equations.get(target_node)
-            se = float(target_eq.residual_std) if target_eq else abs(ite) * 0.15
+            # Shapley: single-treatment — just use the ITE directly
+            shapley = {treatment: ite}
+
+            # SE from residual_std of the target equation (from backend payload)
+            target_eq_data = equations_raw.get(target_node, {})
+            se = float(target_eq_data.get("residual_std", abs(ite) * 0.15))
             ci_lower = ite - 1.96 * se
             ci_upper = ite + 1.96 * se
 
@@ -956,7 +867,7 @@ def run_inference(
                 "ate": ite,
                 "ite": ite,
                 "factual_outcome": factual_outcome,
-                "counterfactual_outcome": cf_outcome,
+                "counterfactual_outcome": cf_outcome_val,
                 "ci_lower": ci_lower,
                 "ci_upper": ci_upper,
                 "probability": min(1.0, abs(ite) / (abs(ite) + se + 1e-9)),
@@ -964,7 +875,7 @@ def run_inference(
                 "counterfactual_value": abs_cf_value,
                 "value_type": value_type,
                 "observed_t": t,
-                "shapley_contributions": _safe_json(shapley),
+                "shapley_contributions": shapley,
                 "ripple_effects": [],
             }
 
@@ -985,36 +896,6 @@ def run_inference(
         return {"status": "error", "detail": str(exc)}
 
 
-# ── API 4: Hierarchical sector causal ────────────────────────────────────
-
-@server.api(
-    name="run_hierarchy",
-    description="Run CrossLevelMPNN hierarchical sector causal graph",
-    concurrency_limit=1,
-)
-def run_hierarchy(
-    symbols: Optional[List[str]] = None,
-) -> Dict[str, Any]:
-    """
-    Build the micro + macro causal hierarchy graph.
-
-    Parameters
-    ----------
-    symbols : Optional override for the symbol list (defaults to top-4 NIFTY tickers)
-
-    Returns
-    -------
-    JSON with micro_graph, macro_graph, sector_embeddings
-    """
-    try:
-        from causal_hierarchy.test_hierarchical_causal_flow import run_hierarchical_flow
-        result = run_hierarchical_flow(symbols=symbols)
-        return _safe_json({"status": "ok", **result})
-    except Exception as exc:
-        logger.exception("run_hierarchy failed")
-        return {"status": "error", "detail": str(exc)}
-
-
 # ─────────────────────────────────────────────────────────────────────────────
 # Entry point
 # ─────────────────────────────────────────────────────────────────────────────
@@ -1023,7 +904,7 @@ if __name__ == "__main__":
     port = int(os.environ.get("GRADIO_SERVER_PORT", os.environ.get("PORT", "7860")))
     host = os.environ.get("GRADIO_SERVER_NAME", "0.0.0.0")
 
-    logger.info(f"Starting CUTS+ Causal Terminal on {host}:{port}")
+    logger.info(f"Starting Iroha Causal Terminal on {host}:{port}")
     logger.info(f"  → Frontend : http://localhost:{port}/")
     logger.info(f"  → API docs : http://localhost:{port}/docs")
 

singular_ticker_causal/.env

@@ -1,3 +0,0 @@
-FIREWORKS_API_KEY=fw_KvkeXQmo8LctbP6xx8A5uA
-NVIDIA_API_KEY=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
-LLM_PROVIDER=nvidia

singular_ticker_causal/algorithms/CUTS_PLUS/cuts_plus.py

@@ -1,834 +0,0 @@
-import os
-from os.path import join as opj
-from os.path import dirname as opd
-
-import tqdm
-import numpy as np
-import argparse
-from omegaconf import OmegaConf
-from copy import deepcopy
-from einops import rearrange
-import torch
-from torch import nn
-import torch.nn.functional as F
-
-from .utils.gumbel_softmax import gumbel_softmax
-from .utils.misc import calc_and_log_metrics, log_time_series, plot_causal_matrix
-from .utils.opt_type import MultiCADopt
-from .utils.logger import MyLogger
-from .model.cuts_plus_net import CUTS_Plus_Net
-from causal_hierarchy.grouping import build_group_matrix
-from causal.cuts_plus.edge_controller import DualEdgeTemperatureController
-
-
-def accounting_prior_loss(
-    G_sequence: torch.Tensor,
-    G_p: torch.Tensor,
-    lambda_s: float,
-    lambda_d: float,
-) -> torch.Tensor:
-    """Compute the combined sparseness + domain-fit prior loss for a lag-indexed graph sequence.
-
-    Implements the plan §5.2 formula::
-
-        p(G) ∝ exp( -λ_s ‖G_{1:L}‖_F² - λ_d ‖G_{1:L} - G^p_{1:L}‖_F² )
-
-    Parameters
-    ----------
-    G_sequence : Tensor (L, D, D) or (D, D)
-        Soft adjacency sequence. When 2-D it is treated as a single-lag graph
-        and the sparseness / domain-fit are applied directly.
-    G_p : Tensor (D, D)
-        Static binary accounting prior mask. Broadcast across lags.
-    lambda_s : float
-        Sparseness regularisation coefficient.
-    lambda_d : float
-        Domain-fit (DuPont structural) regularisation coefficient.
-
-    Returns
-    -------
-    Tensor scalar
-        Combined prior loss term to be added to the CUTS+ objective.
-    """
-    if G_sequence.ndim == 2:
-        G_sequence = G_sequence.unsqueeze(0)  # treat as (1, D, D)
-
-    # Broadcast static prior to (L, D, D)
-    G_p_expanded = G_p.unsqueeze(0).expand_as(G_sequence)
-
-    sparseness = lambda_s * torch.norm(G_sequence, p="fro") ** 2
-    domain_fit = lambda_d * torch.norm(G_sequence - G_p_expanded, p="fro") ** 2
-    return sparseness + domain_fit
-
-
-
-def plot_matrix(name, mat, log, log_step, vmin=None, vmax=None):
-    if len(mat.shape) == 3:
-        mat = np.max(mat, axis=-1)
-    n, m = mat.shape
-
-    # Show Discovered Graph (Probability)
-    sub_cg = plot_causal_matrix(
-        mat,
-        figsize=[1.5*n, 1*n],
-        show_text=False,
-        vmin=vmin, vmax=vmax)
-    log.log_figures(sub_cg, name=name, iters=log_step)
-
-
-def generate_indices(input_step, pred_step, t_length, block_size=None):
-    if block_size is None:
-        block_size = t_length
-        
-    offsets_in_block = np.arange(input_step, block_size-pred_step+1)
-    assert t_length % block_size == 0, "t_length % block_size != 0"
-    random_t_list = []
-    for block_start in range(0, t_length, block_size):
-        random_t_list += (offsets_in_block + block_start).tolist()
-    
-    np.random.shuffle(random_t_list)
-    return random_t_list
-
-
-
-def batch_generater(data, observ_mask, bs, n_nodes, input_step, pred_step, block_size=None):
-    t, n, d = data.shape
-    first_sample_t = input_step
-    random_t_list = generate_indices(input_step, pred_step, t_length=t, block_size=block_size)
-
-    for batch_i in range(len(random_t_list) // bs):
-        x = torch.zeros([bs, n_nodes, input_step, d]).to(data.device)
-        y = torch.zeros([bs, n_nodes, pred_step, d]).to(data.device)
-        t = torch.zeros([bs]).to(data.device).long()
-        mask_x = torch.zeros([bs, n_nodes, input_step, d]).to(data.device)
-        mask_y = torch.zeros([bs, n_nodes, pred_step, d]).to(data.device)
-        for data_i in range(bs):
-            data_t = random_t_list.pop()
-            x[data_i, :, :, :] = rearrange(data[data_t-input_step : data_t, :], "t n d -> n t d")
-            y[data_i, :, :, :] = rearrange(data[data_t : data_t+pred_step, :], "t n d -> n t d")
-            t[data_i] = data_t
-            mask_x[data_i, :, :, :] = rearrange(observ_mask[data_t-input_step : data_t, :], "t n d -> n t d")
-            mask_y[data_i, :, :, :] = rearrange(observ_mask[data_t:data_t+pred_step, :], "t n d -> n t d")
-
-        yield x, y, t, mask_x, mask_y
-        
-        
-
-
-
-class MultiCAD(object):
-    def __init__(
-        self,
-        args: MultiCADopt.MultiCADargs,
-        log,
-        device="cuda",
-        text_data=None,
-        text_mask=None,
-        G_prior=None,
-        denoised_news=None,
-        denoised_mask=None,
-    ):
-        self.log: MyLogger = log
-        self.args = args
-        self.device = device
-
-        self.text_data = text_data.to(device) if text_data is not None else None
-        self.text_mask = text_mask.to(device) if text_mask is not None else None
-        self.denoised_news = denoised_news.to(device) if denoised_news is not None else None
-        self.denoised_mask = denoised_mask.to(device) if denoised_mask is not None else None
-        self.projector = None
-        self.denoised_projector = None
-
-        self.lambda_d = getattr(args, 'lambda_d', 1e-2)
-        if G_prior is not None:
-            self.G_prior = torch.from_numpy(G_prior).float().to(device)
-        else:
-            self.G_prior = torch.zeros(args.n_nodes, args.n_nodes).to(device)
-        hard_edge_mask = getattr(args, "hard_edge_mask", None)
-        if hard_edge_mask is None and bool(getattr(args, "use_hard_prior_edges", False)):
-            hard_edge_mask = G_prior
-        if hard_edge_mask is not None:
-            self.hard_edge_mask_graph = torch.as_tensor(hard_edge_mask, dtype=torch.float32, device=device)
-        else:
-            self.hard_edge_mask_graph = torch.zeros(args.n_nodes, args.n_nodes, device=device)
-        soft_edge_mask = getattr(args, "soft_edge_mask", None)
-        if soft_edge_mask is not None:
-            self.soft_edge_mask_graph = torch.as_tensor(soft_edge_mask, dtype=torch.float32, device=device)
-        else:
-            self.soft_edge_mask_graph = 1.0 - self.hard_edge_mask_graph
-        self.hard_edge_tau = float(getattr(args, "hard_edge_tau", 0.02))
-        self.hard_edge_trainable = bool(getattr(args, "hard_edge_trainable", False))
-
-        # No embedding projector is required when text features are already
-        # represented as low-dimensional sparse event tensors.
-        self.projector = None
-        self.denoised_projector = None
-
-        self.fitting_model = CUTS_Plus_Net(self.args.n_nodes, in_ch=self.args.data_dim,
-                                           n_layers=self.args.data_pred.gru_layers,
-                                           hidden_ch=self.args.data_pred.mlp_hid,
-                                           shared_weights_decoder=self.args.data_pred.shared_weights_decoder,
-                                           concat_h=self.args.data_pred.concat_h,
-                                           ).to(self.device)
-
-        self.data_pred_loss = nn.MSELoss()
-        
-        params = list(self.fitting_model.parameters())
-            
-        self.data_pred_optimizer = torch.optim.Adam(
-            params,
-            lr=self.args.data_pred.lr_data_start,
-            weight_decay=self.args.data_pred.weight_decay
-        )
-        
-        if "every" in self.args.fill_policy:
-            lr_schedule_length = int(self.args.fill_policy.split("_")[-1])
-        else:
-            lr_schedule_length = self.args.total_epoch
-            
-        gamma = (self.args.data_pred.lr_data_end / self.args.data_pred.lr_data_start) ** (1 / lr_schedule_length)
-        self.data_pred_scheduler = torch.optim.lr_scheduler.StepLR(
-            self.data_pred_optimizer, step_size=1, gamma=gamma)
-        
-        self.n_groups = self.args.n_groups
-        print("n_groups: ", self.n_groups)
-        if self.args.group_policy == "None":
-            self.args.group_policy = None
-        self.fixed_group_spec = self._resolve_fixed_group_spec()
-        if self.fixed_group_spec is not None:
-            self.n_groups = self.fixed_group_spec.n_groups
-
-        end_tau, start_tau = self.args.graph_discov.end_tau, self.args.graph_discov.start_tau
-        self.gumbel_tau_gamma = (end_tau / start_tau) ** (1 / self.args.total_epoch)
-        self.gumbel_tau = start_tau
-        self.start_tau = start_tau
-        self.current_epoch = 0
-        self.edge_controller = DualEdgeTemperatureController(
-            G_prior=self.G_prior.detach().cpu().numpy(),
-            tau_start=float(start_tau),
-            tau_end=float(end_tau),
-            tau_hard=self.hard_edge_tau,
-            total_epochs=int(self.args.total_epoch),
-        )
-        
-        end_lmd, start_lmd = self.args.graph_discov.lambda_s_end, self.args.graph_discov.lambda_s_start
-        self.lambda_gamma = (end_lmd / start_lmd) ** (1 / self.args.total_epoch)
-        self.lambda_s = start_lmd
-    
-    def set_graph_optimizer(self, epoch=None):
-        if epoch == None:
-            epoch = 0
-        
-        gamma = (self.args.graph_discov.lr_graph_end / self.args.graph_discov.lr_graph_start) ** (1 / self.args.total_epoch)
-        self.graph_optimizer = torch.optim.Adam([self.GT], lr=self.args.graph_discov.lr_graph_start * gamma ** epoch)
-        self.graph_scheduler = torch.optim.lr_scheduler.StepLR(self.graph_optimizer, step_size=1, gamma=gamma)
-
-    def _resolve_fixed_group_spec(self):
-        policy = getattr(self.args, "group_policy", None)
-        if policy not in {"deterministic", "deterministic_sector", "deterministic_geography"}:
-            return None
-        assignments = getattr(self.args, "group_assignments", None)
-        if assignments is None and policy == "deterministic_sector":
-            ticker_list = getattr(self.args, "ticker_list", None)
-            sector_map = getattr(self.args, "sector_map", None)
-            if ticker_list is not None and sector_map is not None:
-                assignments = [sector_map[ticker] for ticker in ticker_list]
-        if assignments is None:
-            raise ValueError("Deterministic grouping requires opt.group_assignments.")
-        labels = getattr(self.args, "group_labels", None)
-        return build_group_matrix(assignments, labels=labels)
-
-    def _has_fixed_grouping(self) -> bool:
-        return self.fixed_group_spec is not None
-
-    def _init_random_gt(self, n_groups: int) -> torch.Tensor:
-        return torch.ones((n_groups, self.args.n_nodes)) * 0.5 + torch.randn(n_groups, self.args.n_nodes) * 0.01
-
-    def _build_prior_seed_logits(self, n_groups: int) -> torch.Tensor:
-        gt_init = torch.full((n_groups, self.args.n_nodes), -2.0)
-        if n_groups == self.args.n_nodes:
-            grouped_prior = self.G_prior
-        elif self._has_fixed_grouping():
-            grouped_prior = torch.zeros(n_groups, self.args.n_nodes, device=self.device)
-            for group_idx in range(n_groups):
-                members = [
-                    idx for idx, assigned_group in enumerate(self.fixed_group_spec.assignments)
-                    if assigned_group == group_idx
-                ]
-                if members:
-                    grouped_prior[group_idx] = torch.max(self.G_prior[members], dim=0).values
-        else:
-            return self._init_random_gt(n_groups)
-
-        gt_init[grouped_prior > 0.5] = 2.0
-        gt_init += torch.randn_like(gt_init) * 0.05
-        return gt_init
-
-    def _build_graph_prob(self) -> torch.Tensor:
-        return torch.einsum("nm,ml->nl", self.G, torch.sigmoid(self.GT))
-
-    def _compose_graph_parts(self) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
-        base_graph = self._build_graph_prob()
-        soft_graph = base_graph * self.soft_edge_mask_graph
-        if self.hard_edge_trainable:
-            hard_graph = base_graph * self.hard_edge_mask_graph
-        else:
-            hard_graph = self.hard_edge_mask_graph
-        effective_graph = torch.clamp(soft_graph + hard_graph, 0.0, 1.0)
-        return soft_graph, hard_graph, effective_graph
-
-    def _gumbel_sigmoid_sample(self, graph: torch.Tensor, batch_size: int, tau: float) -> torch.Tensor:
-        prob = graph[None, :, :, None].expand(batch_size, -1, -1, -1)
-        logits = torch.concat([prob, (1 - prob)], axis=-1)
-        return gumbel_softmax(logits, tau=tau, hard=True)[:, :, :, 0]
-
-    def _sample_graph_with_controller(self, graph: torch.Tensor) -> torch.Tensor:
-        graph = torch.clamp(torch.nan_to_num(graph, nan=0.5), 1e-6, 1.0 - 1e-6)
-        logits = torch.logit(graph)
-        sampled = self.edge_controller.gumbel_sample(logits, epoch=self.current_epoch, hard=True)
-        if not self.hard_edge_trainable and torch.any(self.hard_edge_mask_graph > 0):
-            sampled = torch.clamp(sampled * self.soft_edge_mask_graph + self.hard_edge_mask_graph, 0.0, 1.0)
-        return sampled[None].expand(self.args.batch_size, -1, -1)
-
-    def _append_context(self, x, y, mask_x, mask_y, t, inp_step: int, pred_step: int):
-        t_vals = t.cpu().tolist()
-        if self.text_data is not None:
-            tx = torch.stack([rearrange(self.text_data[ti - inp_step:ti], "t n d -> n t d") for ti in t_vals])
-            ty = torch.stack([rearrange(self.text_data[ti:ti + pred_step], "t n d -> n t d") for ti in t_vals])
-            d_text = self.text_data.shape[-1]
-            if self.text_mask is not None:
-                if self.text_mask.shape[-1] == 1:
-                    tmx = torch.stack([rearrange(self.text_mask[ti - inp_step:ti].expand(-1, -1, d_text), "t n d -> n t d") for ti in t_vals])
-                    tmy = torch.stack([rearrange(self.text_mask[ti:ti + pred_step].expand(-1, -1, d_text), "t n d -> n t d") for ti in t_vals])
-                else:
-                    tmx = torch.stack([rearrange(self.text_mask[ti - inp_step:ti], "t n d -> n t d") for ti in t_vals])
-                    tmy = torch.stack([rearrange(self.text_mask[ti:ti + pred_step], "t n d -> n t d") for ti in t_vals])
-            else:
-                tmx = torch.ones_like(tx)
-                tmy = torch.ones_like(ty)
-            x = torch.cat([x, tx], dim=-1)
-            y = torch.cat([y, ty], dim=-1)
-            mask_x = torch.cat([mask_x, tmx], dim=-1)
-            mask_y = torch.cat([mask_y, tmy], dim=-1)
-
-        if self.denoised_news is not None:
-            dx = torch.stack([rearrange(self.denoised_news[ti - inp_step:ti], "t n d -> n t d") for ti in t_vals])
-            dy = torch.stack([rearrange(self.denoised_news[ti:ti + pred_step], "t n d -> n t d") for ti in t_vals])
-            d_denoised = self.denoised_news.shape[-1]
-            if self.denoised_mask is not None:
-                dmask_src = self.denoised_mask
-                if dmask_src.shape[-1] == 1:
-                    dmx = torch.stack([rearrange(dmask_src[ti - inp_step:ti].expand(-1, -1, d_denoised), "t n d -> n t d") for ti in t_vals])
-                    dmy = torch.stack([rearrange(dmask_src[ti:ti + pred_step].expand(-1, -1, d_denoised), "t n d -> n t d") for ti in t_vals])
-                else:
-                    dmx = torch.stack([rearrange(dmask_src[ti - inp_step:ti], "t n d -> n t d") for ti in t_vals])
-                    dmy = torch.stack([rearrange(dmask_src[ti:ti + pred_step], "t n d -> n t d") for ti in t_vals])
-            else:
-                dmx = torch.ones_like(dx)
-                dmy = torch.ones_like(dy)
-            x = torch.cat([x, dx], dim=-1)
-            y = torch.cat([y, dy], dim=-1)
-            mask_x = torch.cat([mask_x, dmx], dim=-1)
-            mask_y = torch.cat([mask_y, dmy], dim=-1)
-            
-        return x, y, mask_x, mask_y
-
-    def _sample_graph_for_prediction(self, graph: torch.Tensor, batch_size: int) -> torch.Tensor:
-        sample_matrix = graph[None].expand(batch_size, -1, -1)
-        sample_matrix = torch.clamp(sample_matrix, 0.0, 1.0)
-        sample_matrix = torch.nan_to_num(sample_matrix, nan=0.5)
-        return torch.bernoulli(sample_matrix).float()
-
-    def freeze_accounting_edges(self) -> None:
-        """Stop gradient flow through GT logits that correspond to known prior edges.
-
-        Called after epoch 10 (per plan §5.3). For positions where
-        ``G_prior >= 0.5`` the logit is detached so that subsequent
-        ``graph_optimizer.step()`` calls do not update those weights.
-
-        The method operates in-place by replacing ``self.GT`` with a new
-        ``nn.Parameter`` whose values at prior positions are detached
-        constants while non-prior positions retain full gradient.
-        """
-        if not hasattr(self, "GT"):
-            return
-        with torch.no_grad():
-            gt_data = self.GT.data.clone()
-
-        # Build a mask aligned to GT shape (n_groups × n_nodes)
-        n_groups, n_nodes = self.GT.shape
-        if self.G_prior.shape == (n_nodes, n_nodes) and n_groups == n_nodes:
-            # 1-to-1 mapping: prior mask applies directly
-            prior_mask = (self.G_prior >= 0.5)
-        elif self._has_fixed_grouping():
-            # Map node-level prior to group-level: group is frozen if any member
-            # has a prior edge from that group
-            prior_mask = torch.zeros(n_groups, n_nodes, dtype=torch.bool, device=self.device)
-            for group_idx in range(n_groups):
-                members = [
-                    idx for idx, g in enumerate(self.fixed_group_spec.assignments)
-                    if g == group_idx
-                ]
-                if members:
-                    row_prior = self.G_prior[members].max(dim=0).values
-                    prior_mask[group_idx] = row_prior >= 0.5
-        else:
-            return  # cannot determine mapping — skip freeze
-
-        frozen_vals = gt_data[prior_mask].detach()
-        new_gt = nn.Parameter(gt_data)
-        # Freeze prior positions by zeroing their gradient contribution
-        # via a register_hook that zeroes the grad at those positions.
-        def _freeze_hook(grad: torch.Tensor) -> torch.Tensor:
-            grad = grad.clone()
-            grad[prior_mask] = 0.0
-            return grad
-
-        new_gt.register_hook(_freeze_hook)
-        self.GT = new_gt
-        self.set_graph_optimizer()   # refresh optimizer to point at new GT
-        n_frozen = int(prior_mask.sum().item())
-        print(f"[freeze_accounting_edges] Froze {n_frozen} / {n_groups * n_nodes} GT logit positions.")
-
-
-    def ticker_price_pred(self, x, y, mask_x, mask_y):
-        bs, n, t, d = x.shape
-        self.fitting_model.train()
-        self.data_pred_optimizer.zero_grad()
-
-        _, _, effective_graph = self._compose_graph_parts()
-        graph_sampled = self._sample_graph_for_prediction(effective_graph, self.args.batch_size)
-            
-        y_pred = self.fitting_model(x, mask_x, graph_sampled)
-
-        # print(y_pred.shape, y.shape, observ_mask.shape)
-        loss = self.data_pred_loss(y * mask_y, y_pred * mask_y) / (torch.mean(mask_y) + 1e-8)
-        loss.backward()
-        self.data_pred_optimizer.step()
-        return y_pred, loss
-
-    def graph_discov(self, x, y, mask_x, mask_y):
-        gn, n = self.GT.shape
-        self.graph_optimizer.zero_grad()
-        soft_graph, hard_graph, effective_graph = self._compose_graph_parts()
-
-        graph_sampled = self._sample_graph_with_controller(effective_graph)
-
-        loss_sparsity = torch.linalg.norm(soft_graph.flatten(), ord=1) / (n * n)
-
-        y_pred = self.fitting_model(x, mask_x, graph_sampled)
-        
-        loss_data = self.data_pred_loss(y * mask_y, y_pred * mask_y) / (torch.mean(mask_y) + 1e-8)
-        
-        # DuPont structural prior penalty: push toward known edges, away from impossible ones
-        loss_dupont = torch.linalg.norm((effective_graph - self.G_prior).flatten(), ord=2) ** 2 / (n * n)
-
-        # L2 regularization on raw GT logits to prevent saturation to ±∞
-        loss_l2_gt = torch.linalg.norm(self.GT.flatten(), ord=2) ** 2 / (gn * n)
-        if torch.any(self.hard_edge_mask_graph > 0):
-            hard_edge_density = effective_graph[self.hard_edge_mask_graph > 0].mean()
-        else:
-            hard_edge_density = torch.tensor(0.0, device=self.device)
-        
-        loss = (loss_sparsity * self.lambda_s
-                + loss_data
-                + self.lambda_d * loss_dupont
-                + 1e-3 * loss_l2_gt)   # small L2 keeps logits from drifting to ±∞
-        loss.backward()
-        self.graph_optimizer.step()
-        
-        return loss, loss_sparsity, loss_data, loss_dupont, hard_edge_density
-
-
-
-    def train(self, data, observ_mask, original_data, true_cm=None):
-
-        original_data = torch.from_numpy(original_data).float().to(self.device)
-        observ_mask = torch.from_numpy(observ_mask).float().to(self.device)
-        data = torch.from_numpy(data).float().to(self.device)
-        
-        if self.args.supervision_policy == "masked":
-            print("Using masked supervision for data prediction...")
-        elif self.args.supervision_policy == "full":
-            print("Using full supervision for data prediction......")
-            observ_mask = torch.ones_like(observ_mask)
-        elif "masked_before" in self.args.supervision_policy:
-            print(f"Using masked supervision for data prediction ({self.args.supervision_policy:s})......")
-
-        price_pred_step = 0
-        graph_discov_step = 0
-        pbar = tqdm.tqdm(total=self.args.total_epoch)
-        data_interp = deepcopy(data)
-        original_mask = deepcopy(observ_mask)
-        auc = 0
-        _edges_frozen = False  # track whether freeze_accounting_edges() has run
-        for epoch_i in range(self.args.total_epoch):
-            self.current_epoch = epoch_i
-            # Phase 5 §5.3: freeze accounting edge logits after epoch 10
-            if epoch_i == 10 and not _edges_frozen:
-                self.freeze_accounting_edges()
-                _edges_frozen = True
-            if self._has_fixed_grouping():
-                if epoch_i == 0:
-                    self.G = torch.from_numpy(self.fixed_group_spec.matrix).float().to(self.device)
-                    self.GT = nn.Parameter(self._build_prior_seed_logits(self.fixed_group_spec.n_groups).to(self.device))
-                    self.set_graph_optimizer(epoch_i)
-            elif self.args.group_policy is not None:
-                group_mul = int(self.args.group_policy.split("_")[1])
-                group_every = int(self.args.group_policy.split("_")[3])
-                if epoch_i % group_every == 0 and self.n_groups < self.args.n_nodes:
-                    if epoch_i != 0:
-                        self.n_groups *= group_mul
-                    if self.n_groups > self.args.n_nodes:
-                        self.n_groups = self.args.n_nodes
-                    
-                    self.G = torch.zeros([self.args.n_nodes, self.n_groups]).to(self.device)
-
-                    for i in range(0, self.n_groups):
-                        for j in range(0, self.args.n_nodes // self.n_groups):
-                            self.G[i*(self.args.n_nodes // self.n_groups) + j, i] = 1
-                    for k in range(i*(self.args.n_nodes // self.n_groups) + j, self.args.n_nodes):
-                        self.G[k, i] = 1
-
-                    if hasattr(self, "GT"):
-                        GT_init = torch.sigmoid(self.GT).detach().cpu().repeat_interleave(group_mul, 0)[:self.n_groups, :]
-                        GT_init = 1 - (1 - GT_init)**(1 / group_mul)
-                    else:
-                        GT_init = self._init_random_gt(self.n_groups)
-
-                    self.GT = nn.Parameter(GT_init.to(self.device))
-                    
-                    self.set_graph_optimizer(epoch_i)
-                elif epoch_i == 0 and self.n_groups == self.args.n_nodes:
-                    self.G = torch.eye(self.args.n_nodes).to(self.device)
-                    # Add small noise to break symmetry — identical init → identical gradients
-                    GT_init = torch.ones((self.n_groups, self.args.n_nodes))*0.5 + torch.randn(self.n_groups, self.args.n_nodes)*0.01
-                    self.GT = nn.Parameter(GT_init.to(self.device))
-                    self.set_graph_optimizer(epoch_i)
-            else:
-                if epoch_i == 0:
-                    self.n_groups = self.args.n_nodes
-                    self.G = torch.eye(self.args.n_nodes).to(self.device)
-                    GT_init = self._build_prior_seed_logits(self.n_groups)
-                    self.GT = nn.Parameter(GT_init.to(self.device))
-                    self.set_graph_optimizer(epoch_i)
-                    
-            
-            if "every" in self.args.fill_policy:
-                update_every = int(self.args.fill_policy.split("_")[-1])
-                if (epoch_i+1) % update_every == 0:
-                    data = data_pred
-                    print("Update data!")
-                    # self.graph_optimizer.param_groups[0]['lr'] = self.args.graph_discov.lr_graph_start
-                    self.data_pred_optimizer.param_groups[0]['lr'] = self.args.data_pred.lr_data_start
-                    observ_mask = torch.ones_like(original_mask)
-            elif "rate" in self.args.fill_policy:
-                update_rate = float(self.args.fill_policy.split("_")[1])
-                update_after = int(self.args.fill_policy.split("_")[3])
-                if epoch_i+1 > update_after:
-                    if epoch_i == update_after:
-                        print("Data update started!")
-                    data = data * (1 - update_rate) + data_pred * update_rate
-            else:
-                # no data update
-                pass
-            
-            if "masked_before" in self.args.supervision_policy:
-                masked_before = int(self.args.supervision_policy.split("_")[2])
-                if epoch_i == masked_before:
-                    print("Using full supervision for data prediction......")
-                    observ_mask = torch.ones_like(original_mask)
-                    self.gumbel_tau = self.start_tau
-            
-            # Data Prediction
-            if hasattr(self.args, "data_pred"):
-                if hasattr(self.args, "block_size"):
-                    block_size = self.args.block_size
-                else:
-                    block_size = None
-                    
-                # Always use tech-only data for the batch generator.
-                # If a projector exists, text is projected fresh INSIDE each batch
-                # to avoid stale computation graphs after optimizer.step().
-                batch_gen = batch_generater(data, observ_mask,
-                                            bs=self.args.batch_size, 
-                                            n_nodes=self.args.n_nodes, 
-                                            input_step=self.args.input_step, 
-                                            pred_step=self.args.data_pred.pred_step,
-                                            block_size=block_size)
-                batch_gen = list(batch_gen)
-                
-                data_pred = data.clone().detach()  # tech-only predictions
-                data_pred_all = data.clone().detach()
-                d_tech = data.shape[-1]
-                inp_step = self.args.input_step
-                pred_step = self.args.data_pred.pred_step
-                
-                for x, y, t, mask_x, mask_y in batch_gen:
-                    price_pred_step += self.args.batch_size
-                    
-                    x, y, mask_x, mask_y = self._append_context(x, y, mask_x, mask_y, t, inp_step, pred_step)
-                    
-                    y_pred, loss = self.ticker_price_pred(x, y, mask_x, mask_y)
-                    # Map back only the tech portion
-                    data_pred[t] = (y_pred*(1-mask_y) + y*mask_y).clone().detach()[:,:,0,:d_tech]
-                    data_pred_all[t] = y_pred.clone().detach()[:,:,0,:d_tech]
-                    self.log.log_metrics({"ticker_price_pred/pred_loss": loss.item()}, price_pred_step)
-                    pbar.set_postfix_str(f"S1 loss={loss.item():.2f}, spr=IDLE, auc={auc:.4f}")
-
-                current_data_pred_lr = self.data_pred_optimizer.param_groups[0]['lr']
-                self.log.log_metrics({"graph_discov/lr": current_data_pred_lr}, price_pred_step)
-                self.data_pred_scheduler.step()
-                mse_pred_to_original = self.data_pred_loss(original_data, data_pred)
-                mse_interp_to_original = self.data_pred_loss(original_data, data_interp)
-                
-                self.log.log_metrics({"ticker_price_pred/mse_pred_to_original": mse_pred_to_original,
-                                      "ticker_price_pred/mse_interp_to_original": mse_interp_to_original}, price_pred_step)
-                
-            # Graph Discovery
-            if hasattr(self.args, "graph_discov"):
-                for x, y, t, mask_x, mask_y in batch_gen:
-                    graph_discov_step += self.args.batch_size
-                    if hasattr(self.args, "disable_graph") and self.args.disable_graph:
-                        pass
-                    else:
-                        x, y, mask_x, mask_y = self._append_context(x, y, mask_x, mask_y, t, inp_step, pred_step)
-                        
-                        loss, loss_sparsity, loss_data, loss_dupont, hard_edge_density = self.graph_discov(x, y, mask_x, mask_y)
-                        self.log.log_metrics({"graph_discov/sparsity_loss": loss_sparsity.item(),
-                                            "graph_discov/data_loss": loss_data.item(),
-                                            "graph_discov/prior_loss": loss_dupont.item(),
-                                            "graph_discov/hard_edge_density": hard_edge_density.item(),
-                                            "graph_discov/total_loss": loss.item()}, graph_discov_step)
-                        pbar.set_postfix_str(f"S2 loss={loss_data.item():.2f}, spr={loss_sparsity.item():.2f}, auc={auc:.4f}")
-                    
-                self.graph_scheduler.step()
-                # self.group_scheduler.step()
-                current_graph_disconv_lr = self.graph_optimizer.param_groups[0]['lr']
-                self.log.log_metrics({"graph_discov/lr": current_graph_disconv_lr}, graph_discov_step)
-                self.log.log_metrics({"graph_discov/tau": self.gumbel_tau}, graph_discov_step)
-                self.gumbel_tau *= self.gumbel_tau_gamma
-                self.lambda_s *= self.lambda_gamma
-
-            pbar.update(1)
-            
-            plot_roc = False
-            
-            G_prob = self.G.detach().cpu().numpy()
-            GT_prob = self.GT.detach().cpu().numpy()
-            # Apply sigmoid to match training forward pass (ticker_price_pred/graph_discov use sigmoid)
-            GT_prob_sigmoid = 1 / (1 + np.exp(-GT_prob))
-            Graph = np.einsum("nm,ml->nl", G_prob, GT_prob_sigmoid)
-            if np.any(self.hard_edge_mask_graph.detach().cpu().numpy() > 0):
-                if self.hard_edge_trainable:
-                    Graph = Graph * self.soft_edge_mask_graph.detach().cpu().numpy() + Graph * self.hard_edge_mask_graph.detach().cpu().numpy()
-                else:
-                    Graph = Graph * self.soft_edge_mask_graph.detach().cpu().numpy() + self.hard_edge_mask_graph.detach().cpu().numpy()
-                Graph = np.clip(Graph, 0.0, 1.0)
-            
-            
-            if (epoch_i+1) % self.args.show_graph_every == 0:
-                avg_mask = np.mean(observ_mask.cpu().numpy(), axis=(0,2))
-                if np.min(avg_mask) < 1:
-                    time_series_idx = int(np.argwhere(avg_mask < 1)[0, 0])
-                else:
-                    time_series_idx = 0
-                d_tech = original_data.shape[-1]
-                log_time_series(
-                    original_data.cpu()[-100:,time_series_idx], 
-                    data_interp.cpu()[-100:,time_series_idx], 
-                    data_pred_all.cpu()[-100:,time_series_idx, :d_tech], 
-                    log=self.log, log_step=price_pred_step
-                )
-
-                plot_matrix("G", G_prob, self.log, graph_discov_step, vmin=0, vmax=1)
-                plot_matrix("GT", GT_prob, self.log, graph_discov_step, vmin=0, vmax=1)
-                plot_matrix("Graph", Graph, self.log, graph_discov_step, vmin=0, vmax=1)
-                np.save(os.path.join(self.log.log_dir, 'Graph.npy'), Graph)
-                plot_roc = True
-            
-            # Show TPR FPR AUC ROC
-            if true_cm is not None:
-                Graph = rearrange(Graph, "n m -> m n")
-                auc = calc_and_log_metrics(Graph, true_cm, self.log, graph_discov_step, plot_roc=plot_roc)
-        
-        return Graph
-                
-           
-def prepross_data(data):
-    T, N, D = data.shape
-    new_data = np.zeros_like(data, dtype=float)
-    for i in range(N):
-        node = data[:,i,:]
-        std = np.std(node)
-        # Guard against zero-std (constant) columns to prevent NaN from 0/0
-        new_data[:,i,:] = (node - np.mean(node)) / (std + 1e-8)
-    # Replace any residual NaN/Inf (e.g. from upstream data issues) with 0
-    new_data = np.nan_to_num(new_data, nan=0.0, posinf=0.0, neginf=0.0)
-    return new_data
-
-
-def main(
-    data,
-    mask,
-    true_cm,
-    opt,
-    log,
-    device="cuda",
-    text_data=None,
-    text_mask=None,
-    G_prior=None,
-    denoised_news=None,
-    denoised_mask=None,
-):
-    if opt.n_nodes == "auto":
-        opt.n_nodes = data.shape[1]
-        
-    if len(data.shape) == 2:
-        data = data[:,:,None]
-        mask = mask[:,:,None]
-    data = prepross_data(data)
-    
-    if text_data is not None:
-        text_data_torch = torch.from_numpy(text_data).float()
-        text_mask_torch = torch.from_numpy(text_mask).float()
-    else:
-        text_data_torch = None
-        text_mask_torch = None
-    if denoised_news is not None:
-        denoised_news_torch = torch.from_numpy(denoised_news).float()
-        denoised_mask_torch = torch.from_numpy(denoised_mask).float() if denoised_mask is not None else None
-    else:
-        denoised_news_torch = None
-        denoised_mask_torch = None
-        
-    effective_dim = opt.data_dim
-    projector_output_dim = getattr(opt, 'projector_output_dim', 16)
-    if text_data is not None:
-        effective_dim += projector_output_dim
-    if denoised_news is not None:
-        effective_dim += projector_output_dim
-    opt.data_dim = effective_dim
-    
-    multicad = MultiCAD(
-        opt,
-        log,
-        device=device,
-        text_data=text_data_torch,
-        text_mask=text_mask_torch,
-        G_prior=G_prior,
-        denoised_news=denoised_news_torch,
-        denoised_mask=denoised_mask_torch,
-    )
-    max_refuter_retries = 3
-    refuter_retries = 0
-    falsified = True
-    
-    while falsified and refuter_retries <= max_refuter_retries:
-        Graph = multicad.train(data, mask, data, true_cm)
-        
-        # Run Refuter Validation
-        try:
-            from singular_ticker_causal.causal_inference.causal_model import StructuralCausalModel
-            from singular_ticker_causal.causal_inference.query_engine import CausalQueryEngine
-            
-            data_cpu = data.cpu().numpy() if isinstance(data, torch.Tensor) else data
-            n_nodes = Graph.shape[0]
-            nodes = [f"N_{i}" for i in range(n_nodes)]
-            adj_mask = G_prior if G_prior is not None else np.zeros_like(Graph)
-            
-            scm = StructuralCausalModel(
-                nodes=nodes,
-                adj=Graph,
-                adjacency_mask=adj_mask,
-                data_tech=data_cpu,
-                lag=1
-            ).fit()
-            
-            pywhyllm_enabled = os.environ.get("PYWHYLLM_ENABLED", "").lower() in {"1", "true", "yes", "on"}
-            pywhyllm_max_edges = int(os.environ.get("PYWHYLLM_REFUTER_MAX_EDGES", "3"))
-            engine = CausalQueryEngine(
-                scm,
-                data_tech=data_cpu,
-                pywhyllm_enabled=pywhyllm_enabled,
-            )
-            
-            falsified = False
-            failed_edges = []
-            
-            edges_to_check = []
-            for i in range(n_nodes):
-                for j in range(n_nodes):
-                    if scm.dag_adj[i, j] and (adj_mask[i, j] == 0):
-                        edges_to_check.append((nodes[i], nodes[j], float(Graph[i, j])))
-            edges_to_check.sort(key=lambda edge: abs(edge[2]), reverse=True)
-            
-            validation_reports = []
-            for treatment, outcome, _score in edges_to_check[:pywhyllm_max_edges]:
-                if pywhyllm_enabled:
-                    res = engine.validate_with_pywhyllm_and_dowhy(
-                        treatment,
-                        outcome,
-                        max_edges=pywhyllm_max_edges,
-                    )
-                else:
-                    res = engine.validate_with_dowhy(treatment, outcome)
-                validation_reports.append({"edge": [treatment, outcome], "validation": res})
-                if res.get("falsified"):
-                    falsified = True
-                    failed_edges.append((treatment, outcome, res))
-                    break
-            
-            if falsified:
-                print(f"[Refuter] Graph failed refutation on edges: {failed_edges}. Applying penalty and retrying...")
-                multicad.args.total_epoch = 100
-                multicad.lambda_s *= 1.5
-                
-                if hasattr(multicad, "GT"):
-                    multicad.GT.data = multicad._build_prior_seed_logits(multicad.n_groups).to(device)
-                refuter_retries += 1
-                
-                import json
-                artifact = {
-                    "failed_edges": [
-                        {"treatment": edge[0], "outcome": edge[1], "validation": edge[2]}
-                        for edge in failed_edges
-                    ],
-                    "validation_reports": validation_reports,
-                    "lambda_s_new": float(multicad.lambda_s),
-                    "retry_epoch": 100,
-                    "adj": Graph.tolist()
-                }
-                with open(os.path.join(log.log_dir, f"refuter_failed_artifact_retry_{refuter_retries}.json"), "w") as f:
-                    json.dump(artifact, f)
-            else:
-                print("[Refuter] Graph passed refutation or no testable edges.")
-                
-        except Exception as e:
-            print(f"[Refuter] Validation error: {e}. Bypassing refuter.")
-            falsified = False
-
-    return Graph
-
-
-if __name__ == "__main__":
-    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
-
-    parser = argparse.ArgumentParser(description='Batch Compress')
-    parser.add_argument('-opt', type=str, default=opj(opd(__file__),
-                        'opt/multi_cad_lorenz.yaml'), help='yaml file path')
-    parser.add_argument('-g', help='availabel gpu list', default='2', type=str)
-    parser.add_argument('-debug', action='store_true')
-    parser.add_argument('-log', action='store_true')
-    args = parser.parse_args()
-
-    if args.g == "mps":
-        os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
-        device = "mps"
-    elif args.g == "cpu":
-        device = "cpu"
-    else:
-        os.environ["CUDA_VISIBLE_DEVICES"] = args.g
-        device = "cuda"
-        
-    main(OmegaConf.load(args.opt), device=device)

singular_ticker_causal/algorithms/CUTS_PLUS/data/generate_data_mod.py

@@ -1,430 +0,0 @@
-import numpy as np
-from collections import defaultdict 
-
-def check_stationarity(links):
-    """Returns stationarity according to a unit root test
-
-    Assuming a Gaussian Vector autoregressive process
-
-    Three conditions are necessary for stationarity of the VAR(p) model:
-    - Absence of mean shifts;
-    - The noise vectors are identically distributed;
-    - Stability condition on Phi(t-1) coupling matrix (stabmat) of VAR(1)-version  of VAR(p).
-    """
-
-
-    N = len(links)
-    # Check parameters
-    max_lag = 0
-
-    for j in range(N):
-        for link_props in links[j]:
-            var, lag = link_props[0]
-            # coeff = link_props[1]
-            # coupling = link_props[2]
-
-            max_lag = max(max_lag, abs(lag))
-
-    graph = np.zeros((N,N,max_lag))
-    couplings = []
-
-    for j in range(N):
-        for link_props in links[j]:
-            var, lag = link_props[0]
-            coeff    = link_props[1]
-            coupling = link_props[2]
-            if abs(lag) > 0:
-                graph[j,var,abs(lag)-1] = coeff
-            couplings.append(coupling)
-
-    stabmat = np.zeros((N*max_lag,N*max_lag))
-    index = 0
-
-    for i in range(0,N*max_lag,N):
-        stabmat[:N,i:i+N] = graph[:,:,index]
-        if index < max_lag-1:
-            stabmat[i+N:i+2*N,i:i+N] = np.identity(N)
-        index += 1
-
-    eig = np.linalg.eig(stabmat)[0]
-    # print "----> maxeig = ", np.abs(eig).max()
-    if np.all(np.abs(eig) < 1.):
-        stationary = True
-    else:
-        stationary = False
-
-    if len(eig) == 0:
-        return stationary, 0.
-    else:
-        return stationary, np.abs(eig).max()
-
-
-class Graph(): 
-    def __init__(self,vertices): 
-        self.graph = defaultdict(list) 
-        self.V = vertices 
-  
-    def addEdge(self,u,v): 
-        self.graph[u].append(v) 
-  
-    def isCyclicUtil(self, v, visited, recStack): 
-  
-        # Mark current node as visited and  
-        # adds to recursion stack 
-        visited[v] = True
-        recStack[v] = True
-  
-        # Recur for all neighbours 
-        # if any neighbour is visited and in  
-        # recStack then graph is cyclic 
-        for neighbour in self.graph[v]: 
-            if visited[neighbour] == False: 
-                if self.isCyclicUtil(neighbour, visited, recStack) == True: 
-                    return True
-            elif recStack[neighbour] == True: 
-                return True
-  
-        # The node needs to be poped from  
-        # recursion stack before function ends 
-        recStack[v] = False
-        return False
-  
-    # Returns true if graph is cyclic else false 
-    def isCyclic(self): 
-        visited = [False] * self.V 
-        recStack = [False] * self.V 
-        for node in range(self.V): 
-            if visited[node] == False: 
-                if self.isCyclicUtil(node,visited,recStack) == True: 
-                    return True
-        return False
-  
-    # A recursive function used by topologicalSort 
-    def topologicalSortUtil(self,v,visited,stack): 
-
-      # Mark the current node as visited. 
-      visited[v] = True
-
-      # Recur for all the vertices adjacent to this vertex 
-      for i in self.graph[v]: 
-          if visited[i] == False: 
-              self.topologicalSortUtil(i,visited,stack) 
-
-      # Push current vertex to stack which stores result 
-      stack.insert(0,v) 
-
-    # The function to do Topological Sort. It uses recursive  
-    # topologicalSortUtil() 
-    def topologicalSort(self): 
-        # Mark all the vertices as not visited 
-        visited = [False]*self.V 
-        stack =[] 
-
-        # Call the recursive helper function to store Topological 
-        # Sort starting from all vertices one by one 
-        for i in range(self.V): 
-          if visited[i] == False: 
-              self.topologicalSortUtil(i,visited,stack) 
-
-        return stack
-
-def generate_nonlinear_contemp_timeseries(links, T, noises=None, random_state=None):
-
-    if random_state is None:
-        random_state = np.random
-
-    # links must be {j:[((i, -tau), func), ...], ...}
-    # coeff is coefficient
-    # func is a function f(x) that becomes linear ~x in limit
-    # noises is a random_state.___ function
-    N = len(links.keys())
-    if noises is None:
-        noises = [random_state.randn for j in range(N)]
-
-    if N != max(links.keys())+1 or N != len(noises):
-        raise ValueError("links and noises keys must match N.")
-
-    # Check parameters
-    max_lag = 0
-    contemp = False
-    contemp_dag = Graph(N)
-    causal_order = list(range(N))
-    for j in range(N):
-        for link_props in links[j]:
-            var, lag = link_props[0]
-            coeff = link_props[1]
-            func = link_props[2]
-            if lag == 0: contemp = True
-            if var not in range(N):
-                raise ValueError("var must be in 0..{}.".format(N-1))
-            if 'float' not in str(type(coeff)):
-                raise ValueError("coeff must be float.")
-            if lag > 0 or type(lag) != int:
-                raise ValueError("lag must be non-positive int.")
-            max_lag = max(max_lag, abs(lag))
-
-            # Create contemp DAG
-            if var != j and lag == 0:
-                contemp_dag.addEdge(var, j)
-                # a, b = causal_order.index(var), causal_order.index(j)
-                # causal_order[b], causal_order[a] = causal_order[a], causal_order[b]
-
-    if contemp_dag.isCyclic() == 1: 
-        raise ValueError("Contemporaneous links must not contain cycle.")
-
-    causal_order = contemp_dag.topologicalSort() 
-
-    transient = int(.2*T)
-
-    X = np.zeros((T+transient, N), dtype='float32')
-    for j in range(N):
-        X[:, j] = noises[j](T+transient)
-
-    for t in range(max_lag, T+transient):
-        for j in causal_order:
-            for link_props in links[j]:
-                var, lag = link_props[0]
-                # if abs(lag) > 0:
-                coeff = link_props[1]
-                func = link_props[2]
-
-                X[t, j] += coeff * func(X[t + lag, var])
-
-    X = X[transient:]
-
-    if (check_stationarity(links)[0] == False or 
-        np.any(np.isnan(X)) or 
-        np.any(np.isinf(X)) or
-        # np.max(np.abs(X)) > 1.e4 or
-        np.any(np.abs(np.triu(np.corrcoef(X, rowvar=0), 1)) > 0.999)):
-        nonstationary = True
-    else:
-        nonstationary = False
-
-    return X, nonstationary
-
-
-def generate_random_contemp_model(N, L, 
-    coupling_coeffs, 
-    coupling_funcs, 
-    auto_coeffs, 
-    tau_max, 
-    contemp_fraction=0.,
-    # num_trials=1000,
-    random_state=None):
-
-    def lin(x): return x
-
-    if random_state is None:
-        random_state = np.random
-
-    # print links
-    a_len = len(auto_coeffs)
-    if type(coupling_coeffs) == float:
-        coupling_coeffs = [coupling_coeffs]
-    c_len  = len(coupling_coeffs)
-    func_len = len(coupling_funcs)
-
-    if tau_max == 0:
-        contemp_fraction = 1.
-
-    if contemp_fraction > 0.:
-        contemp = True
-        L_lagged = int((1.-contemp_fraction)*L)
-        L_contemp = L - L_lagged
-        if L==1: 
-            # Randomly assign a lagged or contemp link
-            L_lagged = random_state.randint(0,2)
-            L_contemp = int(L_lagged == False)
-
-    else:
-        contemp = False
-        L_lagged = L
-        L_contemp = 0
-
-
-    # for ir in range(num_trials):
-
-    # Random order
-    causal_order = list(random_state.permutation(N))
-
-    links = dict([(i, []) for i in range(N)])
-
-    # Generate auto-dependencies at lag 1
-    if tau_max > 0:
-        for i in causal_order:
-            a = auto_coeffs[random_state.randint(0, a_len)]
-
-            if a != 0.:
-                links[i].append(((int(i), -1), float(a), lin))
-
-    chosen_links = []
-    # Create contemporaneous DAG
-    contemp_links = []
-    for l in range(L_contemp):
-
-        cause = random_state.choice(causal_order[:-1])
-        effect = random_state.choice(causal_order)
-        while (causal_order.index(cause) >= causal_order.index(effect)
-             or (cause, effect) in chosen_links):
-            cause = random_state.choice(causal_order[:-1])
-            effect = random_state.choice(causal_order)
-        
-        contemp_links.append((cause, effect))
-        chosen_links.append((cause, effect))
-
-    # Create lagged links (can be cyclic)
-    lagged_links = []
-    for l in range(L_lagged):
-
-        cause = random_state.choice(causal_order)
-        effect = random_state.choice(causal_order)
-        while (cause, effect) in chosen_links or cause == effect:
-            cause = random_state.choice(causal_order)
-            effect = random_state.choice(causal_order)
-        
-        lagged_links.append((cause, effect))
-        chosen_links.append((cause, effect))
-
-    # print(chosen_links)
-    # print(contemp_links)
-    for (i, j) in chosen_links:
-
-        # Choose lag
-        if (i, j) in contemp_links:
-            tau = 0
-        else:
-            tau = int(random_state.randint(1, tau_max+1))
-        # print tau
-        # CHoose coupling
-        c = float(coupling_coeffs[random_state.randint(0, c_len)])
-        if c != 0:
-            func = coupling_funcs[random_state.randint(0, func_len)]
-
-            links[j].append(((int(i), -tau), c, func))
-
-    #     # Stationarity check assuming model with linear dependencies at least for large x
-    #     # if check_stationarity(links)[0]:
-    #         # return links
-    #     X, nonstat = generate_nonlinear_contemp_timeseries(links, 
-    #         T=10000, noises=None, random_state=None)
-    #     if nonstat == False:
-    #         return links
-    #     else:
-    #         print("Trial %d: Not a stationary model" % ir)
-
-
-    # print("No stationary models found in {} trials".format(num_trials))
-    return links
-
-def generate_logistic_maps(N, T, links, noise_lev):
-
-    # Check parameters
-    # contemp = False
-    max_lag = 0
-    for j in range(N):
-        for link_props in links[j]:
-            var, lag = link_props[0]
-            max_lag  = max(max_lag, abs(lag))
-
-    transient = int(.2*T)
-
-    # Chaotic logistic map parameter
-    r = 4.
-
-    X = np.random.rand(T+transient, N)
-
-    for t in range(max_lag, T+transient):
-        for j in range(N):
-            added_input = 0.
-            for link_props in links[j]:
-                var, lag = link_props[0]
-                if var != j and abs(lag) > 0:
-                    coeff        = link_props[1]
-                    coupling     = link_props[2]
-                    added_input += coeff*X[t - abs(lag), var]
-
-            X[t, j] = (X[t-1, j] * (r - r*X[t-1, j] - added_input + noise_lev*np.random.rand())) % 1
-                        #func(coeff, X[t+lag, var], coupling)
-
-    X = X[transient:]
-
-    if np.any(np.abs(X) == np.inf) or np.any(X == np.nan):
-        raise ValueError("Data divergent")
-    return X
-
-
-
-def weighted_avg_and_std(values, axis, weights):
-    """Returns the weighted average and standard deviation.
-
-    Parameters
-    ---------
-    values : array
-        Data array of shape (time, variables).
-
-    axis : int
-        Axis to average/std about
-
-    weights : array
-        Weight array of shape (time, variables).
-
-    Returns
-    -------
-    (average, std) : tuple of arrays
-        Tuple of weighted average and standard deviation along axis.
-    """
-
-    values[np.isnan(values)] = 0.
-    average  = np.ma.average(values, axis=axis, weights=weights)
-    variance = np.sum(weights * (values - np.expand_dims(average, axis)
-                                    ) ** 2, axis=axis) / weights.sum(axis=axis)
-
-    return (average, np.sqrt(variance))
-
-def time_bin_with_mask(data, time_bin_length, sample_selector=None):
-    """Returns time binned data where only about non-masked values is averaged.
-
-    Parameters
-    ----------
-    data : array
-        Data array of shape (time, variables).
-
-    time_bin_length : int
-        Length of time bin.
-
-    mask : bool array, optional (default: None)
-        Data mask where True labels masked samples.
-
-    Returns
-    -------
-    (bindata, T) : tuple of array and int
-        Tuple of time-binned data array and new length of array.
-    """
-
-    T = len(data)
-
-    time_bin_length = int(time_bin_length)
-
-    if sample_selector is None:
-        sample_selector = np.ones(data.shape)
-
-    if np.ndim(data) == 1.:
-        data.shape = (T, 1)
-        sample_selector.shape = (T, 1)
-
-    bindata = np.zeros(
-        (T // time_bin_length,) + data.shape[1:], dtype="float32")
-    for index, i in enumerate(range(0, T - time_bin_length + 1,
-                                    time_bin_length)):
-        # print weighted_avg_and_std(fulldata[i:i+time_bin_length], axis=0,
-        # weights=sample_selector[i:i+time_bin_length])[0]
-        bindata[index] = weighted_avg_and_std(data[i:i + time_bin_length],
-                                              axis=0,
-                                              weights=sample_selector[i:i +
-                                              time_bin_length])[0]
-
-    T, grid_size = bindata.shape
-
-    return (bindata.squeeze(), T)
-

singular_ticker_causal/algorithms/CUTS_PLUS/data/simu_data.py

@@ -1,313 +0,0 @@
-import os
-import sys
-from os.path import join as opj
-sys.path.append(opj(os.getcwd(), "../"))
-sys.path.append(os.getcwd())
-
-import csv
-import torch
-import scipy
-from .generate_data_mod import generate_random_contemp_model, generate_nonlinear_contemp_timeseries
-import numpy as np
-from scipy.integrate import odeint
-
-
-######################################
-# Function for loading input data 
-######################################
-def loadTrainingData(inputDataFilePath, device):
-
-    # Load and parse input data (create batch data)
-    inpData = torch.load(inputDataFilePath)
-    Xtrain = torch.zeros(inpData['TsData'].shape[1], inpData['TsData'].shape[0], requires_grad = False, device=device)
-    Xtrain1 = inpData['TsData'].t()
-    Xtrain.data[:,:] = Xtrain1.data[:,:]
-
-    return Xtrain
-
-#######################################################
-# Function for reading ground truth network from file 
-#######################################################
-def loadTrueNetwork(inputFilePath, networkSize):
-
-    with open(inputFilePath) as tsvin:
-        reader = csv.reader(tsvin, delimiter='\t')
-        numrows = 0    
-        for row in reader:
-            numrows = numrows + 1
-
-    network = np.zeros((numrows,2),dtype=np.int16)
-    with open(inputFilePath) as tsvin:
-        reader = csv.reader(tsvin, delimiter='\t')
-        rowcounter = 0
-        for row in reader:
-            network[rowcounter][0] = int(row[0][1:])
-            network[rowcounter][1] = int(row[1][1:])
-            rowcounter = rowcounter + 1 
-
-    Gtrue = np.zeros((networkSize,networkSize), dtype=np.int16)
-    for row in range(0,len(network),1):
-        Gtrue[network[row][1]-1][network[row][0]-1] = 1   
-    
-    return Gtrue
-
-
-def load_dream_data(dataset_id):
-    device = "cpu"
-
-    if(dataset_id == 0):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Ecoli1.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Ecoli1.tsv"
-    elif(dataset_id == 1):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Ecoli2.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Ecoli2.tsv"
-    elif(dataset_id == 2):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Yeast1.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Yeast1.tsv"
-    elif(dataset_id == 3):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Yeast2.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Yeast2.tsv"
-    elif(dataset_id == 4):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size100Yeast3.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize100-Yeast3.tsv"
-    elif(dataset_id == 5):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Ecoli1.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Ecoli1.tsv"
-    elif(dataset_id == 6):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Ecoli2.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Ecoli2.tsv"
-    elif(dataset_id == 7):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Yeast1.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Yeast1.tsv"
-    elif(dataset_id == 8):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Yeast2.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Yeast2.tsv"
-    elif(dataset_id == 9):
-        InputDataFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/Dream3TensorData/Size10Yeast3.pt"
-        RefNetworkFilePath = "causal_discov_sota/SRU_for_GCI/data/dream3/TrueGeneNetworks/InSilicoSize10-Yeast3.tsv"
-    else:
-        print("Error while loading gene training data")    
-
-    Xtrain = loadTrainingData(InputDataFilePath, device)
-    n = Xtrain.shape[0]
-    Gref = loadTrueNetwork(RefNetworkFilePath, n)
-    
-    Xtrain = Xtrain.numpy().T
-    # Gref = Gref.T
-    
-    return Xtrain, Gref
-
-
-
-def links_to_matrix(links):
-    N = len(links)
-    cm = np.zeros([N, N])
-    for i, effect_node in links.items():
-        for (j, _), _, _ in effect_node:
-            cm[i, j] += 1
-    return cm
-
-
-class noise_model:
-    def __init__(self, sigma=1, seed=0):
-        self.random_state = np.random.RandomState(seed)
-        self.sigma = sigma
-
-    def gaussian(self, T):
-        # Get zero-mean unit variance gaussian distribution
-        return self.sigma*self.random_state.randn(T)
-
-    def weibull(self, T):
-        # Get zero-mean sigma variance weibull distribution
-        a = 2
-        mean = scipy.special.gamma(1./a + 1)
-        variance = scipy.special.gamma(
-            2./a + 1) - scipy.special.gamma(1./a + 1)**2
-        return self.sigma*(self.random_state.weibull(a=a, size=T) - mean)/np.sqrt(variance)
-
-    def uniform(self, T):
-        # Get zero-mean sigma variance uniform distribution
-        mean = 0.5
-        variance = 1./12.
-        return self.sigma*(self.random_state.uniform(size=T) - mean)/np.sqrt(variance)
-
-
-def lin_f(x): return x
-def f2(x): return (x + 5. * x**2 * np.exp(-x**2 / 20.))
-
-
-def simulate_random_var(seed, T, N, L, coef=[0.2, 0.8], auto_corr=[0.4,0.9], tau_max=5, noise_sigma=[0.01, 0.01]):
-
-    if True:
-        coupling_funcs = [lin_f]
-        noise_types = ['gaussian']  # , 'weibull', 'uniform']
-        # noise_sigma = (0.1, 0.3)
-
-    couplings = list(np.arange(coef[0], coef[1]+1e-5, coef[2]))
-    couplings += [-c for c in couplings]
-
-    # auto_deps = list(np.arange(max(0., auto_corr-0.6), auto_corr+0.01, 0.05))
-    auto_deps = list(np.arange(auto_corr[0], auto_corr[1]+1e-5, auto_corr[2]))
-
-    # Models may be non-stationary. Hence, we iterate over a number of seeds
-    # to find a stationary one regarding network topology, noises, etc
-
-    ir = 0
-    model_seed = seed
-    while True:
-        ir += 1
-        # np.random.seed(model_seed)
-        random_state = np.random.RandomState(model_seed)
-
-        links = generate_random_contemp_model(
-            N=N, L=L,
-            coupling_coeffs=couplings,
-            coupling_funcs=coupling_funcs,
-            auto_coeffs=auto_deps,
-            tau_max=tau_max,
-            contemp_fraction=0.,
-            # num_trials=1000,
-            random_state=random_state)
-
-        noises = []
-        for j in links:
-            noise_type = random_state.choice(noise_types)
-            sigmas = list(np.arange(noise_sigma[0], noise_sigma[1]+1e-5, noise_sigma[2]))
-            sigma = random_state.choice(sigmas)
-            # sigma = noise_sigma[0] + (noise_sigma[1]-noise_sigma[0])*random_state.rand()
-            noises.append(getattr(noise_model(sigma=sigma, seed=seed), noise_type))
-
-        data_all_check, nonstationary = generate_nonlinear_contemp_timeseries(
-            links=links, T=100, noises=noises, random_state=random_state)
-
-        # If the model is stationary, break the loop
-        if not nonstationary:
-            data, nonstationary_full = generate_nonlinear_contemp_timeseries(
-                links=links, T=T, noises=noises, random_state=random_state)
-            if not nonstationary_full:
-                break
-        else:
-            print("Trial %d: Not a stationary model" % ir)
-            model_seed += 10000
-
-    cm = links_to_matrix(links)
-    return data, cm
-
-
-def simulate_var_from_links(links, T, seed=0, noise_sigma=[0.1, 0.2], noise_type="gaussian", func_name="lin_f"):
-    """
-    links_coeffs = {0: [((0, -1), 0.7), ((1, -1), -0.8)],
-                    1: [((1, -1), 0.8), ((3, -1), 0.8)],
-                    2: [((2, -1), 0.5), ((1, -2), 0.5), ((3, -3), 0.6)],
-                    3: [((3, -1), 0.4)],
-                    }
-    """
-    def get_func(func_name):
-        if func_name == "lin_f":
-            return lin_f
-        else:
-            raise NotImplementedError
-
-    random_state = np.random.RandomState(seed)
-    noises = []
-
-    new_links = {}
-    for j in range(len(links)):
-        sigma = noise_sigma[0] + \
-            (noise_sigma[1]-noise_sigma[0])*random_state.rand()
-        noises.append(getattr(noise_model(sigma=sigma, seed=seed), noise_type))
-        new_links[j] = []
-        for props in links[j]:
-            new_links[j].append(
-                (tuple(props[0:2]), props[2], get_func(props[3]),))
-    data, nonstationary = generate_nonlinear_contemp_timeseries(
-        links=new_links, T=T, noises=noises, random_state=random_state)
-    if nonstationary:
-        print("Model nonstationay!")
-
-    cm = links_to_matrix(new_links)
-    return data, cm
-
-
-def make_var_stationary(beta, radius=0.97):
-    '''Rescale coefficients of VAR model to make stable.'''
-    p = beta.shape[0]
-    lag = beta.shape[1] // p
-    bottom = np.hstack((np.eye(p * (lag - 1)), np.zeros((p * (lag - 1), p))))
-    beta_tilde = np.vstack((beta, bottom))
-    eigvals = np.linalg.eigvals(beta_tilde)
-    max_eig = max(np.abs(eigvals))
-    nonstationary = max_eig > radius
-    if nonstationary:
-        # print(f"Nonstationary, beta={str(beta):s}, max_eig={max_eig:.4f}")
-        return make_var_stationary((beta / max_eig) * 0.7, radius)
-    else:
-        # print(f"Stationary, beta={str(beta):s}")
-        return beta
-
-
-def simulate_var(p, T, lag, sparsity=0.2, beta_value=1.0, auto_corr=3.0, sd=0.1, seed=0):
-    if seed is not None:
-        np.random.seed(seed)
-
-    # Set up coefficients and Granger causality ground truth.
-    GC = np.eye(p, dtype=int)
-    beta = np.eye(p) * auto_corr
-
-    num_nonzero = int(p * sparsity) - 1
-    for i in range(p):
-        choice = np.random.choice(p - 1, size=num_nonzero, replace=False)
-        choice[choice >= i] += 1
-        beta[i, choice] = beta_value
-        GC[i, choice] = 1
-
-    beta = np.hstack([beta for _ in range(lag)])
-    beta = make_var_stationary(beta)
-
-    # Generate data.
-    burn_in = 100
-    errors = np.random.normal(loc=0, scale=sd, size=(p, T + burn_in))
-    X = np.ones((p, T + burn_in))
-    X[:, :lag] = errors[:, :lag]
-    for t in range(lag, T + burn_in):
-        X[:, t] = np.dot(beta, X[:, (t-lag):t].flatten(order='F'))
-        X[:, t] += errors[:, t-1]
-        
-    data = X.T[burn_in:, :]
-    return data, beta, GC
-
-
-
-
-
-def lorenz(x, t, F):
-    '''Partial derivatives for Lorenz-96 ODE.'''
-    p = len(x)
-    dxdt = np.zeros(p)
-    for i in range(p):
-        dxdt[i] = (x[(i+1) % p] - x[(i-2) % p]) * x[(i-1) % p] - x[i] + F
-
-    return dxdt
-
-
-def simulate_lorenz_96(p, T, F=10.0, delta_t=0.1, sd=0.1, burn_in=1000,
-                       seed=0):
-    if seed is not None:
-        np.random.seed(seed)
-
-    # Use scipy to solve ODE.
-    x0 = np.random.normal(scale=0.01, size=p)
-    t = np.linspace(0, (T + burn_in) * delta_t, T + burn_in)
-    X = odeint(lorenz, x0, t, args=(F,))
-    X += np.random.normal(scale=sd, size=(T + burn_in, p))
-
-    # Set up Granger causality ground truth.
-    GC = np.zeros((p, p), dtype=int)
-    for i in range(p):
-        GC[i, i] = 1
-        GC[i, (i + 1) % p] = 1
-        GC[i, (i - 1) % p] = 1
-        GC[i, (i - 2) % p] = 1
-
-    return X[burn_in:, :], GC
-

singular_ticker_causal/algorithms/CUTS_PLUS/data/utils.py

@@ -1,136 +0,0 @@
-import numpy as np
-import pandas as pd
-
-from sklearn.metrics.pairwise import haversine_distances
-
-
-def compute_mean(x, index=None):
-    """Compute the mean values for each datetime. The mean is first computed hourly over the week of the year.
-    Further NaN values are computed using hourly mean over the same month through the years. If other NaN are present,
-    they are removed using the mean of the sole hours. Hoping reasonably that there is at least a non-NaN entry of the
-    same hour of the NaN datetime in all the dataset."""
-    if isinstance(x, np.ndarray) and index is not None:
-        shape = x.shape
-        x = x.reshape((shape[0], -1))
-        df_mean = pd.DataFrame(x, index=index)
-    else:
-        df_mean = x.copy()
-    cond0 = [df_mean.index.year, df_mean.index.isocalendar().week, df_mean.index.hour]
-    cond1 = [df_mean.index.year, df_mean.index.month, df_mean.index.hour]
-    conditions = [cond0, cond1, cond1[1:], cond1[2:]]
-    while df_mean.isna().values.sum() and len(conditions):
-        nan_mean = df_mean.groupby(conditions[0]).transform(np.nanmean)
-        df_mean = df_mean.fillna(nan_mean)
-        conditions = conditions[1:]
-    if df_mean.isna().values.sum():
-        df_mean = df_mean.fillna(method='ffill')
-        df_mean = df_mean.fillna(method='bfill')
-    if isinstance(x, np.ndarray):
-        df_mean = df_mean.values.reshape(shape)
-    return df_mean
-
-
-def geographical_distance(x=None, to_rad=True):
-    """
-    Compute the as-the-crow-flies distance between every pair of samples in `x`. The first dimension of each point is
-    assumed to be the latitude, the second is the longitude. The inputs is assumed to be in degrees. If it is not the
-    case, `to_rad` must be set to False. The dimension of the data must be 2.
-
-    Parameters
-    ----------
-    x : pd.DataFrame or np.ndarray
-        array_like structure of shape (n_samples_2, 2).
-    to_rad : bool
-        whether to convert inputs to radians (provided that they are in degrees).
-
-    Returns
-    -------
-    distances :
-        The distance between the points in kilometers.
-    """
-    _AVG_EARTH_RADIUS_KM = 6371.0088
-
-    # Extract values of X if it is a DataFrame, else assume it is 2-dim array of lat-lon pairs
-    latlon_pairs = x.values if isinstance(x, pd.DataFrame) else x
-
-    # If the input values are in degrees, convert them in radians
-    if to_rad:
-        latlon_pairs = np.vectorize(np.radians)(latlon_pairs)
-
-    distances = haversine_distances(latlon_pairs) * _AVG_EARTH_RADIUS_KM
-
-    # Cast response
-    if isinstance(x, pd.DataFrame):
-        res = pd.DataFrame(distances, x.index, x.index)
-    else:
-        res = distances
-
-    return res
-
-
-def infer_mask(df, infer_from='next'):
-    """Infer evaluation mask from DataFrame. In the evaluation mask a value is 1 if it is present in the DataFrame and
-    absent in the `infer_from` month.
-
-    @param pd.DataFrame df: the DataFrame.
-    @param str infer_from: denotes from which month the evaluation value must be inferred.
-    Can be either `previous` or `next`.
-    @return: pd.DataFrame eval_mask: the evaluation mask for the DataFrame
-    """
-    mask = (~df.isna()).astype('uint8')
-    eval_mask = pd.DataFrame(index=mask.index, columns=mask.columns, data=0).astype('uint8')
-    if infer_from == 'previous':
-        offset = -1
-    elif infer_from == 'next':
-        offset = 1
-    else:
-        raise ValueError('infer_from can only be one of %s' % ['previous', 'next'])
-    months = sorted(set(zip(mask.index.year, mask.index.month)))
-    length = len(months)
-    for i in range(length):
-        j = (i + offset) % length
-        year_i, month_i = months[i]
-        year_j, month_j = months[j]
-        mask_j = mask[(mask.index.year == year_j) & (mask.index.month == month_j)]
-        mask_i = mask_j.shift(1, pd.DateOffset(months=12 * (year_i - year_j) + (month_i - month_j)))
-        mask_i = mask_i[~mask_i.index.duplicated(keep='first')]
-        mask_i = mask_i[np.in1d(mask_i.index, mask.index)]
-        eval_mask.loc[mask_i.index] = ~mask_i.loc[mask_i.index] & mask.loc[mask_i.index]
-    return eval_mask
-
-
-def disjoint_months(dataset, months=None):
-    idxs = np.arange(len(dataset))
-    months = ensure_list(months)
-    # divide indices according to window or horizon
-    start, end = 0, dataset.window - 1
-    # after idxs
-    start_in_months = np.in1d(dataset.index[dataset._indices + start].month, months)
-    end_in_months = np.in1d(dataset.index[dataset._indices + end].month, months)
-    idxs_in_months = start_in_months & end_in_months
-    after_idxs = idxs[idxs_in_months]
-    # previous idxs
-    months = np.setdiff1d(np.arange(1, 13), months)
-    start_in_months = np.in1d(dataset.index[dataset._indices + start].month, months)
-    end_in_months = np.in1d(dataset.index[dataset._indices + end].month, months)
-    idxs_in_months = start_in_months & end_in_months
-    prev_idxs = idxs[idxs_in_months]
-    return prev_idxs, after_idxs
-
-
-def thresholded_gaussian_kernel(x, theta=None, threshold=None, threshold_on_input=False):
-    if theta is None:
-        theta = np.std(x)
-    weights = np.exp(-np.square(x / theta))
-    if threshold is not None:
-        mask = x > threshold if threshold_on_input else weights < threshold
-        weights[mask] = 0.
-    return weights
-
-
-def ensure_list(obj):
-    if isinstance(obj, (list, tuple)):
-        return list(obj)
-    else:
-        return [obj]
-

singular_ticker_causal/algorithms/CUTS_PLUS/lagged_graph.py

@@ -1,95 +0,0 @@
-from __future__ import annotations
-
-from copy import deepcopy
-from dataclasses import dataclass
-from typing import Any
-
-import numpy as np
-
-from .cuts_plus import main as cuts_plus_main
-
-
-@dataclass(frozen=True)
-class LaggedGraphBundle:
-    graphs: np.ndarray
-    summary_graph: np.ndarray
-    priors: np.ndarray
-    input_steps: np.ndarray
-
-
-def broadcast_prior_to_lags(G_prior: np.ndarray | None, num_lags: int) -> np.ndarray | None:
-    if G_prior is None:
-        return None
-    G_prior = np.asarray(G_prior, dtype=np.float32)
-    if G_prior.ndim == 2:
-        return np.repeat(G_prior[None, :, :], num_lags, axis=0)
-    if G_prior.ndim == 3 and G_prior.shape[0] == num_lags:
-        return G_prior
-    raise ValueError("G_prior must have shape (N, N) or (L, N, N)")
-
-
-def aggregate_lagged_graphs(graphs: np.ndarray, reducer: str = "max") -> np.ndarray:
-    if reducer == "max":
-        return np.max(graphs, axis=0)
-    if reducer == "mean":
-        return np.mean(graphs, axis=0)
-    if reducer == "last":
-        return graphs[-1]
-    raise ValueError(f"Unsupported lagged graph reducer: {reducer}")
-
-
-def discover_lagged_graphs(
-    data: np.ndarray,
-    mask: np.ndarray,
-    opt: Any,
-    log: Any,
-    device: str = "cpu",
-    text_data: np.ndarray | None = None,
-    text_mask: np.ndarray | None = None,
-    G_prior: np.ndarray | None = None,
-    num_lags: int = 1,
-    reducer: str = "max",
-) -> LaggedGraphBundle:
-    if num_lags < 1:
-        raise ValueError("num_lags must be >= 1")
-
-    lag_priors = broadcast_prior_to_lags(G_prior, num_lags)
-    graphs = []
-    input_steps = []
-    previous_graph = None
-
-    for lag_idx in range(num_lags):
-        cfg = deepcopy(opt)
-        cfg.input_step = max(int(opt.input_step), lag_idx + 1)
-        input_steps.append(cfg.input_step)
-
-        current_prior = None
-        if lag_priors is not None:
-            current_prior = lag_priors[lag_idx]
-        elif previous_graph is not None:
-            current_prior = previous_graph
-
-        graph = cuts_plus_main(
-            data=data,
-            mask=mask,
-            true_cm=None,
-            opt=cfg,
-            log=log,
-            device=device,
-            text_data=text_data,
-            text_mask=text_mask,
-            G_prior=current_prior,
-        )
-        previous_graph = graph
-        graphs.append(np.asarray(graph, dtype=np.float32))
-
-    stacked = np.stack(graphs, axis=0)
-    summary = aggregate_lagged_graphs(stacked, reducer=reducer)
-    if lag_priors is None:
-        lag_priors = np.zeros_like(stacked)
-    return LaggedGraphBundle(
-        graphs=stacked,
-        summary_graph=summary.astype(np.float32),
-        priors=lag_priors.astype(np.float32),
-        input_steps=np.asarray(input_steps, dtype=np.int32),
-    )

singular_ticker_causal/algorithms/CUTS_PLUS/model/cuts_plus_net.py

@@ -1,153 +0,0 @@
-import torch
-from einops import rearrange
-from torch import nn
-
-class GRUCell(nn.Module):
-
-    def __init__(self, d_in, num_units, n_nodes, concat_h=False, activation='tanh'):
-        super(GRUCell, self).__init__()
-        self.activation_fn = getattr(torch, activation)
-
-        mpnn_channel = d_in*n_nodes+num_units if concat_h else d_in*n_nodes
-        self.forget_gate = MPNN(c_in=mpnn_channel, c_out=num_units, concat_h=concat_h)
-        self.update_gate = MPNN(c_in=mpnn_channel, c_out=num_units, concat_h=concat_h)
-        self.c_gate = MPNN(c_in=mpnn_channel, c_out=num_units, concat_h=concat_h)
-
-    def forward(self, x, h, adj):
-        """
-        :param x: (B, input_dim, num_nodes)
-        :param h: (B, num_units, num_nodes)
-        :param adj: (num_nodes, num_nodes)
-        :return:
-        """
-        # we start with bias 1.0 to not reset and not update
-        r = torch.sigmoid(self.forget_gate(x, h, adj))
-        u = torch.sigmoid(self.update_gate(x, h, adj))
-        c = self.c_gate(x, r * h, adj)  # batch_size, self._num_nodes * output_size
-        c = self.activation_fn(c)
-        return u * h + (1. - u) * c
-
-
-class MPNN(nn.Module):
-    def __init__(self, c_in, c_out, concat_h=True):
-        super(MPNN, self).__init__()
-        self.concat_h = concat_h
-        self.mlp = nn.Conv1d(c_in, c_out, kernel_size=1)
-        
-    def forward(self, x, h, graph):
-        b, c, n = x.shape
-        
-        x_repeat = x[:, :, :, None].expand(-1, -1, -1, n) # [b, c, n, n]
-        # graph = rearrange(graph, 'b n m -> b m n')
-        x_messages = torch.einsum('bcmn,bmn->bcmn', (x_repeat, graph))
-        x_messages = rearrange(x_messages, 'b c m n -> b (c m) n')
-        
-        if self.concat_h:
-            out = self.mlp(torch.cat([x_messages, h], dim=1))
-        else:
-            out = self.mlp(x_messages)
-        return out
-
-
-class LocalConv1D(nn.Module):
-    def __init__(self, in_channels, out_channels, kernel_size, n_nodes):
-        super(LocalConv1D, self).__init__()
-        self.out_channel = out_channels
-        self.conv_list = nn.ModuleList([
-            nn.Conv1d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size) for _ in range(n_nodes)
-        ])
-    
-    def forward(self, x): # x: [batch, features, nodes]
-        b, h, n = x.shape
-        out = torch.zeros((b, self.out_channel, n)).to(x.device)
-        for i in range(n):
-            x_local_in = x[..., i].unsqueeze(-1)
-            x_local_out = self.conv_list[i](x_local_in)
-            out[..., i] = x_local_out.squeeze(-1)
-        return out
-
-
-class CUTS_Plus_Net(nn.Module):
-    def __init__(self, n_nodes,
-                 in_ch=1,
-                 hidden_ch=32,
-                 n_layers=1,
-                 shared_weights_decoder=False,
-                 concat_h=False,):
-        super().__init__()
-        self.in_ch = in_ch
-        self.hidden_ch = hidden_ch
-        self.n_layers = n_layers
-        
-        self.conv_encoder1 = nn.Conv1d(in_channels=hidden_ch, out_channels=hidden_ch, kernel_size=1)
-        self.conv_encoder2 = nn.Conv1d(in_channels=2*hidden_ch, out_channels=hidden_ch, kernel_size=1)
-        if shared_weights_decoder:
-            self.decoder = nn.Sequential(
-                nn.Conv1d(in_channels=2*hidden_ch, out_channels=in_ch, kernel_size=1),
-                # nn.LeakyReLU(),
-                # nn.Conv1d(in_channels=hidden_ch, out_channels=hidden_ch, kernel_size=1),
-                # nn.LeakyReLU(),
-                # nn.Conv1d(in_channels=hidden_ch, out_channels=in_ch, kernel_size=1),
-                # nn.LeakyReLU(),
-            )
-        else:
-            self.decoder = nn.Sequential(
-                LocalConv1D(in_channels=2*hidden_ch, out_channels=in_ch, kernel_size=1, n_nodes=n_nodes),
-                # nn.LeakyReLU(),
-                # LocalConv1D(in_channels=hidden_ch, out_channels=hidden_ch, kernel_size=1, n_nodes=n_nodes),
-                # nn.LeakyReLU(),
-                # LocalConv1D(in_channels=hidden_ch, out_channels=in_ch, kernel_size=1, n_nodes=n_nodes),
-                # nn.LeakyReLU(),
-            )
-        # self.act = nn.PReLU()
-        self.act = nn.LeakyReLU()
-        
-        self.cells = nn.ModuleList()
-        for i in range(self.n_layers):
-            self.cells.append(GRUCell(d_in=in_ch if i==0 else hidden_ch, 
-                                      num_units=hidden_ch, 
-                                      n_nodes=n_nodes,
-                                      concat_h=concat_h))
-                
-        self.h0 = self.init_state(n_nodes)
-    
-    def init_state(self, n_nodes):
-        h = []
-        for layer in range(self.n_layers):
-            h.append(nn.parameter.Parameter(torch.zeros([self.hidden_ch, n_nodes])))
-        return nn.ParameterList(h)
-    
-    def update_state(self, x, h, graph):
-        rnn_in = x
-        for layer in range(self.n_layers):
-            rnn_in = h[layer] = self.cells[layer](rnn_in, h[layer], graph)
-        return h
-    
-    def forward(self, x, mask, fwd_graph):
-        x = rearrange(x, 'b n s c -> b c n s')
-        # fwd_graph = torch.ones_like(fwd_graph)
-        # mask = torch.ones_like(x).byte()
-        bs, in_ch, n_nodes, steps = x.shape
-        
-        h = [h_.expand(bs, -1, -1) for h_ in self.h0.to(x.device)]
-        
-        pred = []
-        for step in range(steps):
-            x_now = x[..., step] # [batches, in_ch, nodes]
-            
-            """Update state"""
-            h = self.update_state(x_now, h, fwd_graph)
-            h_now = h[-1]
-            
-            """Prediction"""
-            x_repr = self.act(self.conv_encoder1(h_now)) # [batches, hidden_ch, nodes]
-            x_repr = self.act(self.conv_encoder2(torch.cat([x_repr, h_now], dim=1))) # [batches, hidden_ch, nodes]
-            x_repr = torch.cat([x_repr, h_now], dim=1) # [batches, 2*hidden_ch, nodes]
-            x_hat2 = self.decoder(x_repr) # [batches, in_ch, nodes]
-            pred.append(x_hat2)
-            
-        
-        pred = torch.stack(pred, dim=-1)
-        pred = rearrange(pred, 'b c n s -> b n s c')
-        return pred[:, :, -1:]
-

singular_ticker_causal/algorithms/CUTS_PLUS/utils/causal_plot.py

@@ -1,57 +0,0 @@
-from re import X
-import matplotlib
-matplotlib.use('Agg')
-import matplotlib.pyplot as plt
-import tqdm
-import numpy as np
-import matplotlib.cm as cm
-
-def save_causal_graph(save_path, causal_matrix: np.ndarray, thres_percentile=100, colormap="gnuplot2"):
-    causal_matrix = np.max(causal_matrix)
-    print(causal_matrix.shape)
-    
-    n_node = causal_matrix.shape[0]
-    image_size = [100, 100]
-    n_node_dim = n_node**0.5
-    causal_thres = np.percentile(causal_matrix, 100-thres_percentile)
-    
-    colormap = cm.get_cmap(colormap)
-
-    plt.figure(figsize=[10,10], facecolor='black', edgecolor='black')
-
-        
-    for node_i_from in tqdm.tqdm(range(n_node)):
-        x_from = image_size[0] // n_node_dim * (node_i_from // n_node_dim + 0.5)
-        y_from = image_size[0] // n_node_dim * (node_i_from % n_node_dim + 0.5)
-        plt.text(y=x_from, x=y_from, s=f"{node_i_from:d}", size=20, color="#ffffff")
-        for node_i_to in range(n_node):
-            if not node_i_from == node_i_to:
-                x_to = image_size[0] // n_node_dim * (node_i_to // n_node_dim + 0.5)
-                y_to = image_size[0] // n_node_dim * (node_i_to % n_node_dim + 0.5)
-                causal_effect = causal_matrix[node_i_from, node_i_to]
-                if causal_effect > causal_thres:
-                    width = max(0.01, 1*causal_effect)
-                    arrow_length = ((x_to-x_from)**2 + (y_to-y_from)**2)**0.5
-                    plt.arrow(
-                        y=x_from+(x_to-x_from)*width/arrow_length,
-                        x=y_from+(y_to-y_from)*width/arrow_length,
-                        dy=(x_to-x_from)*(arrow_length-5*width)/arrow_length,
-                        dx=(y_to-y_from)*(arrow_length-5*width)/arrow_length,
-                        width=width,
-                        head_length=4*width,
-                        facecolor=colormap(causal_effect)[:3]+(causal_effect,),
-                        edgecolor="#00000000"
-                    )
-        
-        # fig.add_annotation(text=f"{node_i_from:d}", x=x_from, y=y_from, showarrow=False
-    
-    ax=plt.gca()
-    ax.patch.set_facecolor("black")
-    ax.xaxis.set_ticks_position('top') 
-    ax.invert_yaxis()
-    plt.savefig(save_path)
-    
-    
-if __name__=="__main__":
-    causal_matrix = np.load("outputs/tsgae_2022_0716_191203_262072/w.npy")
-    save_causal_graph("outputs/pic/causal.png", causal_matrix, thres_percentile=100)

singular_ticker_causal/algorithms/CUTS_PLUS/utils/exp_utils.py

@@ -1,286 +0,0 @@
-import os
-from os.path import join as opj
-from os.path import dirname as opd
-
-from .opt_type import MultiCADopt
-from .misc import omegaconf2dict
-
-import re
-import numpy as np
-import matplotlib
-matplotlib.use('Agg')
-import matplotlib.pyplot as plt
-
-from datetime import datetime
-from omegaconf import OmegaConf
-import glob
-import tqdm
-from tensorboard.backend.event_processing import event_accumulator
-
-PROPER_NAME = {"NFGR":"BRIEF", "h265":"H.265", "h264":"H.264", "jpg":"JPEG", "aoi-2000":"AoI", "vvc":"H.266"}
-
-MAIN_COLOR = {"NFGR":"#e64b35", "h265":"#48c9b0", "h264":"#5599c7", "jpg":"#c39bd2", "DVC":"#e6b0aa", "SGA":"#f8c370", "vvc":"#b1babb",
-              "GOOD":"#239954", "BAD":"#cc6155"}
-SECONDARY_COLOR = {"NFGR":"#d98880", "h265":"#76d7c3", "SGA":"#f9d7a0", "h264":"#7fb3d5", "DVC":"#e6b0aa"}
-
-
-def get_time_stamp():
-    return str(datetime.now().strftime("%m-%d-%H%M%S-%f"))
-
-def show_single_scores(x_arr, y_arr, label="exp", suffix="", 
-                       scatter=False, log_axis=True, 
-                       xlim=None, ylim=None):
-    
-    '''Plotting figures'''
-    cvs = FigureCanvas(name=label, figsize=[30,20])
-    fig_idx = 0
-    for idx,data_path in enumerate(y_arr.labels[2]): # range(score_arr.shape[2]):
-        if fig_idx == 40:
-            break
-        elif np.isnan(np.nanmean(y_arr[:,:,data_path])):
-            continue
-        
-        fig_idx += 1
-        if log_axis:
-            ax = plt.subplot(8,5,fig_idx, xscale="log")
-        else:
-            ax = plt.subplot(8,5,fig_idx)
-        ax.set_title("DATA_{:02d}_".format(idx) + data_path[-100:-60] + "\n" + data_path[-60:])
-        # ax.set_title(dim_marks[2][data_i])
-        plt.set_cmap("rainbow")
-        for i, dim0 in enumerate(y_arr.labels[0]):
-            x_nan = x_arr[dim0,:,data_path]
-            y_nan = y_arr[dim0,:,data_path]
-            x = x_nan[np.isfinite(x_nan + y_nan)]
-            y = y_nan[np.isfinite(x_nan + y_nan)]
-            
-            if len(x) > 0 and len(y) > 0:
-                x, y = sort_lists(x, y)
-                
-                plt.plot(x, y, color=plt.get_cmap("tab20")(i), label=dim0)
-                plt.scatter(x, y, color=plt.get_cmap("tab20")(i))
-            
-        plt.legend()
-        if ylim is not None:
-            plt.ylim(ylim)
-    
-    ax=plt.gca()
-    ax.xaxis.set_major_locator(plt.LogLocator(base=10, numticks=5))
-    ax.yaxis.set_major_locator(plt.MaxNLocator(5))
-            
-    cvs.save_fig(suffix=suffix, time_stamp=False, save_format=".pdf")
-
-
-def sort_lists(*lists):
-    sorted_index = np.argsort(lists[0]).astype(int)
-    results = []
-    for l in lists:
-        sorted_list = np.array([l[i] for i in sorted_index])
-        results.append(sorted_list)
-    return results
-
-
-def show_averge_scores(x_arr, y_arr, label="exp", suffix="", percentile=25,
-                       scatter=False, log_axis=True, figsize=[4,3],
-                       xlim=None, ylim=None, legend=False, grid=False, std=False):
-    cvs = FigureCanvas(name=label, figsize=figsize)
-    if log_axis:
-        plt.xscale("log")
-        
-    fig_idx = 0
-    if scatter:
-        for data_i,data_path in enumerate(y_arr.labels[2]):
-            if np.isnan(np.nanmean(y_arr[:,:,data_path])):
-                continue
-            
-            for i, dim0 in enumerate(y_arr.labels[0]):
-                x_nan = x_arr[dim0,:,data_path]
-                y_nan = y_arr[dim0,:,data_path]
-                x = x_nan[np.isfinite(x_nan + y_nan)]
-                y = y_nan[np.isfinite(x_nan + y_nan)]
-                
-                if len(x) > 0 and len(y) > 0:
-                    x, y = sort_lists(x, y)
-                    
-                    # plt.plot(x, y, color=plt.get_cmap("tab20")(i), label=dim0, marker="v")
-                    plt.scatter(x, y, color=MAIN_COLOR[dim0], alpha=0.5, marker="v", edgecolors='none',
-                                s=80 if "NFGR" in dim0 else 50)
-                
-            # plt.legend(edgecolors='none')
-            if ylim is not None:
-                plt.ylim(ylim)
-                
-    if ylim is not None:
-        full_range = ylim[1] - ylim[0]  
-    else:
-        full_range = np.nanmax(np.nanmean(y_arr.arr, axis=2)) - np.nanmin(np.nanmean(y_arr.arr, axis=2))
-    max_std = np.nanmax(np.nanstd(y_arr["NFGR",:,:], axis=1))
-            
-        
-    for idx, dim0 in enumerate(sorted(y_arr.labels[0], key=lambda item:item == "NFGR")):
-        # if "jpg" in dim0:
-        #     continue
-        
-        x_nan = np.nanmean(x_arr[dim0,:], axis=1)
-        y_nan = np.nanmean(y_arr[dim0,:], axis=1)
-        # y_l = np.nanpercentile(y_arr[dim0,:], percentile, axis=1)
-        # y_u = np.nanpercentile(y_arr[dim0,:], 100-percentile, axis=1)
-        y_std = np.nanstd(y_arr[dim0,:], axis=1) # / max_std * full_range * 0.07
-        
-        print(1 / max_std * full_range * 0.07)
-        
-        x = x_nan[np.isfinite(x_nan + y_nan)]
-        y = y_nan[np.isfinite(x_nan + y_nan)]
-        # y_l = y_l[np.isfinite(x_nan + y_nan)]
-        # y_u = y_u[np.isfinite(x_nan + y_nan)]
-        y_std = y_std[np.isfinite(x_nan + y_nan)]
-        
-        x, y, y_std = sort_lists(x, y, y_std)
-        
-        if std:
-            plt.fill_between(x, y-y_std/2, y+y_std/2, color=MAIN_COLOR[dim0], alpha=0.25, edgecolors="none")
-        plt.plot(x, y, color=MAIN_COLOR[dim0], label=name(dim0), 
-                 lw=2 if "NFGR" in dim0 else 1.5)
-        plt.scatter(x, y, color=MAIN_COLOR[dim0], 
-                 s=30 if "NFGR" in dim0 else 20)
-        
-    if ylim is not None:
-        plt.ylim(ylim)
-    if xlim is not None:
-        plt.xlim(xlim)
-    if legend:
-        plt.legend(loc='lower left', bbox_to_anchor=(0.1, 0.1), fancybox=False)
-    
-    ax=plt.gca()
-    if log_axis:
-        ax.xaxis.set_major_locator(plt.LogLocator(base=10, numticks=5))
-    else:
-        ax.xaxis.set_major_locator(plt.MaxNLocator(5))
-    ax.yaxis.set_major_locator(plt.MaxNLocator(5))
-    
-    if grid:
-        ax.spines['right'].set_visible(True)
-        ax.spines['top'].set_visible(True)
-        # plt.xticks(np.arange(0.4, 1.8, 0.28))
-        # plt.yticks(np.arange(50, 500, 40))
-        plt.grid(axis='both', c="#cacaca", which="major")
-    
-    cvs.save_fig(suffix=suffix, time_stamp=False, save_format=".pdf")
-    
-    
-def name(alias):
-    if alias in PROPER_NAME:
-        return PROPER_NAME[alias]
-    else:
-        print("Cannot find proper name.")
-        return alias
-
-class FigureCanvas(object):
-    
-    def __init__(self, name="ex1", figsize=[14,9]):
-        self.name = name
-        plt.close('all')
-        fig = plt.figure(figsize=figsize)
-        ax = plt.axes()
-        ax.spines['right'].set_visible(False)
-        ax.spines['top'].set_visible(False)
-        plt.tight_layout()
-    
-    def show_fig(self, save_format=".png", suffix="", time_stamp=True):
-        save_path = "./exp/figs/%s/%s_%s%s"%(
-            self.name, 
-            get_time_stamp() if time_stamp else "plt", 
-            suffix, save_format)
-        if not os.path.exists(opd(save_path)):
-            os.makedirs(opd(save_path))
-        plt.savefig(save_path, bbox_inches='tight')
-        plt.show()
-        
-    def save_fig(self, save_format=".png", suffix="", time_stamp=True, save_root="./exp/figs/"):
-        save_path = opj(save_root, "%s/%s_%s%s"%(
-            self.name, 
-            get_time_stamp() if time_stamp else "plt", 
-            suffix, save_format))
-        if not os.path.exists(opd(save_path)):
-            os.makedirs(opd(save_path))
-        plt.savefig(save_path, bbox_inches='tight')
-
-
-def find_lineprofile_cmp(im_list):    
-    for x in range(0, im_list[0].shape[0], 10):
-        for y in range(0, im_list[0].shape[1], 10):
-            lp = [im[x][y] for im in im_list]
-            if np.max(lp) > 2500 and np.max(lp) < 3000:
-                return lp
-
-
-def get_decompressed_path(opt_path):
-    res_root = opd(opt_path)
-    max_step = 0
-    for dirn in os.listdir(res_root):
-        if dirn == "decompressed":
-            return glob.glob(res_root + "/decompressed/*.tif")[0]
-        elif "steps" in dirn:
-            step_n = int(dirn[5:])
-            if step_n > max_step:
-                max_step = step_n
-    search_list = glob.glob(res_root + "/steps" + str(max_step) + "/decompressed/*.*")
-    if len(search_list) > 0:
-        return search_list[0]
-    return None
-
-def load_scalars(event_path):
-    try:
-        event_path = glob.glob(event_path)[0]
-    except:
-        print("No event file found.")
-        return None
-    ea = event_accumulator.EventAccumulator(event_path)
-    ea.Reload()
-    # print("Available scalars: ", ea.scalars.Keys())
-    scalars = {}
-    for criterion in ea.scalars.Keys():
-        val_scalar = ea.scalars.Items(criterion)
-        val_curve = ([(i.step, i.value) for i in val_scalar])
-        scalars[criterion] = val_curve
-    return scalars
-    
-def load_scalars_cached(root_path, cache_dir="exp/cache", reload_data=False):
-    
-    if root_path[-1] == "/":
-        root_path = root_path[:-1]
-    
-    # csv_file = glob.glob(root_path + "/*.csv")[0]
-    # exp_list = load_csv(csv_file)
-    root_name = "".join(re.split("/|\\\\", root_path)[-2:])
-    cache_path = opj(cache_dir, root_name + ".npy")
-    
-    # read from cached data
-    if os.path.exists(cache_path) and not reload_data:
-        print("Loading cached result...")
-        loaded_res = np.load(cache_path, allow_pickle=True)
-    else:
-        if not os.path.exists(cache_dir):
-            os.makedirs(cache_dir)
-        
-        loaded_res = []
-        for dirn in tqdm.tqdm(os.listdir(root_path)):
-            if os.path.isdir(opj(root_path, dirn)):
-                if len(glob.glob(root_path + "/%s/events.out.tfevents*"%(dirn))) > 0:
-                    res_fname = glob.glob(root_path + "/%s/events.out.tfevents*"%(dirn))[0]
-                    opt_fname = glob.glob(root_path + "/%s/opt.yaml"%(dirn))[0]
-                    scores = load_scalars(res_fname)
-                    opt: MultiCADopt = omegaconf2dict(OmegaConf.load(opt_fname), sep=".")
-                    loaded_res.append((res_fname, scores, opt))
-        
-        np.save(cache_path, loaded_res)
-    
-    return loaded_res
-
-
-# if __name__=="__main__":
-#     # scalars = load_scalars("cyx_exp/experiments_outputs/ex2_1227/*/exp_00000/events.out.tfevents*")
-#     # print(scalars)
-    
-#     res = load_scalars_cached("cyx_exp/experiments_outputs/ex2_1227/", reload_data=False)
-#     print("")

singular_ticker_causal/algorithms/CUTS_PLUS/utils/gumbel_softmax.py

@@ -1,66 +0,0 @@
-import torch
-from torch import Tensor
-import warnings
-
-def gumbel_softmax(logits: Tensor, tau: float = 1, hard: bool = False, eps: float = 1e-10, dim: int = -1) -> Tensor:
-    r"""
-    Samples from the Gumbel-Softmax distribution (`Link 1`_  `Link 2`_) and optionally discretizes.
-
-    Args:
-      logits: `[..., num_features]` unnormalized log probabilities
-      tau: non-negative scalar temperature
-      hard: if ``True``, the returned samples will be discretized as one-hot vectors,
-            but will be differentiated as if it is the soft sample in autograd
-      dim (int): A dimension along which softmax will be computed. Default: -1.
-
-    Returns:
-      Sampled tensor of same shape as `logits` from the Gumbel-Softmax distribution.
-      If ``hard=True``, the returned samples will be one-hot, otherwise they will
-      be probability distributions that sum to 1 across `dim`.
-
-    .. note::
-      This function is here for legacy reasons, may be removed from nn.Functional in the future.
-
-    .. note::
-      The main trick for `hard` is to do  `y_hard - y_soft.detach() + y_soft`
-
-      It achieves two things:
-      - makes the output value exactly one-hot
-      (since we add then subtract y_soft value)
-      - makes the gradient equal to y_soft gradient
-      (since we strip all other gradients)
-
-    Examples::
-        >>> logits = torch.randn(20, 32)
-        >>> # Sample soft categorical using reparametrization trick:
-        >>> F.gumbel_softmax(logits, tau=1, hard=False)
-        >>> # Sample hard categorical using "Straight-through" trick:
-        >>> F.gumbel_softmax(logits, tau=1, hard=True)
-
-    .. _Link 1:
-        https://arxiv.org/abs/1611.00712
-    .. _Link 2:
-        https://arxiv.org/abs/1611.01144
-    """
-    if eps != 1e-10:
-        warnings.warn("`eps` parameter is deprecated and has no effect.")
-
-    gumbels = (
-        -torch.empty_like(logits, memory_format=torch.legacy_contiguous_format).exponential_().log()
-    )  # ~Gumbel(0,1)
-    gumbels = (logits + gumbels) / tau  # ~Gumbel(logits,tau)
-    y_soft = gumbels.softmax(dim)
-
-    if hard:
-        # Straight through.
-        index = y_soft.max(dim, keepdim=True)[1]
-        y_hard = torch.zeros_like(logits, memory_format=torch.legacy_contiguous_format).scatter_(dim, index, 1.0)
-        ret = y_hard - y_soft.detach() + y_soft
-    else:
-        # Reparametrization trick.
-        ret = y_soft
-    return ret
-
-if __name__=="__main__":
-    a = torch.tensor([[2.0, 0.7]]*10)
-    print(gumbel_softmax(a, tau=10))

singular_ticker_causal/algorithms/CUTS_PLUS/utils/logger.py

@@ -1,74 +0,0 @@
-import sys
-from omegaconf import OmegaConf
-import os
-from os.path import join as opj
-import numpy as np
-from os.path import dirname as opd
-from typing import Dict
-from torch.utils.tensorboard import SummaryWriter
-from .misc import omegaconf2list
-
-
-class MyLogger():
-    def __init__(self, log_dir: str, stderr: bool = True, tensorboard: bool = True, stdout: bool = True):
-        self.log_dir = log_dir
-        if not os.path.exists(self.log_dir):
-            os.makedirs(self.log_dir)
-        self.logger_dict: Dict[str] = {}
-        if stdout:
-            stdout_handler = open(opj(self.log_dir, 'stdout.log'), 'w')
-            sys.stdout = stdout_handler
-        if stderr:
-            stderr_handler = open(opj(self.log_dir, 'stderr.log'), 'w')
-            sys.stderr = stderr_handler
-        if tensorboard:
-            self.tblogger = SummaryWriter(self.log_dir)
-            self.logger_dict['tblogger'] = self.tblogger
-
-    def log_opt(self, opt):
-        OmegaConf.save(config=opt, f=opj(self.log_dir, 'opt.yaml'))
-        opt_log = omegaconf2list(opt, sep='/')
-        for logger_name in self.logger_dict.keys():
-            if logger_name == 'tblogger':
-                for idx, opt in enumerate(opt_log):
-                    self.logger_dict[logger_name].add_text('hparam', opt, idx)
-
-    def log_metrics(self, metrics_dict: Dict[str, float], iters):
-        for logger_name in self.logger_dict.keys():
-            if logger_name == 'csvlogger':
-                self.logger_dict[logger_name].log_metrics(metrics_dict, iters)
-                self.logger_dict[logger_name].save()
-            elif logger_name == 'clearml_logger':
-                for k in metrics_dict.keys():
-                    self.logger_dict[logger_name].report_scalar(
-                        k, k, metrics_dict[k], iters)
-            elif logger_name == 'tblogger':
-                for k in metrics_dict.keys():
-                    self.logger_dict[logger_name].add_scalar(
-                        k, metrics_dict[k], iters)
-    
-    def log_figures(self, figure, name="figure.png", iters=None, exclude_logger=[]):
-        for logger_name in self.logger_dict.keys():
-            if logger_name == 'tblogger':
-                if logger_name not in exclude_logger:
-                    self.logger_dict[logger_name].add_figure(tag=name, figure=figure, global_step=iters)
-        
-        if iters is None:
-            save_path = opj(self.log_dir, "figures")
-        else:
-            save_path = opj(self.log_dir, f"iter_{iters:d}", name)
-        os.makedirs(opd(save_path), exist_ok=True)
-        figure.savefig(save_path)
-        
-    def log_npz(self, data: Dict, name="data.npz", iters=None):
-        if iters is None:
-            save_path = opj(self.log_dir)
-        else:
-            save_path = opj(self.log_dir, f"iter_{iters:d}", name)
-        os.makedirs(save_path, exist_ok=True)
-        np.savez(opj(save_path, "graph.npz"), **data)
-
-    def close(self):
-        for logger_name in self.logger_dict.keys():
-            if logger_name == 'tblogger':
-                self.logger_dict[logger_name].close()

singular_ticker_causal/algorithms/CUTS_PLUS/utils/misc.py

@@ -1,292 +0,0 @@
-from copy import deepcopy
-import numpy as np
-import matplotlib
-matplotlib.use('Agg')
-import matplotlib.pyplot as plt
-import itertools
-import random
-import numpy as np
-import torch
-import omegaconf
-from sklearn.metrics import roc_curve, roc_auc_score
-
-
-def log_time_series(original_data, data_interp, data_pred, log, log_step):
-    fig = plt.figure(figsize=[10,10])
-    plt.plot(np.arange(0, original_data.shape[0], 1), original_data, label="original")
-    plt.plot(np.arange(0, data_interp.shape[0], 1), data_interp, label="interp")
-    plt.plot(np.arange(0, data_pred.shape[0], 1), data_pred, label="pred")
-    plt.legend()
-    log.log_figures(fig, name="Predicted Latent Data", iters=log_step)
-
-
-def calc_and_log_metrics(time_prob_mat, true_cm, log, log_step, threshold=0.5, plot_roc=True):
-    if len(time_prob_mat.shape) == 3:
-        graph = np.max(time_prob_mat, axis=2)
-    else:
-        graph = time_prob_mat
-    causal_graph = graph > threshold
-    tp = np.mean(causal_graph * true_cm)
-    tn = np.mean((1-causal_graph) * (1-causal_graph))
-    fp = np.mean(causal_graph * (1-true_cm))
-    fn = np.mean((1-causal_graph) * true_cm)
-    tpr = tp / (tp + fn)
-    fpr = fp / (fp + tn)
-    acc = (tp + tn) / (tp + tn + fp + fn)
-    log.log_metrics({"metrics/tpr": tpr}, log_step)
-    log.log_metrics({"metrics/fpr": fpr}, log_step)
-    log.log_metrics({"metrics/accuracy": acc}, log_step)
-
-    if plot_roc:
-        fpr, tpr, thres = roc_curve(true_cm.reshape(-1) > 0.5, 
-                                    graph.reshape(-1), pos_label=1)
-        fig = plt.figure(figsize=[4, 4])
-        plt.plot(fpr, tpr)
-        log.tblogger.add_figure(tag="ROC", figure=fig, global_step=log_step)
-        
-        log.log_npz(name="graph", 
-                    data={"true_cm":true_cm, "pred_cm":graph},
-                    iters=log_step)
-
-    auc = roc_auc_score(true_cm.reshape(-1)>0.5,
-                        graph.reshape(-1))
-    log.log_metrics({"metrics/auc": auc}, log_step)
-    return auc
-
-def sigmoid(z):
-    return 1/(1 + np.exp(-z))
-
-def plot_causal_matrix_in_training(time_coef, name, log, log_step, threshold=0.5, plot_each_time=False):
-    if time_coef is None:
-        return
-    
-    if np.max(time_coef) - np.min(time_coef) > 0.01:
-        time_coef = (time_coef - np.min(time_coef)) / (np.max(time_coef) - np.min(time_coef))
-    n, m, t = time_coef.shape
-    
-    # # Show Discovered Graph (Coefficiency)
-    # sub_cg = plot_causal_matrix(
-    #     np.max(time_coef, axis=2),
-    #     figsize=[1.5*time_coef.shape[0], 1*n])
-    # log.log_figures(sub_cg, name="Discovered Graph Coef/" + name, iters=log_step)
-    
-    # # Graph for Each Time Lag
-    # if plot_each_time:
-    #     for ti in range(t):
-    #         sub_cg = plot_causal_matrix(
-    #             time_coef[:, :, ti],
-    #             figsize=[1.5*n, 1*n],
-    #             vmin=0, vmax=1)
-    #         log.log_figures(sub_cg, name=f"Discovered Prob T-{t-ti:d}", 
-    #                         iters=log_step, exclude_logger="tblogger")
-
-    # Show Discovered Graph (Probability)
-    time_graph = time_coef
-    sub_cg = plot_causal_matrix(
-        np.max(time_graph, axis=2),
-        figsize=[1.5*n, 1*n],
-        vmin=0, vmax=1)
-    log.log_figures(sub_cg, name="Discovered Prob/" + name, iters=log_step)
-
-    # Show Thresholded Graph
-    time_thres = np.max(time_graph, axis=2) > threshold
-    sub_cg = plot_causal_matrix(
-        time_thres,
-        figsize=[1.5*n, 1*n])
-    log.log_figures(sub_cg, name="Discovered Graph/" + name, iters=log_step)
-    log.log_npz({"Discovered Graph Coef": time_coef, "Discovered Prob": time_graph, "Discovered Graph": time_thres}, 
-                name="Graph.npz", iters=log_step)
-
-
-def plot_causal_matrix(cmtx, class_names=None, figsize=None, vmin=None, vmax=None, show_text=True, cmap="magma"):
-    """
-    A function to create a colored and labeled causal matrix matplotlib figure
-    given true labels and preds.
-    Args:
-        cmtx (ndarray): causal matrix.
-        num_classes (int): total number of nodes.
-        class_names (Optional[list of strs]): a list of node names.
-        figsize (Optional[float, float]): the figure size of the causal matrix.
-            If None, default to [6.4, 4.8].
-
-    Returns:
-        img (figure): matplotlib figure.
-    """
-    num_classes = cmtx.shape[0]
-    if class_names is None or type(class_names) != list:
-        class_names = [str(i) for i in range(num_classes)]
-
-    
-    figsize[0] = 30 if figsize[0] > 30 else figsize[0]
-    figsize[1] = 20 if figsize[1] > 20 else figsize[1]
-    
-    plt.clf()
-    plt.close("all")
-    figure = plt.figure(figsize=figsize)
-    plt.imshow(cmtx, interpolation="nearest",
-               cmap=cmap, vmin=vmin, vmax=vmax)
-    plt.title("Causal matrix")
-    plt.colorbar()
-    # tick_marks = np.arange(len(class_names))
-    # plt.xticks(tick_marks, class_names, rotation=45)
-    # plt.yticks(tick_marks, class_names)
-
-    # Use white text if squares are dark; otherwise black.
-    threshold = cmtx.max() / 2.0
-    for i, j in itertools.product(range(cmtx.shape[0]), range(cmtx.shape[1])):
-        color = "white" if cmtx[i, j] < threshold else "black"
-        if cmtx.shape[0] < 20 and show_text:
-            plt.text(j, i, format(cmtx[i, j], ".2e") if cmtx[i, j] != 0 else ".",
-                    horizontalalignment="center", color=color,)
-
-    plt.tight_layout()
-    plt.ylabel("True label")
-    plt.xlabel("Predicted label")
-
-    return figure
-
-
-def reproduc(seed, benchmark=False, deterministic=True):
-    """Make experiments reproducible
-    """
-    random.seed(seed)
-    np.random.seed(seed)
-    torch.manual_seed(seed)
-    torch.cuda.manual_seed_all(seed)
-    torch.backends.cudnn.benchmark = benchmark
-    torch.backends.cudnn.deterministic = deterministic
-
-
-def omegaconf2list(opt, prefix='', sep='.'):
-    notation_list = []
-    for k, v in opt.items():
-        k = str(k)
-        if isinstance(v, omegaconf.listconfig.ListConfig):
-            notation_list.append("{}{}={}".format(prefix, k, v))
-            # if k in ['iter_list','step_list']: # do not sparse list
-            #     dot_notation_list.append("{}{}={}".format(prefix,k,v))
-            # else:
-            #     templist = []
-            #     for v_ in v:
-            #         templist.append('{}{}={}'.format(prefix,k,v_))
-            #     dot_notation_list.append(templist)
-        elif isinstance(v, (float, str, int,)):
-            notation_list.append("{}{}={}".format(prefix, k, v))
-        elif v is None:
-            notation_list.append("{}{}=~".format(prefix, k,))
-        elif isinstance(v, omegaconf.dictconfig.DictConfig):
-            nested_flat_list = omegaconf2list(v, prefix + k + sep, sep=sep)
-            if nested_flat_list:
-                notation_list.extend(nested_flat_list)
-        else:
-            raise NotImplementedError
-    return notation_list
-
-
-def omegaconf2dotlist(opt, prefix='',):
-    return omegaconf2list(opt, prefix, sep='.')
-
-
-def omegaconf2dict(opt, sep):
-    notation_list = omegaconf2list(opt, sep=sep)
-    dict = {notation.split('=', maxsplit=1)[0]: notation.split(
-        '=', maxsplit=1)[1] for notation in notation_list}
-    return dict
-
-
-# def read_video(video_path: str):
-#     if ops(video_path)[-1] == ".tif":
-#         data = tifffile.imread(video_path)
-#         data = (data / np.max(data) * 255).astype(np.uint8)
-#         if len(data.shape) == 3:
-#             data = data[:, :, :, None]
-#         return data
-#     else:
-#         cap = cv2.VideoCapture(video_path)
-#         frames = []
-#         while cap.isOpened():
-#             # get a frame
-#             ret, frame = cap.read()
-#             if not ret:
-#                 break
-#             frames.append(np.array(frame)[None])
-
-#         cap.release()
-#         return np.concatenate(frames, axis=0)
-
-
-# def save_video(video_path: str, data):
-#     skvideo.io.vwrite(video_path, data)
-
-
-
-class LabelArray(object):
-    # def __init__(self, array, labels):
-    #     self.arr = array
-    #     self.labels = labels
-    #     self.marks = dim_marks
-    #     assert [len[label_list] for label_list in labels] == self.arr.shape
-        
-    def __init__(self, dim, labels=None):
-        if labels is not None:
-            if len(dim) != dim:
-                raise "The length of labels has to be equal to dim if defined"
-            else:
-                self.labels = deepcopy(labels)
-        else:
-            self.labels = [[] for _ in range(dim)]
-        self.arr = None
-        self.update_arr()
-        
-    def update_arr(self):
-        if self.arr is not None:
-            oldarr = self.arr
-            self.arr = np.zeros([len(dim) for dim in self.labels]) * np.nan
-            self.arr[tuple([slice(0,sh_dim,1) for sh_dim in oldarr.shape])] = oldarr
-        else:
-            self.arr = np.zeros([len(dim) for dim in self.labels]) * np.nan
-        self.shape = self.arr.shape
-        
-        
-    def __getitem__(self, label_list):
-        index_list = []
-        for dim,label in enumerate(label_list):
-            if isinstance(label, str):
-                index_list.append(self.labels[dim].index(label))
-            elif isinstance(label, slice):
-                index_list.append(label)
-            elif isinstance(label, int):
-                index_list.append(label)
-            else:
-                raise NotImplementedError
-            
-        return self.arr[tuple(index_list)]
-    
-    def __setitem__(self, label_list, val):    
-        index_list = []
-        for dim,label in enumerate(label_list):
-            if isinstance(label, str):
-                if not label in self.labels[dim]:
-                    self.labels[dim].append(label)
-                    self.update_arr()
-                index_list.append(self.labels[dim].index(label))
-            elif isinstance(label, slice):
-                index_list.append(label)
-            elif isinstance(label, int):
-                index_list.append(label)
-            else:
-                raise NotImplementedError
-            
-        self.arr[tuple(index_list)] = val
-    
-    def __str__(self):
-        return str(self.arr) + "\n--------------------------\n" + str(self.labels)
-    
-    
-    def to_np(self):
-        return self.arr
-    
-    def from_np(self, array):
-        assert [len[label_list] for label_list in self.labels] == self.arr.shape
-        self.arr = array
-        

singular_ticker_causal/algorithms/CUTS_PLUS/utils/opt_type.py

@@ -1,83 +0,0 @@
-from dataclasses import dataclass
-from typing import Any
-
-@dataclass
-class ReproducOpt:    
-    seed: int
-    benchmark: bool
-    deterministic: bool
-    
-@dataclass
-class NetworkOpt:
-    name: str
-    network_param: Any
-    
-@dataclass
-class TrainOpt:
-  batch_size: int
-  total_epoch: int
-  time_window: int
-    
-    
-@dataclass
-class TsGAEopt:
-    dir_name: str
-    task_name: str
-    optimizer: Any
-    reproduc: ReproducOpt
-    network: NetworkOpt
-    train: TrainOpt
-    log: Any
-    causal_thres: str
-    
-@dataclass
-class MultiCADopt:
-    dir_name: str
-    task_name: str
-    
-    @dataclass
-    class MultiCADargs:
-        n_nodes: int
-        input_step: int
-        window_step: int
-        stride: int
-        batch_size: int
-        sample_per_epoch: int
-        data_dim: int
-        total_epoch: int
-        
-        patience: int
-        warmup: Any
-        
-        show_graph_every: int
-        val_every: int
-        
-        n_groups: int
-        group_policy: Any
-        causal_thres: str
-        
-        @dataclass
-        class data_pred:
-            model: str
-            merge_policy: str
-            lr_data_start: float
-            lr_data_end: float
-            weight_decay: int
-            prob: bool
-
-        @dataclass
-        class graph_discov:
-            lr_graph_start: float
-            lr_graph_end: float
-            lambda_s_start: float
-            lambda_s_end: float
-            tau_start: float
-            tau_end: float
-            disable_bwd: bool
-            separate_bwd: bool
-            disable_ind: bool
-            disable_graph: bool
-            use_true_graph: bool
-    
-    reproduc: ReproducOpt
-    log: Any

singular_ticker_causal/algorithms/__init__.py

@@ -1,2 +0,0 @@
-# CAMEF algorithms subpackage — GPT4MTS and dataloader have been retired.
-# This package is intentionally empty pending removal of the CAMEF directory.

singular_ticker_causal/causal_inference/__init__.py

@@ -1,15 +0,0 @@
-from .causal_model import StructuralCausalModel
-from .identification import IdentificationStrategy, find_adjustment_set, is_identifiable
-from .pywhyllm_assumptions import CausalAssumptionReport, PyWhyLLMConfig, PyWhyLLMAssumptionService
-from .query_engine import CausalQueryEngine
-
-__all__ = [
-    "CausalAssumptionReport",
-    "CausalQueryEngine",
-    "IdentificationStrategy",
-    "PyWhyLLMConfig",
-    "PyWhyLLMAssumptionService",
-    "StructuralCausalModel",
-    "find_adjustment_set",
-    "is_identifiable",
-]

singular_ticker_causal/causal_inference/abduction.py

@@ -1,127 +0,0 @@
-from __future__ import annotations
-
-from typing import Dict
-
-import numpy as np
-
-from .causal_model import StructuralCausalModel
-
-
-def abduct_noise(scm: StructuralCausalModel, observed_values: Dict[str, float]) -> Dict[str, float]:
-    noise: Dict[str, float] = {}
-    for node_idx in scm.topological_indices:
-        node = scm.nodes[node_idx]
-        if node not in observed_values:
-            continue
-        obs = float(observed_values[node])
-        eq = scm.equations[node]
-        if not eq.parents or eq.equation_type == "exogenous":
-            noise[node] = obs - eq.intercept
-            continue
-        parent_vals = []
-        for p in eq.parents:
-            if p not in observed_values:
-                parent_vals = []
-                break
-            parent_vals.append(float(observed_values[p]))
-        if not parent_vals:
-            continue
-        coef_vec = np.array([eq.coefficients[p] for p in eq.parents], dtype=float)
-        pred = eq.intercept + float(np.array(parent_vals, dtype=float) @ coef_vec)
-        noise[node] = obs - pred
-    return noise
-
-
-def counterfactual_predict(
-    scm: StructuralCausalModel,
-    observed_values: Dict[str, float],
-    treatment: str,
-    counterfactual_value: float,
-    target: str,
-) -> Dict[str, float]:
-    if treatment not in scm.node_to_idx:
-        raise ValueError(f"Unknown treatment node: {treatment}")
-    if target not in scm.node_to_idx:
-        raise ValueError(f"Unknown target node: {target}")
-
-    noises = abduct_noise(scm, observed_values)
-    state = scm.data_level[-1].copy()
-    for node, value in observed_values.items():
-        if node in scm.node_to_idx:
-            state[scm.node_to_idx[node]] = float(value)
-
-    cf = state.copy()
-    t_idx = scm.node_to_idx[treatment]
-    cf[t_idx] = float(counterfactual_value)
-
-    for node_idx in scm.topological_indices:
-        node = scm.nodes[node_idx]
-        if node_idx == t_idx:
-            continue
-        eq = scm.equations[node]
-        if not eq.parents or eq.equation_type == "exogenous":
-            cf[node_idx] = eq.intercept + noises.get(node, 0.0)
-            continue
-        parent_vals = cf[eq.parent_indices]
-        coef_vec = np.array([eq.coefficients[p] for p in eq.parents], dtype=float)
-        pred = eq.intercept + float(parent_vals @ coef_vec)
-        cf[node_idx] = pred + noises.get(node, 0.0)
-
-    y_idx = scm.node_to_idx[target]
-    factual = float(state[y_idx])
-    counterfactual = float(cf[y_idx])
-    ite = counterfactual - factual
-    pct_change = ite / (abs(factual) + 1e-12)
-    return {
-        "factual_outcome": factual,
-        "counterfactual_outcome": counterfactual,
-        "ite": float(ite),
-        "pct_change": float(pct_change),
-    }
-
-
-def counterfactual_predict_multi(
-    scm: StructuralCausalModel,
-    observed_values: Dict[str, float],
-    counterfactual_values: Dict[str, float],
-    target: str,
-) -> Dict[str, float]:
-    """Compute a joint counterfactual for multiple simultaneous interventions."""
-    for node in counterfactual_values:
-        if node not in scm.node_to_idx:
-            raise ValueError(f"Unknown counterfactual node: {node}")
-
-    noises = abduct_noise(scm, observed_values)
-    state = scm.data_level[-1].copy()
-    for node, value in observed_values.items():
-        if node in scm.node_to_idx:
-            state[scm.node_to_idx[node]] = float(value)
-
-    cf = state.copy()
-    for node, value in counterfactual_values.items():
-        cf[scm.node_to_idx[node]] = float(value)
-
-    for node_idx in scm.topological_indices:
-        node = scm.nodes[node_idx]
-        if node in counterfactual_values:
-            continue
-        eq = scm.equations[node]
-        if not eq.parents or eq.equation_type == "exogenous":
-            cf[node_idx] = eq.intercept + noises.get(node, 0.0)
-            continue
-        parent_vals = cf[eq.parent_indices]
-        coef_vec = np.array([eq.coefficients[p] for p in eq.parents], dtype=float)
-        pred = eq.intercept + float(parent_vals @ coef_vec)
-        cf[node_idx] = pred + noises.get(node, 0.0)
-
-    y_idx = scm.node_to_idx[target]
-    factual = float(state[y_idx])
-    counterfactual = float(cf[y_idx])
-    ite = counterfactual - factual
-    pct_change = ite / (abs(factual) + 1e-12)
-    return {
-        "factual_outcome": factual,
-        "counterfactual_outcome": counterfactual,
-        "ite": float(ite),
-        "pct_change": float(pct_change),
-    }

singular_ticker_causal/causal_inference/causal_model.py

@@ -1,314 +0,0 @@
-from __future__ import annotations
-
-from dataclasses import dataclass, field
-from typing import Dict, List, Optional, Sequence, Tuple
-
-import numpy as np
-
-
-@dataclass
-class StructuralEquation:
-    node: str
-    parents: List[str]
-    parent_indices: List[int]
-    intercept: float
-    coefficients: Dict[str, float]
-    residual_mean: float
-    residual_std: float
-    r_squared: float
-    n_obs: int
-    equation_type: str  # "linear" | "exogenous"
-
-
-@dataclass
-class StructuralCausalModel:
-    nodes: Sequence[str]
-    adj: np.ndarray
-    adjacency_mask: np.ndarray
-    data_tech: np.ndarray
-    mask_tech: Optional[np.ndarray] = None
-    prohibition_mask: Optional[np.ndarray] = None
-    threshold: float = 0.5
-    lag: int = 1
-    min_obs: int = 12
-    ridge_alpha: float = 1e-4
-    dag_adj: np.ndarray = field(init=False)
-    topological_indices: List[int] = field(init=False, default_factory=list)
-    equations: Dict[str, StructuralEquation] = field(init=False, default_factory=dict)
-    removed_cycle_edges: List[Dict[str, float]] = field(init=False, default_factory=list)
-
-    def __post_init__(self) -> None:
-        self.nodes = list(self.nodes)
-        self.node_to_idx = {node: i for i, node in enumerate(self.nodes)}
-        self.data_level = self._extract_level_data(self.data_tech)
-        self.mask_level = self._extract_mask_level(self.mask_tech)
-        self._validate_shapes()
-
-    @property
-    def n_nodes(self) -> int:
-        return len(self.nodes)
-
-    @property
-    def t_steps(self) -> int:
-        return self.data_level.shape[0]
-
-    def fit(self) -> "StructuralCausalModel":
-        self.dag_adj = self._build_dag()
-        self._enforce_acyclic()
-        self.topological_indices = self._topological_sort(self.dag_adj)
-        self._fit_equations()
-        return self
-
-    def density(self) -> float:
-        n = self.n_nodes
-        max_edges = n * (n - 1)
-        if max_edges == 0:
-            return 0.0
-        return float(np.sum(self.dag_adj) / max_edges)
-
-    def parents_of(self, node: str) -> List[str]:
-        j = self.node_to_idx[node]
-        return [self.nodes[i] for i in np.where(self.dag_adj[:, j])[0]]
-
-    def has_directed_path(self, source: str, target: str) -> bool:
-        s = self.node_to_idx[source]
-        t = self.node_to_idx[target]
-        stack = [s]
-        visited = set()
-        while stack:
-            u = stack.pop()
-            if u == t:
-                return True
-            if u in visited:
-                continue
-            visited.add(u)
-            children = np.where(self.dag_adj[u])[0].tolist()
-            stack.extend(children)
-        return False
-
-    def descendants_of(self, node: str) -> List[str]:
-        start = self.node_to_idx[node]
-        stack = [start]
-        visited = set()
-        while stack:
-            u = stack.pop()
-            children = np.where(self.dag_adj[u])[0].tolist()
-            for v in children:
-                if v not in visited:
-                    visited.add(v)
-                    stack.append(v)
-        visited.discard(start)
-        return [self.nodes[i] for i in sorted(visited)]
-
-    def _validate_shapes(self) -> None:
-        n = len(self.nodes)
-        if self.adj.shape != (n, n):
-            raise ValueError(f"adj shape mismatch: expected {(n, n)}, got {self.adj.shape}")
-        if self.adjacency_mask.shape != (n, n):
-            raise ValueError(
-                f"adjacency_mask shape mismatch: expected {(n, n)}, got {self.adjacency_mask.shape}"
-            )
-        if self.prohibition_mask is not None and self.prohibition_mask.shape != (n, n):
-            raise ValueError(
-                f"prohibition_mask shape mismatch: expected {(n, n)}, got {self.prohibition_mask.shape}"
-            )
-        if self.data_level.ndim != 2 or self.data_level.shape[1] != n:
-            raise ValueError(
-                f"data_level shape mismatch: expected (T, {n}), got {self.data_level.shape}"
-            )
-        if self.mask_level is not None and self.mask_level.shape != self.data_level.shape:
-            raise ValueError(
-                "mask_tech shape mismatch after extraction: expected shape "
-                f"{self.data_level.shape}, got {self.mask_level.shape}"
-            )
-        if self.lag < 1:
-            raise ValueError("lag must be >= 1")
-
-    def _extract_level_data(self, data_tech: np.ndarray) -> np.ndarray:
-        if data_tech.ndim == 3:
-            return np.asarray(data_tech[:, :, 0], dtype=float)
-        if data_tech.ndim == 2:
-            return np.asarray(data_tech, dtype=float)
-        raise ValueError(f"Unsupported data_tech ndim={data_tech.ndim}; expected 2 or 3.")
-
-    def _extract_mask_level(self, mask_tech: Optional[np.ndarray]) -> Optional[np.ndarray]:
-        if mask_tech is None:
-            return None
-        if mask_tech.ndim == 3:
-            return np.asarray(mask_tech[:, :, 0], dtype=float)
-        if mask_tech.ndim == 2:
-            return np.asarray(mask_tech, dtype=float)
-        raise ValueError(f"Unsupported mask_tech ndim={mask_tech.ndim}; expected 2 or 3.")
-
-    def _build_dag(self) -> np.ndarray:
-        cuts_edges = self.adj >= self.threshold
-        prior_edges = self.adjacency_mask > 0
-        dag = np.logical_or(cuts_edges, prior_edges)
-        if self.prohibition_mask is not None:
-            prohibited = self.prohibition_mask <= 0
-            dag = np.where(prohibited, False, dag)
-        np.fill_diagonal(dag, False)
-        return dag.astype(bool)
-
-    def _enforce_acyclic(self) -> None:
-        while True:
-            cycle_edges = self._find_cycle_edges(self.dag_adj)
-            if not cycle_edges:
-                return
-
-            removable = []
-            for src, dst in cycle_edges:
-                mandatory = bool(self.adjacency_mask[src, dst] > 0)
-                score = float(self.adj[src, dst])
-                removable.append((mandatory, score, src, dst))
-
-            non_mandatory = [r for r in removable if not r[0]]
-            choice = min(non_mandatory or removable, key=lambda x: x[1])
-            _, score, src, dst = choice
-
-            self.dag_adj[src, dst] = False
-            self.removed_cycle_edges.append(
-                {
-                    "source": self.nodes[src],
-                    "target": self.nodes[dst],
-                    "adj_score": score,
-                }
-            )
-
-    def _topological_sort(self, dag_adj: np.ndarray) -> List[int]:
-        n = dag_adj.shape[0]
-        indegree = np.sum(dag_adj, axis=0).astype(int)
-        queue = [i for i in range(n) if indegree[i] == 0]
-        order: List[int] = []
-
-        while queue:
-            node = queue.pop(0)
-            order.append(node)
-            children = np.where(dag_adj[node])[0]
-            for child in children:
-                indegree[child] -= 1
-                if indegree[child] == 0:
-                    queue.append(int(child))
-
-        if len(order) != n:
-            raise RuntimeError("DAG still contains a cycle after pruning.")
-        return order
-
-    def _find_cycle_edges(self, dag_adj: np.ndarray) -> List[Tuple[int, int]]:
-        n = dag_adj.shape[0]
-        state = np.zeros(n, dtype=int)  # 0=unvisited, 1=visiting, 2=done
-        parent = -np.ones(n, dtype=int)
-
-        def dfs(u: int) -> Optional[List[Tuple[int, int]]]:
-            state[u] = 1
-            for v in np.where(dag_adj[u])[0]:
-                v = int(v)
-                if state[v] == 0:
-                    parent[v] = u
-                    found = dfs(v)
-                    if found:
-                        return found
-                elif state[v] == 1:
-                    nodes = [v]
-                    cur = u
-                    while cur != v and cur != -1:
-                        nodes.append(cur)
-                        cur = int(parent[cur])
-                    nodes.append(v)
-                    nodes.reverse()
-                    return [(nodes[i], nodes[i + 1]) for i in range(len(nodes) - 1)]
-            state[u] = 2
-            return None
-
-        for start in range(n):
-            if state[start] == 0:
-                result = dfs(start)
-                if result:
-                    return result
-        return []
-
-    def _fit_equations(self) -> None:
-        T = self.t_steps
-        for idx in self.topological_indices:
-            node = self.nodes[idx]
-            parent_idx = [int(i) for i in np.where(self.dag_adj[:, idx])[0]]
-            parent_names = [self.nodes[i] for i in parent_idx]
-            valid_t = self._valid_timesteps(idx, parent_idx)
-
-            if len(valid_t) == 0:
-                self.equations[node] = StructuralEquation(
-                    node=node,
-                    parents=parent_names,
-                    parent_indices=parent_idx,
-                    intercept=0.0,
-                    coefficients={},
-                    residual_mean=0.0,
-                    residual_std=1.0,
-                    r_squared=0.0,
-                    n_obs=0,
-                    equation_type="exogenous",
-                )
-                continue
-
-            y = self.data_level[valid_t, idx]
-
-            if not parent_idx or len(valid_t) < self.min_obs:
-                mu = float(np.mean(y))
-                residuals = y - mu
-                self.equations[node] = StructuralEquation(
-                    node=node,
-                    parents=parent_names,
-                    parent_indices=parent_idx,
-                    intercept=mu,
-                    coefficients={},
-                    residual_mean=float(np.mean(residuals)) if residuals.size else 0.0,
-                    residual_std=float(np.std(residuals)) if residuals.size else 1.0,
-                    r_squared=0.0,
-                    n_obs=int(len(valid_t)),
-                    equation_type="exogenous",
-                )
-                continue
-
-            X = self.data_level[valid_t - self.lag][:, parent_idx]
-            coef, intercept = self._fit_ridge(X, y)
-            y_hat = intercept + (X @ coef)
-            residuals = y - y_hat
-            ss_res = float(np.sum(residuals ** 2))
-            ss_tot = float(np.sum((y - np.mean(y)) ** 2))
-            r2 = 1.0 - (ss_res / ss_tot) if ss_tot > 1e-12 else 0.0
-
-            coeffs = {name: float(coef[i]) for i, name in enumerate(parent_names)}
-            self.equations[node] = StructuralEquation(
-                node=node,
-                parents=parent_names,
-                parent_indices=parent_idx,
-                intercept=float(intercept),
-                coefficients=coeffs,
-                residual_mean=float(np.mean(residuals)),
-                residual_std=float(np.std(residuals)),
-                r_squared=r2,
-                n_obs=int(len(valid_t)),
-                equation_type="linear",
-            )
-
-    def _valid_timesteps(self, node_idx: int, parent_indices: List[int]) -> np.ndarray:
-        valid_t = np.arange(self.lag, self.t_steps, dtype=int)
-        if self.mask_level is None:
-            return valid_t
-
-        valid = self.mask_level[valid_t, node_idx] > 0
-        for p_idx in parent_indices:
-            valid &= self.mask_level[valid_t - self.lag, p_idx] > 0
-        return valid_t[valid]
-
-    def _fit_ridge(self, X: np.ndarray, y: np.ndarray) -> Tuple[np.ndarray, float]:
-        X_mean = np.mean(X, axis=0)
-        y_mean = float(np.mean(y))
-        Xc = X - X_mean
-        yc = y - y_mean
-
-        p = X.shape[1]
-        reg = self.ridge_alpha * np.eye(p)
-        beta = np.linalg.solve(Xc.T @ Xc + reg, Xc.T @ yc)
-        intercept = y_mean - float(X_mean @ beta)
-        return beta, intercept

singular_ticker_causal/causal_inference/estimator.py

@@ -1,125 +0,0 @@
-from __future__ import annotations
-
-from typing import Dict, Optional, Set
-
-import numpy as np
-
-from .causal_model import StructuralCausalModel
-from .mutilator import propagate_intervention
-
-
-def estimate_ate(
-    scm: StructuralCausalModel,
-    treatment: str,
-    outcome: str,
-    treatment_value: Optional[float] = None,
-    adjustment_set: Optional[Set[str]] = None,
-    horizon: int = 5,
-) -> Dict[str, object]:
-    t_idx = scm.node_to_idx[treatment]
-    y_idx = scm.node_to_idx[outcome]
-    z_nodes = sorted(adjustment_set or set())
-    z_idx = [scm.node_to_idx[z] for z in z_nodes]
-
-    valid_t = np.arange(scm.lag, scm.t_steps, dtype=int)
-    if scm.mask_level is not None:
-        valid = (scm.mask_level[valid_t, y_idx] > 0) & (scm.mask_level[valid_t - scm.lag, t_idx] > 0)
-        for zi in z_idx:
-            valid &= scm.mask_level[valid_t - scm.lag, zi] > 0
-        valid_t = valid_t[valid]
-
-    if len(valid_t) < 5:
-        raise ValueError(
-            f"Insufficient observations for ATE estimation of {treatment}->{outcome}: {len(valid_t)} rows."
-        )
-
-    y = scm.data_level[valid_t, y_idx]
-    x_treat = scm.data_level[valid_t - scm.lag, t_idx]
-    X_parts = [np.ones((len(valid_t), 1)), x_treat.reshape(-1, 1)]
-    if z_idx:
-        X_parts.append(scm.data_level[valid_t - scm.lag][:, z_idx])
-    X = np.concatenate(X_parts, axis=1)
-
-    beta = np.linalg.pinv(X.T @ X) @ (X.T @ y)
-    y_hat = X @ beta
-    resid = y - y_hat
-    dof = max(1, len(y) - X.shape[1])
-    sigma2 = float(np.sum(resid ** 2) / dof)
-    cov = sigma2 * np.linalg.pinv(X.T @ X)
-    se = float(np.sqrt(max(cov[1, 1], 0.0)))
-    ate = float(beta[1])
-
-    baseline = float(np.mean(np.abs(y))) + 1e-12
-    ate_normalized = ate / baseline
-    ci_95 = (ate - 1.96 * se, ate + 1.96 * se)
-
-    path_contributions = _path_contributions(scm, treatment, outcome)
-
-    horizon_effects = []
-    if treatment_value is not None:
-        baseline_t = scm.t_steps - 1
-        shocked = propagate_intervention(scm, treatment, treatment_value, horizon=horizon, baseline_t=baseline_t)
-        base_outcome = float(scm.data_level[baseline_t, y_idx])
-        for h in range(horizon):
-            horizon_effects.append(float(shocked[h, y_idx] - base_outcome))
-
-    return {
-        "ate": ate,
-        "ate_normalized": float(ate_normalized),
-        "ci_95": (float(ci_95[0]), float(ci_95[1])),
-        "n_obs": int(len(valid_t)),
-        "path_contributions": path_contributions,
-        "horizon_effects": horizon_effects,
-    }
-
-
-def _path_contributions(
-    scm: StructuralCausalModel,
-    treatment: str,
-    outcome: str,
-    max_paths: int = 100,
-) -> Dict[str, float]:
-    start = scm.node_to_idx[treatment]
-    target = scm.node_to_idx[outcome]
-    paths = []
-
-    def dfs(node: int, path: list[int], seen: set[int]) -> None:
-        if len(paths) >= max_paths:
-            return
-        if node == target:
-            paths.append(path.copy())
-            return
-        for child in np.where(scm.dag_adj[node])[0]:
-            child = int(child)
-            if child in seen:
-                continue
-            seen.add(child)
-            path.append(child)
-            dfs(child, path, seen)
-            path.pop()
-            seen.remove(child)
-
-    dfs(start, [start], {start})
-
-    contributions: Dict[str, float] = {}
-    for path in paths:
-        coeff_product = 1.0
-        valid = True
-        for i in range(len(path) - 1):
-            src = scm.nodes[path[i]]
-            dst = scm.nodes[path[i + 1]]
-            eq = scm.equations.get(dst)
-            if eq is None:
-                valid = False
-                break
-            coef = eq.coefficients.get(src)
-            if coef is None:
-                valid = False
-                break
-            coeff_product *= coef
-        if not valid:
-            continue
-        label = " -> ".join(scm.nodes[i] for i in path)
-        contributions[label] = float(coeff_product)
-
-    return contributions

singular_ticker_causal/causal_inference/identification.py

@@ -1,41 +0,0 @@
-from __future__ import annotations
-
-from enum import Enum
-from typing import Optional, Set, Tuple
-
-from .causal_model import StructuralCausalModel
-
-
-class IdentificationStrategy(Enum):
-    DIRECT = "direct"
-    BACKDOOR = "backdoor"
-    NOT_IDENTIFIABLE = "not_identifiable"
-
-
-def find_adjustment_set(
-    scm: StructuralCausalModel,
-    treatment: str,
-    outcome: str,
-) -> Tuple[IdentificationStrategy, Optional[Set[str]]]:
-    if treatment not in scm.node_to_idx:
-        raise ValueError(f"Unknown treatment node: {treatment}")
-    if outcome not in scm.node_to_idx:
-        raise ValueError(f"Unknown outcome node: {outcome}")
-    if treatment == outcome:
-        return IdentificationStrategy.DIRECT, set()
-
-    if not scm.has_directed_path(treatment, outcome):
-        return IdentificationStrategy.NOT_IDENTIFIABLE, None
-
-    parents = set(scm.parents_of(treatment))
-    descendants = set(scm.descendants_of(treatment))
-    adjustment = parents - descendants - {outcome}
-
-    if adjustment:
-        return IdentificationStrategy.BACKDOOR, adjustment
-    return IdentificationStrategy.DIRECT, set()
-
-
-def is_identifiable(scm: StructuralCausalModel, treatment: str, outcome: str) -> bool:
-    strategy, _ = find_adjustment_set(scm, treatment, outcome)
-    return strategy != IdentificationStrategy.NOT_IDENTIFIABLE

singular_ticker_causal/causal_inference/mutilator.py

@@ -1,62 +0,0 @@
-from __future__ import annotations
-
-from copy import deepcopy
-from typing import Dict, Optional, Sequence
-
-import numpy as np
-
-from .causal_model import StructuralCausalModel
-
-
-def mutilate_graph(scm: StructuralCausalModel, treatment: str, value: float) -> StructuralCausalModel:
-    mutilated = deepcopy(scm)
-    t_idx = mutilated.node_to_idx[treatment]
-    mutilated.dag_adj[:, t_idx] = False
-    mutilated.pinned_values = {treatment: float(value)}
-    return mutilated
-
-
-def propagate_intervention(
-    scm: StructuralCausalModel,
-    treatment: str,
-    value: float,
-    targets: Optional[Sequence[str]] = None,
-    horizon: int = 5,
-    baseline_t: int = -1,
-) -> np.ndarray:
-    if horizon < 1:
-        raise ValueError("horizon must be >= 1")
-
-    baseline_idx = baseline_t if baseline_t >= 0 else (scm.t_steps + baseline_t)
-    if baseline_idx < 0 or baseline_idx >= scm.t_steps:
-        raise ValueError(f"baseline_t {baseline_t} resolves out of bounds for T={scm.t_steps}")
-
-    t_idx = scm.node_to_idx[treatment]
-    prev = scm.data_level[baseline_idx].copy()
-    forecasts = np.zeros((horizon, scm.n_nodes), dtype=float)
-
-    for h in range(horizon):
-        nxt = prev.copy()
-        nxt[t_idx] = float(value)
-        for node_idx in scm.topological_indices:
-            if node_idx == t_idx:
-                continue
-            node = scm.nodes[node_idx]
-            eq = scm.equations[node]
-            if not eq.parent_indices or eq.equation_type == "exogenous":
-                continue
-            parent_vals = prev[eq.parent_indices]
-            coef_vec = np.array([eq.coefficients[p] for p in eq.parents], dtype=float)
-            nxt[node_idx] = eq.intercept + float(parent_vals @ coef_vec)
-        forecasts[h] = nxt
-        prev = nxt
-
-    if targets:
-        missing = [n for n in targets if n not in scm.node_to_idx]
-        if missing:
-            raise ValueError(f"Unknown targets: {missing}")
-    return forecasts
-
-
-def target_series(forecasts: np.ndarray, scm: StructuralCausalModel, targets: Sequence[str]) -> Dict[str, list[float]]:
-    return {t: [float(v) for v in forecasts[:, scm.node_to_idx[t]]] for t in targets}

singular_ticker_causal/causal_inference/pywhyllm_assumptions.py

@@ -1,338 +0,0 @@
-from __future__ import annotations
-
-import hashlib
-import json
-import os
-from dataclasses import asdict, dataclass, field
-from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
-
-import numpy as np
-
-
-def _json_safe(value: Any) -> Any:
-    if isinstance(value, np.ndarray):
-        return value.tolist()
-    if isinstance(value, (np.integer,)):
-        return int(value)
-    if isinstance(value, (np.floating,)):
-        return float(value)
-    if isinstance(value, dict):
-        return {str(k): _json_safe(v) for k, v in value.items()}
-    if isinstance(value, (list, tuple, set)):
-        return [_json_safe(v) for v in value]
-    if isinstance(value, (str, int, float, bool)) or value is None:
-        return value
-    return str(value)
-
-
-@dataclass
-class PyWhyLLMConfig:
-    enabled: bool = False
-    model: str = "gpt-4"
-    max_edges: int = 25
-    cache_dir: str = os.path.join(
-        os.path.dirname(__file__),
-        "..",
-        "debug_data",
-        "pywhyllm_cache",
-    )
-
-
-@dataclass
-class CausalAssumptionReport:
-    available: bool = True
-    reason: Optional[str] = None
-    domain_expertises: List[str] = field(default_factory=list)
-    suggested_confounders: List[str] = field(default_factory=list)
-    suggested_backdoor_sets: List[List[str]] = field(default_factory=list)
-    suggested_mediators: List[str] = field(default_factory=list)
-    suggested_ivs: List[str] = field(default_factory=list)
-    negative_controls: List[str] = field(default_factory=list)
-    latent_confounders: List[str] = field(default_factory=list)
-    edge_critiques: Any = field(default_factory=list)
-    accepted_edges: List[Tuple[str, str]] = field(default_factory=list)
-    rejected_edges: List[Tuple[str, str]] = field(default_factory=list)
-    warnings: List[str] = field(default_factory=list)
-
-    def to_dict(self) -> Dict[str, Any]:
-        payload = asdict(self)
-        payload["accepted_edges"] = [list(edge) for edge in self.accepted_edges]
-        payload["rejected_edges"] = [list(edge) for edge in self.rejected_edges]
-        return _json_safe(payload)
-
-    @classmethod
-    def from_dict(cls, payload: Dict[str, Any]) -> "CausalAssumptionReport":
-        data = dict(payload)
-        data["accepted_edges"] = [tuple(edge) for edge in data.get("accepted_edges", [])]
-        data["rejected_edges"] = [tuple(edge) for edge in data.get("rejected_edges", [])]
-        return cls(**data)
-
-
-def _dedupe_strings(values: Iterable[Any]) -> List[str]:
-    seen = set()
-    result: List[str] = []
-    for value in values or []:
-        text = str(value).strip()
-        if text and text not in seen:
-            seen.add(text)
-            result.append(text)
-    return result
-
-
-def _normalise_suggestion(value: Any) -> List[str]:
-    if value is None:
-        return []
-    if isinstance(value, tuple) and len(value) == 2:
-        return _normalise_suggestion(value[1])
-    if isinstance(value, dict):
-        keys = [k for k, v in value.items() if isinstance(k, str) and v]
-        if keys:
-            return _dedupe_strings(keys)
-        flattened: List[str] = []
-        for item in value.values():
-            flattened.extend(_normalise_suggestion(item))
-        return _dedupe_strings(flattened)
-    if isinstance(value, (list, set, tuple)):
-        flattened = []
-        for item in value:
-            if isinstance(item, (list, set, tuple, dict)):
-                flattened.extend(_normalise_suggestion(item))
-            else:
-                flattened.append(item)
-        return _dedupe_strings(flattened)
-    return _dedupe_strings([value])
-
-
-class PyWhyLLMAssumptionService:
-    def __init__(
-        self,
-        config: Optional[PyWhyLLMConfig] = None,
-        model_suggester: Any = None,
-        identification_suggester: Any = None,
-        validation_suggester: Any = None,
-        relationship_strategy: Any = None,
-    ):
-        self.config = config or PyWhyLLMConfig()
-        self._model_suggester = model_suggester
-        self._identification_suggester = identification_suggester
-        self._validation_suggester = validation_suggester
-        self._relationship_strategy = relationship_strategy
-
-    @property
-    def enabled(self) -> bool:
-        return bool(self.config.enabled)
-
-    def analyze(
-        self,
-        *,
-        nodes: Sequence[str],
-        dag_adj: np.ndarray,
-        treatment: str,
-        outcome: str,
-        max_edges: Optional[int] = None,
-    ) -> CausalAssumptionReport:
-        if not self.enabled:
-            return CausalAssumptionReport(
-                available=False,
-                reason="PyWhy-LLM is disabled. Set PYWHYLLM_ENABLED=true or pass pywhyllm_enabled=True.",
-            )
-
-        cache_path = self._cache_path(nodes, dag_adj, treatment, outcome)
-        cached = self._read_cache(cache_path)
-        if cached is not None:
-            return cached
-
-        try:
-            modeler, identifier, validator, relationship_strategy = self._suggesters()
-        except Exception as exc:
-            return CausalAssumptionReport(
-                available=False,
-                reason=f"PyWhy-LLM is not installed or failed to initialize: {exc}",
-            )
-
-        all_factors = list(nodes)
-        edges = self._edges(nodes, dag_adj, max_edges or self.config.max_edges)
-        warnings: List[str] = []
-
-        domain_expertises: List[str] = []
-        suggested_confounders: List[str] = []
-        suggested_backdoor_sets: List[List[str]] = []
-        suggested_mediators: List[str] = []
-        suggested_ivs: List[str] = []
-        negative_controls: List[str] = []
-        latent_confounders: List[str] = []
-        edge_critiques: Any = []
-        suggested_dag: Any = edges
-
-        try:
-            domain_expertises = _normalise_suggestion(modeler.suggest_domain_expertises(all_factors))
-        except Exception as exc:
-            warnings.append(f"domain_expertises failed: {exc}")
-
-        try:
-            suggested_confounders = _normalise_suggestion(
-                modeler.suggest_confounders(treatment, outcome, all_factors, domain_expertises)
-            )
-        except Exception as exc:
-            warnings.append(f"confounder suggestion failed: {exc}")
-
-        try:
-            suggested_dag = modeler.suggest_relationships(
-                treatment,
-                outcome,
-                all_factors,
-                domain_expertises,
-                relationship_strategy,
-            )
-        except Exception as exc:
-            warnings.append(f"relationship suggestion failed: {exc}")
-
-        try:
-            backdoor = identifier.suggest_backdoor(treatment, outcome, all_factors, domain_expertises)
-            backdoor_nodes = _normalise_suggestion(backdoor)
-            if backdoor_nodes:
-                suggested_backdoor_sets = [backdoor_nodes]
-        except Exception as exc:
-            warnings.append(f"backdoor suggestion failed: {exc}")
-
-        try:
-            suggested_mediators = _normalise_suggestion(
-                identifier.suggest_mediators(treatment, outcome, all_factors, domain_expertises)
-            )
-        except Exception as exc:
-            warnings.append(f"mediator suggestion failed: {exc}")
-
-        try:
-            suggested_ivs = _normalise_suggestion(
-                identifier.suggest_ivs(treatment, outcome, all_factors, domain_expertises)
-            )
-        except Exception as exc:
-            warnings.append(f"iv suggestion failed: {exc}")
-
-        try:
-            edge_critiques = validator.critique_graph(
-                all_factors,
-                suggested_dag,
-                domain_expertises,
-                relationship_strategy,
-            )
-        except Exception as exc:
-            warnings.append(f"edge critique failed: {exc}")
-
-        try:
-            latent_confounders = _normalise_suggestion(
-                validator.suggest_latent_confounders(treatment, outcome, all_factors, domain_expertises)
-            )
-        except Exception as exc:
-            warnings.append(f"latent confounder suggestion failed: {exc}")
-
-        try:
-            negative_controls = _normalise_suggestion(
-                validator.suggest_negative_controls(treatment, outcome, all_factors, domain_expertises)
-            )
-        except Exception as exc:
-            warnings.append(f"negative control suggestion failed: {exc}")
-
-        accepted_edges, rejected_edges = self._classify_edges(edges, edge_critiques)
-        report = CausalAssumptionReport(
-            available=True,
-            domain_expertises=domain_expertises,
-            suggested_confounders=suggested_confounders,
-            suggested_backdoor_sets=suggested_backdoor_sets,
-            suggested_mediators=suggested_mediators,
-            suggested_ivs=suggested_ivs,
-            negative_controls=negative_controls,
-            latent_confounders=latent_confounders,
-            edge_critiques=edge_critiques,
-            accepted_edges=accepted_edges,
-            rejected_edges=rejected_edges,
-            warnings=warnings,
-        )
-        self._write_cache(cache_path, report)
-        return report
-
-    def _suggesters(self) -> Tuple[Any, Any, Any, Any]:
-        if self._model_suggester and self._identification_suggester and self._validation_suggester:
-            return (
-                self._model_suggester,
-                self._identification_suggester,
-                self._validation_suggester,
-                self._relationship_strategy,
-            )
-
-        from pywhyllm import RelationshipStrategy
-        from pywhyllm.suggesters.identification_suggester import IdentificationSuggester
-        from pywhyllm.suggesters.model_suggester import ModelSuggester
-        from pywhyllm.suggesters.validation_suggester import ValidationSuggester
-
-        relationship_strategy = self._relationship_strategy or RelationshipStrategy.Pairwise
-        return (
-            self._model_suggester or ModelSuggester(self.config.model),
-            self._identification_suggester or IdentificationSuggester(self.config.model),
-            self._validation_suggester or ValidationSuggester(self.config.model),
-            relationship_strategy,
-        )
-
-    def _cache_path(
-        self,
-        nodes: Sequence[str],
-        dag_adj: np.ndarray,
-        treatment: str,
-        outcome: str,
-    ) -> str:
-        payload = {
-            "nodes": list(nodes),
-            "dag_adj": np.asarray(dag_adj, dtype=int).tolist(),
-            "treatment": treatment,
-            "outcome": outcome,
-            "model": self.config.model,
-        }
-        digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode("utf-8")).hexdigest()
-        return os.path.join(os.path.abspath(self.config.cache_dir), f"{digest}.json")
-
-    def _read_cache(self, path: str) -> Optional[CausalAssumptionReport]:
-        if not os.path.exists(path):
-            return None
-        try:
-            with open(path) as f:
-                return CausalAssumptionReport.from_dict(json.load(f))
-        except Exception:
-            return None
-
-    def _write_cache(self, path: str, report: CausalAssumptionReport) -> None:
-        try:
-            os.makedirs(os.path.dirname(path), exist_ok=True)
-            with open(path, "w") as f:
-                json.dump(report.to_dict(), f, indent=2)
-        except Exception:
-            pass
-
-    def _edges(self, nodes: Sequence[str], dag_adj: np.ndarray, max_edges: int) -> List[Tuple[str, str]]:
-        found: List[Tuple[str, str, float]] = []
-        for src_idx, src in enumerate(nodes):
-            for dst_idx, dst in enumerate(nodes):
-                if bool(dag_adj[src_idx, dst_idx]):
-                    found.append((src, dst, float(dag_adj[src_idx, dst_idx])))
-        found.sort(key=lambda edge: abs(edge[2]), reverse=True)
-        return [(src, dst) for src, dst, _ in found[:max_edges]]
-
-    def _classify_edges(
-        self,
-        edges: List[Tuple[str, str]],
-        edge_critiques: Any,
-    ) -> Tuple[List[Tuple[str, str]], List[Tuple[str, str]]]:
-        critique_text = str(edge_critiques).lower()
-        rejected: List[Tuple[str, str]] = []
-        for edge in edges:
-            src, dst = edge
-            edge_tokens = [
-                f"{src}->{dst}".lower(),
-                f"{src} -> {dst}".lower(),
-                f"{src}, {dst}".lower(),
-            ]
-            if any(token in critique_text for token in edge_tokens) and any(
-                marker in critique_text for marker in ["reject", "unlikely", "invalid", "implausible"]
-            ):
-                rejected.append(edge)
-        accepted = [edge for edge in edges if edge not in rejected]
-        return accepted, rejected

singular_ticker_causal/causal_inference/query_engine.py

@@ -1,505 +0,0 @@
-from __future__ import annotations
-
-import os
-import re
-from typing import Any, Dict, List, Optional, Sequence
-
-import numpy as np
-import pandas as pd
-
-from .abduction import counterfactual_predict, counterfactual_predict_multi
-from .causal_model import StructuralCausalModel
-from .estimator import estimate_ate
-from .identification import find_adjustment_set
-from .mutilator import mutilate_graph, propagate_intervention, target_series
-from .pywhyllm_assumptions import PyWhyLLMConfig, PyWhyLLMAssumptionService
-
-
-class CausalQueryEngine:
-    def __init__(
-        self,
-        scm: StructuralCausalModel,
-        data_tech: Optional[np.ndarray] = None,
-        data_text: Optional[np.ndarray] = None,
-        pywhyllm_service: Optional[PyWhyLLMAssumptionService] = None,
-        pywhyllm_enabled: bool = False,
-    ):
-        self.scm = scm
-        self.data_tech = data_tech
-        self.data_text = data_text
-        self.pywhyllm_enabled = pywhyllm_enabled
-        self.pywhyllm_service = pywhyllm_service
-
-    def assert_edge(self, treatment: str, outcome: str) -> Dict[str, object]:
-        strategy, z = find_adjustment_set(self.scm, treatment, outcome)
-        if strategy.value == "not_identifiable":
-            return {
-                "ate": 0.0,
-                "ci_95": (0.0, 0.0),
-                "strategy": strategy.name,
-                "adjustment_set": [],
-                "identifiable": False,
-                "path_contributions": {},
-            }
-
-        est = estimate_ate(
-            self.scm,
-            treatment=treatment,
-            outcome=outcome,
-            adjustment_set=z,
-            treatment_value=None,
-        )
-        return {
-            "ate": est["ate"],
-            "ci_95": est["ci_95"],
-            "strategy": strategy.name,
-            "adjustment_set": sorted(z or set()),
-            "identifiable": True,
-            "path_contributions": est["path_contributions"],
-        }
-
-    def intervene(self, treatment: str, value: float, targets: List[str], horizon: int = 5) -> Dict[str, object]:
-        mutilated = mutilate_graph(self.scm, treatment=treatment, value=value)
-        forecasts = propagate_intervention(
-            mutilated,
-            treatment=treatment,
-            value=value,
-            targets=targets,
-            horizon=horizon,
-            baseline_t=-1,
-        )
-        predicted = target_series(forecasts, self.scm, targets)
-        ate_per_target = {}
-        for target in targets:
-            tidx = self.scm.node_to_idx[target]
-            base = float(self.scm.data_level[-1, tidx])
-            ate_per_target[target] = float(forecasts[0, tidx] - base)
-        return {
-            "mutilated_adj": mutilated.dag_adj.copy(),
-            "predicted_values": predicted,
-            "ate_per_target": ate_per_target,
-            "horizon": horizon,
-        }
-
-    def _counterfactual_outcome(
-        self,
-        observed: Dict[str, float],
-        counterfactual_values: Dict[str, float],
-        target: str,
-    ) -> Dict[str, float]:
-        if len(counterfactual_values) == 1:
-            treatment, value = next(iter(counterfactual_values.items()))
-            return counterfactual_predict(
-                self.scm,
-                observed_values=observed,
-                treatment=treatment,
-                counterfactual_value=value,
-                target=target,
-            )
-        return counterfactual_predict_multi(
-            self.scm,
-            observed_values=observed,
-            counterfactual_values=counterfactual_values,
-            target=target,
-        )
-
-    def _shapley_contributions(
-        self,
-        observed: Dict[str, float],
-        interventions: Dict[str, float],
-        target: str,
-        mc_samples: int = 1000,
-        n_jobs: int = -1,
-    ) -> Dict[str, float]:
-        import random
-        from math import factorial
-        from joblib import Parallel, delayed
-        import os
-
-        treatments = list(interventions.keys())
-        n = len(treatments)
-        if n == 1:
-            return {treatments[0]: self._counterfactual_outcome(observed, interventions, target)["ite"]}
-
-        contributions: Dict[str, float] = {t: 0.0 for t in treatments}
-        
-        # Use exact if n <= 10, else Monte Carlo
-        use_exact = n <= 10
-        if n_jobs < 0:
-            n_jobs = os.cpu_count() or 4
-
-        def marginal_contribution(k: str, subset: list[str]) -> float:
-            with_k = {**{t: interventions[t] for t in subset}, k: interventions[k]}
-            without_k = {t: interventions[t] for t in subset}
-            v_with = self._counterfactual_outcome(observed, with_k, target)["counterfactual_outcome"]
-            v_without = self._counterfactual_outcome(observed, without_k, target)["counterfactual_outcome"]
-            return v_with - v_without
-
-        if use_exact:
-            all_factorial = float(factorial(n))
-            tasks = []
-            
-            for k in treatments:
-                others = [t for t in treatments if t != k]
-                for r in range(len(others) + 1):
-                    for subset in __import__("itertools").combinations(others, r):
-                        subset_list = list(subset)
-                        weight = float(factorial(len(subset_list)) * factorial(n - len(subset_list) - 1) / all_factorial)
-                        tasks.append((k, subset_list, weight))
-            
-            results = Parallel(n_jobs=n_jobs, backend="threading")(
-                delayed(marginal_contribution)(task[0], task[1]) for task in tasks
-            )
-            for task, res in zip(tasks, results):
-                k, _, weight = task
-                contributions[k] += weight * res
-        else:
-            # Monte Carlo approximation
-            tasks = []
-            for _ in range(mc_samples):
-                perm = treatments.copy()
-                random.shuffle(perm)
-                for i, k in enumerate(perm):
-                    subset_list = perm[:i]
-                    tasks.append((k, subset_list))
-                    
-            results = Parallel(n_jobs=n_jobs, backend="threading")(
-                delayed(marginal_contribution)(task[0], task[1]) for task in tasks
-            )
-            for task, res in zip(tasks, results):
-                k, _ = task
-                contributions[k] += res / mc_samples
-
-        return contributions
-
-    def _build_dowhy_graph(self) -> str:
-        edges = []
-        for src_idx, src in enumerate(self.scm.nodes):
-            for dst_idx, dst in enumerate(self.scm.nodes):
-                if self.scm.dag_adj[src_idx, dst_idx]:
-                    edges.append(f"{src} -> {dst}")
-        return "digraph{" + "; ".join(edges) + "}"
-
-    def _default_pywhyllm_service(self) -> PyWhyLLMAssumptionService:
-        enabled = self.pywhyllm_enabled or os.environ.get("PYWHYLLM_ENABLED", "").lower() in {
-            "1",
-            "true",
-            "yes",
-            "on",
-        }
-        return PyWhyLLMAssumptionService(
-            PyWhyLLMConfig(
-                enabled=enabled,
-                model=os.environ.get("PYWHYLLM_MODEL", "gpt-4"),
-                max_edges=int(os.environ.get("PYWHYLLM_MAX_EDGES", "25")),
-                cache_dir=os.environ.get(
-                    "PYWHYLLM_CACHE_DIR",
-                    PyWhyLLMConfig().cache_dir,
-                ),
-            )
-        )
-
-    def _get_pywhyllm_service(self) -> PyWhyLLMAssumptionService:
-        if self.pywhyllm_service is None:
-            self.pywhyllm_service = self._default_pywhyllm_service()
-        return self.pywhyllm_service
-
-    def analyze_assumptions_with_pywhyllm(
-        self,
-        treatment: str,
-        outcome: str,
-        max_edges: Optional[int] = None,
-    ) -> Dict[str, object]:
-        report = self._get_pywhyllm_service().analyze(
-            nodes=self.scm.nodes,
-            dag_adj=self.scm.dag_adj,
-            treatment=treatment,
-            outcome=outcome,
-            max_edges=max_edges,
-        )
-        return report.to_dict()
-
-    def _valid_nodes(self, candidates: Sequence[Any]) -> List[str]:
-        valid = set(self.scm.nodes)
-        result = []
-        for candidate in candidates or []:
-            node = str(candidate).strip()
-            if node in valid and node not in result:
-                result.append(node)
-        return result
-
-    def _valid_backdoor_sets(self, report: Dict[str, Any], treatment: str, outcome: str) -> List[List[str]]:
-        result = []
-        blocked = {treatment, outcome}
-        for suggested_set in report.get("suggested_backdoor_sets") or []:
-            valid_set = [node for node in self._valid_nodes(suggested_set) if node not in blocked]
-            if valid_set and valid_set not in result:
-                result.append(valid_set)
-        confounders = [node for node in self._valid_nodes(report.get("suggested_confounders") or []) if node not in blocked]
-        if confounders and confounders not in result:
-            result.append(confounders)
-        return result
-
-    def _coerce_causal_model(self, causal_model_cls: Any, data: pd.DataFrame, treatment: str, outcome: str, graph: str):
-        try:
-            return causal_model_cls(df=data, treatment=treatment, outcome=outcome, graph=graph)
-        except TypeError:
-            return causal_model_cls(data=data, treatment=treatment, outcome=outcome, graph=graph)
-
-    def _parse_p_value(self, value: Any) -> Optional[float]:
-        if value is None:
-            return None
-        if isinstance(value, (int, float, np.floating)):
-            return float(value)
-        if isinstance(value, (list, tuple)) and value:
-            return self._parse_p_value(value[0])
-        if isinstance(value, dict):
-            for key in ("p_value", "p-value", "p value"):
-                if key in value:
-                    return self._parse_p_value(value[key])
-            return None
-        match = re.search(r"p[-_ ]?value[^0-9<>=-]*[<>=: ]+\s*([0-9]*\.?[0-9]+)", str(value), re.I)
-        if match:
-            return float(match.group(1))
-        return None
-
-    def _as_optional_float(self, value: Any) -> Optional[float]:
-        if value is None:
-            return None
-        try:
-            arr = np.asarray(value, dtype=float)
-            if arr.size == 1:
-                return float(arr.reshape(-1)[0])
-        except Exception:
-            pass
-        try:
-            return float(value)
-        except Exception:
-            return None
-
-    def _parse_refuter_result(self, method: str, refute: Any, alpha: float = 0.05) -> Dict[str, object]:
-        text = str(refute)
-        result_attr = getattr(refute, "refutation_result", None)
-        estimated_effect = getattr(refute, "estimated_effect", None)
-        new_effect = getattr(refute, "new_effect", None)
-        p_value = self._parse_p_value(result_attr)
-        if p_value is None:
-            p_value = self._parse_p_value(text)
-        lower_text = text.lower()
-
-        if method == "placebo_treatment":
-            if p_value is not None:
-                falsified = p_value < alpha
-            elif "not statistically significant" in lower_text:
-                falsified = False
-            elif "statistically significant" in lower_text:
-                falsified = True
-            else:
-                falsified = False
-        else:
-            if p_value is not None:
-                falsified = p_value < alpha
-            elif "not statistically significant" in lower_text:
-                falsified = False
-            elif "statistically significant" in lower_text:
-                falsified = True
-            else:
-                falsified = False
-
-        return {
-            "method": method,
-            "refute": text,
-            "estimated_effect": self._as_optional_float(estimated_effect),
-            "new_effect": self._as_optional_float(new_effect),
-            "p_value": p_value,
-            "passed": not falsified,
-            "falsified": falsified,
-        }
-
-    def _run_dowhy_validation(
-        self,
-        causal_model_cls: Any,
-        treatment: str,
-        outcome: str,
-        treatment_value: float = 1.0,
-        adjustment_candidates: Optional[List[List[str]]] = None,
-        negative_controls: Optional[List[str]] = None,
-    ) -> Dict[str, object]:
-        data = pd.DataFrame(self.scm.data_level, columns=self.scm.nodes)
-        graph = self._build_dowhy_graph()
-        model = self._coerce_causal_model(causal_model_cls, data, treatment, outcome, graph)
-        identified_estimand = model.identify_effect()
-        estimate = model.estimate_effect(identified_estimand, method_name="backdoor.linear_regression")
-
-        refutations = []
-        for method in ["placebo_treatment", "random_common_cause"]:
-            try:
-                kwargs = {"method_name": method}
-                if method == "placebo_treatment":
-                    kwargs["placebo_type"] = "permute"
-                refute = model.refute_estimate(identified_estimand, estimate, **kwargs)
-                refutations.append(self._parse_refuter_result(method, refute))
-            except TypeError:
-                try:
-                    kwargs = {"method_name": method}
-                    if method == "placebo_treatment":
-                        kwargs["placebo_type"] = "permute"
-                    refute = model.refute_estimate(estimate, **kwargs)
-                    refutations.append(self._parse_refuter_result(method, refute))
-                except Exception as exc:
-                    refutations.append({"method": method, "error": str(exc), "passed": False, "falsified": False})
-            except Exception as exc:
-                refutations.append({"method": method, "error": str(exc), "passed": False, "falsified": False})
-
-        negative_control_checks = []
-        for control in negative_controls or []:
-            if control in {treatment, outcome}:
-                continue
-            try:
-                nc_model = self._coerce_causal_model(causal_model_cls, data, treatment, control, graph)
-                nc_identified = nc_model.identify_effect()
-                nc_estimate = nc_model.estimate_effect(nc_identified, method_name="backdoor.linear_regression")
-                negative_control_checks.append(
-                    {
-                        "control": control,
-                        "identified_estimand": str(nc_identified),
-                        "estimate": str(nc_estimate),
-                    }
-                )
-            except Exception as exc:
-                negative_control_checks.append({"control": control, "error": str(exc)})
-
-        falsified = any(bool(item.get("falsified")) for item in refutations)
-        return {
-            "available": True,
-            "falsified": falsified,
-            "identified_estimand": str(identified_estimand),
-            "estimate": str(estimate),
-            "refutations": refutations,
-            "adjustment_candidates": adjustment_candidates or [],
-            "negative_control_checks": negative_control_checks,
-        }
-
-    def validate_with_dowhy(
-        self,
-        treatment: str,
-        outcome: str,
-        treatment_value: float = 1.0,
-        num_placebo: int = 5,
-    ) -> Dict[str, object]:
-        try:
-            from dowhy import CausalModel
-        except Exception as exc:
-            return {
-                "available": False,
-                "reason": str(exc),
-                "falsified": False,
-                "summary": "DoWhy is not installed or failed to import.",
-            }
-
-        try:
-            return self._run_dowhy_validation(
-                CausalModel,
-                treatment=treatment,
-                outcome=outcome,
-                treatment_value=treatment_value,
-            )
-        except Exception as exc:
-            return {
-                "available": False,
-                "reason": str(exc),
-                "falsified": False,
-                "summary": "DoWhy refutation failed.",
-            }
-
-    def validate_with_pywhyllm_and_dowhy(
-        self,
-        treatment: str,
-        outcome: str,
-        treatment_value: float = 1.0,
-        max_edges: Optional[int] = None,
-    ) -> Dict[str, object]:
-        warnings: List[str] = []
-        pywhyllm_report = self.analyze_assumptions_with_pywhyllm(treatment, outcome, max_edges=max_edges)
-        adjustment_candidates = self._valid_backdoor_sets(pywhyllm_report, treatment, outcome)
-        negative_controls = [
-            node
-            for node in self._valid_nodes(pywhyllm_report.get("negative_controls") or [])
-            if node not in {treatment, outcome}
-        ]
-
-        try:
-            from dowhy import CausalModel
-        except Exception as exc:
-            return {
-                "pywhyllm": pywhyllm_report,
-                "dowhy": {
-                    "available": False,
-                    "reason": str(exc),
-                    "falsified": False,
-                    "summary": "DoWhy is not installed or failed to import.",
-                    "adjustment_candidates": adjustment_candidates,
-                    "negative_controls": negative_controls,
-                },
-                "falsified": False,
-                "warnings": warnings,
-            }
-
-        try:
-            dowhy_report = self._run_dowhy_validation(
-                CausalModel,
-                treatment=treatment,
-                outcome=outcome,
-                treatment_value=treatment_value,
-                adjustment_candidates=adjustment_candidates,
-                negative_controls=negative_controls,
-            )
-        except Exception as exc:
-            dowhy_report = {
-                "available": False,
-                "reason": str(exc),
-                "falsified": False,
-                "summary": "DoWhy refutation failed.",
-                "adjustment_candidates": adjustment_candidates,
-                "negative_controls": negative_controls,
-            }
-
-        warnings.extend(pywhyllm_report.get("warnings") or [])
-        return {
-            "pywhyllm": pywhyllm_report,
-            "dowhy": dowhy_report,
-            "falsified": bool(dowhy_report.get("falsified")),
-            "warnings": warnings,
-        }
-
-    def counterfactual(
-        self,
-        observed_t: int,
-        treatment: Optional[str] = None,
-        cf_value: Optional[float] = None,
-        target: str = "",
-        treatments: Optional[Dict[str, float]] = None,
-    ) -> Dict[str, object]:
-        t = observed_t if observed_t >= 0 else (self.scm.t_steps + observed_t)
-        if t < 0 or t >= self.scm.t_steps:
-            raise ValueError(f"observed_t {observed_t} resolves out of bounds for T={self.scm.t_steps}")
-
-        if treatments is None:
-            if treatment is None or cf_value is None:
-                raise ValueError("Either treatment/cf_value or treatments must be provided.")
-            treatments = {treatment: cf_value}
-        elif treatment is not None or cf_value is not None:
-            raise ValueError("Provide either treatment/cf_value or treatments, not both.")
-
-        observed = {node: float(self.scm.data_level[t, i]) for i, node in enumerate(self.scm.nodes)}
-        result = self._counterfactual_outcome(observed, treatments, target)
-        result["explanation"] = (
-            f"Counterfactual computed at t={t}: set {treatments} "
-            f"and propagated structural equations with abducted residuals."
-        )
-        if len(treatments) > 1:
-            result["shapley_contributions"] = self._shapley_contributions(observed, treatments, target)
-        else:
-            result["shapley_contributions"] = {next(iter(treatments)): result["ite"]}
-        result["ite_total"] = result["ite"]
-        return result

singular_ticker_causal/causal_inference/tests/test_causal_queries.py

@@ -1,102 +0,0 @@
-import importlib.util
-import numpy as np
-
-from causal_inference.causal_model import StructuralCausalModel
-from causal_inference.query_engine import CausalQueryEngine
-
-
-def _synthetic_inputs():
-    nodes = ["A", "B", "C"]
-    T = 40
-    data = np.zeros((T, 3, 1), dtype=float)
-    rng = np.random.default_rng(42)
-
-    a = rng.normal(0, 1, size=T)
-    b = 0.6 * np.roll(a, 1) + rng.normal(0, 0.1, size=T)
-    c = 0.7 * np.roll(b, 1) + rng.normal(0, 0.1, size=T)
-    b[0] = rng.normal()
-    c[0] = rng.normal()
-    data[:, 0, 0] = a
-    data[:, 1, 0] = b
-    data[:, 2, 0] = c
-
-    # A->B, B->C and a weak cycle C->A to exercise pruning.
-    adj = np.array(
-        [
-            [0.0, 0.9, 0.0],
-            [0.0, 0.0, 0.8],
-            [0.1, 0.0, 0.0],
-        ],
-        dtype=float,
-    )
-    prior = np.zeros((3, 3), dtype=float)
-    mask = np.ones((T, 3), dtype=float)
-    return nodes, adj, prior, data, mask
-
-
-def test_scm_fit_and_cycle_prune():
-    nodes, adj, prior, data, mask = _synthetic_inputs()
-    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask).fit()
-    assert scm.dag_adj.shape == (3, 3)
-    assert len(scm.topological_indices) == 3
-    assert np.sum(scm.dag_adj) <= 2
-
-
-def test_intervention_and_counterfactual_shapes():
-    nodes, adj, prior, data, mask = _synthetic_inputs()
-    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask).fit()
-    engine = CausalQueryEngine(scm)
-
-    inter = engine.intervene("A", value=0.5, targets=["B", "C"], horizon=3)
-    assert set(inter["predicted_values"].keys()) == {"B", "C"}
-    assert len(inter["predicted_values"]["B"]) == 3
-
-    cf = engine.counterfactual(observed_t=-1, treatment="A", cf_value=0.8, target="C")
-    assert "factual_outcome" in cf
-    assert "counterfactual_outcome" in cf
-    assert "ite" in cf
-
-
-def test_counterfactual_shapley_contributions():
-    nodes, adj, prior, data, mask = _synthetic_inputs()
-    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask).fit()
-    engine = CausalQueryEngine(scm)
-
-    cf = engine.counterfactual(
-        observed_t=-1,
-        treatments={"A": 0.8, "B": -0.3},
-        target="C",
-    )
-
-    assert "shapley_contributions" in cf
-    assert set(cf["shapley_contributions"].keys()) == {"A", "B"}
-    assert abs(sum(cf["shapley_contributions"].values()) - cf["ite"]) < 1e-6
-
-
-def test_dowhy_validation_is_optional():
-    nodes, adj, prior, data, mask = _synthetic_inputs()
-    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask).fit()
-    engine = CausalQueryEngine(scm)
-
-    result = engine.validate_with_dowhy(treatment="A", outcome="C")
-    assert "available" in result
-    if importlib.util.find_spec("dowhy") is None:
-        assert result["available"] is False
-    else:
-        assert "falsified" in result
-
-
-def test_exogenous_fallback_prevention():
-    nodes, adj, prior, data, mask = _synthetic_inputs()
-    # Force fallback to exogenous by setting min_obs to a very large number (e.g. 100)
-    scm = StructuralCausalModel(nodes=nodes, adj=adj, adjacency_mask=prior, prohibition_mask=prior, data_tech=data, mask_tech=mask, min_obs=100).fit()
-    engine = CausalQueryEngine(scm)
-
-    inter = engine.intervene("A", value=0.5, targets=["B", "C"], horizon=3)
-    assert set(inter["predicted_values"].keys()) == {"B", "C"}
-    assert len(inter["predicted_values"]["B"]) == 3
-
-    cf = engine.counterfactual(observed_t=-1, treatment="A", cf_value=0.8, target="C")
-    assert "factual_outcome" in cf
-    assert "counterfactual_outcome" in cf
-    assert "ite" in cf

singular_ticker_causal/causal_inference/tests/test_pywhyllm_assumptions.py

@@ -1,193 +0,0 @@
-import builtins
-import sys
-import types
-
-import numpy as np
-
-from singular_ticker_causal.causal_inference.causal_model import StructuralCausalModel
-from singular_ticker_causal.causal_inference.pywhyllm_assumptions import (
-    PyWhyLLMConfig,
-    PyWhyLLMAssumptionService,
-)
-from singular_ticker_causal.causal_inference.query_engine import CausalQueryEngine
-
-
-def _synthetic_scm():
-    nodes = ["A", "B", "C"]
-    data = np.zeros((24, 3, 1), dtype=float)
-    rng = np.random.default_rng(11)
-    data[:, 0, 0] = rng.normal(size=24)
-    data[:, 1, 0] = rng.normal(size=24)
-    data[:, 2, 0] = rng.normal(size=24)
-    adj = np.array(
-        [
-            [0.0, 0.8, 0.7],
-            [0.0, 0.0, 0.9],
-            [0.0, 0.0, 0.0],
-        ],
-        dtype=float,
-    )
-    prior = np.ones((3, 3), dtype=float)
-    np.fill_diagonal(prior, 0.0)
-    return StructuralCausalModel(
-        nodes=nodes,
-        adj=adj,
-        adjacency_mask=np.zeros((3, 3), dtype=float),
-        prohibition_mask=prior,
-        data_tech=data,
-    ).fit()
-
-
-class FakeModelSuggester:
-    def suggest_domain_expertises(self, all_factors):
-        return ["financial accounting"]
-
-    def suggest_confounders(self, treatment, outcome, all_factors, domain_expertises):
-        return ({("B", treatment): 1, ("B", outcome): 1}, ["B", "Missing_Node"])
-
-    def suggest_relationships(self, treatment, outcome, all_factors, domain_expertises, strategy):
-        return [(treatment, outcome), ("B", outcome)]
-
-
-class FakeIdentificationSuggester:
-    def suggest_backdoor(self, treatment, outcome, all_factors, domain_expertises):
-        return ["B", "Missing_Node"]
-
-    def suggest_mediators(self, treatment, outcome, all_factors, domain_expertises):
-        return ["B"]
-
-    def suggest_ivs(self, treatment, outcome, all_factors, domain_expertises):
-        return ["A"]
-
-
-class FakeValidationSuggester:
-    def critique_graph(self, all_factors, suggested_dag, domain_expertises, strategy):
-        return "A -> C accepted; B -> C implausible, reject"
-
-    def suggest_latent_confounders(self, treatment, outcome, all_factors, domain_expertises):
-        return ["market regime"]
-
-    def suggest_negative_controls(self, treatment, outcome, all_factors, domain_expertises):
-        return ["B", "Missing_Node"]
-
-
-def _fake_service(tmp_path):
-    return PyWhyLLMAssumptionService(
-        PyWhyLLMConfig(enabled=True, cache_dir=str(tmp_path)),
-        model_suggester=FakeModelSuggester(),
-        identification_suggester=FakeIdentificationSuggester(),
-        validation_suggester=FakeValidationSuggester(),
-        relationship_strategy="pairwise",
-    )
-
-
-def test_pywhyllm_assumption_service_uses_fakes_and_cache(tmp_path):
-    scm = _synthetic_scm()
-    service = _fake_service(tmp_path)
-
-    report = service.analyze(nodes=scm.nodes, dag_adj=scm.dag_adj, treatment="A", outcome="C")
-
-    assert report.available is True
-    assert report.domain_expertises == ["financial accounting"]
-    assert report.suggested_confounders == ["B", "Missing_Node"]
-    assert report.suggested_backdoor_sets == [["B", "Missing_Node"]]
-    assert report.negative_controls == ["B", "Missing_Node"]
-    assert ("B", "C") in report.rejected_edges
-
-    cached = service.analyze(nodes=scm.nodes, dag_adj=scm.dag_adj, treatment="A", outcome="C")
-    assert cached.to_dict() == report.to_dict()
-
-
-def test_missing_pywhyllm_returns_unavailable(monkeypatch, tmp_path):
-    original_import = builtins.__import__
-
-    def fake_import(name, *args, **kwargs):
-        if name.startswith("pywhyllm"):
-            raise ImportError("blocked pywhyllm")
-        return original_import(name, *args, **kwargs)
-
-    monkeypatch.setattr(builtins, "__import__", fake_import)
-    scm = _synthetic_scm()
-    service = PyWhyLLMAssumptionService(PyWhyLLMConfig(enabled=True, cache_dir=str(tmp_path)))
-
-    report = service.analyze(nodes=scm.nodes, dag_adj=scm.dag_adj, treatment="A", outcome="C")
-
-    assert report.available is False
-    assert "blocked pywhyllm" in report.reason
-
-
-def test_engine_filters_pywhyllm_adjustments_and_negative_controls(tmp_path):
-    scm = _synthetic_scm()
-    engine = CausalQueryEngine(scm, pywhyllm_service=_fake_service(tmp_path), pywhyllm_enabled=True)
-
-    report = engine.analyze_assumptions_with_pywhyllm("A", "C")
-    adjustment_sets = engine._valid_backdoor_sets(report, "A", "C")
-    negative_controls = engine._valid_nodes(report["negative_controls"])
-
-    assert adjustment_sets == [["B"]]
-    assert negative_controls == ["B"]
-
-
-def test_placebo_not_statistically_significant_passes():
-    scm = _synthetic_scm()
-    engine = CausalQueryEngine(scm)
-
-    parsed = engine._parse_refuter_result(
-        "placebo_treatment",
-        "Refute: Use a Placebo Treatment. The result is not statistically significant.",
-    )
-
-    assert parsed["passed"] is True
-    assert parsed["falsified"] is False
-
-
-def test_combined_validation_reports_unavailable_dowhy(monkeypatch, tmp_path):
-    monkeypatch.setitem(sys.modules, "dowhy", None)
-    scm = _synthetic_scm()
-    engine = CausalQueryEngine(scm, pywhyllm_service=_fake_service(tmp_path), pywhyllm_enabled=True)
-
-    result = engine.validate_with_pywhyllm_and_dowhy("A", "C")
-
-    assert result["pywhyllm"]["available"] is True
-    assert result["dowhy"]["available"] is False
-    assert result["dowhy"]["adjustment_candidates"] == [["B"]]
-
-
-def test_combined_validation_uses_fake_dowhy(monkeypatch, tmp_path):
-    class FakeRefute:
-        estimated_effect = 1.0
-        new_effect = 0.0
-        refutation_result = {"p_value": 0.8}
-
-        def __str__(self):
-            return "not statistically significant"
-
-    class FakeEstimate:
-        def __str__(self):
-            return "estimate"
-
-    class FakeCausalModel:
-        def __init__(self, **kwargs):
-            self.kwargs = kwargs
-
-        def identify_effect(self):
-            return "estimand"
-
-        def estimate_effect(self, identified_estimand, method_name):
-            return FakeEstimate()
-
-        def refute_estimate(self, identified_estimand, estimate, **kwargs):
-            return FakeRefute()
-
-    fake_dowhy = types.ModuleType("dowhy")
-    fake_dowhy.CausalModel = FakeCausalModel
-    monkeypatch.setitem(sys.modules, "dowhy", fake_dowhy)
-
-    scm = _synthetic_scm()
-    engine = CausalQueryEngine(scm, pywhyllm_service=_fake_service(tmp_path), pywhyllm_enabled=True)
-
-    result = engine.validate_with_pywhyllm_and_dowhy("A", "C")
-
-    assert result["dowhy"]["available"] is True
-    assert result["dowhy"]["falsified"] is False
-    assert result["dowhy"]["negative_control_checks"][0]["control"] == "B"

singular_ticker_causal/data_sources/__init__.py

@@ -1,4 +0,0 @@
-from .fetcher import Fetcher
-from .gdelt_client import GDELTClient
-from .news_client import NewsClient
-from .sebi_reg30_client import SEBIREG30Client

singular_ticker_causal/data_sources/bsedata/__init__.py

@@ -1,27 +0,0 @@
-"""
-
-    MIT License
-
-    Copyright (c) 2018 - 2024 Shrey Dabhi
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-
-name = "bsedata"

singular_ticker_causal/data_sources/bsedata/bhavcopy.py

@@ -1,58 +0,0 @@
-import os
-import io
-import csv
-import requests
-import tempfile
-import datetime
-from zipfile import ZipFile
-from .exceptions import BhavCopyNotFound
-from .helpers import COMMON_REQUEST_HEADERS
-
-
-def loadBhavCopyData(statsDate: datetime.date) -> list:
-    tempDir = os.path.join(tempfile.gettempdir(), "bsedata")
-    zipfileName = f"EQ{statsDate.strftime('%d%m%y')}_CSV.ZIP"
-    r = requests.get(
-        f"https://www.bseindia.com/download/BhavCopy/Equity/{zipfileName}",
-        headers=COMMON_REQUEST_HEADERS,
-    )
-
-    if r.status_code != 200:
-        raise BhavCopyNotFound()
-
-    try:
-        os.makedirs(tempDir)
-    except FileExistsError:
-        pass
-
-    f_zip = open(os.path.join(tempDir, zipfileName), "wb+")
-    f_zip.write(r.content)
-    f_zip.close()
-
-    output = []
-
-    with ZipFile(os.path.join(tempDir, zipfileName)) as bhavCopyZip:
-        with bhavCopyZip.open(f"EQ{statsDate.strftime('%d%m%y')}.CSV") as bhavCopyFile:
-            reader = csv.DictReader(io.TextIOWrapper(bhavCopyFile))
-            for row in reader:
-                output.append(mapBhavCopyRowToDict(row))
-
-    return output
-
-
-def mapBhavCopyRowToDict(row: dict) -> dict:
-    SC_TYPE_MAP = {"B": "bond", "Q": "equity", "D": "debenture", "P": "preference"}
-    return {
-        "scripCode": row["SC_CODE"],
-        "open": row["OPEN"],
-        "high": row["HIGH"],
-        "low": row["LOW"],
-        "close": row["CLOSE"],
-        "last": row["LAST"],
-        "prevClose": row["PREVCLOSE"],
-        "totalTrades": row["NO_TRADES"],
-        "totalSharesTraded": row["NO_OF_SHRS"],
-        "netTurnover": row["NET_TURNOV"],
-        "scripType": SC_TYPE_MAP[row["SC_TYPE"]],
-        "securityID": row["SC_NAME"].strip(),
-    }

singular_ticker_causal/data_sources/bsedata/bse.py

@@ -1,150 +0,0 @@
-"""
-
-    MIT License
-
-    Copyright (c) 2018 - 2024 Shrey Dabhi
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-
-from .gainers import getGainers
-from .losers import getLosers
-from .bhavcopy import loadBhavCopyData
-from .quote import quote
-from .indices import indices
-import datetime
-import requests
-import json
-
-
-class BSE(object):
-    """
-    Class which implements the functionality for
-    Bombay Stock Exchange (BSE)
-    """
-
-    def __init__(self, update_codes=False):
-        self.__update_codes = update_codes
-        if update_codes:
-            self.updateScripCodes()
-
-    def topGainers(self):
-        """
-        :returns: A sorted list of codes of top gainers
-        """
-        return getGainers()
-
-    def topLosers(self):
-        """
-        :returns: A sorted list of codes of top losers
-        """
-        return getLosers()
-
-    def getQuote(self, scripCode):
-        """
-        :param scripCode: A stock code
-        :returns: A dictionary which contain details about the stock
-        :raises InvalidStockException: Raised for stocks which have been suspended or no longer trading on BSE
-        """
-        return quote(scripCode)
-
-    def getIndices(self, category):
-        """
-        :param category: A category of indices
-        :returns: A dictionary with details about the indices belonging to the given category
-        """
-        return indices(category)
-
-    def updateScripCodes(self):
-        """
-        Download a fresh copy of the scrip code listing
-
-        :returns: None
-        """
-        r = requests.get("https://pub-87b187a07d9c42109c9e6999439a583f.r2.dev/stk.json")
-        f_stk = open("stk.json", "w+")
-        f_stk.write(json.dumps(r.json()))
-        f_stk.close()
-        return
-    
-    def getBhavCopyData(self, statsDate: datetime.date):
-        """
-        Get historical OHLCV data from Bhav Copy released by BSE everyday after market closing.
-        The columns available in the data and their description is as given below.
-
-        .. list-table::
-            :widths: 25 75
-            :header-rows: 1
-
-            * - Dictionary Field
-              - Description
-            * - scripCode
-              - Unique code assigned to a scrip of a company by BSE
-            * - open
-              - The price at which the security first trades on a given trading day
-            * - high
-              - The highest intra-day price of a stock
-            * - low
-              - The lowest intra-day price of a stock
-            * - close
-              - The final price at which a security is traded on a given trading day
-            * - last
-              - The last trade price of the stock
-            * - prevClose
-              - The closing price of the stock for the previous trading day
-            * - totalTrades
-              - The total number of trades of a scrip
-            * - totalSharesTraded
-              - The total number of shares transacted of a scrip
-            * - netTurnover
-              - Total turnover of a scrip
-            * - scripType
-              - Scrip category: Equity, Preference, Debenture or Bond
-            * - securityID
-              - Name of the company
-
-        The Bhav Copy files have been mapped to the above mentioned custom fields. The complete documentation for Bhav Copy can be found here: https://www.bseindia.com/markets/MarketInfo/BhavCopy.aspx.
-
-        
-        :param statsDate: A `datetime.date` object for the for which you want to fetch the data
-        :returns: A list of dictionaries which contains OHLCV data for that day for all scrip codes active on that day
-        :raises BhavCopyNotFound: Raised when Bhav Copy file is not found on BSE
-        """
-        return loadBhavCopyData(statsDate)
-
-    def getScripCodes(self):
-        """
-        :returns: A dictionary with scrip codes as keys and company names as values
-        """
-        f = open("stk.json", "r")
-        return json.loads(f.read())
-
-    def verifyScripCode(self, code):
-        """
-        :returns: Company name if it is a valid stock code, else None
-        """
-        data = self.getScripCodes()
-        return data.get(code)
-
-    def __str__(self):
-        return "Driver Class for Bombay Stock Exchange (BSE)"
-
-    def __repr__(self):
-        return f"<{self.__class__.__name__}: update_codes={self.__update_codes}> Driver Class for Bombay Stock Exchange (BSE)"

singular_ticker_causal/data_sources/bsedata/exceptions.py

@@ -1,51 +0,0 @@
-"""
-
-    MIT License
-
-    Copyright (c) 2018 - 2024 Shrey Dabhi
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-
-
-class InvalidStockException(Exception):
-    """
-    Exception raised for stocks which have been suspended or no longer trading on BSE.
-
-    :param status: the status of the stock as mentioned on BSE website
-    """
-
-    def __init__(self, status: str = "Inactive stock"):
-        if status == "":
-            self.status = "Inactive stock"
-        else:
-            self.status = status
-        super().__init__(self.status)
-
-
-class BhavCopyNotFound(Exception):
-    """
-    Exception raised when the BhavCopy file is not found on BSE website.
-    """
-
-    def __init__(self):
-        super().__init__(
-            """The BhavCopy file was not found on the BSE website. You are probably trying to get data for a trading holiday."""
-        )

singular_ticker_causal/data_sources/bsedata/gainers.py

@@ -1,57 +0,0 @@
-"""
-
-    MIT License
-
-    Copyright (c) 2018 - 2024 Shrey Dabhi
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-
-from .helpers import COMMON_REQUEST_HEADERS
-from bs4 import BeautifulSoup as bs
-import requests
-
-
-def getGainers() -> dict:
-    baseurl = """https://m.bseindia.com"""
-    res = requests.get(baseurl, headers=COMMON_REQUEST_HEADERS)
-    c = res.content
-    soup = bs(c, "lxml")
-    for tag in soup("div"):
-        try:
-            if tag["id"] == "divGainers":
-                resSoup = tag
-                break
-        except KeyError:
-            continue
-    children = list(resSoup.table.contents)
-    children = children[1:]
-    gainers = []
-    for tr in children:
-        td = tr.contents
-        gainer = {
-            "securityID": str(td[0].a.string),
-            "scripCode": str(tr.td.a["href"].split("=")[1]),
-            "LTP": str(td[1].string),
-            "change": str(td[2].string),
-            "pChange": str(td[3].string),
-        }
-        gainers.append(gainer)
-    return gainers

singular_ticker_causal/data_sources/bsedata/helpers.py

@@ -1,3 +0,0 @@
-COMMON_REQUEST_HEADERS = {
-    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36 Edg/83.0.478.45"
-}

singular_ticker_causal/data_sources/bsedata/indices.py

@@ -1,112 +0,0 @@
-"""
-
-    MIT License
-
-    Copyright (c) 2018 - 2024 Shrey Dabhi
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-
-from .helpers import COMMON_REQUEST_HEADERS
-from bs4 import BeautifulSoup as bs
-import requests
-
-
-def indices(category: str) -> dict:
-    cat = {
-        "market_cap/broad": "1,2",
-        "sector_and_industry": "2,2",
-        "thematics": "3,2",
-        "strategy": "4,2",
-        "sustainability": "5,2",
-        "volatility": "6,1",
-        "composite": "7,1",
-        "government": "8,1",
-        "corporate": "9,1",
-        "money_market": "10,1",
-    }
-    try:
-        ddl_category = cat[category]
-    except KeyError:
-        print(
-            """
-### Invalid category ###
-Use one of the categories mentioned below:
-
-market_cap/broad
-sector_and_industry
-thematics
-strategy
-sustainability
-volatility
-composite
-government
-corporate
-money_market
-        """
-        )
-        return
-    baseurl = """https://m.bseindia.com/IndicesView_New.aspx"""
-    res = requests.get(baseurl, headers=COMMON_REQUEST_HEADERS)
-    c = res.content
-    soup = bs(c, "lxml")
-    options = {
-        "__EVENTTARGET": "ddl_Category",
-        "__VIEWSTATEENCRYPTED": "",
-        "__EVENTARGUMENT": "",
-        "__LASTFOCUS": "",
-        "__VIEWSTATEGENERATOR": "162C96CD",
-        "UcHeaderMenu1$txtGetQuote": "",
-        "__EVENTVALIDATION": "",
-        "__VIEWSTATE": "",
-    }
-    for input in soup("input"):
-        try:
-            if input["type"] == "hidden":
-                if input["id"] == "__VIEWSTATE":
-                    options["__VIEWSTATE"] = input["value"]
-                elif input["id"] == "__EVENTVALIDATION":
-                    options["__EVENTVALIDATION"] = input["value"]
-        except KeyError:
-            continue
-    options["ddl_Category"] = ddl_category
-    res = requests.post(url=baseurl, data=options, headers=COMMON_REQUEST_HEADERS)
-    c = res.content
-    soup = bs(c, "lxml")
-    index_list = []
-    for td in soup("td"):
-        try:
-            if td["class"][0] == "TTRow_left":
-                index = {}
-                index["currentValue"] = td.next_sibling.string.strip()
-                index["change"] = td.next_sibling.next_sibling.string.strip()
-                index[
-                    "pChange"
-                ] = td.next_sibling.next_sibling.next_sibling.string.strip()
-                index["scripFlag"] = td.a["href"].strip().split("=")[1]
-                index["name"] = td.a.string.strip().replace(";", "")
-                index_list.append(index)
-        except KeyError:
-            continue
-    results = {}
-    for span in soup("span", id="inddate"):
-        results["updatedOn"] = span.string[6:].split("|")[0].strip()
-    results["indices"] = index_list
-    return results

singular_ticker_causal/data_sources/bsedata/losers.py

@@ -1,57 +0,0 @@
-"""
-
-    MIT License
-
-    Copyright (c) 2018 - 2024 Shrey Dabhi
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-
-from .helpers import COMMON_REQUEST_HEADERS
-from bs4 import BeautifulSoup as bs
-import requests
-
-
-def getLosers() -> dict:
-    baseurl = """https://m.bseindia.com"""
-    res = requests.get(baseurl, headers=COMMON_REQUEST_HEADERS)
-    c = res.content
-    soup = bs(c, "lxml")
-    for tag in soup("div"):
-        try:
-            if tag["id"] == "divLosers":
-                resSoup = tag
-                break
-        except KeyError:
-            continue
-    children = list(resSoup.table.contents)
-    children = children[1:]
-    losers = []
-    for tr in children:
-        td = tr.contents
-        loser = {
-            "securityID": str(td[0].a.string),
-            "scripCode": str(tr.td.a["href"].split("=")[1]),
-            "LTP": str(td[1].string),
-            "change": str(td[2].string),
-            "pChange": str(td[3].string),
-        }
-        losers.append(loser)
-    return losers

singular_ticker_causal/data_sources/bsedata/quote.py

@@ -1,176 +0,0 @@
-"""
-
-    MIT License
-
-    Copyright (c) 2018 - 2024 Shrey Dabhi
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-
-from .exceptions import InvalidStockException
-from .helpers import COMMON_REQUEST_HEADERS
-from datetime import datetime as dt
-from bs4 import BeautifulSoup as bs
-import requests
-
-
-def quote(scripCode: str) -> dict:
-    baseurl = """https://m.bseindia.com/StockReach.aspx?scripcd="""
-    res = requests.get(baseurl + scripCode, headers=COMMON_REQUEST_HEADERS)
-    c = res.content
-    soup = bs(c, "lxml")
-
-    res = {}
-
-    for span in soup("span"):
-        updt_date = soup.find("span", id="strongDate").text.split("-")[1].strip()
-        updt_diff = dt.strptime(updt_date, "%d %b %y | %I:%M %p") - dt.now()
-        if updt_diff.days < -7:
-            error_text = ""
-            error_text_element = soup.find("td", id="tdDispTxt")
-            if error_text_element is not None:
-                error_text = error_text_element.text
-            raise InvalidStockException(status=error_text)
-        try:
-            if span["class"][0] == "srcovalue":
-                try:
-                    if span["id"] == "spanchangVal":
-                        res["change"] = span.string.split("(")[0].strip()
-                        res["pChange"] = span.string.split("(")[1].strip()[:-2]
-                except KeyError:
-                    res["currentValue"] = span.strong.string
-            elif span["class"][0] == "companyname":
-                res["companyName"] = span.string
-        except KeyError:
-            try:
-                if span["id"] == "lblPBdate":
-                    try:
-                        res["priceBand"] = span.string.split(":")[1].strip()
-                    except AttributeError:
-                        res["priceBand"] = ""
-                elif span["id"] == "strongDate":
-                    res["updatedOn"] = span.string.split("-")[1].strip()
-            except KeyError:
-                continue
-
-    for td in soup("td"):
-        try:
-            if td["id"] == "tdCShortName":
-                res["securityID"] = td.string.strip()
-            elif td["id"] == "tdscripcode":
-                res["scripCode"] = td.string.strip()
-            elif td["id"] == "tdgroup":
-                res["group"] = td.string.strip()
-            elif td["id"] == "tdfacevalue":
-                res["faceValue"] = td.string.strip()
-            elif td["id"] == "tdIndustry":
-                res["industry"] = td.string.strip()
-            elif td["id"] == "tdpcloseopen":
-                res["previousClose"] = td.string.split("/")[0].strip()
-                res["previousOpen"] = td.string.split("/")[1].strip()
-            elif td["id"] == "tdDHL":
-                res["dayHigh"] = td.string.split("/")[0].strip()
-                res["dayLow"] = td.string.split("/")[1].strip()
-            elif td["id"] == "td52WHL":
-                res["52weekHigh"] = td.string.split("/")[0].strip()
-                res["52weekLow"] = td.string.split("/")[1].strip()
-            elif td["id"] == "tdWAp":
-                res["weightedAvgPrice"] = td.string.strip()
-            elif td["id"] == "tdTTV":
-                res["totalTradedValue"] = td.string.strip() + " Cr."
-            elif td["id"] == "tdTTQW":
-                res["totalTradedQuantity"] = td.string.split("/")[0].strip() + " Lakh"
-                res["2WeekAvgQuantity"] = td.string.split("/")[1].strip() + " Lakh"
-            elif td["id"] == "tdMktCapVal":
-                res["marketCapFull"] = td.string.split("/")[0].strip() + " Cr."
-                res["marketCapFreeFloat"] = td.string.split("/")[1].strip() + " Cr."
-        except KeyError:
-            continue
-
-    if res.get("priceBand", "") != "":
-        for tbody in soup("tbody"):
-            try:
-                if tbody["id"] == "PBtablebody":
-                    data = tbody.contents[2]
-                    res["upperPriceBand"] = data.contents[1].string.strip()
-                    res["lowerPriceBand"] = data.contents[2].string.strip()
-            except KeyError:
-                continue
-
-    buy = {}
-    sell = {}
-    for td in soup("td"):
-        try:
-            if td["id"] == "tdBQ1":
-                buy["1"] = {
-                    "quantity": td.string,
-                    "price": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdBQ2":
-                buy["2"] = {
-                    "quantity": td.string,
-                    "price": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdBQ3":
-                buy["3"] = {
-                    "quantity": td.string,
-                    "price": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdBQ4":
-                buy["4"] = {
-                    "quantity": td.string,
-                    "price": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdBQ5":
-                buy["5"] = {
-                    "quantity": td.string,
-                    "price": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdSP1":
-                sell["1"] = {
-                    "price": td.string,
-                    "quantity": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdSP2":
-                sell["2"] = {
-                    "price": td.string,
-                    "quantity": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdSP3":
-                sell["3"] = {
-                    "price": td.string,
-                    "quantity": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdSP4":
-                sell["4"] = {
-                    "price": td.string,
-                    "quantity": td.next_sibling.next_sibling.string,
-                }
-            elif td["id"] == "tdSP5":
-                sell["5"] = {
-                    "price": td.string,
-                    "quantity": td.next_sibling.next_sibling.string,
-                }
-        except KeyError:
-            continue
-    res["buy"] = buy
-    res["sell"] = sell
-
-    return res

singular_ticker_causal/data_sources/fetcher.py

@@ -1,219 +0,0 @@
-import pandas as pd
-import numpy as np
-import yfinance as yf
-import logging
-from typing import Optional
-from singular_ticker_causal.services.schema import INCOME_STATEMENT_NODES, BALANCE_SHEET_NODES, STRATEGIC_NODES
-IND_AS_NODES = INCOME_STATEMENT_NODES + BALANCE_SHEET_NODES + STRATEGIC_NODES
-from .nseconnect.nse import Nse
-from .bsedata.bse import BSE
-
-
-logger = logging.getLogger(__name__)
-
-
-class Fetcher:
-    """
-    Tiered fundamental data fetcher.
-    Priority: NSE/BSE → XBRL → IndianAPI → yfinance.
-    """
-    def __init__(self, ticker: str, api_key: Optional[str] = None):
-        self.ticker = ticker if ticker.endswith(".NS") or ticker.endswith(".BO") else f"{ticker}.NS"
-        self.api_key = api_key
-
-    def fetch(self, start: str, end: str) -> pd.DataFrame:
-        logger.info(f"Fetching fundamentals for {self.ticker} from {start} to {end}")
-        
-        df = None
-        if self.ticker.endswith(".NS"):
-            logger.info("Attempting NSE fetch...")
-            df = self._try_nse(start, end)
-        elif self.ticker.endswith(".BO"):
-            logger.info("Attempting BSE fetch...")
-            df = self._try_bse(start, end)
-
-        if df is None or df.empty:
-            logger.info("Attempting Tier 1: XBRL...")
-            df = self._try_xbrl(start, end)
-        if df is None or df.empty:
-            logger.info("Tier 1 failed or returned empty. Attempting Tier 2: IndianAPI...")
-            df = self._try_indianapi(start, end)
-        if df is None or df.empty:
-            logger.info("Tier 2 failed or returned empty. Attempting Tier 3: yfinance...")
-            df = self._try_yfinance(start, end)
-        
-        if df is not None and not df.empty:
-            df = df.sort_index()
-            # Forward fill balance sheet items as they are point-in-time and usually stable
-            # We do this AFTER mapping in _try_yfinance, so we use mapped names
-            bs_cols = ["Total_Assets", "Shareholders_Equity", "Inventory", "Accounts_Payable", "Total_Debt", "PPE", "CWIP", "Intangible_Assets"]
-            available_bs_cols = [c for c in bs_cols if c in df.columns]
-            if available_bs_cols:
-                df[available_bs_cols] = df[available_bs_cols].ffill()
-        
-        if df is None or df.empty:
-            logger.warning(f"No fundamental data found for {self.ticker}")
-            return pd.DataFrame(columns=IND_AS_NODES)
-
-        logger.info("Deriving strategic nodes...")
-        df = self._derive_strategic_nodes(df)
-        
-        # Ensure all IND_AS_NODES are present
-        for node in IND_AS_NODES:
-            if node not in df.columns:
-                df[node] = np.nan
-                
-        return df[IND_AS_NODES].sort_index()
-
-    def _try_nse(self, start: str, end: str) -> Optional[pd.DataFrame]:
-        # Using nseconnect for high-fidelity NSE data
-        try:
-            # Clean ticker (e.g. RELIANCE.NS -> RELIANCE)
-            clean_ticker = self.ticker.split('.')[0]
-            nse = Nse()
-            logger.info(f"Attempting to fetch NSE data for {clean_ticker}...")
-            # Note: nseconnect is primarily for quotes; 
-            # for full fundamentals we still rely on yfinance or XBRL.
-            # We return None here to let it fall back, but the plumbing is now real.
-            quote = nse.get_quote(clean_ticker)
-            if quote:
-                logger.info(f"Successfully connected to NSE for {clean_ticker}")
-            return None
-        except Exception as e:
-            logger.error(f"Error fetching from NSE: {e}")
-            return None
-
-    def _try_bse(self, start: str, end: str) -> Optional[pd.DataFrame]:
-        # Using bsedata for high-fidelity BSE data
-        try:
-            # TODO: Implement mapping from alphabetic ticker to numeric BSE scrip code
-            bse = BSE()
-            logger.info(f"Attempting to fetch BSE data for {self.ticker}...")
-            # Currently limited to quotes; returning None to fall back to yfinance
-            return None
-        except Exception as e:
-            logger.error(f"Error fetching from BSE: {e}")
-            return None
-
-    def _try_xbrl(self, start: str, end: str) -> Optional[pd.DataFrame]:
-        # Tier 1 extraction via python-xbrl / Arelle is currently in development
-        return None
-
-    def _try_indianapi(self, start: str, end: str) -> Optional[pd.DataFrame]:
-        # Tier 2 integration for IndianAPI.in / FinEdge API is currently in development
-        return None
-
-    def _try_yfinance(self, start: str, end: str) -> Optional[pd.DataFrame]:
-        try:
-            t = yf.Ticker(self.ticker)
-            q_fin = t.quarterly_financials.T
-            q_bs = t.quarterly_balance_sheet.T
-            q_cf = t.quarterly_cashflow.T
-            
-            if q_fin.empty and q_bs.empty and q_cf.empty:
-                logger.warning("All yfinance statements (financials, balance_sheet, cashflow) are empty.")
-                return None
-                
-            # Merge all three statements
-            logger.info(f"Merging yfinance statements: q_fin={q_fin.shape}, q_bs={q_bs.shape}, q_cf={q_cf.shape}")
-            df = pd.concat([q_fin, q_bs, q_cf], axis=1)
-            
-            
-            df = df.loc[:, ~df.columns.duplicated()] # Remove duplicate columns if any
-            logger.info(f"Merged shape after removing duplicates: {df.shape}")
-            
-            # Map yfinance columns to IND_AS_NODES (Simplified mapping for MVP)
-            mapping = {
-                "Total Revenue": "Revenue",
-                "Cost Of Revenue": "COGS",
-                "Operating Expense": "Operating_Expenses",
-                "Operating Income": "EBIT",
-                "EBIT": "EBIT",
-                "EBITDA": "EBITDA",
-                "Interest Expense": "Interest_Expense",
-                "Pretax Income": "EBT",
-                "Tax Provision": "Tax_Expense",
-                "Net Income": "PAT",
-                "Total Assets": "Total_Assets",
-                "Stockholders Equity": "Shareholders_Equity",
-                "Depreciation And Amortization": "D_A",
-                "Inventory": "Inventory",
-                "Accounts Payable": "Accounts_Payable",
-                "Total Debt": "Total_Debt",
-                "Operating Cash Flow": "Operating_Cash_Flow",
-                "Capital Expenditure": "Capex",
-                "Net PPE": "PPE",
-                "Construction In Progress": "CWIP",
-                "Goodwill And Other Intangible Assets": "Intangible_Assets",
-            }
-            
-            df = df.rename(columns=mapping)
-            df = df.loc[:, ~df.columns.duplicated()] # Remove duplicates after rename
-            df.index = pd.to_datetime(df.index)
-            return df
-        except Exception as e:
-            logger.error(f"Error fetching from yfinance: {e}")
-            return None
-
-
-    def _derive_strategic_nodes(self, df: pd.DataFrame) -> pd.DataFrame:
-        """
-        Compute Layer 3 nodes and other derived fields.
-        """
-        df = df.copy()
-        
-        # Helper to safely get a series from a column that might be a DataFrame
-        def get_series(name):
-            if name not in df.columns:
-                return None
-            col = df[name]
-            if isinstance(col, pd.DataFrame):
-                logger.warning(f"Column '{name}' is a DataFrame with multiple columns: {col.columns.tolist()}. Taking the first.")
-                return col.iloc[:, 0]
-            return col
-
-        # 1. Average Assets & Equity (Rolling 2-period mean)
-        assets = get_series("Total_Assets")
-        if assets is not None:
-            df["Average_Total_Assets"] = assets.rolling(window=2).mean().fillna(assets)
-            
-        equity = get_series("Shareholders_Equity")
-        if equity is not None:
-            df["Average_Shareholders_Equity"] = equity.rolling(window=2).mean().fillna(equity)
-
-        # 2. Basic derivations
-        rev = get_series("Revenue")
-        cogs = get_series("COGS")
-        if rev is not None and cogs is not None:
-            df["Gross_Profit"] = rev - cogs.fillna(0)
-            
-        ebit = get_series("EBIT")
-        da = get_series("D_A")
-        if ebit is not None:
-            # Proper EBITDA = EBIT + Depreciation & Amortization
-            df["EBITDA"] = ebit + da.fillna(0) if da is not None else ebit
-            
-        pat = get_series("PAT")
-        if pat is not None and rev is not None:
-            df["Net_Profit_Margin"] = pat / rev.replace(0, np.nan)
-            
-        avg_assets = get_series("Average_Total_Assets")
-        if rev is not None and avg_assets is not None:
-            df["Asset_Turnover"] = rev / avg_assets.replace(0, np.nan)
-            
-        avg_equity = get_series("Average_Shareholders_Equity")
-        if avg_assets is not None and avg_equity is not None:
-            df["Equity_Multiplier"] = avg_assets / avg_equity.replace(0, np.nan)
-            
-        npm = get_series("Net_Profit_Margin")
-        at = get_series("Asset_Turnover")
-        em = get_series("Equity_Multiplier")
-        if npm is not None and at is not None and em is not None:
-            df["ROE"] = npm * at * em
-            
-        ocf = get_series("Operating_Cash_Flow")
-        capex = get_series("Capex")
-        if ocf is not None and capex is not None:
-            df["Free_Cash_Flow"] = ocf - capex.abs().fillna(0)
-            
-        return df

singular_ticker_causal/data_sources/gdelt_client.py

@@ -1,400 +0,0 @@
-import time
-import random
-import logging
-import requests
-import yfinance as yf
-from datetime import datetime, timezone
-from typing import List, Dict, Any
-from singular_ticker_causal.utils.llm_client import LLMClient
-
-
-logger = logging.getLogger(__name__)
-
-
-# ==============================================================
-# GDELT DOC 2.0 FETCHER
-# ==============================================================
-
-def _fmt_gdelt_dt(dt: datetime) -> str:
-    """
-    Convert datetime -> GDELT YYYYMMDDHHMMSS (UTC).
-    """
-    if dt.tzinfo is not None:
-        dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
-
-    return dt.strftime("%Y%m%d%H%M%S")
-
-
-def _dedupe_articles(articles: list[dict]) -> list[dict]:
-    """
-    Deduplicate by URL/title combination.
-    """
-    seen = set()
-    deduped = []
-
-    for article in articles:
-        key = (
-            article.get("link", "").strip(),
-            article.get("title", "").strip().lower(),
-        )
-
-        if key in seen:
-            continue
-
-        seen.add(key)
-        deduped.append(article)
-
-    return deduped
-
-
-class GDELTClient:
-    """
-    Thin wrapper around GDELT Doc API for macro context.
-    """
-    GDELT_DOC_API: str = "https://api.gdeltproject.org/api/v2/doc/doc"
-    GDELT_MAX_RECORDS: int = 250
-    GDELT_MAX_WINDOW_DAYS: int = 90
-    USER_AGENT: str = (
-        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) "
-        "Gecko/20100101 Firefox/128.0"
-    )
-
-    def __init__(self):
-        pass
-
-    def fetch(
-        self,
-        keyword: str,
-        from_dt: datetime,
-        to_dt: datetime,
-        ticker: str | None = None,
-        **kwargs,
-    ) -> List[Dict[str, Any]]:
-        """
-        Main entry point for GDELT fetching.
-        If 'ticker' is provided, performs a broad generic search using yfinance context.
-        """
-        if ticker:
-            return self.fetch_ticker_context(ticker, from_dt, to_dt, **kwargs)
-
-        logger.info(
-            "Fetching GDELT news for %s from %s to %s",
-            keyword,
-            from_dt,
-            to_dt,
-        )
-
-        try:
-            articles = self.fetch_gdelt_window(
-                keyword=keyword,
-                from_dt=from_dt,
-                to_dt=to_dt,
-                **kwargs,
-            )
-
-            for article in articles:
-                article["credibility_weight"] = 0.5
-                article["source_type"] = "gdelt"
-
-            return articles
-
-        except Exception as e:
-            logger.exception("Error fetching from GDELT: %s", e)
-            return []
-
-    def _extract_search_terms(self, summary: str, company_name: str) -> Dict[str, List[str]]:
-        """Extract product, partner, and industry search terms from business summary using LLM."""
-        if not summary:
-            return {"products": [], "partners": [], "industry": []}
-
-        llm = LLMClient()
-        prompt = f"""
-        Extract major products/services, strategic partners, and industry-related details from the following summary of {company_name}.
-        
-        Summary: {summary}
-        
-        Important instructions:
-        1. For partners, include a few words describing the specific connection or nature of the relationship between {company_name} and the partner (e.g., "{company_name} strategic collaboration with [Partner]").
-        2. For industry, extract a few words describing the sector or niche {company_name} is connected to and its specific role/connection (e.g., "{company_name} provides [Services] in the [Industry] sector").
-        3. Keep it to the top 5 most relevant terms for products and partners, and top 3 for industry.
-        
-        Return a JSON object with:
-        "products": ["product1", "product2", ...],
-        "partners": ["{company_name} [connection] [partner1]", ...],
-        "industry": ["{company_name} [role/connection] [industry]", ...]
-        
-        Return ONLY valid JSON.
-        """
-        try:
-            result = llm.chat_json([{"role": "user", "content": prompt}])
-            return {
-                "products": result.get("products", []),
-                "partners": result.get("partners", []),
-                "industry": result.get("industry", [])
-            }
-        except Exception as e:
-            logger.error(f"[GDELT] LLM term extraction failed: {e}")
-            return {"products": [], "partners": [], "industry": []}
-
-    def fetch_ticker_context(
-        self,
-        ticker_symbol: str,
-        from_dt: datetime,
-        to_dt: datetime,
-        **kwargs
-    ) -> List[Dict[str, Any]]:
-        """
-        Generic fetcher that uses yfinance and LLM to build a broad context.
-        Sequence: products, partners, company name, company heads.
-        """
-        logger.info(f"[GDELT] Fetching generic context for ticker: {ticker_symbol}")
-        
-        try:
-            ticker = yf.Ticker(ticker_symbol)
-            info = ticker.info
-        except Exception as e:
-            logger.error(f"[GDELT] yfinance failed for {ticker_symbol}: {e}")
-            return []
-        
-        summary = info.get("longBusinessSummary", "")
-        company_name = info.get("longName") or info.get("shortName") or ticker_symbol
-        officers = info.get("companyOfficers", [])
-        
-        # 1. LLM extraction
-        terms = self._extract_search_terms(summary, company_name)
-        products = terms.get("products", [])
-        partners = terms.get("partners", [])
-        industry = terms.get("industry", [])
-        
-        # 2. Officer names
-        heads = [o.get("name") for o in officers if o.get("name")]
-        
-        # 3. Execution sequence
-        all_articles = []
-        search_plan = [
-            ("products", products),
-            ("partners", partners),
-            ("industry", industry),
-            ("company_name", [company_name]),
-            ("company_heads", heads)
-        ]
-        
-        for category, keywords in search_plan:
-            for kw in keywords:
-                if not kw: continue
-                logger.info(f"[GDELT] Scraping category '{category}': {kw}")
-                results = self.fetch_gdelt_window(
-                    keyword=kw,
-                    from_dt=from_dt,
-                    to_dt=to_dt,
-                    **kwargs
-                )
-                all_articles.extend(results)
-                # Polite pause to avoid aggressive rate limiting
-                time.sleep(random.uniform(2.0, 5.0))
-                
-        return _dedupe_articles(all_articles)
-
-    def fetch_gdelt_window(
-        self,
-        keyword: str,
-        from_dt: datetime,
-        to_dt: datetime,
-        max_records: int = 250,
-        exact_phrase: bool = False,
-        source_country: str | None = None,
-        source_lang: str | None = None,
-        theme: str | None = None,
-        domain: str | None = None,
-        extra_query: str | None = None,
-    ) -> list:
-        """
-        Fetch historical articles from the GDELT DOC 2.0 API.
-        Includes robust retry logic with 5-minute cooldown for timeouts/rate-limits.
-        """
-        if from_dt >= to_dt:
-            raise ValueError("from_dt must be earlier than to_dt")
-
-        window_days = (to_dt - from_dt).days
-        if window_days > self.GDELT_MAX_WINDOW_DAYS:
-            raise ValueError(
-                f"GDELT DOC API only supports ~{self.GDELT_MAX_WINDOW_DAYS} days history"
-            )
-
-        max_records = min(max_records, self.GDELT_MAX_RECORDS)
-        query = self._build_gdelt_query(
-            keyword=keyword,
-            exact_phrase=exact_phrase,
-            source_country=source_country,
-            source_lang=source_lang,
-            theme=theme,
-            domain=domain,
-            extra_query=extra_query,
-        )
-
-        params = {
-            "query": query,
-            "mode": "artlist",
-            "format": "json",
-            "maxrecords": max_records,
-            "sort": "DateDesc",
-            "STARTDATETIME": _fmt_gdelt_dt(from_dt),
-            "ENDDATETIME": _fmt_gdelt_dt(to_dt),
-        }
-
-        max_retries = 5
-        for attempt in range(max_retries):
-            try:
-                resp = requests.get(
-                    self.GDELT_DOC_API,
-                    params=params,
-                    timeout=45,
-                    headers={
-                        "User-Agent": self.USER_AGENT,
-                        "Accept": "application/json",
-                    },
-                )
-
-                if resp.status_code == 429:
-                    logger.warning(
-                        "[GDELT] Rate limited (429). Cooling down for 5 minutes..."
-                    )
-                    time.sleep(305)
-                    continue
-
-                resp.raise_for_status()
-                
-                try:
-                    data = resp.json()
-                except Exception as e:
-                    logger.error("[GDELT] Invalid JSON response (Status %d): %s", resp.status_code, e)
-                    # Log the body to see what GDELT is actually returning (likely HTML error)
-                    body_snippet = resp.text[:500] if resp.text else "[Empty Response]"
-                    logger.error("[GDELT] Response body snippet: %s", body_snippet)
-                    
-                    # If it's an HTML error page, we might be blocked or throttled in a way that doesn't return 429
-                    if "<html" in body_snippet.lower():
-                        logger.warning("[GDELT] Received HTML instead of JSON. Cooling down for 5 minutes...")
-                        time.sleep(305)
-                    else:
-                        time.sleep(10 * (attempt + 1))
-                    continue
-
-                raw_articles = data.get("articles", [])
-                if not isinstance(raw_articles, list):
-                    return []
-
-                results = []
-                for article in raw_articles:
-                    title = article.get("title", "") or ""
-                    url = article.get("url", "") or ""
-                    if not title or not url:
-                        continue
-
-                    normalized = {
-                        "feed_name": "GDELT",
-                        "title": title.strip(),
-                        "link": url.strip(),
-                        "published": article.get("seendate", ""),
-                        "summary": title.strip(),
-                        "language": article.get("language", ""),
-                        "sourcecountry": article.get("sourcecountry", ""),
-                        "domain": article.get("domain", ""),
-                        "socialimage": article.get("socialimage", ""),
-                    }
-                    results.append(normalized)
-
-                results = _dedupe_articles(results)
-                logger.info(
-                    "[GDELT] Retrieved %d articles for '%s'",
-                    len(results),
-                    keyword,
-                )
-                return results
-
-            except (requests.exceptions.Timeout, requests.exceptions.ConnectionError) as e:
-                logger.warning(
-                    "[GDELT] Timeout/Connection error on attempt %d/%d: %s. Cooling down 5 mins...",
-                    attempt + 1, max_retries, e
-                )
-                time.sleep(305)
-                continue
-
-            except requests.exceptions.RequestException as e:
-                wait_time = min(60, (2 ** attempt) * 5 + random.uniform(0, 3))
-                logger.warning(
-                    "[GDELT] Request error on attempt %d/%d: %s. Retrying in %.2fs",
-                    attempt + 1, max_retries, e, wait_time
-                )
-                time.sleep(wait_time)
-
-        logger.error("[GDELT] Failed after %d retries", max_retries)
-        return []
-
-    def _build_gdelt_query(
-        self,
-        keyword: str,
-        exact_phrase: bool = False,
-        source_country: str | None = None,
-        source_lang: str | None = None,
-        theme: str | None = None,
-        domain: str | None = None,
-        extra_query: str | None = None,
-    ) -> str:
-        """Build a valid GDELT DOC API query string."""
-        query_parts = []
-        if keyword:
-            if exact_phrase and " " in keyword:
-                query_parts.append(f'"{keyword}"')
-            else:
-                query_parts.append(keyword)
-
-        if source_country:
-            query_parts.append(f"sourcecountry:{source_country.lower()}")
-        if source_lang:
-            query_parts.append(f"sourcelang:{source_lang.lower()}")
-        if theme:
-            query_parts.append(f"theme:{theme}")
-        if domain:
-            query_parts.append(f"domain:{domain}")
-        if extra_query:
-            query_parts.append(extra_query)
-
-        return " ".join(query_parts)
-
-
-if __name__ == "__main__":
-    import json
-    from datetime import timedelta
-
-    # Configure logging
-    logging.basicConfig(
-        level=logging.INFO,
-        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
-    )
-
-    client = GDELTClient()
-
-    # Test parameters
-    ticker = "RELIANCE.NS"
-    end_date = datetime.now(timezone.utc)
-    start_date = end_date - timedelta(days=3)
-
-    print(f"\n--- Testing GDELT Ticker Context Fetch: {ticker} ---")
-    try:
-        articles = client.fetch(
-            keyword="",
-            from_dt=start_date,
-            to_dt=end_date,
-            ticker=ticker
-        )
-
-        output_file = "gdelt.json"
-        with open(output_file, "w") as f:
-            json.dump(articles, f, indent=4)
-        
-        print(f"Successfully fetched {len(articles)} articles.")
-        print(f"Output saved to {output_file}")
-
-    except Exception as e:
-        print(f"Test failed: {e}")

singular_ticker_causal/data_sources/news_client.py

@@ -1,460 +0,0 @@
-"""
-singular_ticker_causal/data_sources/news_client.py
-
-Fetches quarterly RSS news for a single ticker over a multi-year window.
-  - Operates on a single ticker (not a universe)
-  - Fetches quarterly windows across the full fundamental date range
-  - Returns articles structured for DenoisedNewsEncoder consumption
-
-Sources:
-  • LiveMint RSS feeds
-  • CNBC-TV18 RSS feeds
-  • Other RSS feeds (Business Standard, Forbes India, Zee News, Economic Times, etc.)
-  • Trading Economics (Selenium — stream, economy, markets, India news)
-  • Zerodha Pulse (requests + BeautifulSoup)
-"""
-
-import os
-import json
-import time
-import logging
-import datetime
-import requests
-import feedparser
-import yfinance as yf
-from typing import List
-from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.firefox.options import Options as FirefoxOptions
-from selenium.webdriver.firefox.service import Service as FirefoxService
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support.ui import WebDriverWait
-from selenium.webdriver.support import expected_conditions as EC
-
-
-logger = logging.getLogger(__name__)
-
-
-# ==============================================================
-# SCRAPER SOURCES
-# ==============================================================
-
-LIVEMINT_FEEDS = [
-    "https://www.livemint.com/rss/companies",
-    "https://www.livemint.com/rss/opinion",
-    "https://www.livemint.com/rss/money",
-    "https://www.livemint.com/rss/politics",
-    "https://www.livemint.com/rss/science",
-    "https://www.livemint.com/rss/industry",
-    "https://www.livemint.com/rss/education",
-    "https://www.livemint.com/rss/sports",
-    "https://www.livemint.com/rss/technology",
-    "https://www.livemint.com/rss/news",
-    "https://www.livemint.com/rss/markets",
-    "https://www.livemint.com/rss/AI",
-    "https://www.livemint.com/rss/insurance",
-    "https://www.livemint.com/rss/budget",
-    "https://www.livemint.com/rss/elections",
-]
-
-CNBC18_FEEDS = [
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/latest.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/india.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/economy.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/market.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/business.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/sports.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/politics.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/world.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/education.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/travel.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/auto.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/technology.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/personal-finance.xml",
-    "https://www.cnbctv18.com/commonfeeds/v1/cne/rss/web-stories.xml",
-]
-
-OTHER_FEEDS = [
-    "http://www.business-standard.com/rss/todays-paper.rss",
-    "https://www.forbesindia.com/commonfeeds/v1/frb/rss/blog.xml",
-    "http://zeenews.india.com/rss/business.xml",
-    "https://economictimes.indiatimes.com/rssfeedsdefault.cms",
-    "https://news.google.com/rss?cf=all&hl=en-IN&topic=b&gl=IN&ceid=IN:en",
-    "https://cfo.economictimes.indiatimes.com/rss/topstories",
-    "https://cfo.economictimes.indiatimes.com/rss/recentstories",
-    "https://cfo.economictimes.indiatimes.com/rss/corporate-finance",
-    "https://cfo.economictimes.indiatimes.com/rss/esg",
-    "https://cfo.economictimes.indiatimes.com/rss/cfo-tech",
-    "https://cfo.economictimes.indiatimes.com/rss/governance-risk-compliance",
-    "https://cfo.economictimes.indiatimes.com/rss/lateststories",
-]
-
-TE_SOURCES = {
-    "te_stream":  "https://tradingeconomics.com/stream",
-    "te_economy": "https://tradingeconomics.com/stream?i=economy",
-    "te_markets": "https://tradingeconomics.com/stream?i=markets",
-    "te_india":   "https://tradingeconomics.com/india/news",
-}
-
-ZERODHA_PULSE_URL = "https://pulse.zerodha.com/"
-
-
-# ==============================================================
-# TRADING ECONOMICS Selenium Scraper
-# ==============================================================
-
-JS_SCROLL_DOWN = "window.scrollTo(0, document.body.scrollHeight); return document.body.scrollHeight;"
-
-JS_EXTRACT_TE_NEWS = """
-var items = [];
-var listItems = document.querySelectorAll('li[id]');
-listItems.forEach(function(li) {
-    var titleLink = li.querySelector('a[href]');
-    if (!titleLink) return;
-    var titleText = '';
-    var bTag = titleLink.querySelector('b');
-    if (bTag) { titleText = bTag.textContent.trim(); }
-    else { titleText = titleLink.textContent.trim(); }
-    if (!titleText) return;
-    var url = titleLink.getAttribute('href') || '';
-    if (url && !url.startsWith('http')) { url = 'https://tradingeconomics.com' + url; }
-    var descEl = li.querySelector('.te-stream-item-description, span[style]');
-    var description = descEl ? descEl.textContent.trim() : '';
-    var dateEl = li.querySelector('small');
-    var dateText = dateEl ? dateEl.textContent.trim() : '';
-    var countryEl = li.querySelector('.te-stream-country');
-    var country = countryEl ? countryEl.textContent.trim() : '';
-    var categoryEl = li.querySelector('.te-stream-category');
-    var category = categoryEl ? categoryEl.textContent.trim() : '';
-    items.push({ title: titleText, description: description, date: dateText, url: url, country: country, category: category });
-});
-return JSON.stringify(items);
-"""
-
-
-class NewsClient:
-    """
-    Fetches news for a single ticker across a multi-year window by
-    chunking into quarterly GDELT calls. Implements the same cache-first
-    pattern as causal.test_causal_flow.test_causal_with_text.
-
-    Usage:
-        client = NewsClient("RELIANCE")
-        articles = client.fetch("2022-01-01", "2026-04-30")
-        # articles: List[dict] with keys: title, content, published, url, credibility_weight, source
-    """
-    USER_AGENT: str = (
-        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) "
-        "Gecko/20100101 Firefox/128.0"
-    )
-
-    def __init__(self, ticker: str):
-        self.ticker = ticker.upper().replace(".NS", "")
-        self.keyword = self._get_ticker_keyword(self.ticker)
-
-    def _get_ticker_keyword(self, ticker: str) -> str:
-        """Fetch a search-friendly name for the ticker from yfinance."""
-        symbol = ticker if "." in ticker else f"{ticker}.NS"
-        try:
-            t = yf.Ticker(symbol)
-            info = t.info
-            name = info.get("shortName") or info.get("longName") or ticker
-            # Clean up the name for better matching
-            for suffix in [" Limited", " Ltd.", " Ltd", " Corp.", " Corp", " Inc.", " Inc"]:
-                name = name.replace(suffix, "")
-            return name.strip()
-        except Exception as e:
-            logger.warning(f"Failed to fetch yfinance info for {symbol}: {e}")
-            return ticker
-
-    def _normalise(self, raw: dict) -> dict:
-        """Convert the raw dict format from news.py into a pipeline-ready article dict."""
-        title = raw.get("title", "").strip()
-        summary = raw.get("summary", "") or raw.get("description", "") or ""
-        content = f"{title}. {summary}".strip(". ") if summary and summary != title else title
-        pub_raw = raw.get("published", "") or raw.get("seendate", "")
-        # Try to parse the published field
-        pub_dt = None
-        if pub_raw:
-            try:
-                from dateutil import parser as dp
-                pub_dt = dp.parse(pub_raw)
-            except Exception:
-                pub_dt = None
-        if pub_dt is None:
-            pub_dt = datetime.datetime.now(datetime.timezone.utc)
-        elif pub_dt.tzinfo is None:
-            pub_dt = pub_dt.replace(tzinfo=datetime.timezone.utc)
-
-        return {
-            "title": title,
-            "content": content,
-            "published": pub_dt.isoformat(),
-            "url": raw.get("link", "") or raw.get("url", ""),
-        }
-
-    def fetch(
-        self,
-        start: str,
-        end: str,
-        include_pulse: bool = False,
-        include_te: bool = False,
-    ) -> List[dict]:
-        """
-        Fetch all news for self.ticker between start and end.
-        Aggregates RSS, Pulse, and (optionally) Trading Economics.
-        """
-
-        logger.info(
-            f"Fetching news for {self.ticker} ('{self.keyword}') "
-            f"from {start} to {end}..."
-        )
-
-        all_raw: List[dict] = []
-
-        # 1. RSS Feeds
-        logger.info("Searching RSS feeds...")
-        rss_articles = (
-            self.search_rss_feeds(LIVEMINT_FEEDS, self.keyword, "LiveMint")
-            + self.search_rss_feeds(CNBC18_FEEDS, self.keyword, "CNBC18")
-            + self.search_rss_feeds(OTHER_FEEDS, self.keyword, "OtherFeeds")
-        )
-        all_raw.extend(rss_articles)
-
-        # 2. Zerodha Pulse
-        if include_pulse:
-            logger.info("Fetching Zerodha Pulse...")
-            pulse_news = self.scrape_pulse()
-            all_raw.extend(pulse_news)
-
-        # 3. Trading Economics (Optional)
-        if include_te:
-            logger.info("Fetching Trading Economics...")
-            te_data = self.scrape_all_te(headless=True, scroll_count=1)
-            for items_list in te_data.values():
-                all_raw.extend(items_list)
-
-        # # Filter by date window
-        # start_dt = datetime.datetime(
-        #     *[int(x) for x in start.split("-")], tzinfo=datetime.timezone.utc
-        # )
-        # end_dt = datetime.datetime(
-        #     *[int(x) for x in end.split("-")], tzinfo=datetime.timezone.utc
-        # )
-        # in_window = self._filter_by_window(all_raw, start_dt, end_dt)
-        # logger.info(f"Collected {len(in_window)} articles in window.")
-
-        # # Normalise and deduplicate by title
-        # seen_titles: set = set()
-        # articles: List[dict] = []
-        # for raw in all_raw:
-        #     title = raw.get("title", "").strip().lower()
-        #     if not title or title in seen_titles:
-        #         continue
-        #     seen_titles.add(title)
-        #     articles.append(self._normalise(raw))
-
-        # Sort chronologically
-        all_raw.sort(key=lambda a: str(a.get("published") or a.get("date") or ""))
-
-        logger.info(
-            f"Final corpus: {len(all_raw)} unique articles for {self.ticker}."
-        )
-
-        return all_raw
-        
-    def search_rss_feeds(self, feeds: list, search_keyword: str, feed_name: str) -> list:
-        """Search for a keyword across a list of RSS feed URLs."""
-        print(f"[DEBUG] search_rss_feeds - Searching {feed_name} for keyword: '{search_keyword}'")
-        results = []
-        for feed_url in feeds:
-            print(f"[DEBUG] search_rss_feeds - Parsing URL: {feed_url}")
-            try:
-                feed = feedparser.parse(feed_url)
-                if not feed.entries:
-                    print(f"[DEBUG] search_rss_feeds - No entries found for {feed_url}")
-                for entry in feed.entries:
-                    title = entry.get("title", "")
-                    summary = entry.get("summary", "")
-                    if not title:
-                        continue
-                    if search_keyword.lower() in title.lower() or search_keyword.lower() in summary.lower():
-                        print(f"[DEBUG] search_rss_feeds - Match found: {title[:60]}...")
-                        results.append({
-                            "feed_name": feed_name,
-                            "title": title,
-                            "link": entry.get("link", ""),
-                            "published": entry.get("published", ""),
-                            "summary": summary,
-                        })
-            except Exception as e:
-                print(f"[news.py] RSS error ({feed_url}): {e}")
-        print(f"[DEBUG] search_rss_feeds - {feed_name} done. Found {len(results)} items.")
-        return results
-
-    def scrape_all_te(self, headless: bool = True, scroll_count: int = 3) -> dict:
-        """Scrape all Trading Economics news sources. Returns dict of source_key -> list of items."""
-        driver = self._create_driver(headless=headless)
-        results = {}
-        scraped_at = datetime.datetime.now(datetime.timezone.utc).isoformat()
-        try:
-            for key, url in TE_SOURCES.items():
-                try:
-                    items = self._scrape_te_page(driver, url, scroll_count=scroll_count)
-                    for item in items:
-                        item["source"] = key
-                        item["scraped_at"] = scraped_at
-                    results[key] = items
-                except Exception as e:
-                    print(f"  [TE] Error scraping {key}: {e}")
-                    results[key] = []
-                    try:
-                        driver.quit()
-                    except Exception:
-                        pass
-                    driver = self._create_driver(headless=headless)
-        finally:
-            try:
-                driver.quit()
-            except Exception:
-                pass
-        return results
-
-    def scrape_pulse(self) -> list:
-        """Scrape latest news from Zerodha Pulse using requests + BeautifulSoup."""
-        print("[news.py] Scraping Zerodha Pulse...")
-        headers = {
-            "User-Agent": self.USER_AGENT,
-            "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
-            "Accept-Language": "en-US,en;q=0.5",
-        }
-        try:
-            resp = requests.get(ZERODHA_PULSE_URL, headers=headers, timeout=30)
-            resp.raise_for_status()
-        except Exception as e:
-            print(f"[news.py] Zerodha Pulse error: {e}")
-            return []
-
-        soup = BeautifulSoup(resp.text, "html.parser")
-        items = []
-        scraped_at = datetime.datetime.now(datetime.timezone.utc).isoformat()
-
-        for li in soup.select("li.item"):
-            title_el = li.select_one("h2.title a")
-            if not title_el:
-                continue
-            title = title_el.get_text(strip=True)
-            url = title_el.get("href", "")
-            desc_el = li.select_one("div.desc")
-            description = desc_el.get_text(strip=True) if desc_el else ""
-            date_el = li.select_one("span.date")
-            date_text = date_el.get_text(strip=True) if date_el else ""
-            feed_el = li.select_one("span.feed")
-            feed = feed_el.get_text(strip=True) if feed_el else ""
-            items.append({
-                "title": title,
-                "description": description,
-                "published": date_text,
-                "link": url,
-                "summary": description,
-                "publisher": feed,
-                "scraped_at": scraped_at,
-                "source": "pulse"
-            })
-
-        print(f"[news.py] Zerodha Pulse: extracted {len(items)} items.")
-        return items
-
-    def _filter_by_window(
-        self,
-        articles: list,
-        from_dt: datetime.datetime,
-        to_dt: datetime.datetime,
-    ) -> list:
-        """
-        Drop articles whose parsed `published` timestamp falls outside [from_dt, to_dt].
-        Articles with unparseable or missing dates are kept (conservative).
-        """
-        from dateutil import parser as dp
-
-        def _to_utc(dt):
-            if dt.tzinfo is None:
-                return dt.replace(tzinfo=datetime.timezone.utc)
-            return dt.astimezone(datetime.timezone.utc)
-
-        from_utc = _to_utc(from_dt)
-        to_utc   = _to_utc(to_dt)
-
-        filtered = []
-        for art in articles:
-            pub = art.get("published", "")
-            if not pub:
-                filtered.append(art)
-                continue
-            try:
-                dt = _to_utc(dp.parse(pub))
-                if from_utc <= dt <= to_utc:
-                    filtered.append(art)
-            except Exception:
-                filtered.append(art)  # keep on parse failure
-        return filtered
-
-    def _create_driver(self, headless: bool = True):
-        """Create a headless Firefox webdriver."""
-        options = FirefoxOptions()
-        if headless:
-            options.add_argument("--headless")
-        options.set_preference("general.useragent.override", self.USER_AGENT)
-        options.set_preference("dom.webdriver.enabled", False)
-        options.set_preference("useAutomationExtension", False)
-        service = FirefoxService(log_output=os.devnull)
-        driver = webdriver.Firefox(options=options, service=service)
-        driver.set_page_load_timeout(60)
-        return driver
-
-    def _scrape_te_page(self, driver, url: str, scroll_count: int = 3, scroll_pause: float = 2.0) -> list:
-        """Scrape a single Trading Economics news page."""
-        print(f"  [TE] Loading: {url}")
-        driver.get(url)
-        WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.TAG_NAME, "body")))
-        time.sleep(3)
-        prev_height = 0
-        for i in range(scroll_count):
-            new_height = driver.execute_script(JS_SCROLL_DOWN)
-            if new_height == prev_height:
-                break
-            prev_height = new_height
-            time.sleep(scroll_pause)
-        raw = driver.execute_script(JS_EXTRACT_TE_NEWS)
-        items = json.loads(raw)
-        print(f"  [TE] Extracted {len(items)} items from {url}")
-        return items
-
-
-if __name__ == "__main__":
-    """Simple test runner for NewsClient."""
-    logging.basicConfig(
-        level=logging.INFO,
-        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
-    )
-    
-    ticker = "RELIANCE"
-    client = NewsClient(ticker)
-    
-    # Test Fetch (Aggregates RSS and Pulse)
-    print(f"\n--- Testing Fetch for {ticker} ---")
-    start_date = (datetime.datetime.now() - datetime.timedelta(days=7)).strftime("%Y-%m-%d")
-    end_date = datetime.datetime.now().strftime("%Y-%m-%d")
-    
-    # We include_te=False by default as it requires Selenium/Firefox
-    articles = client.fetch(start_date, end_date)
-    print(f"Fetched {len(articles)} articles.")
-    
-    if articles:
-        # Save results to debug_data/
-        with open("news.json", "w", encoding="utf-8") as f:
-            json.dump(articles, f, indent=4, ensure_ascii=False)
-        for a in articles[:3]:
-            print(f"  - [{a['published']}] {a['title']}")
-    else:
-        print("No articles found in the specified window.")

singular_ticker_causal/data_sources/nseconnect/__init__.py

@@ -1,25 +0,0 @@
-"""
-    The MIT License (MIT)
-
-    Copyright (c) 2014 Noufal Nazar
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-"""
-__VERSION__='2.0.1'
-from .nse import Nse

singular_ticker_causal/data_sources/nseconnect/bases.py

@@ -1,72 +0,0 @@
-"""
-    The MIT License (MIT)
-
-    Copyright (c) 2014 Noufal Nazar
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-"""
-
-from abc import ABCMeta, abstractmethod
-import six
-
-
-class AbstractBaseExchange(six.with_metaclass(ABCMeta, object)):
-
-    @abstractmethod
-    def get_stock_codes(self):
-        """
-        :return: list of tuples with stock code and stock name
-        """
-        raise NotImplementedError
-
-    @abstractmethod
-    def is_valid_code(self, code):
-        """
-        :return: True, if it is a valid stock code, else False
-        """
-        raise NotImplementedError
-
-    @abstractmethod
-    def get_quote(self, code):
-        """
-        :param code: a stock code
-        :return: a dictionary which contain detailed stock code.
-        """
-        raise NotImplementedError
-
-    @abstractmethod
-    def get_top_gainers(self):
-        """
-        :return: a sorted list of codes of top gainers
-        """
-        raise NotImplementedError
-
-    @abstractmethod
-    def get_top_losers(self):
-        """
-        :return: a sorted list of codes of top losers
-        """
-        raise NotImplementedError
-
-    @abstractmethod
-    def __str__(self):
-        """
-        :return: market name
-        """
-        raise NotImplementedError

singular_ticker_causal/data_sources/nseconnect/cleaners.py

@@ -1,51 +0,0 @@
-"""
-Module for various data structure cleaning tasks
-"""
-from datetime import datetime
-dirty_data = """
-{
-    "fname": "Jon",
-    "lname": "Doe",
-    "age": 20,
-    "str_age": "20",
-    "pi": 3.1415927,
-    "str_pi": "3.1415927",
-    "dob": "01-Jan-2023",
-    "mobile": [
-        {
-            "id": "Home",
-            "number": "123456789"
-        }, 
-        {
-            "id": "office", 
-            "number": "987645321"
-        }
-    ] 
-}
-"""
-
-def parse_values(obj):
-    for key, value in obj.items():
-        if isinstance(value, str):
-            # Try to parse as datetime if the string matches the format
-            date_formats = ["%d-%b-%Y", "%d-%m-%Y", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d"]
-            for date_format in date_formats:
-                try:
-                    obj[key] = datetime.strptime(value, date_format)
-                    break
-                except ValueError:
-                    pass
-            else:
-                # If the string couldn't be parsed as datetime, try numeric conversion
-                try:
-                    obj[key] = int(value)
-                except ValueError:
-                    try:
-                        obj[key] = float(value)
-                    except ValueError:
-                        pass
-        elif isinstance(value, dict):
-            obj[key] = parse_values(value)
-        elif isinstance(value, list):
-            obj[key] = [parse_values(item) if isinstance(item, dict) else item for item in value]
-    return obj

singular_ticker_causal/data_sources/nseconnect/datemgr.py

@@ -1,104 +0,0 @@
-import datetime as dt
-from dateutil.relativedelta import relativedelta
-from dateutil.parser import parse
-from dateutil import rrule
-from .errors import DateFormatError
-
-
-def get_nearest_business_day(d):
-    """ takes datetime object"""
-    if d.isoweekday() == 7 or d.isoweekday() == 6:
-        d = d - relativedelta(days=1)
-        return get_nearest_business_day(d)
-
-    # republic day
-    elif d.month == 1 and d.day == 26:
-        d = d - relativedelta(days=1)
-        return get_nearest_business_day(d)
-    # labour day
-    elif d.month == 5 and d.day == 1:
-        d = d - relativedelta(days=1)
-        return get_nearest_business_day(d)
-    # independece day
-    elif d.month == 8 and d.day == 15:
-        d = d - relativedelta(days=1)
-        return get_nearest_business_day(d)
-    # Gandhi Jayanti
-    elif d.month == 10 and d.day == 2:
-        d = d - relativedelta(days=1)
-        return get_nearest_business_day(d)
-    # chirstmas
-    elif d.month == 12 and d.day == 25:
-        d = d - relativedelta(days=1)
-        return get_nearest_business_day(d)
-    else:
-        return d
-
-def is_known_holiday(d):
-    """accepts datetime/date object and returns boolean"""
-    if type(d) == dt.datetime:
-        d = d.date()
-    elif type(d) != dt.date:
-        raise DateFormatError("only date objects or datetime objects")
-    else:
-        # fine do nothing
-        pass 
-
-    # declare the list of holidays here.
-    # republic day.
-    if d.month  == 1 and d.day == 26:
-        return True
-    # labour day
-    elif d.month == 5 and d.day == 1:
-        d = d - relativedelta(days=1)
-        return get_nearest_business_day(d)
-    # independence day
-    elif d.month == 8 and d.day == 15:
-        return True
-    # gandhi jayanti
-    elif d.month == 10 and d.day == 2:
-        return True
-    # christmas
-    elif d.month == 12 and d.day == 25:
-        return True
-    else:
-        return False
-
-def mkdate(d):
-    """tries its best to return a valid date. it can accept pharse like today,
-    yesterday, day before yesterday etc. 
-    """
-    # check if the it == a string
-    return_date = ""
-    if type(d) is str:
-        if d == "today":
-            return_date = dt.date.today()
-        elif d == "yesterday":
-            return_date = dt.date.today() - relativedelta(days=1)
-        elif d == "day before yesterday":
-            return_date = dt.date.today() - relativedelta(days=2)
-        else:
-            return_date = parse(d, dayfirst=True).date()
-    elif type(d) == dt.datetime:
-        return_date = d.date()
-    elif type(d) == dt.date:
-        return d
-    else:
-        raise DateFormatError("wrong date format %s" % str(d))
-    # check if future date.
-    return return_date
-
-def usable_date(d):
-    """accepts fuzzy format and returns most sensible date"""
-    return get_nearest_business_day(mkdate(d))
-
-def get_date_range(frm, to, skip_dates=[]):
-    """accepts fuzzy format date and returns business adjusted date ranges"""
-    # for x in rrule.rrule(rrule.DAILY, dtstart=s, until=dt.datetime.now(), byweekday=[0, 1, 2, 3, 4]): print(x)
-    frm = usable_date(frm)
-    to = usable_date(to)
-    datelist = []
-    for date in rrule.rrule(rrule.DAILY, dtstart=frm, until=to, byweekday=[0, 1, 2, 3, 4]):
-        if not is_known_holiday(date):
-            datelist.append(date.date())
-    return datelist

singular_ticker_causal/data_sources/nseconnect/downloader.py

@@ -1,116 +0,0 @@
-"""
-    The MIT License (MIT)
-
-    Copyright (c) 2014 Noufal Nazar
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-import io
-import os
-import zipfile
-import datetime as dt
-from urllib.request import Request
-from .datemgr import mkdate, get_date_range
-from .nse import Nse
-from abc import ABCMeta, abstractmethod
-
-class BaseBhavcopyDownloader(metaclass=ABCMeta):
-    """Base class for all types of bhavcopy downloader"""
-    def __init__(self, from_date, to_date=dt.datetime.now().date(), skip_dates=[]):
-        """accepts date in fuzzy format"""
-        self.bhavcopy_base_url = "https://www.nseindia.com/content/historical/EQUITIES/%s/%s/cm%s%s%sbhav.csv.zip"
-        self.bhavcopy_base_filename = "cm%s%s%sbhav.csv"
-        self.from_date = from_date
-        self.to_date = to_date
-        self.skip_dates = skip_dates
-        self.nse = Nse()
-        self.dates = self.generate_dates()
-
-    def generate_dates(self):
-        return get_date_range(self.from_date, self.to_date, skip_dates=self.skip_dates)
-    
-    def get_bhavcopy_url(self, d):
-        """accept date and return bhavcopy url"""
-        day_of_month = d.strftime("%d")
-        mon = d.strftime("%b").upper()
-        year = d.year
-        url = self.bhavcopy_base_url % (year, mon, day_of_month, mon, year)
-        return url
-
-    def get_bhavcopy_filename(self, d):
-        """for a given date generate bhavcopy filename"""
-        day_of_month = d.strftime("%d")
-        mon = d.strftime("%b").upper()
-        year = d.year
-        filename = self.bhavcopy_base_filename % (day_of_month, mon, year)
-        return filename
-
-    def download_one(self, d):
-        """download bhavcopy for the given date"""
-        # this will keep this method usable for any arbitrary date.
-        d = mkdate(d)
-        # ex_url = "https://www.nseindia.com/content/historical/EQUITIES/2011/NOV/cm08NOV2011bhav.csv.zip"
-        url = self.get_bhavcopy_url(d)
-        print(url)
-        filename = self.get_bhavcopy_filename(d)
-        # response = requests.get(url, headers=self.headers)
-        response = self.nse.opener.open(Request(url, None, self.nse.headers))
-        zip_file_handle = io.BytesIO(response.read())
-        zf = zipfile.ZipFile(zip_file_handle)
-        return zf.read(filename).decode("utf-8")
-
-    @abstractmethod
-    def download(self):
-        pass 
-    
-    @abstractmethod
-    def update(self):
-        pass 
-
-
-class BhavcopyFileSystemDownloader(BaseBhavcopyDownloader):
-    def __init__(self, directory, *args, **kwargs):
-        if (os.path.exists(directory) and os.path.isdir(directory) and os.access(directory, os.W_OK)):
-            super().__init__(*args, **kwargs)
-            self.directory = directory
-        else:
-            raise Exception("directory path must be valid and writtable, please check manually")
-
-    def download(self):
-        for date in self.dates:
-            print("downloading for " + str(date))
-            try:
-                content = self.download_one(date)
-            except Exception as err:
-                print("unable to download for the date: %s" %  date.strftime("%Y-%m-%d"))
-            else:
-                fh = open(self.directory + "/" + date.strftime("%Y-%m-%d") + ".csv", "w")
-                fh.write(content)
-                fh.close()
-    
-    def update(self):
-        pass 
-
-
-if __name__ == '__main__':
-    b = BhavcopyFileSystemDownloader(directory="/tmp/bhavcopy", from_date="01-01-2018")
-    b.download()
-
-# https://stackoverflow.com/questions/49183801/ssl-certificate-verify-failed-with-urllib

singular_ticker_causal/data_sources/nseconnect/errors.py

@@ -1,8 +0,0 @@
-class BhavcopyNotAvailableError(Exception):
-    """this error could occur in case you download bhavcopy for the dates
-    when the market was close"""
-    pass
-
-class DateFormatError(Exception):
-    """in case the date format is errorneous"""
-    pass

singular_ticker_causal/data_sources/nseconnect/nse.py

@@ -1,624 +0,0 @@
-"""
-    The MIT License (MIT)
-
-    Copyright (c) 2014 Noufal Nazar
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-
-import csv
-from .bases import AbstractBaseExchange
-from .urls import (
-    STOCKS_CSV_URL, STOCKS_IN_INDEX_URL, QUOTE_API_URL, QUOTE_DRIVATIVE_URL,
-    TOP_GAINERS_URL, TOP_LOSERS_URL, ALL_INDICES_URL,
-    FIFTYTWO_WEEK_HIGH_URL, FIFTYTWO_WEEK_LOW_URL
-)
-from .ua import Session
-from .utils import cast_intfloat_string_values_to_intfloat
-
-class Nse(AbstractBaseExchange):
-    """
-    class which implements all the functionality for
-    National Stock Exchange
-    """
-    __CODECACHE__ = None
-
-    def __init__(self, session_refresh_interval=120):
-        """Initialize a new NSE object.
-        Initializes a session management for making API calls to NSE (National Stock Exchange).
-        Args:
-            session_refresh_interval (int, optional): Time interval in seconds after which the session 
-                should be refreshed. Defaults to 120 seconds.
-        Note:
-            The session refresh interval helps maintain an active connection with NSE servers by
-            periodically creating a new session to prevent timeouts.
-        """
-        
-        self.session_refresh_interval = session_refresh_interval 
-        self.session = Session(session_refresh_interval)
-
-    #############################
-    ###      STOCKS APIS      ###
-    #############################
-    
-    def get_stock_codes(self):
-        """Gets a list of stock codes traded in NSE.
-
-        This function fetches stock data from NSE's CSV endpoint and extracts the stock symbols.
-
-        Returns:
-            list: A list of strings containing stock symbols traded on NSE.
-
-        Example:
-            >>> nse = Nse()
-            >>> codes = nse.get_stock_codes()
-            >>> print(codes[:5])
-            ['20MICRONS', '3IINFOTECH', '3MINDIA', '3PLAND', '63MOONS']
-        """
-        res = self.session.fetch(STOCKS_CSV_URL)
-        csv_content = res.text.splitlines()
-        symbols = []
-        csv_reader = csv.DictReader(csv_content)
-        for row in csv_reader:
-            symbols.append(row['SYMBOL'])
-        return symbols
-
-    def is_valid_code(self, code):
-        """Checks if a given stock code is valid.
-
-        This method validates whether the provided stock code exists in the list of valid
-        stock codes from NSE (National Stock Exchange).
-
-        Args:
-            code (str): Stock code/symbol to validate.
-
-        Returns:
-            bool: True if the code is valid, False otherwise.
-
-        Example:
-            >>> nse = NSE()
-            >>> nse.is_valid_code("INFY")
-            True
-            >>> nse.is_valid_code("INVALID")
-            False
-        """
-        stock_codes = self.get_stock_codes()
-        return code.upper() in stock_codes
-
-    def get_quote(self, code, all_data=False):
-        """Gets the stock quote for a given NSE stock symbol.
-
-        This function fetches real-time or delayed quote data from NSE for the specified stock code.
-
-        Args:
-            code (str): NSE stock symbol/code for which quote is to be fetched
-            all_data (bool, optional): If True returns complete quote data, if False returns only price info. 
-            Defaults to False.
-
-        Returns:
-            dict: A dictionary containing quote data.
-
-        Raises:
-            requests.exceptions.RequestException: If there is an error in HTTP request
-            ValueError: If the response JSON is invalid
-
-        Example:
-            >>> nse = Nse()
-            >>> nse.get_quote('abb')
-            {
-            'lastPrice': 5189.1,
-            'change': 70.55,
-            'pChange': 1.38,
-            'previousClose': 5118.55,
-            'open': 5160,
-            'close': 5187.65,
-            'vwap': 5162.91,
-            'stockIndClosePrice': 0,
-            'lowerCP': 4606.7,
-            'upperCP': 5630.4,
-            'pPriceBand': 'No Band',
-            'basePrice': 5118.55,
-            'intraDayHighLow': {'min': 5101, 'max': 5218.45, 'value': 5189.1},
-            'weekHighLow': {'min': 4890}
-            }
-        """
-        code = code.upper()
-        # TODO: implement if the code is valid
-        res = self.session.fetch(QUOTE_API_URL % code)
-        res = res.json()['priceInfo'] if all_data is False else res.json()
-        return cast_intfloat_string_values_to_intfloat(res)
-    
-    def get_52_week_high(self):
-        """Retrieves a list of stocks that have hit their 52-week high.
-
-        This method fetches data for stocks that have reached new 52-week high prices on the NSE.
-
-        Returns:
-            list[dict]: A list of dictionaries containing 52-week high data.
-
-        Example:
-            >>> nse.get_52_week_high()
-            [{'symbol': 'AVANTIFEED',
-              'series': 'EQ', 
-              'comapnyName': 'Avanti Feeds Limited',
-              'new52WHL': 899,
-              'prev52WHL': 849.9,
-              'prevHLDate': '13-Mar-2025',
-              'ltp': 887,
-              'prevClose': 842.55,
-              'change': 44.45,
-              'pChange': 5.28},
-                {...}
-            ]
-        """
-        res = self.session.fetch(FIFTYTWO_WEEK_HIGH_URL)
-        json_response = res.json()
-        # Handle the new API response structure which has dataLtpGreater20 and dataLtpLess20 fields
-        data = cast_intfloat_string_values_to_intfloat(json_response)
-        
-        # Check if the old structure with 'data' key exists
-        if 'data' in data:
-            return data['data']
-        
-        # Otherwise, extract and combine the lists from the new structure
-        result = []
-        if 'dataLtpGreater20' in data:
-            result.extend(data['dataLtpGreater20'])
-        if 'dataLtpLess20' in data:
-            result.extend(data['dataLtpLess20'])
-        return result
-    
-    def get_52_week_low(self):
-        """Retrieves a list of stocks that have hit their 52-week low.
-
-        This method fetches data for stocks that have reached new 52-week low prices on the NSE.
-
-        Returns:
-            list[dict]: A list of dictionaries containing 52-week low data.
-
-        Example:
-            >>> nse.get_52_week_low()
-            [{'symbol': 'AVANTIFEED',
-              'series': 'EQ', 
-              'comapnyName': 'Avanti Feeds Limited',
-              'new52WHL': 899,
-              'prev52WHL': 849.9,
-              'prevHLDate': '13-Mar-2025',
-              'ltp': 887,
-              'prevClose': 842.55,
-              'change': 44.45,
-              'pChange': 5.28},
-                {...}
-            ]
-        """
-        res = self.session.fetch(FIFTYTWO_WEEK_LOW_URL)
-        json_response = res.json()
-        # Handle the new API response structure which has dataLtpGreater20 and dataLtpLess20 fields
-        data = cast_intfloat_string_values_to_intfloat(json_response)
-        
-        # Check if the old structure with 'data' key exists
-        if 'data' in data:
-            return data['data']
-        
-        # Otherwise, extract and combine the lists from the new structure
-        result = []
-        if 'dataLtpGreater20' in data:
-            result.extend(data['dataLtpGreater20'])
-        if 'dataLtpLess20' in data:
-            result.extend(data['dataLtpLess20'])
-        return result
-    
-    #############################
-    ###       INDEX APIS      ###
-    #############################
-    
-    def get_index_quote(self, index="NIFTY 50"):
-        """Gets the quote for a specific index from NSE.
-
-        This function retrieves detailed quote information for a given index code from the
-        National Stock Exchange (NSE) of India.
-
-        Args:
-            index (str): The index code/symbol (e.g. "NIFTY 50", "BANKNIFTY", etc.)
-
-        Returns:
-            dict: A dictionary containing index quote details
-
-        Raises:
-            Exception: If the provided index code is invalid or not found
-
-        Example:
-            >>> nse = NSE()
-            >>> nse.get_index_quote("NIFTY 50")
-            {
-            'key': 'BROAD MARKET INDICES',
-            'index': 'NIFTY 50', 
-            'last': 22508.75,
-            'variation': 111.55,
-            'percentChange': 0.5,
-            'open': 22353.15,
-            'high': 22577.0,
-            'low': 22353.15,
-            'previousClose': 22397.2,
-            'yearHigh': 26277.35,
-            'yearLow': 21281.45,
-            # ... additional fields omitted for brevity
-            }
-        """
-        
-        url = ALL_INDICES_URL
-        all_index_quote = self.get_all_index_quote()
-        index_list = [ i['indexSymbol'] for i in all_index_quote]
-        index = index.upper()
-        index = ' '.join(index.split())
-        if index in index_list:
-            response = list(filter(lambda idx: idx['indexSymbol'] == index, all_index_quote))[0]
-            return cast_intfloat_string_values_to_intfloat(response)
-        else:
-            raise Exception('Wrong index code')
-    
-    def get_index_list(self):
-        """Gets a list of all NSE index symbols.
-
-        This method fetches all available NSE (National Stock Exchange) index symbols by
-        extracting the 'indexSymbol' from the complete index quote data.
-
-        Returns:
-            list: A list of strings containing index symbols (e.g., ['NIFTY 50', 'NIFTY BANK', ...])
-
-        Examples:
-            >>> nse = Nse()
-            >>> indices = nse.get_index_list()
-            >>> print(indices)
-            ['NIFTY 50', 'NIFTY BANK', 'NIFTY IT', ...]
-        """
-        return [ i['indexSymbol'] for i in self.get_all_index_quote()]
-    
-    def get_all_index_quote(self):
-        """Gets information for all NSE indices in one request.
-
-        This method fetches quotes and information for all available indices on the
-        National Stock Exchange (NSE) through a single API call.
-
-        Returns:
-            list[dict]: A list of dictionaries where each dictionary contains quote
-            information for an index. The quote information includes details like
-            index name, current value, change, percentage change etc.
-
-        Example:
-            >>> nse = Nse()
-            >>> quotes = nse.get_all_index_quote()
-            >>> quotes  # Sample output
-            [
-                {
-                'key': 'BROAD MARKET INDICES',
-                'index': 'NIFTY 50',
-                'indexSymbol': 'NIFTY 50', 
-                'last': 22508.75,
-                'variation': 111.55,
-                'percentChange': 0.5,
-                'open': 22353.15,
-                ...
-                },
-                # ... additional indices follow
-            ]
-
-        Raises:
-            URLError: If there is an error accessing the NSE API endpoint
-            ValueError: If the response JSON cannot be parsed properly
-        """
-        url = ALL_INDICES_URL
-        res = self.session.fetch(url)
-        return res.json()['data']
-    
-    def get_top_gainers(self, index="NIFTY"):
-        """Gets the list of top gaining stocks for the specified index.
-
-        This function retrieves real-time data for stocks that have gained the most value
-        during the current trading day. It can filter results by different indices.
-
-        Args:
-            index (str, optional): The index to get top gainers for. Defaults to "NIFTY".
-            Valid values are:
-            - NIFTY: Nifty 50 index
-            - BANKNIFTY: Bank Nifty index 
-            - NIFTYNEXT50: Nifty Next 50 index
-            - SecGtr20: Securities greater than 20 
-            - SecLwr20: Securities lower than 20
-            - FNO: Futures & Options
-            - ALL: All stocks
-
-        Returns:
-            list[dict]: List of dictionaries containing top gainer details.
-            
-        Raises:
-            ConnectionError: If unable to fetch data from NSE
-            
-        Example:
-            >>> nse = Nse()
-            >>> gainers = nse.get_top_gainers()
-            >>> gainers[0]  # Sample output
-            {
-            'symbol': 'DRREDDY',
-            'series': 'EQ',
-            'open_price': 1107.9,
-            'high_price': 1154.1,
-            'low_price': 1101.5,
-            'ltp': 1151.5,
-            'prev_price': 1107.95,
-            'net_price': 3.93,
-            'trade_quantity': 2714559,
-            'turnover': 31016.01,
-            'market_type': 'N',
-            'ca_ex_dt': '28-Oct-2024',
-            'ca_purpose': 'Face Value Split (Sub-Division) - From Rs 5/- Per Share To Re 1/- Per Share',
-            'perChange': 3.93
-            }
-        """
-        return self._get_top_gainers_losers('gainers', index)
-
-    def get_top_losers(self, index="NIFTY"):  # Changed from None to "NIFTY"
-        """Gets the top losers from specified index from NSE.
-
-        The function fetches real-time data for stocks that have declined the most in terms
-        of percentage change compared to their previous closing price.
-
-        Args:
-            index (str, optional): Index name for which top losers are to be fetched.
-                Available options:
-                    - NIFTY (Default)
-                    - BANKNIFTY
-                    - NIFTYNEXT50  
-                    - SecGtr20
-                    - SecLwr20
-                    - FNO
-                    - ALL
-
-        Returns:
-            list: List of dictionaries containing stock information with following keys:
-        
-        Raises:
-            URLError: When unable to connect to NSE
-            ValueError: When invalid index is provided
-
-        Examples:
-            >>> from nseconnect import Nse
-            >>> nse = Nse()
-            >>> losers = nse.get_top_losers()
-            >>> losers[0]
-            {'symbol': 'TATAMOTORS', 'series': 'EQ', 'openPrice': 375.0, ...}
-        """
-        return self._get_top_gainers_losers('losers', index)  # Changed from 'gainers' to 'losers'
-    
-    def get_advances_declines(self, index='nifty 50'):
-        """Gets the advances/declines data for given index.
-        This method provides the number of stocks advancing and declining in a given index
-        on NSE at any given point of time.
-        Args:
-            index (str, optional): Name of the index. Defaults to 'nifty 50'.
-                Valid values include 'NIFTY 50', 'NIFTY BANK', etc.
-        Returns:
-            dict: A dictionary with two keys:
-                - 'advances': Number of advancing stocks in the index
-                - 'declines': Number of declining stocks in the index
-        Examples:
-            >>> nse = Nse()
-            >>> nse.get_advances_declines(index="NIFTY BANK")
-            {'advances': 7, 'declines': 4}
-        Note:
-            The method is case-insensitive for the index parameter.
-        """
-        
-        # fixing this
-        index = index.upper()
-        index_quote = self.get_index_quote(index)
-        return {'advances': index_quote['advances'], 'declines': index_quote['declines']}
-    
-    def get_stocks_in_index(self, index="NIFTY 50"):
-        """Gets the list of symbols of stocks included in the specified NSE index.
-        The function retrieves the current constituents of a given NSE index like NIFTY 50, 
-        NIFTY BANK etc. and returns their stock symbols.
-        Args:
-            index (str, optional): Name of the NSE index. Defaults to "NIFTY 50".
-                Possible values: "NIFTY 50", "NIFTY BANK", "NIFTY IT" etc.
-        Returns:
-            list: List of stock symbols (str) that are part of the specified index.
-        Raises:
-            URLError: If unable to connect to NSE server
-            ValueError: If invalid index name is provided
-        Examples:
-            >>> nse = Nse()
-            >>> nse.get_stocks_in_index("NIFTY 50")
-            ['ADANIPORTS', 'ASIANPAINT', 'AXISBANK', 'BAJAJ-AUTO', 'BAJFINANCE', ...]
-            >>> nse.get_stocks_in_index("NIFTY BANK") 
-            ['AUBANK', 'AXISBANK', 'BANDHANBNK', 'FEDERALBNK', 'HDFCBANK', ...]
-        """
-        
-        index = index.upper()
-        url = STOCKS_IN_INDEX_URL % index
-        res = self.session.fetch(url)
-        res_dict = res.json()
-        return  [stock['symbol'] for stock in res_dict['data']][1:]
-    
-    def get_stock_quote_in_index(self, index="NIFTY 50", include_index=False):
-        """Gets stock quotes for all stocks in a given index.
-        This function fetches real-time quotes for all stocks that are part of the specified index
-        from NSE (National Stock Exchange).
-        Args:
-            index (str, optional): The name of the index. Defaults to "NIFTY 50".
-            include_index (bool, optional): Whether to include the index itself in results.
-                If True, includes both stocks and index. If False, returns only stocks.
-                Defaults to False.
-        Returns:
-            list: A list of dictionaries containing stock quote data.
-                Each dictionary contains various fields including:
-                - symbol: Stock symbol
-                - open: Opening price
-                - high: High price
-                - low: Low price
-                - lastPrice: Last traded price
-                - change: Change in price
-                - pChange: Percentage change
-                And other relevant trading information.
-        Raises:
-            URLError: If unable to connect to NSE servers
-            ValueError: If invalid index name is provided
-        Example:
-            >>> nse = Nse()
-            >>> nifty_quotes = nse.get_stock_quote_in_index("NIFTY 50")
-            >>> nifty_quotes_with_index = nse.get_stock_quote_in_index("NIFTY 50", include_index=True)
-        """
-        
-        index = index.upper()
-        url = STOCKS_IN_INDEX_URL % index
-        res = self.session.fetch(url)
-        res_dict = res.json()
-        res_dict = cast_intfloat_string_values_to_intfloat(res_dict)
-        if include_index is False:
-            return  [record for record in res_dict['data'] if record['priority'] == 0]
-        else:
-            return res_dict['data']
-
-    def _get_top_gainers_losers(self, direction, index):
-        """Internal method to fetch top gainers or losers for a given index.
-
-        Args:
-            direction (str): Either 'gainers' or 'losers'
-            index (str): Index name - one of NIFTY, BANKNIFTY, NIFTYNEXT50, SecGtr20, SecLwr20, FNO, ALL
-
-        Returns:
-            list: List of dictionaries containing top gainers/losers data for the specified index
-
-        Raises:
-            ValueError: If invalid index name is provided
-        """
-        index = index or 'NIFTY'  # Default to NIFTY if None
-        index = index.upper()
-        index = {
-            "NIFTY": "NIFTY",
-            "NIFTY 50": "NIFTY",
-            "NIFTY BANK": "BANKNIFTY",
-            "BANKNIFTY": "BANKNIFTY",
-            "NIFTYNEXT50": "NIFTYNEXT50",
-            "NIFTY NEXT 50": "NIFTYNEXT50",
-            "SECGTR20": "SecGtr20",
-            "SECLWR20": "SecLwr20",
-            "FNO": "FOSec",
-            "ALL": "allSec"
-        }.get(index)
-        if index is None:
-            raise ValueError("Index must be one of NIFTY 50, NIFTY BANK, NIFTY NEXT 50, SecGtr20, SecLwr20, FNO, ALL")
-        url = TOP_GAINERS_URL if direction == 'gainers' else TOP_LOSERS_URL
-        res = self.session.fetch(url)
-        return cast_intfloat_string_values_to_intfloat(res.json())[index]['data']
-
-    #############################
-    ###    DERIVATIVE APIS    ###
-    #############################
-
-    def get_future_quote(self, code, expiry_date=None):
-        """Get future quote for given stock code.
-
-        This function fetches futures trading data for a given stock code from NSE's derivatives segment.
-        If expiry date is provided, returns data for that specific expiry, else returns data for all
-        available expiry dates.
-
-        Args:
-            code (str): Stock code for which futures data needs to be fetched
-            expiry_date (str, optional): Expiry date in format DD-MMM-YYYY (e.g. "27-Mar-2025"). 
-                           Defaults to None.
-
-        Returns:
-            Union[dict, list]: If expiry_date provided returns dict with futures data for that expiry,
-                      else returns list of dicts with data for all expiries.
-
-        Example:
-            >>> nse = Nse()
-            >>> nse.get_future_quote('RELIANCE') 
-            [{'expiryDate': '27-Mar-2025',
-              'lastPrice': 1246,
-              'premium': 4.45,
-              'openPrice': 1245.25,
-              'highPrice': 1260.85,
-              'lowPrice': 1236.2,
-              'openInterest': 257812,
-              'changeInOpenInterest': 7144,
-              ...},
-             {...}]
-        """
-
-        url = QUOTE_DRIVATIVE_URL % code.upper()
-        res = self.session.fetch(url)
-        res_dict = res.json()
-        # list containing all options and futures data
-        data = res_dict['stocks']
-        # filter out only future data
-        future_data = [s for s in data if s['metadata']['instrumentType'] == "Stock Futures"]
-        # future data is very convoluted, so flatten-out the desired data
-        # !! there is bug in spelling of the key 'dailyvolatility', it is not camel cased
-        # fixing that in my code for uniformity
-        filtered_data = [
-            {
-                'expiryDate': record['metadata']['expiryDate'],
-                'lastPrice': record['metadata']['lastPrice'],
-                'premium': record['metadata']['lastPrice'] - record['underlyingValue'],
-                'openPrice': record['metadata']['openPrice'],
-                'highPrice': record['metadata']['highPrice'],
-                'lowPrice': record['metadata']['lowPrice'],
-                'closePrice': record['metadata']['closePrice'],
-                'prevClose': record['metadata']['prevClose'],
-                'change': record['metadata']['change'],
-                'pChange': record['metadata']['pChange'],
-                'numberOfContractsTraded': record['metadata']['numberOfContractsTraded'],
-                'totalTurnover': record['metadata']['totalTurnover'],
-                'underlyingValue': record['underlyingValue'],
-                'tradedVolume': record['marketDeptOrderBook']['tradeInfo']['tradedVolume'],
-                'openInterest': record['marketDeptOrderBook']['tradeInfo']['openInterest'],
-                'changeInOpenInterest': record['marketDeptOrderBook']['tradeInfo']['changeinOpenInterest'],
-                'pchangeinOpenInterest': record['marketDeptOrderBook']['tradeInfo']['pchangeinOpenInterest'],
-                'marketLot': record['marketDeptOrderBook']['tradeInfo']['marketLot'],
-                'dailyVolatility': record['marketDeptOrderBook']['otherInfo']['dailyvolatility'],
-                'annualisedVolatility': record['marketDeptOrderBook']['otherInfo']['annualisedVolatility']
-            }
-            for record in future_data
-        ]
-        # if expiry_date is provided, filter out data for that expiry date
-        if expiry_date:
-            matching_records = [record for record in filtered_data if record['expiryDate'] == expiry_date]
-            if matching_records:
-                return matching_records[0]  # Return the first matching record
-            else:
-                # Return an empty dictionary if no records found for the given expiry date
-                return {}
-        return filtered_data
-    
-    def __str__(self):
-        """Returns a string representation of the NSE driver class.
-        Returns:
-            str: A descriptive string identifying this as the NSE driver class.
-        """
-        
-        return 'Driver Class for National Stock Exchange (NSE)'
-
-
-if __name__ == "__main__":
-    n = Nse()
-    # data = n.download_bhavcopy("14th Dec")
-    n.get_quote('reliance')

singular_ticker_causal/data_sources/nseconnect/ua.py

@@ -1,137 +0,0 @@
-import requests
-import random
-from datetime import datetime as dt
-from .urls import NSE_MAIN
-from time import sleep
-
-
-class Session():
-    __CACHE__ = {}
-
-    def __init__(self, session_refresh_interval=60, cache_timeout=60):
-        """Initialize the class instance with session and cache parameters.
-        Args:
-            session_refresh_interval (int, optional): Time interval in seconds to refresh session. Defaults to 60.
-            cache_timeout (int, optional): Cache timeout duration in seconds. Defaults to 20.
-        Attributes:
-            session_refresh_interval (int): Time interval for session refresh.
-            cache_timeout (int): Duration for cache timeout.
-        """
-
-        self.session_refresh_interval = session_refresh_interval
-        self.cache_timeout = cache_timeout  # cache timeout in seconds
-        self._session = None  # Initialize _session attribute to None
-        self.create_session()
-        self.flush()
-    
-    def nse_headers(self):
-        """Returns a dictionary of headers required for making requests to NSE (National Stock Exchange).
-        These headers are designed to mimic a web browser request to prevent request blocking.
-        Returns:
-            dict: A dictionary containing HTTP headers with the following keys:
-                - Accept: Acceptable content types
-                - Accept-Language: Preferred language for response
-                - user-agent: Browser identification string
-                - X-Requested-With: Identifies AJAX requests
-        """
-        
-        return {
-                "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
-                "Accept-Language": "en-US,en;q=0.9",
-                "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
-                "X-Requested-With": "XMLHttpRequest",
-                "Referer": "https://www.nseindia.com/",
-                "Origin": "https://www.nseindia.com",
-                "Connection": "keep-alive",
-                "Sec-Fetch-Dest": "empty",
-                "Sec-Fetch-Mode": "cors",
-                "Sec-Fetch-Site": "same-origin"
-                }
-    
-    def create_session(self):
-        """Creates and initializes a new HTTP session for NSE (National Stock Exchange) API requests.
-        This method sets up a requests.Session object with appropriate headers for NSE and initializes
-        it by making a GET request to the NSE home page. The session is used for subsequent API calls.
-        Returns:
-            None
-        Side Effects:
-            - Sets self._session with configured requests.Session object
-            - Sets self._session_init_time with current timestamp
-        """
-
-        # Clean up old session if it exists
-        if hasattr(self, '_session') and self._session is not None:
-            old_session = self._session
-            self._session = None
-            # Explicitly delete old session to ensure garbage collection
-            del old_session
-            
-        # Create a completely new session object
-        self._session = requests.Session()
-        self._session.headers.update(self.nse_headers())
-        
-        # First visit NSE home page to get cookies
-        self._session.get(NSE_MAIN)
-        # Small delay to mimic human behavior
-        sleep(1)
-        # Visit the market page to get additional cookies
-        self._session.get(f"{NSE_MAIN}/market-data/live-equity-market")
-        
-        self._session_init_time = dt.now()
-    
-    def flush(self):
-        """Flushes the cached user agent data.
-        This method clears the internal cache dictionary storing user agent information
-        by resetting the class's __CACHE__ attribute to an empty dictionary.
-        Returns:
-            None
-        """
-        
-        self.__class__.__CACHE__ = {}
-
-    def fetch(self, url):
-        """Fetches data from a given URL with caching and session management.
-        This method implements a caching mechanism and session refresh logic to optimize 
-        network requests. It also includes random delays to prevent rate limiting.
-        Args:
-            url (str): The URL to fetch data from.
-        Returns:
-            requests.Response: The response object from the request.
-        Note:
-            - Uses class-level cache to store responses
-            - Implements random delays between 0-300ms before making requests
-            - Auto-refreshes session if expired based on session_refresh_interval
-        """
-
-        # Check cache first
-        if url in self.__class__.__CACHE__:
-            cache_time, response = self.__class__.__CACHE__[url]
-            if (dt.now() - cache_time).seconds < self.cache_timeout:
-                # print("serving from cache")
-                return response
-
-        # Only check session expiry if we need to make a network request
-        time_diff = dt.now() - self._session_init_time
-        if time_diff.seconds >= self.session_refresh_interval:
-            # print("re-initing the session because of expiry")
-            self.create_session()
-
-        # Add random delay before making request
-        sleep_time = random.uniform(0, 0.3)  # Random delay between 0-300ms
-        # print(f"Adding random delay of {sleep_time:.3f} seconds")
-        sleep(sleep_time)
-
-        # Make actual request if not in cache or cache expired
-        try:
-            response = self._session.get(url)
-            # Force a 401 response to retry with a fresh session
-            if response.status_code == 401:
-                self.create_session()
-                response = self._session.get(url)
-        except requests.RequestException:
-            # Try again with a fresh session on any request exception
-            self.create_session()
-            response = self._session.get(url)
-            
-        self.__class__.__CACHE__[url] = (dt.now(), response)
-        return response

singular_ticker_causal/data_sources/nseconnect/urls.py

@@ -1,35 +0,0 @@
-"""
-URL constants for NSE related operations
-"""
-
-# Base URLs
-NSE_HOME = "https://www.nseindia.com"
-NSE_MAIN = "https://www.nseindia.com"
-NSE_LEGACY = "https://www.nseindia.com"
-
-# Quote URLs
-QUOTE_EQUITY_URL = f"{NSE_MAIN}/get-quotes/equity?symbol=%s"
-QUOTE_API_URL = f"{NSE_MAIN}/api/quote-equity?symbol=%s"
-
-# Stock list URLs
-STOCKS_CSV_URL = f"https://archives.nseindia.com/content/equities/EQUITY_L.csv"
-
-# Market movers URLs
-TOP_GAINERS_URL = f"{NSE_MAIN}/api/live-analysis-variations?index=gainers"
-TOP_LOSERS_URL = f"{NSE_MAIN}/api/live-analysis-variations?index=loosers"
-TOP_FNO_GAINER_URL = f"{NSE_MAIN}/api/market-data-pre-open?key=FO"
-TOP_FNO_LOSER_URL = f"{NSE_MAIN}/api/market-data-pre-open?key=FO"
-FIFTYTWO_WEEK_HIGH_URL = f"{NSE_MAIN}/api/live-analysis-52Week?index=high"
-FIFTYTWO_WEEK_LOW_URL = f"{NSE_MAIN}/api/live-analysis-52Week?index=low"
-
-# Index URLs
-ALL_INDICES_URL = f"{NSE_MAIN}/api/allIndices"
-STOCKS_IN_INDEX_URL = f"{NSE_MAIN}/api/equity-stockIndices?index=%s"
-
-
-# Historical data URLs
-BHAVCOPY_BASE_URL = f"{NSE_MAIN}/archives/equities-bhavcopy/%s"
-BHAVCOPY_BASE_FILENAME = "cm%s%s%sbhav.csv"
-
-# Drivative URLs
-QUOTE_DRIVATIVE_URL = f"{NSE_MAIN}/api/quote-derivative?symbol=%s"

singular_ticker_causal/data_sources/nseconnect/utils.py

@@ -1,373 +0,0 @@
-"""
-    The MIT License (MIT)
-
-    Copyright (c) 2014 Noufal Nazar
-
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
-
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
-
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE.
-
-"""
-import six
-import re
-import operator
-
-def byte_adaptor(fbuffer):
-    """ provides py3 compatibility by converting byte based
-    file stream to string based file stream
-
-    Arguments:
-        fbuffer: file like objects containing bytes
-
-    Returns:
-        string buffer
-    """
-    if six.PY3:
-        strings = fbuffer.read().decode('latin-1')
-        fbuffer = six.StringIO(strings)
-        return fbuffer
-    else:
-        return fbuffer
-
-
-def js_adaptor(buffer):
-    """ convert javascript objects like true, none, NaN etc. to
-    quoted word.
-
-    Arguments:
-        buffer: string to be converted
-
-    Returns:
-        string after conversion
-    """
-    buffer = re.sub('true', 'True', buffer)
-    buffer = re.sub('false', 'False', buffer)
-    buffer = re.sub('none', 'None', buffer)
-    buffer = re.sub('NaN', '"NaN"', buffer)
-    return buffer
-
-def cast_intfloat_string_values_to_intfloat(data, round_digits=2):
-    """Recursively converts string representations of numbers to integers or floats in nested data structures.
-    This function traverses through dictionaries and lists, converting string values that represent
-    numbers into their corresponding numeric types (int or float). For float values, it rounds to
-    the specified number of decimal places.
-    Args:
-        data (Union[dict, list]): The input data structure containing values to be converted.
-            Can be either a dictionary or a list, potentially nested.
-        round_digits (int, optional): Number of decimal places to round float values to.
-            Defaults to 2.
-    Returns:
-        Union[dict, list]: A new data structure of the same type as input, with string
-            representations of numbers converted to their numeric types.
-    Example:
-        >>> data = {'a': '1', 'b': '2.5', 'c': 'text', 'd': {'e': '3.14'}}
-        >>> cast_intfloat_string_values_to_intfloat(data)
-        {'a': 1, 'b': 2.5, 'c': 'text', 'd': {'e': 3.14}}
-    """
-
-    if isinstance(data, dict):
-        data = data.copy()
-        for key, value in data.items():
-            if isinstance(value, str):
-                try:
-                    data[key] = int(value)
-                except ValueError:
-                    try:
-                        data[key] = round(float(value), round_digits)
-                    except ValueError:
-                        pass
-            elif isinstance(value, (dict, list)):
-                data[key] = cast_intfloat_string_values_to_intfloat(value, round_digits)
-            elif isinstance(value, float):
-                data[key] = round(value, round_digits)
-    elif isinstance(data, list):
-        data = data[:]
-        for i, value in enumerate(data):
-            if isinstance(value, str):
-                try:
-                    data[i] = int(value)
-                except ValueError:
-                    try:
-                        data[i] = round(float(value), round_digits)
-                    except ValueError:
-                        pass
-            elif isinstance(value, (dict, list)):
-                data[i] = cast_intfloat_string_values_to_intfloat(value, round_digits)
-            elif isinstance(value, float):
-                data[i] = round(value, round_digits)
-    return data
-
-def camel_to_title(camel_str):
-    """Converts a camel case string to title case.
-    This function takes a camel case string and converts it to title case by adding
-    spaces before capital letters and capitalizing the first letter of each word.
-    Args:
-        camel_str (str): The camel case string to be converted.
-    Returns:
-        str: The converted string in title case format.
-    Examples:
-        >>> camel_to_title("camelCaseString")
-        'Camel Case String'
-        >>> camel_to_title("thisIsATest")
-        'This Is A Test'
-    """
-    
-    return re.sub(r'(?<!^)(?=[A-Z])', ' ', camel_str).title()
-
-def _resolve_path(data, path, case_insensitive=True):
-    """Helper function to resolve dot notation paths in dictionaries."""
-    if not path:
-        return data
-    
-    parts = path.split('.')
-    current = data
-    
-    for part in parts:
-        if isinstance(current, dict):
-            if case_insensitive:
-                key_map = {k.lower(): k for k in current.keys()}
-                part_lower = part.lower()
-                if part_lower in key_map:
-                    current = current[key_map[part_lower]]
-                else:
-                    return None
-            else:
-                current = current.get(part)
-        else:
-            return None
-    return current
-
-def _parse_query(query_str):
-    """Parse query string into (path, operator, value) tuple."""
-    operators = {
-        '==': operator.eq,
-        '!=': operator.ne,
-        '>=': operator.ge,
-        '<=': operator.le,
-        '>': operator.gt,
-        '<': operator.lt
-    }
-    
-    for op_str, op_func in operators.items():
-        if op_str in query_str:
-            path, value = query_str.split(op_str)
-            path = path.strip()
-            value = value.strip()
-            
-            # Try to convert value to number if possible
-            try:
-                value = int(value)
-            except ValueError:
-                try:
-                    value = float(value)
-                except ValueError:
-                    # Keep as string if not numeric
-                    pass
-                    
-            return path, op_func, value
-            
-    return None, None, None
-
-def dict_to_table(data, title="Data Table", filter=None, ignore=None, sort=None, direction="desc", query=None):
-    """Converts dictionary or list of dictionaries to a formatted table using Rich library.
-    This function takes either a dictionary or a list of dictionaries and displays it as a
-    formatted table in the console. It supports filtering specific keys, ignoring keys, and
-    applies special formatting for negative numbers.
-    Args:
-        data (Union[dict, List[dict]]): The data to be displayed. Can be either a dictionary
-            or a list of dictionaries.
-        title (str, optional): The title to display above the table. Defaults to "Data Table".
-        filter (List[str], optional): List of keys to include in the output. If provided,
-            only these keys will be displayed. Keys are matched case-insensitively.
-            Defaults to None.
-        ignore (List[str], optional): List of keys to exclude from the output. Keys are
-            matched case-insensitively. Defaults to None.
-        sort (str, optional): Key to sort by. Case-insensitive. Will sort numerically 
-            for numeric values and alphabetically for string values. Defaults to None.
-        direction (str, optional): Sort direction - "asc" for ascending or "desc" for 
-            descending. Defaults to "desc".
-        query (str, optional): Filter rows using dot notation path and comparison.
-            Supports operators: ==, !=, >, <, >=, <=
-            Example: "market.price>100" or "status.active==True"
-            Keys are matched case-insensitively. Defaults to None.
-    """
-    from rich.console import Console
-    from rich.table import Table
-
-    console = Console()
-    table = Table(title=title)
-
-    if not data:
-        console.print("[red]No data to display![/red]")
-        return
-
-    # Parse query if provided
-    query_path = None
-    query_op = None
-    query_value = None
-    if query:
-        query_path, query_op, query_value = _parse_query(query)
-        if not all([query_path, query_op, query_value]):
-            console.print("[red]Invalid query format![/red]")
-            return
-
-    # Validate direction
-    if direction not in ["asc", "desc"]:
-        console.print("[red]Direction must be 'asc' or 'desc'![/red]")
-        return
-
-    # Normalize filter, ignore and sort keys
-    if filter:
-        if not isinstance(filter, list):
-            console.print("[red]Filter should be a list of keys![/red]")
-            return
-        filter = [str(key).lower() for key in filter]
-    
-    if ignore:
-        if not isinstance(ignore, list):
-            console.print("[red]Ignore should be a list of keys![/red]")
-            return
-        ignore = [str(key).lower() for key in ignore]
-    else:
-        ignore = []
-
-    if sort:
-        sort = str(sort).lower()
-
-    # Check if data is a list of dicts
-    if isinstance(data, list) and all(isinstance(i, dict) for i in data):
-        # Get all unique keys and create key mapping
-        keys = set()
-        for item in data:
-            keys.update(item.keys())
-        key_map = {k.lower(): k for k in keys}
-
-        # Validate sort key if provided
-        if sort and sort not in key_map:
-            console.print(f"[red]Sort key '{sort}' not found in data![/red]")
-            return
-
-        # Create ordered keys list
-        if filter:
-            ordered_keys = [key_map[f] for f in filter if f in key_map and f not in ignore]
-        else:
-            ordered_keys = [key_map[k.lower()] for k in keys if k.lower() not in ignore]
-
-        if not ordered_keys:
-            console.print("[red]No matching keys found![/red]")
-            return
-
-        # Apply query filter before sorting
-        if query:
-            filtered_data = []
-            for item in data:
-                item_value = _resolve_path(item, query_path)
-                if item_value is not None:
-                    try:
-                        if query_op(item_value, query_value):
-                            filtered_data.append(item)
-                    except TypeError:
-                        # Handle type mismatch gracefully
-                        continue
-            data = filtered_data
-            
-            if not data:
-                console.print("[red]No data matches the query![/red]")
-                return
-
-        # Sort data if sort key is provided
-        if sort and sort in key_map:
-            original_key = key_map[sort]
-            # Try numeric sort first
-            try:
-                sorted_data = sorted(
-                    data,
-                    key=lambda x: float(x.get(original_key, 0)),
-                    reverse=(direction == "desc")
-                )
-            except (ValueError, TypeError):
-                # Fall back to string sort
-                sorted_data = sorted(
-                    data,
-                    key=lambda x: str(x.get(original_key, "")),
-                    reverse=(direction == "desc")
-                )
-        else:
-            sorted_data = data
-
-        # Add columns and display table
-        for key in ordered_keys:
-            table.add_column(camel_to_title(key), style="bright_white")
-
-        for item in sorted_data:
-            row = []
-            for key in ordered_keys:
-                value = item.get(key, "")
-                if isinstance(value, (int, float)) and value < 0:
-                    row.append(f"[red]{value}[/red]")
-                else:
-                    row.append(f"[bright_white]{value}[/bright_white]")
-            table.add_row(*row)
-
-    elif isinstance(data, dict):
-        # Single dict can't be queried for rows
-        if query:
-            console.print("[red]Query is only supported for list of dictionaries![/red]")
-            return
-
-        # Filter and ignore the dictionary data
-        filtered_data = {}
-        key_map = {k.lower(): k for k in data.keys()}
-        
-        if filter:
-            # Add keys in filter order if they exist and not in ignore
-            for f in filter:
-                if f in key_map and f not in ignore:
-                    original_key = key_map[f]
-                    value = data[original_key]
-                    if not isinstance(value, (dict, list, tuple, set)):
-                        filtered_data[original_key] = value
-        else:
-            # If no filter, exclude ignored and nested items
-            filtered_data = {k: v for k, v in data.items() 
-                           if not isinstance(v, (dict, list, tuple, set))
-                           and k.lower() not in ignore}
-
-        if not filtered_data:
-            console.print("[red]No matching key-value pairs to display![/red]")
-            return
-
-        # Add columns
-        table.add_column("Key", style="cyan", no_wrap=True)
-        table.add_column("Value", style="bright_white")
-
-        # Add rows
-        for key, value in filtered_data.items():
-            if isinstance(value, (int, float)) and value < 0:
-                value_str = f"[red]{value}[/red]"
-            else:
-                value_str = f"[bright_white]{value}[/bright_white]"
-            table.add_row(camel_to_title(key), value_str)
-
-    else:
-        console.print("[red]Unsupported data format![/red]")
-        return
-
-    console.print(table)
-
-
-