# Data Sources

================================================================================
INDEX
================================================================================

| Data | Sources |
|------|----------|
| OHLCV daily | DuckDB (cached), Alpaca (live), Yahoo (live), DefeatBeta (cached) |
| minute bars | Alpaca (live), DuckDB cache (cached) |
| hour bars | Alpaca (live) |
| gainers/losers | Webull (live, API + scraper), Finviz (live), TradingView (live) |
| premarket/afterhours | Webull (live, API + scraper) |
| Nueva: | sector | NASDAQ API (live), DuckDB (cached), Finviz (live), StockAnalysis (live) |
| Nueva: | industry | NASDAQ API (live), DuckDB (cached), Finviz (live), StockAnalysis (live) |
| Nueva: | market_cap | NASDAQ API (live), DuckDB (cached), Finviz (live), StockAnalysis (live) |
| symbols list | NASDAQ API (live) | symbol, name, price, change, volume, market_cap, country, ipoyear, industry, sector |
| deficient/non-compliant listing | NASDAQ API (live) | non-compliant issuers, deficiency type, market tier, notification date |
| catalyst (why moving) | Finviz API (live) |
| low float stocks | LowFloat (live) |
| high short interest | HighShortInterest (live) |
| hot penny stocks | PennyStockFlow (live) |
| reddit mentions | ChartExchange (live) |
| Nueva: | social sentiment (bullish/bearish %) | StockTwits (live) |
| Nueva: | message volume / social buzz | StockTwits (live) |
| Nueva: | trending symbols | StockTwits (live) |
| Nueva: | watchlist count (social interest) | StockTwits (live) |
| earnings date | Yahoo (live), SEC 10-K (live), DefeatBeta (cached) |
| earnings history | DefeatBeta (cached) |
| news | Yahoo (live), DefeatBeta (cached) |
| news headline | StockTitan overview (live) |
| short interest | StockTitan (live), StockAnalysis (live) |
| press releases | DefeatBeta (cached) |
| insider trades | SEC Form 4 (live), OpenInsider (live), StockTitan (live), DefeatBeta (cached) |
| Nueva: | employees | Finviz (live), StockAnalysis (live) |
| Nueva: | company description | Finviz (live), StockAnalysis (live) |
| fundamentals | Yahoo (live), Finviz (live), StockAnalysis (live), DefeatBeta (cached) |
| Nueva: | valuation ratios (PE, PB, PS, PEG...) | StockAnalysis (live), DefeatBeta (cached) |
| Nueva: | financial position (current ratio, debt/equity...) | StockAnalysis (live) |
| Nueva: | efficiency (ROE, ROIC, WACC...) | StockAnalysis (live), DefeatBeta (cached) |
| Nueva: | balance sheet (cash, debt, equity, book value...) | StockAnalysis (live), DefeatBeta (cached) |
| Nueva: | cash flow (OCF, FCF, capex...) | StockAnalysis (live), DefeatBeta (cached) |
| Nueva: | share stats (insider %, institution %, float) | StockAnalysis (live) |
| Nueva: | dividends (yield, growth, payout) | StockAnalysis (live), Yahoo (live), Alpaca (cached), DefeatBeta (cached) |
| Nueva: | stock splits (date, ratio, type) | StockAnalysis (live), Yahoo (live), Alpaca (cached), SEC 8-K (live), DefeatBeta (cached) |
| Nueva: | scores (Altman Z, Piotroski F) | StockAnalysis (live) |
| Nueva: | fair value (Lynch, Graham) | StockAnalysis (live) |
| Nueva: | key executives | StockAnalysis (live), DefeatBeta (cached) |
| Nueva: | contact (address, phone, website) | StockAnalysis (live) |
| Nueva: | stock metadata (CUSIP, ISIN, SIC, fiscal year...) | StockAnalysis (live), SEC (live) |
| 8-K | SEC (live) |
| S-1/S-3/S-8 | SEC (live) — offerings / dilution |
| 10-K/10-Q | SEC (live) |
| Form 4 | SEC (live) |
| 13D/13G | SEC (live) — activist filings |
| reverse splits | Yahoo (live), Alpaca (cached), SEC 8-K (live) |
| Nueva: | short sale restriction (SSR / Rule 201 list) | NASDAQ Trader (live) — ssrlist.com is a viewer |
| dividends | Yahoo (live), Alpaca (cached) |
| Nueva: | CIK lookup | SEC (live), StockAnalysis (live) |
| trading calendar | DuckDB (cached), Alpaca (live) |
| supports / resistances | DuckDB (cached) + Alpaca minute (live) |
| cost to borrow (fee, available, history) | iBorrowDesk (live) |
| offering risk rating | AskEdgar (live) |
| dilution risk (rating, ATM, capital structure, runway) | SmallCapLab (live, API + scraper) |
| pump timing (pump→filing correlation) | SmallCapLab (live, API) |
| institutional ownership | AskEdgar (live) |
| share float / outstanding shares | AskEdgar (live) |

**Access types:**
- **cached**: obtenido periódicamente y almacenado en local
- **live**: obtenido al vuelo cuando se necesita

================================================================================
SOURCE DETAILS
================================================================================

## DuckDB (Local — Cached)

| File | Data |
|------|------|
| `data/alpaca_merged.parquet` | Daily OHLCV + computed |
| `data/market_calendar.csv` | Trading days |
| `data/corporate_actions.csv` | Historical corporate actions |
| `data/symbol_metadata.csv` | Stock metadata |
| `data/metadata.db` (table: tickers) | CIK mapping |
| `data/cache/{SYMBOL}_{DATE}.parquet` | Minute bars cache (TTL: 24h) |

---

## Alpaca (Live)

| Endpoint | Data |
|----------|------|
| `/v2/stocks/bars` | OHLCV daily/minute/hour |
| `/v2/corporate_actions` | splits, dividends, mergers, spin-offs |
| `/v1/calendar` | trading calendar |

---

## Yahoo Finance (Live)

| Data | Endpoint / Resource |
|------|---------------------|
| OHLCV, volumes, pre/post market | `/v8/finance/chart/{symbol}` |
| recent news | news feed |
| earnings calendar | earnings history |
| splits, dividends | actions feed |
| press releases | press releases feed |

---

## Webull (Live — Scraper + API)

Webull expone dos tipos de datos: **scraping HTML** (páginas públicas) y **API interna** (endpoints JSON).

**Base URL:** `https://quotes-gw.webullfintech.com`

### HTML Scraper

| Endpoint | Data |
|----------|------|
| `/quote/us/gainers` | today gainers |
| `/quote/us/gainers/1d` | 1-day gainers |
| `/quote/us/gainers/5d` | 5-day gainers |
| `/quote/us/gainers/1m` | 1-month gainers |
| `/quote/us/gainers/pre` | premarket |
| `/quote/us/gainers/after` | afterhours |
| `/quote/us/actives/rvol10d` | relative volume 10d |
| `/quote/us/actives/turnover_ratio` | turnover ratio |
| `/quote/us/actives/range` | range % |

### Internal API — Rankings

| Endpoint | Data |
|----------|------|
| `/api/wlas/ranking/v9/rise` | gainers |
| `/api/wlas/ranking/v9/fall` | losers |

**Parameters:** `regionId=6` (US market), `rankType` (timeframe: 1d, 5d, 1m, 3m, 52w, 3min, 5min, gainers.preMarket, gainers.afterMarket), `pageIndex`, `pageSize`

### Internal API — Micro Trend

| Endpoint | Data |
|----------|------|
| `/api/bgw/quote/microTrend?ids={tickerIds}` | Intraday micro-trend (minute-by-minute price data for current session) |

---

## Finviz (Live — Scraper + API)

**Base URL:** `https://finviz.com`

### Views

| Vista ID | Vista | Descripción |
|----------|-------|--------------|
| `v=110` | Overview | Datos básicos (Ticker, Company, Sector, Market Cap, etc.) |
| `v=120` | Valuation | P/E, Forward P/E, PEG, P/S, P/B, etc. |
| `v=160` | Financial | EPS, Revenue, Margins, ROE, ROA, etc. |
| `v=130` | Ownership | Insider Ownership, Institutional Ownership, Short Interest, etc. |
| `v=140` | Performance | Perf Week, Perf Month, Perf YTD, Perf Year, etc. |
| `v=170` | Technical | RSI, SMA20/50/200, ATR, Volatility, etc. |
| `v=180` | ETF | ETF-specific data (AUM, Holdings, Flows, Expense Ratio) |
| `v=150` | **Custom** | Columnas personalizadas vía parámetro `c=` |

### Endpoints

| Endpoint | Data |
|----------|------|
| `/screener.ashx?v=111` | overview signals |
| `/screener.ashx?v=171` | technical signals |
| `/screener.ashx?v=150&c={cols}&t={tickers}` | custom screener |
| `/api/stocks-why-moving/boxover/{symbol}` | catalyst (headline, summary, sentiment) |
| `/quote.ashx?t={symbol}` | employees, sector, industry, company description |

### Custom Screener (v=150)

**URL:** `https://finviz.com/screener?v=150&t={tickers}&c={column_ids}`

**Parameters:** `v=150` (custom), `t=` (tickers, comma-separated), `c=` (column IDs, comma-separated), `r=` (optional signal filters)

---

## SEC EDGAR (Live)

**Base URL:** `https://www.sec.gov`

| Endpoint | Data |
|----------|------|
| `/files/company_tickers.json` | symbol → CIK lookup |
| `/submissions/CIK{cik}.json` | all filings |

**Form types:** Form 4 (insider trades), 8-K (material events), 6-K (foreign events), S-1/S-3/S-8/S-4 (offerings), F-1/F-3 (foreign offerings), 13D/13G (activist filings), 10-K/10-Q/20-F (periodic reports), DEF 14A (proxy statements).

**Filing HTML URL format:** `https://www.sec.gov/Archives/edgar/data/{CIK}/{accessionNumberWithoutDashes}/{primaryDocument}`

---

## OpenInsider (Live — Scraper)

| Endpoint | Data |
|----------|------|
| `/screener` | insider trades (symbol filtered from page) |
| `/cluster-buy` | cluster buys |
| `/ceo-trades` | CEO-only trades |

---

## NASDAQ API (Live)

**Base URL:** `https://api.nasdaq.com`

| Endpoint | Data |
|----------|------|
| `/api/screener/stocks?tableonly=false&download=true` | Full symbols list with metadata |
| `/api/quote/list-type-extended/listing?queryString=deficient` | Non-compliant / deficient companies |

### Screener — `/api/screener/stocks`

Returns: symbol, name, lastsale (price), netchange, pctchange, volume, marketCap, country, ipoyear, industry, sector, url.

Note: NASDAQ only. NYSE/AMEX not included.

### Non-Compliant / Deficient — `/api/quote/list-type-extended/listing`

Lists NASDAQ issuers non-compliant with listing requirements. Parameter: `queryString=deficient` (required). No pagination — returns full list (~350 issuers).

**Deficiency types observed:** Bid Price, Equity, Market Value of Listed Securities, Delinquent filings, Market Value of Publicly Held Shares, Shareholders, Annual Shareholder Meeting, Audit Committee Composition, Net Income, Board Independence, and others.

**Market tiers:** NCM (Capital Market), NGM (Global Market), NGS (Global Select).

---

## SSR — Short Sale Restriction (NASDAQ Trader / Rule 201)

**Data:** Lista de valores sujetos a la restricción de venta en corto de la SEC **Rule 201** (la "Alternative Uptick Rule"). Se activa cuando una acción cae ≥10% respecto al cierre previo; aplica el resto de ese día + el siguiente día hábil. **No prohíbe** vender en corto: solo restringe shorts en dowtick (baja de precio). Útil para filtrar nombres "en play" y detectar presión de cortos reducida en caídas.

**Origen:** NASDAQ Trader (oficial). **ssrlist.com es un viewer de terceros** sobre este feed — JS-rendered, sin API pública. No scrapear ssrlist.com; traer de NASDAQ directo.

**Base URL:** `https://www.nasdaqtrader.com`

| Endpoint | Data | Access | Notes |
|----------|------|--------|-------|
| `/trader.aspx?id=shortSaleCircuitBreaker&symbol=ALL` | SSR list completa (tabla HTML) | live | Página **ASP.NET postback**: la tabla NO está en el HTML estático; hay que replicar el postback (`__VIEWSTATE`/`__EVENTVALIDATION`) o manejar sesión. Frágil. |
| `/rss.aspx?feed=SSRCircuitBreaker` | SSR list (RSS 2.0) | live | Feed canónico machine-readable, pero devuelve la página de error de NASDAQ en un GET simple (requiere sesión/postback). Verificar desde el server de deploy antes de construir sobre esto. |
| `/dynamic/symdir/shorthalts/shorthaltsYYYYMMDD.txt` (relacionado) | Volatility-halt list (CSV) | live | Prueba que los `.txt` CSV estáticos SÍ funcionan en este host. Formato: `Symbol,Security Name,Market Category,Trigger Time`. El path equivalente de SSR aún no está confirmado/descubierto. |

**Columnas SSR esperadas** (según NASDAQ / ssrlist.com): Symbol, Security Name, Market Category (Q=NASDAQ, N=NYSE, A=AMEX, Z=BATS, V=IEX, R=OTC/Regional), Trigger Time.

**Access type:** live (re-fetch por día hábil; la lista cambia intraday a medida que se disparan nombres).

**Integración recomendada** (espeja providers existentes):
- `core/ssr_provider.py` → `SSRProvider.fetch(date?)` devolviendo `[{symbol, name, market, trigger_time, date}]`, con caché TTL (como `SplitsProvider`/`SmallCapLabProvider`).
- Registro en `core/shared/deps.py`.
- Endpoint `GET /api/ssr` en `modules/lists/routes.py` (count + lista), igual que `/api/reverse-splits`.

**Gotchas (verificados 2026-07-13):**
1. ssrlist.com no tiene API y es client-rendered → scrapearlo exige headless browser. Capa equivocada.
2. La página SSR de NASDAQ requiere postback ASP.NET para devolver la tabla (un fetch estático solo trae el chrome de la página).
3. El feed `rss.aspx?feed=SSRCircuitBreaker` tira 404 / error page en GET simple desde sandbox; podría funcionar desde el server de deploy con sesión correcta — confirmar antes de usarlo.
4. Los `.txt` CSV estáticos SÍ funcionan en `nasdaqtrader.com` (verificado con `shorthalts20260713.txt`), así que un `.txt` de SSR probablemente existe — hallar el path/nombre exacto antes de elegir la estrategia de parser.

---

## TradingView (Live)

| Data | Description |
|------|-------------|
| premarket signals | premarket change and volume |
| postmarket signals | postmarket change and volume |

**Columns:** close, premarket_change, postmarket_change, premarket_volume, volume, market_cap_basic, float_shares_outstanding, relative_volume_10d_calc, average_volume_30d_calc

---

## StockTitan (Live — Scraper)

| URL | Data |
|-----|------|
| `/overview/{symbol}/` | news headline, SEC filing title, price impact |
| `/sec-filings/{symbol}/form-4.html` | insider trades |
| `/sec-filings/{symbol}` | filings summary |
| `/short-interest/{symbol}` | short_percent, days_to_cover |
| `/momentum` | stocks with % change (momentum alerts) |

---

## LowFloat (Live — Scraper)

| URL | Data | Frequency |
|-----|------|-----------|
| `/all/` | all exchanges | Weekly |
| `/all_with_otcbb/` | all + OTCBB | Weekly |
| `/nasdaq/` | Nasdaq only | Weekly |
| `/nyse/` | NYSE only | Weekly |
| `/amex/` | AMEX only | Weekly |
| `/otcbb/` | OTCBB only | Weekly |

**Columns:** Ticker, Company, Exchange, Float, Outstd, ShortInt, Industry
**Filter:** Float < 10M shares

---

## HighShortInterest (Live — Scraper)

**Base URL:** `https://www.highshortinterest.com`

| URL | Data | Frequency |
|-----|------|-----------|
| `/all/` | all exchanges | Weekly |
| `/nasdaq/` | Nasdaq only | Weekly |
| `/nyse/` | NYSE only | Weekly |
| `/amex/` | AMEX only | Weekly |

**Columns:** Ticker, Company, Exchange, ShortInt, Float, Outstd, Industry
**Filter:** Short interest > 20%

---

## PennyStockFlow (Live — Scraper)

| URL | Data | Frequency |
|-----|------|-----------|
| `/` | Today's hot penny stocks | Daily / Intradía |

**Columns:** Ticker, Price, Change, $ Volume, Volume, # Trades

---

## ChartExchange (Live — Scraper)

| Endpoint | Data |
|----------|------|
| `/trends/reddit/mentions/{subreddits}/` | Reddit mentions and sentiment from 232 subreddits |

**Subreddit groups:** pennystocks (default), wallstreetbets, crypto, stocks, all, or custom comma-separated list.

**Data fields:** Ticker, Type (stock/crypto), Name, Mention Count, Mention Change %, Sentiment, 1d Price Change %

---

## StockAnalysis (Live — Scraper)

**Base URL:** `https://stockanalysis.com/stocks/{symbol}/`

### Pages

| Page | URL Path | Data |
|------|----------|------|
| Overview | `/stocks/{symbol}/` | price, market cap, volume, PE, EPS, shares, beta, dividends, analyst rating, price target, earnings date |
| Statistics | `/stocks/{symbol}/statistics/` | valuation, financial position, efficiency, margins, dividends, FCF, share stats, short interest, splits, scores, fair value |
| Company | `/stocks/{symbol}/company/` | description, country, industry, sector, employees, CEO, contact, stock metadata (CIK, CUSIP, ISIN, SIC, fiscal year) |
| Financials | `/stocks/{symbol}/financials/` | income statement, balance sheet, cash flow (annual/quarterly/TTM) |
| Forecast | `/stocks/{symbol}/forecast/` | analyst price targets, estimates |
| Ratings | `/stocks/{symbol}/ratings/` | analyst ratings detail |
| Filings | `/stocks/{symbol}/filings/` | SEC filings summary |
| Employees | `/stocks/{symbol}/employees/` | employee count |
| Market Cap | `/stocks/{symbol}/market-cap/` | market cap breakdown |
| History | `/stocks/{symbol}/history/` | historical price data |
| Transcripts | `/stocks/{symbol}/transcripts/` | earnings call transcripts |

### Statistics — Fields

| Category | Fields |
|-----------|--------|
| **Valuation** | PE, Forward PE, PS, PB, PEG, P/FCF, P/OCF, EV/EBITDA, EV/EBIT, EV/FCF |
| **Financial Position** | Current Ratio, Quick Ratio, Debt/Equity, Debt/EBITDA, Interest Coverage |
| **Efficiency** | ROE, ROA, ROIC, ROCE, WACC, Asset Turnover, Inventory Turnover, Revenue/Employee, Profits/Employee |
| **Balance Sheet** | Cash, Total Debt, Net Cash, Net Cash Per Share, Equity (Book Value), Book Value Per Share, Working Capital |
| **Cash Flow** | Operating Cash Flow, FCF, FCF Per Share, Net Borrowing, Depreciation & Amortization |
| **Share Stats** | Shares Outstanding, Shares Change YoY/QoQ, Insider %, Institution %, Float |
| **Short Selling** | Short Interest, Short Previous Month, Short % of Shares Out, Short Ratio |
| **Dividends** | Dividend Per Share, Yield, Growth YoY, Payout Ratio, Buyback Yield, Shareholder Yield, Earnings Yield, FCF Yield |
| **Stock Splits** | Last Split Date, Split Type, Split Ratio |
| **Scores** | Altman Z-Score, Piotroski F-Score |
| **Fair Value** | Lynch Fair Value, Graham Number, Lynch/Graham Upside |
| **Stock Price** | Beta, 52-Week Change, 50/200-Day MA, RSI (14), Avg Volume (20D) |

### Company — Fields

Name, Description, Country, Industry, Sector, Employees, CEO, Address, Phone, Website, Ticker Symbol, Exchange, Fiscal Year, Reporting Currency, CIK Code, CUSIP Number, ISIN Number, Employer ID, SIC Code.

### Financials — Fields

| Statement | Line items |
|-----------|------------|
| **Income Statement** | Revenue, SG&A, R&D, Operating Expenses, EBIT, Interest Income/Expense, Pretax Income, Net Income, EPS, Shares Outstanding, EBITDA, Free Cash Flow |
| **Balance Sheet** | Cash, Total Debt, Net Cash, Equity (Book Value), Working Capital |
| **Cash Flow** | Operating Cash Flow, Capital Expenditures, Depreciation & Amortization, Net Borrowing, Free Cash Flow |

Modes: Annual, Quarterly, TTM

---

## iBorrowDesk (Live — API)

| URL | Data |
|-----|------|
| `https://iborrowdesk.com/api/ticker/{ticker}` | cost to borrow: fee, available, updated timestamp, 5-day history |

---

## AskEdgar (Live — API)

| URL | Data |
|-----|------|
| `https://api.askedgar.io/v1/fmp/company/{ticker}/profile` | company profile: price, change, market cap, sector, industry, exchange, range, volAvg |
| `https://api.askedgar.io/v1/fmp/company/{ticker}/share-float` | float shares, outstanding shares |
| `https://api.askedgar.io/v1/fmp/form13f/inst-ownership?symbol={ticker}` | institutional ownership % |
| `https://api.askedgar.io/v1/screener/{ticker}/overview-page` | short interest (shares), short float % |
| `https://app.askedgar.io/ticker/{ticker}` (POST) | offering risk rating |

---

## SmallCapLab (Live — API + Scraper)

**Base URL:** `https://www.smallcaplab.com`

Specialized dilution risk analysis and pump-timing detection for small-cap stocks.

### Internal API

In addition to the public HTML page, SmallCapLab exposes internal JSON endpoints:

| Endpoint | Data |
|----------|------|
| `/ticker/{ticker}` | Public HTML page — dilution risk, capital structure, filings, offering mechanisms |
| `/api/pump-timing/{ticker}` | Pump-and-dump pattern detection correlated to SEC filings |

#### `/api/pump-timing/{ticker}`

Detects price pumps and correlates them with subsequent dilution filings (ATM, offerings). Returns:

| Field | Type | Description |
|-------|------|-------------|
| `totalPumps` | int | Total pump events detected |
| `matchedPumps` | int | Pumps followed by a dilution filing within matchWindow |
| `hitRatePct` | int | % of pumps that preceded a filing |
| `medianDays` | int | Median days from pump to filing |
| `avgDays` | int | Average days from pump to filing |
| `fastestDays` / `slowestDays` | int | Min/max gap |
| `links[]` | array | Each pump with `pump` (startDate, peakDate, returnPct, durationDays, peakVolMult) and optional `match` (date, form, label, confidence) |
| `allFilings[]` | array | All dilution-relevant SEC filings (date, form, label, confidence) |
| `priceHistory[]` | array | Daily close prices (date, close) — ~2yr lookback |
| `cfg` | object | Detection config: thresholdPct, lookbackBars, volumeMultiple, cooldownBars, matchWindowDays, maxPumps, priceHistoryDays |

**Confidence levels for matches:** `high`, `medium`, `low`

**Form types detected:** 424B1 (Prospectus), 424B3 (Resale Prospectus), 424B4 (Follow-on Offering), 424B5 (ATM Filing), S-8 (Stock Plan), 13G (>5% Holder)

### HTML Page Data Fields

Scraped from the public ticker page at `/ticker/{ticker}`:

| Category | Fields |
|----------|--------|
| **Dilution Risk Score** | Overall rating (Low/Moderate/High), Structural score (1-10), Near-Term score (1-10), Severity score (1-8), ATM Drip flag |
| **Survival / Financing Pressure** | Cash on hand (SEC), quarterly burn, estimated runway (months), 30d ADTV, ATM capacity ($/day) |
| **Dilution Activity** | YoY share growth %, Share CAGR (2Y), capital raises count (12mo/24mo/total), last raise (days ago) |
| **Capital Structure** | Charter authorized shares, shares outstanding (SEC), unissued authorized headroom, authorized÷outstanding ratio |
| **Active Offering Mechanisms** | ATM offering (Active/Inactive + 424B5 on file), Shelf Registration (S-3), Resale Prospectus (424B3), Private Placement (Form D), Reverse Split Signal |
| **Trading Relevance** | Shares outstanding, 30d ADTV |
| **Historical Capital Raises** | Prospectus/offering filings timeline (date, transaction type, SEC document link) |
| **Catalyst Timeline** | 6-month lookback / 12-month outlook with filing dates |
| **SEC Filing History** | Dilution-relevant filings (424B, S-1/S-3/S-8, 13G, Form D) with dates and document links |

**Source:** SEC EDGAR XBRL data (cash, burn, shares outstanding, authorized shares, filing history) + market data (ADTV).

**Note:** Free tier shows current ticker data. Some historical data and API access may require subscription.

---

## StockTwits (Live — Internal v2 API)

Social network for traders/investors. The distinctive value vs. every other source here is **retail sentiment**: an aggregated bullish/bearish poll per symbol returned as a daily historical series. Also exposes a public message stream (social buzz), trending symbols, and per-symbol watchlist counts.

**Web base URL:** `https://stocktwits.com`
**API base URL:** `https://api.stocktwits.com/api/2/`

> ⚠️ These are StockTwits' **internal v2 endpoints** (the ones the web app itself calls), NOT the official developer API. The legacy v1 developer API (`api.stocktwits.com/api/1/...`) is deprecated and now requires an approved token. The v2 endpoints currently need **no API key**, but:
> - You MUST send a browser `User-Agent` header, or they return 404.
> - Stability is not guaranteed — internal endpoints can change without notice and may be rate-limited. Treat as fragile (same risk class as a scraper).

### Endpoints

| Endpoint | Data |
|----------|------|
| `/api/2/symbols/{symbol}/sentiment.json` | **Social sentiment**: daily `bullish`% / `bearish`% poll series (most recent first) |
| `/api/2/streams/symbol/{symbol}.json` | Message stream for a symbol → body, user, `created_at`, `symbols`, `prices`; `cursor` for pagination |
| `/api/2/streams/trending.json` | Trending symbols stream (messages referencing currently-trending tickers) |
| `/api/2/symbols/{symbol}.json` | Symbol metadata (returns 404 for some symbols — use the `symbol` object inside stream responses instead) |

### Sentiment — `/api/2/symbols/{symbol}/sentiment.json`

Returns `data[]` (newest first), each element:

| Field | Type | Description |
|-------|------|-------------|
| `bullish` | float | % of bullish votes (0–100) |
| `bearish` | float | % of bearish votes (0–100) |
| `timestamp` | string | ISO date (`YYYY-MM-DD`) |

Example (SKYQ, 2026-07-12): `bullish: 94.03, bearish: 5.97`. The series gives a sentiment trend over time — useful as a contrarian or momentum social signal.

### Streams — fields of interest

Within `/api/2/streams/symbol/{symbol}.json` and `/api/2/streams/trending.json`:

| Field | Use |
|-------|-----|
| `messages[].body` / `tokenized_body` | Post text (cashtags preserved) |
| `messages[].created_at` | Timestamp (ISO 8601, UTC) — count posts per window to derive **message volume / social buzz** |
| `messages[].user` | Author (`username`, `followers`, `classification`, `badges`) |
| `messages[].symbols[]` | Referenced tickers (`symbol`, `title`) |
| `messages[].prices[]` | Price tagged at post time |
| `symbol.watchlist_count` | How many users watch the symbol — **social-interest** metric |
| `cursor` | Opaque pagination token for the next page |

### Gotchas (verified)

1. **Per-message `sentiment` is `null`** in public streams — users rarely tag bull/bear on individual posts. Do NOT use per-message sentiment. The reliable signal is the **aggregated `/sentiment` poll**.
2. **Crypto tickers use a `.X` suffix** (e.g. `ETH.X`, `XRP.X`) in streams and trending.
3. **No auth today, but fragile** — internal endpoint, browser UA required, rate limits possible. Wrap in retry/backoff + cache.
4. `watchlist_count` and message volume are the best proxies for retail attention; the poll is the best proxy for retail *direction*.

---

## DefeatBeta API (Cached — Parquet via DuckDB HTTPFS)

**GitHub:** https://github.com/defeat-beta/defeatbeta-api
**PyPI:** `defeatbeta-api`
**Dataset:** https://huggingface.co/datasets/defeatbeta/yahoo-finance-data
**License:** Apache 2.0
**Python:** 3.11+
**Stars:** ~683 (Jul 2026)

### Overview

El **DefeatBeta API** es una alternativa open-source a Yahoo Finance con mayor confiabilidad. En lugar de depender de scraping de Yahoo (frágil, rate-limited), hostea todo el dataset de Yahoo Finance transformado en **Parquet files** en Hugging Face, y usa **DuckDB + cache_httpfs** para consultarlos con SQL directo sobre HTTP. No necesita API key, no tiene rate limits, y da respuestas sub-segundo.

### Arquitectura Interna

```
┌──────────────┐    ┌────────────────────┐    ┌──────────────────────┐
│   Ticker()   │───→│  HuggingFaceClient │───→│ HF dataset (Parquet) │
│              │    │  (HTTP metadata)    │    │  (read-only, hosted) │
│  price()     │    └────────────────────┘    └────────┬─────────────┘
│  news()      │                                      │
│  info()      │    ┌────────────────────┐            │  HTTP range requests
│  transcripts │───→│   DuckDBClient     │←───────────┘  (cache_httpfs)
│  statements  │    │  (in-memory OLAP)  │
│  ...         │    │  SQL on parquet    │
└──────────────┘    └────────────────────┘
```

**Flujo de datos:**
1. `HuggingFaceClient` obtiene metadatos (update_time desde `spec.json`) y genera URLs de Parquet vía API REST de HuggingFace.
2. `DuckDBClient` conecta DuckDB en memoria y ejecuta SQL directamente sobre los archivos `.parquet` via HTTP range requests.
3. `cache_httpfs` (extensión DuckDB) cachea los bytes localmente → segundas consultas son instantáneas.
4. Las SQL queries viven en archivos `.sql` en `defeatbeta_api/data/sql/`, cargadas por `sql_loader.py` con templates tipo `{ticker}`, `{url}`.

### Caché Local

- **Ubicación:** `~/.cache/httpfs/` (configurable)
- **Validación automática:** Al iniciar, compara `update_time` del `spec.json` remoto vs local. Si cambió, limpia el caché.
- **TTL:** Determinado por el update_time del dataset (se actualiza cada ~24h)

### Datasets Disponibles (15 tablas Parquet)

Cada tabla es un archivo `.parquet` en Hugging Face datasets (`defeatbeta/yahoo-finance-data`):

| Constante Interna | Dataset (Parquet) | Contenido | Período |
|---|---|---|---|
| `stock_prices` | `stock_prices.parquet` | OHLCV diario (open, high, low, close, volume) | ~15+ años |
| `stock_profile` | `stock_profile.parquet` | Perfil empresa: sector, industry, employees, descripción, website | Actual |
| `stock_officers` | `stock_officers.parquet` | Ejecutivos clave (CEO, CFO, etc.) | Actual |
| `stock_earning_calendar` | `stock_earning_calendar.parquet` | Calendario de earnings (fecha reporte, estimados) | Próximos quarters |
| `stock_tailing_eps` | `stock_tailing_eps.parquet` | TTM EPS histórico | ~15+ años |
| `stock_statement` | `stock_statement.parquet` | Financial statements (income, balance, cash flow) en formato largo con template_type | ~10+ años |
| `stock_dividend_events` | `stock_dividend_events.parquet` | Dividendos históricos (monto, frecuencia) | ~15+ años |
| `stock_split_events` | `stock_split_events.parquet` | Stock splits (ratio, fecha) | ~15+ años |
| `stock_earning_call_transcripts` | `stock_earning_call_transcripts.parquet` | Transcripciones earnings calls (texto completo) | ~15+ años |
| `stock_news` | `stock_news.parquet` | Noticias financieras (título, publisher, cuerpo, uuid) | ~5+ años |
| `stock_revenue_breakdown` | `stock_revenue_breakdown.parquet` | Revenue por segmento/geografía/producto | ~5+ años |
| `stock_shares_outstanding` | `stock_shares_outstanding.parquet` | Shares outstanding histórico | ~15+ años |
| `exchange_rate` | `exchange_rate.parquet` | FX rates diarios | ~15+ años |
| `daily_treasury_yield` | `daily_treasury_yield.parquet` | Treasury yields diarios | ~15+ años |
| `stock_sec_filing` | `stock_sec_filing.parquet` | SEC filings metadata | ~15+ años |

### API Python - Todas las Métricas Disponibles

#### Categoría 0 - Precio

| Método | Returns | Descripción |
|---|---|---|
| `ticker.price()` | DataFrame | OHLCV diario completo |
| `ticker.beta(period, benchmark)` | DataFrame | Beta vs benchmark (5y default) |

#### Categoría 1 - Financieros (Statements)

| Método | Returns | Descripción |
|---|---|---|
| `ticker.quarterly_income_statement()` | `Statement` object | Quarterly income statement |
| `ticker.annual_income_statement()` | `Statement` object | Annual income statement |
| `ticker.quarterly_balance_sheet()` | `Statement` object | Quarterly balance sheet |
| `ticker.annual_balance_sheet()` | `Statement` object | Annual balance sheet |
| `ticker.quarterly_cash_flow()` | `Statement` object | Quarterly cash flow |
| `ticker.annual_cash_flow()` | `Statement` object | Annual cash flow |
| `ticker.calendar()` | DataFrame | Earnings calendar |
| `ticker.splits()` | DataFrame | Split events |
| `ticker.dividends()` | DataFrame | Dividend events |
| `ticker.shares()` | DataFrame | Shares outstanding |
| `ticker.ttm_eps()` | DataFrame | TTM EPS |
| `ticker.ttm_revenue()` | DataFrame | TTM Revenue (conversión USD) |
| `ticker.ttm_fcf()` | DataFrame | TTM Free Cash Flow (conversión USD) |
| `ticker.ttm_net_income_common_stockholders()` | DataFrame | TTM Net Income (conversión USD) |
| `ticker.ttm_ebitda()` | DataFrame | TTM EBITDA (conversión USD) |
| `ticker.quarterly_revenue_by_breakdown()` | DataFrame | Revenue breakdown (segment/geo/product) largo |
| `ticker.trailing_revenue_by_breakdown()` | DataFrame | TTM revenue breakdown largo |

#### Categoría 2 - Valoración (Valuation)

| Método | Returns | Descripción |
|---|---|---|
| `ticker.ttm_pe()` | DataFrame | TTM P/E (merge asof price + TTM EPS) |
| `ticker.market_capitalization()` | DataFrame | Market cap histórico (price × shares) |
| `ticker.ps_ratio()` | DataFrame | P/S ratio histórico |
| `ticker.pb_ratio()` | DataFrame | P/B ratio histórico |
| `ticker.peg_ratio()` | DataFrame | PEG ratio (P/E ÷ EPS growth) |
| `ticker.enterprise_value()` | DataFrame | EV = market cap + debt + minority + preferred - cash |
| `ticker.enterprise_to_revenue()` | DataFrame | EV/Revenue |
| `ticker.enterprise_to_ebitda()` | DataFrame | EV/EBITDA |
| `ticker.net_debt_ttm()` | DataFrame | Net debt TTM |
| `ticker.debt_to_equity()` | DataFrame | Debt/Equity |
| `ticker.rope()` | DataFrame | ROE calculado |
| `ticker.roa()` | DataFrame | ROA calculado |
| `ticker.roic()` | DataFrame | ROIC calculado |
| `ticker.roce()` | DataFrame | ROCE calculado |
| `ticker.wacc()` | DataFrame | WACC (con treasury, beta, S&P500) |
| `ticker.equity_multiplier()` | DataFrame | Equity multiplier (ROE/ROA) |
| `ticker.asset_turnover()` | DataFrame | Asset turnover (ROA/net margin) |

#### Categoría 3 - Crecimiento (Growth)

| Método | Returns | Descripción |
|---|---|---|
| `ticker.quarterly_revenue_yoy_growth()` | DataFrame | Revenue YoY % |
| `ticker.annual_revenue_yoy_growth()` | DataFrame | Revenue YoY % anual |
| `ticker.quarterly_operating_income_yoy_growth()` | DataFrame | Operating income YoY % |
| `ticker.annual_ebitda_yoy_growth()` | DataFrame | EBITDA YoY % anual |
| `ticker.quarterly_net_income_yoy_growth()` | DataFrame | Net income YoY % |
| `ticker.annual_fcf_yoy_growth()` | DataFrame | FCF YoY % anual |
| `ticker.quarterly_eps_yoy_growth()` | DataFrame | EPS YoY % |
| `ticker.quarterly_ttm_eps_yoy_growth()` | DataFrame | TTM EPS YoY % |

#### Categoría 4 - Rentabilidad (Profitability)

| Método | Returns | Descripción |
|---|---|---|
| `ticker.quarterly_gross_margin()` | DataFrame | Gross margin % |
| `ticker.annual_gross_margin()` | DataFrame | Gross margin % anual |
| `ticker.quarterly_operating_margin()` | DataFrame | Operating margin % |
| `ticker.annual_net_margin()` | DataFrame | Net margin % anual |
| `ticker.quarterly_ebitda_margin()` | DataFrame | EBITDA margin % |
| `ticker.annual_fcf_margin()` | DataFrame | FCF margin % anual |

#### Categoría 5 - Fundamentales (Info & News)

| Método | Returns | Descripción |
|---|---|---|
| `ticker.info()` | DataFrame | Company profile completo |
| `ticker.sec_filing()` | DataFrame | SEC filings metadata |
| `ticker.officers()` | DataFrame | Key executives |
| `ticker.currency(symbol)` | DataFrame | FX rates históricos |
| `ticker.earning_call_transcripts()` | `Transcripts` object | Earnings call transcripts (get_transcripts_list, get_transcript) |
| `ticker.news()` | `News` object | Financial news (get_news_list, get_news by uuid) |

#### Categoría 6 - Comparación por Industria

| Método | Descripción |
|---|---|
| `ticker.industry_ttm_pe()` | Industry TTM P/E |
| `ticker.industry_ps_ratio()` | Industry P/S |
| `ticker.industry_pb_ratio()` | Industry P/B |
| `ticker.industry_roe()` | Industry ROE |
| `ticker.industry_roic()` | Industry ROIC |
| `ticker.industry_quarterly_gross_margin()` | Industry gross margin |

#### Batch - `Tickers` Class

`Tickers(['AAPL', 'TSLA', 'NVDA'])` ejecuta queries en paralelo vía `ThreadPoolExecutor`. Cada método retorna o un DataFrame combinado o un dict `{symbol: object}`.

### Uso en LT (Nuestro Proyecto)

**Wrapper:** `core/defeatbeta_provider.py` — `DefeatBetaProvider` como wrapper DI.

**Endpoints expuestos:**
- `GET /api/news/{symbol}` → `ticker.news().get_news_list()` (news headlines)

**Uso actual:**
- Enriquecimiento de calendario con `has_news` + headline.
- Enriquecimiento de scan results con headline + publisher.
- Startup pre-warm: `fetch_news_batch` (WHERE symbol IN (...)) vía DuckDB directo.

**Instalación / Dependencias:**
```toml
# pyproject.toml
[project.dependencies]
defeatbeta-api = ">=0.0.43"
```

**Override de pandas** (por conflicto con tradingview-screener):
```toml
[tool.uv]
override-dependencies = [
    "pandas>=3.0.1",
]
```

### Ventajas sobre Yahoo Finance Directo

| Aspecto | Yahoo Finance (directo) | DefeatBeta API |
|---|---|---|
| Rate limits | Sí (~200 req/min) | No (Parquet estáticos) |
| Confiabilidad | Baja (scraping, caídas frecuentes) | Alta (HF datasets) |
| Velocidad | 200-500ms por request | <100ms (cached) / 2-5s (cold, primer query) |
| Datos históricos | Limitado a 2-3 años (intraday) | 15+ años completo |
| Earnings transcripts | No disponible nativo | Sí, completos |
| Revenue breakdown | No disponible | Sí, por segmento/geo/producto |
| DCF / Excel report | No | Sí (integrados) |
| LLM analysis | No | Sí (OpenAI integrado) |
| Windows support | N/A | Sí (v0.0.60+) |

### Limitaciones Conocidas

1. **Cold start:** La primera query después de instalar o limpiar caché tarda 2-5s (descarga bytes del parquet via HTTP range).
2. **DuckDB in-memory:** Solo singleton; no recomendado para workloads multi-process pesados.
3. **Datos diarios únicamente:** No hay barras intraday (minute, hour). Solo daily OHLCV.
4. **Sin streaming:** No soporta datos en tiempo real; es un batch histórico actualizado ~24h.
5. **Ticker único por instancia:** Cada `Ticker` se instancia con un símbolo; usar `Tickers` para batch.
6. **Windows:** Nativo desde v0.0.60; versiones anteriores requerían WSL.