docs/data-sources.md

Data Sources

================================================================================ INDEX

Data	Sources
OHLCV daily	DuckDB (cached), Alpaca (live), Yahoo (live), DefeatBeta (cached)
minute bars	Alpaca (live), DuckDB cache (cached)
hour bars	Alpaca (live)
gainers/losers	Webull (live, API + scraper), Finviz (live), TradingView (live)
premarket/afterhours	Webull (live, API + scraper)
Nueva:	sector
Nueva:	industry
Nueva:	market_cap
symbols list	NASDAQ API (live)
deficient/non-compliant listing	NASDAQ API (live)
catalyst (why moving)	Finviz API (live)
low float stocks	LowFloat (live)
high short interest	HighShortInterest (live)
hot penny stocks	PennyStockFlow (live)
reddit mentions	ChartExchange (live)
Nueva:	social sentiment (bullish/bearish %)
Nueva:	message volume / social buzz
Nueva:	trending symbols
Nueva:	watchlist count (social interest)
earnings date	Yahoo (live), SEC 10-K (live), DefeatBeta (cached)
earnings history	DefeatBeta (cached)
news	Yahoo (live), DefeatBeta (cached)
news headline	StockTitan overview (live)
short interest	StockTitan (live), StockAnalysis (live)
press releases	DefeatBeta (cached)
insider trades	SEC Form 4 (live), OpenInsider (live), StockTitan (live), DefeatBeta (cached)
Nueva:	employees
Nueva:	company description
fundamentals	Yahoo (live), Finviz (live), StockAnalysis (live), DefeatBeta (cached)
Nueva:	valuation ratios (PE, PB, PS, PEG...)
Nueva:	financial position (current ratio, debt/equity...)
Nueva:	efficiency (ROE, ROIC, WACC...)
Nueva:	balance sheet (cash, debt, equity, book value...)
Nueva:	cash flow (OCF, FCF, capex...)
Nueva:	share stats (insider %, institution %, float)
Nueva:	dividends (yield, growth, payout)
Nueva:	stock splits (date, ratio, type)
Nueva:	scores (Altman Z, Piotroski F)
Nueva:	fair value (Lynch, Graham)
Nueva:	key executives
Nueva:	contact (address, phone, website)
Nueva:	stock metadata (CUSIP, ISIN, SIC, fiscal year...)
8-K	SEC (live)
S-1/S-3/S-8	SEC (live) — offerings / dilution
10-K/10-Q	SEC (live)
Form 4	SEC (live)
13D/13G	SEC (live) — activist filings
reverse splits	Yahoo (live), Alpaca (cached), SEC 8-K (live)
Nueva:	short sale restriction (SSR / Rule 201 list)
dividends	Yahoo (live), Alpaca (cached)
Nueva:	CIK lookup
trading calendar	DuckDB (cached), Alpaca (live)
supports / resistances	DuckDB (cached) + Alpaca minute (live)
cost to borrow (fee, available, history)	iBorrowDesk (live)
offering risk rating	AskEdgar (live)
dilution risk (rating, ATM, capital structure, runway)	SmallCapLab (live, API + scraper)
pump timing (pump→filing correlation)	SmallCapLab (live, API)
institutional ownership	AskEdgar (live)
share float / outstanding shares	AskEdgar (live)

Access types:

• cached: obtenido periódicamente y almacenado en local
• live: obtenido al vuelo cuando se necesita

================================================================================ SOURCE DETAILS

DuckDB (Local — Cached)

File	Data
data/alpaca_merged.parquet	Daily OHLCV + computed
data/market_calendar.csv	Trading days
data/corporate_actions.csv	Historical corporate actions
data/symbol_metadata.csv	Stock metadata
data/metadata.db (table: tickers)	CIK mapping
data/cache/{SYMBOL}_{DATE}.parquet	Minute bars cache (TTL: 24h)

---

Alpaca (Live)

Endpoint	Data
/v2/stocks/bars	OHLCV daily/minute/hour
/v2/corporate_actions	splits, dividends, mergers, spin-offs
/v1/calendar	trading calendar

---

Yahoo Finance (Live)

Data	Endpoint / Resource
OHLCV, volumes, pre/post market	/v8/finance/chart/{symbol}
recent news	news feed
earnings calendar	earnings history
splits, dividends	actions feed
press releases	press releases feed

---

Webull (Live — Scraper + API)

Webull expone dos tipos de datos: scraping HTML (páginas públicas) y API interna (endpoints JSON).

Base URL: https://quotes-gw.webullfintech.com

HTML Scraper

Endpoint	Data
/quote/us/gainers	today gainers
/quote/us/gainers/1d	1-day gainers
/quote/us/gainers/5d	5-day gainers
/quote/us/gainers/1m	1-month gainers
/quote/us/gainers/pre	premarket
/quote/us/gainers/after	afterhours
/quote/us/actives/rvol10d	relative volume 10d
/quote/us/actives/turnover_ratio	turnover ratio
/quote/us/actives/range	range %

Internal API — Rankings

Endpoint	Data
/api/wlas/ranking/v9/rise	gainers
/api/wlas/ranking/v9/fall	losers

Parameters: regionId=6 (US market), rankType (timeframe: 1d, 5d, 1m, 3m, 52w, 3min, 5min, gainers.preMarket, gainers.afterMarket), pageIndex, pageSize

Internal API — Micro Trend

Endpoint	Data
/api/bgw/quote/microTrend?ids={tickerIds}	Intraday micro-trend (minute-by-minute price data for current session)

---

Finviz (Live — Scraper + API)

Base URL: https://finviz.com

Views

Vista ID	Vista	Descripción
v=110	Overview	Datos básicos (Ticker, Company, Sector, Market Cap, etc.)
v=120	Valuation	P/E, Forward P/E, PEG, P/S, P/B, etc.
v=160	Financial	EPS, Revenue, Margins, ROE, ROA, etc.
v=130	Ownership	Insider Ownership, Institutional Ownership, Short Interest, etc.
v=140	Performance	Perf Week, Perf Month, Perf YTD, Perf Year, etc.
v=170	Technical	RSI, SMA20/50/200, ATR, Volatility, etc.
v=180	ETF	ETF-specific data (AUM, Holdings, Flows, Expense Ratio)
v=150	Custom	Columnas personalizadas vía parámetro c=

Endpoints

Endpoint	Data
/screener.ashx?v=111	overview signals
/screener.ashx?v=171	technical signals
/screener.ashx?v=150&c={cols}&t={tickers}	custom screener
/api/stocks-why-moving/boxover/{symbol}	catalyst (headline, summary, sentiment)
/quote.ashx?t={symbol}	employees, sector, industry, company description

Custom Screener (v=150)

URL: https://finviz.com/screener?v=150&t={tickers}&c={column_ids}

Parameters: v=150 (custom), t= (tickers, comma-separated), c= (column IDs, comma-separated), r= (optional signal filters)

---

SEC EDGAR (Live)

Base URL: https://www.sec.gov

Endpoint	Data
/files/company_tickers.json	symbol → CIK lookup
/submissions/CIK{cik}.json	all filings

Form types: Form 4 (insider trades), 8-K (material events), 6-K (foreign events), S-1/S-3/S-8/S-4 (offerings), F-1/F-3 (foreign offerings), 13D/13G (activist filings), 10-K/10-Q/20-F (periodic reports), DEF 14A (proxy statements).

Filing HTML URL format: https://www.sec.gov/Archives/edgar/data/{CIK}/{accessionNumberWithoutDashes}/{primaryDocument}

---

OpenInsider (Live — Scraper)

Endpoint	Data
/screener	insider trades (symbol filtered from page)
/cluster-buy	cluster buys
/ceo-trades	CEO-only trades

---

NASDAQ API (Live)

Base URL: https://api.nasdaq.com

Endpoint	Data
/api/screener/stocks?tableonly=false&download=true	Full symbols list with metadata
/api/quote/list-type-extended/listing?queryString=deficient	Non-compliant / deficient companies

Screener — /api/screener/stocks

Returns: symbol, name, lastsale (price), netchange, pctchange, volume, marketCap, country, ipoyear, industry, sector, url.

Note: NASDAQ only. NYSE/AMEX not included.

Non-Compliant / Deficient — /api/quote/list-type-extended/listing

Lists NASDAQ issuers non-compliant with listing requirements. Parameter: queryString=deficient (required). No pagination — returns full list (~350 issuers).

Deficiency types observed: Bid Price, Equity, Market Value of Listed Securities, Delinquent filings, Market Value of Publicly Held Shares, Shareholders, Annual Shareholder Meeting, Audit Committee Composition, Net Income, Board Independence, and others.

Market tiers: NCM (Capital Market), NGM (Global Market), NGS (Global Select).

---

SSR — Short Sale Restriction (NASDAQ Trader / Rule 201)

Data: Lista de valores sujetos a la restricción de venta en corto de la SEC Rule 201 (la "Alternative Uptick Rule"). Se activa cuando una acción cae ≥10% respecto al cierre previo; aplica el resto de ese día + el siguiente día hábil. No prohíbe vender en corto: solo restringe shorts en dowtick (baja de precio). Útil para filtrar nombres "en play" y detectar presión de cortos reducida en caídas.

Origen: NASDAQ Trader (oficial). ssrlist.com es un viewer de terceros sobre este feed — JS-rendered, sin API pública. No scrapear ssrlist.com; traer de NASDAQ directo.

Base URL: https://www.nasdaqtrader.com

Endpoint	Data	Access	Notes
/trader.aspx?id=shortSaleCircuitBreaker&symbol=ALL	SSR list completa (tabla HTML)	live	Página ASP.NET postback: la tabla NO está en el HTML estático; hay que replicar el postback (__VIEWSTATE/__EVENTVALIDATION) o manejar sesión. Frágil.
/rss.aspx?feed=SSRCircuitBreaker	SSR list (RSS 2.0)	live	Feed canónico machine-readable, pero devuelve la página de error de NASDAQ en un GET simple (requiere sesión/postback). Verificar desde el server de deploy antes de construir sobre esto.
/dynamic/symdir/shorthalts/shorthaltsYYYYMMDD.txt (relacionado)	Volatility-halt list (CSV)	live	Prueba que los .txt CSV estáticos SÍ funcionan en este host. Formato: Symbol,Security Name,Market Category,Trigger Time. El path equivalente de SSR aún no está confirmado/descubierto.

Columnas SSR esperadas (según NASDAQ / ssrlist.com): Symbol, Security Name, Market Category (Q=NASDAQ, N=NYSE, A=AMEX, Z=BATS, V=IEX, R=OTC/Regional), Trigger Time.

Access type: live (re-fetch por día hábil; la lista cambia intraday a medida que se disparan nombres).

Integración recomendada (espeja providers existentes):

• core/ssr_provider.py → SSRProvider.fetch(date?) devolviendo [{symbol, name, market, trigger_time, date}], con caché TTL (como SplitsProvider/SmallCapLabProvider).
• Registro en core/shared/deps.py.
• Endpoint GET /api/ssr en modules/lists/routes.py (count + lista), igual que /api/reverse-splits.

Gotchas (verificados 2026-07-13):

1. ssrlist.com no tiene API y es client-rendered → scrapearlo exige headless browser. Capa equivocada.
2. La página SSR de NASDAQ requiere postback ASP.NET para devolver la tabla (un fetch estático solo trae el chrome de la página).
3. El feed rss.aspx?feed=SSRCircuitBreaker tira 404 / error page en GET simple desde sandbox; podría funcionar desde el server de deploy con sesión correcta — confirmar antes de usarlo.
4. Los .txt CSV estáticos SÍ funcionan en nasdaqtrader.com (verificado con shorthalts20260713.txt), así que un .txt de SSR probablemente existe — hallar el path/nombre exacto antes de elegir la estrategia de parser.

---

TradingView (Live)

Data	Description
premarket signals	premarket change and volume
postmarket signals	postmarket change and volume

Columns: close, premarket_change, postmarket_change, premarket_volume, volume, market_cap_basic, float_shares_outstanding, relative_volume_10d_calc, average_volume_30d_calc

---

StockTitan (Live — Scraper)

URL	Data
/overview/{symbol}/	news headline, SEC filing title, price impact
/sec-filings/{symbol}/form-4.html	insider trades
/sec-filings/{symbol}	filings summary
/short-interest/{symbol}	short_percent, days_to_cover
/momentum	stocks with % change (momentum alerts)

---

LowFloat (Live — Scraper)

URL	Data	Frequency
/all/	all exchanges	Weekly
/all_with_otcbb/	all + OTCBB	Weekly
/nasdaq/	Nasdaq only	Weekly
/nyse/	NYSE only	Weekly
/amex/	AMEX only	Weekly
/otcbb/	OTCBB only	Weekly

Columns: Ticker, Company, Exchange, Float, Outstd, ShortInt, Industry Filter: Float < 10M shares

---

HighShortInterest (Live — Scraper)

Base URL: https://www.highshortinterest.com

URL	Data	Frequency
/all/	all exchanges	Weekly
/nasdaq/	Nasdaq only	Weekly
/nyse/	NYSE only	Weekly
/amex/	AMEX only	Weekly

Columns: Ticker, Company, Exchange, ShortInt, Float, Outstd, Industry Filter: Short interest > 20%

---

PennyStockFlow (Live — Scraper)

URL	Data	Frequency
/	Today's hot penny stocks	Daily / Intradía

Columns: Ticker, Price, Change, $ Volume, Volume, # Trades

---

ChartExchange (Live — Scraper)

Endpoint	Data
/trends/reddit/mentions/{subreddits}/	Reddit mentions and sentiment from 232 subreddits

Subreddit groups: pennystocks (default), wallstreetbets, crypto, stocks, all, or custom comma-separated list.

Data fields: Ticker, Type (stock/crypto), Name, Mention Count, Mention Change %, Sentiment, 1d Price Change %

---

StockAnalysis (Live — Scraper)

Base URL: https://stockanalysis.com/stocks/{symbol}/

Pages

Page	URL Path	Data
Overview	/stocks/{symbol}/	price, market cap, volume, PE, EPS, shares, beta, dividends, analyst rating, price target, earnings date
Statistics	/stocks/{symbol}/statistics/	valuation, financial position, efficiency, margins, dividends, FCF, share stats, short interest, splits, scores, fair value
Company	/stocks/{symbol}/company/	description, country, industry, sector, employees, CEO, contact, stock metadata (CIK, CUSIP, ISIN, SIC, fiscal year)
Financials	/stocks/{symbol}/financials/	income statement, balance sheet, cash flow (annual/quarterly/TTM)
Forecast	/stocks/{symbol}/forecast/	analyst price targets, estimates
Ratings	/stocks/{symbol}/ratings/	analyst ratings detail
Filings	/stocks/{symbol}/filings/	SEC filings summary
Employees	/stocks/{symbol}/employees/	employee count
Market Cap	/stocks/{symbol}/market-cap/	market cap breakdown
History	/stocks/{symbol}/history/	historical price data
Transcripts	/stocks/{symbol}/transcripts/	earnings call transcripts

Statistics — Fields

Category	Fields
Valuation	PE, Forward PE, PS, PB, PEG, P/FCF, P/OCF, EV/EBITDA, EV/EBIT, EV/FCF
Financial Position	Current Ratio, Quick Ratio, Debt/Equity, Debt/EBITDA, Interest Coverage
Efficiency	ROE, ROA, ROIC, ROCE, WACC, Asset Turnover, Inventory Turnover, Revenue/Employee, Profits/Employee
Balance Sheet	Cash, Total Debt, Net Cash, Net Cash Per Share, Equity (Book Value), Book Value Per Share, Working Capital
Cash Flow	Operating Cash Flow, FCF, FCF Per Share, Net Borrowing, Depreciation & Amortization
Share Stats	Shares Outstanding, Shares Change YoY/QoQ, Insider %, Institution %, Float
Short Selling	Short Interest, Short Previous Month, Short % of Shares Out, Short Ratio
Dividends	Dividend Per Share, Yield, Growth YoY, Payout Ratio, Buyback Yield, Shareholder Yield, Earnings Yield, FCF Yield
Stock Splits	Last Split Date, Split Type, Split Ratio
Scores	Altman Z-Score, Piotroski F-Score
Fair Value	Lynch Fair Value, Graham Number, Lynch/Graham Upside
Stock Price	Beta, 52-Week Change, 50/200-Day MA, RSI (14), Avg Volume (20D)

Company — Fields

Name, Description, Country, Industry, Sector, Employees, CEO, Address, Phone, Website, Ticker Symbol, Exchange, Fiscal Year, Reporting Currency, CIK Code, CUSIP Number, ISIN Number, Employer ID, SIC Code.

Financials — Fields

Statement	Line items
Income Statement	Revenue, SG&A, R&D, Operating Expenses, EBIT, Interest Income/Expense, Pretax Income, Net Income, EPS, Shares Outstanding, EBITDA, Free Cash Flow
Balance Sheet	Cash, Total Debt, Net Cash, Equity (Book Value), Working Capital
Cash Flow	Operating Cash Flow, Capital Expenditures, Depreciation & Amortization, Net Borrowing, Free Cash Flow

Modes: Annual, Quarterly, TTM

---

iBorrowDesk (Live — API)

URL	Data
https://iborrowdesk.com/api/ticker/{ticker}	cost to borrow: fee, available, updated timestamp, 5-day history

---

AskEdgar (Live — API)

URL	Data
https://api.askedgar.io/v1/fmp/company/{ticker}/profile	company profile: price, change, market cap, sector, industry, exchange, range, volAvg
https://api.askedgar.io/v1/fmp/company/{ticker}/share-float	float shares, outstanding shares
https://api.askedgar.io/v1/fmp/form13f/inst-ownership?symbol={ticker}	institutional ownership %
https://api.askedgar.io/v1/screener/{ticker}/overview-page	short interest (shares), short float %
https://app.askedgar.io/ticker/{ticker} (POST)	offering risk rating

---

SmallCapLab (Live — API + Scraper)

Base URL: https://www.smallcaplab.com

Specialized dilution risk analysis and pump-timing detection for small-cap stocks.

Internal API

In addition to the public HTML page, SmallCapLab exposes internal JSON endpoints:

Endpoint	Data
/ticker/{ticker}	Public HTML page — dilution risk, capital structure, filings, offering mechanisms
/api/pump-timing/{ticker}	Pump-and-dump pattern detection correlated to SEC filings

/api/pump-timing/{ticker}

Detects price pumps and correlates them with subsequent dilution filings (ATM, offerings). Returns:

Field	Type	Description
totalPumps	int	Total pump events detected
matchedPumps	int	Pumps followed by a dilution filing within matchWindow
hitRatePct	int	% of pumps that preceded a filing
medianDays	int	Median days from pump to filing
avgDays	int	Average days from pump to filing
fastestDays / slowestDays	int	Min/max gap
links[]	array	Each pump with pump (startDate, peakDate, returnPct, durationDays, peakVolMult) and optional match (date, form, label, confidence)
allFilings[]	array	All dilution-relevant SEC filings (date, form, label, confidence)
priceHistory[]	array	Daily close prices (date, close) — ~2yr lookback
cfg	object	Detection config: thresholdPct, lookbackBars, volumeMultiple, cooldownBars, matchWindowDays, maxPumps, priceHistoryDays

Confidence levels for matches: high, medium, low

Form types detected: 424B1 (Prospectus), 424B3 (Resale Prospectus), 424B4 (Follow-on Offering), 424B5 (ATM Filing), S-8 (Stock Plan), 13G (>5% Holder)

HTML Page Data Fields

Scraped from the public ticker page at /ticker/{ticker}:

Category	Fields
Dilution Risk Score	Overall rating (Low/Moderate/High), Structural score (1-10), Near-Term score (1-10), Severity score (1-8), ATM Drip flag
Survival / Financing Pressure	Cash on hand (SEC), quarterly burn, estimated runway (months), 30d ADTV, ATM capacity ($/day)
Dilution Activity	YoY share growth %, Share CAGR (2Y), capital raises count (12mo/24mo/total), last raise (days ago)
Capital Structure	Charter authorized shares, shares outstanding (SEC), unissued authorized headroom, authorized÷outstanding ratio
Active Offering Mechanisms	ATM offering (Active/Inactive + 424B5 on file), Shelf Registration (S-3), Resale Prospectus (424B3), Private Placement (Form D), Reverse Split Signal
Trading Relevance	Shares outstanding, 30d ADTV
Historical Capital Raises	Prospectus/offering filings timeline (date, transaction type, SEC document link)
Catalyst Timeline	6-month lookback / 12-month outlook with filing dates
SEC Filing History	Dilution-relevant filings (424B, S-1/S-3/S-8, 13G, Form D) with dates and document links

Source: SEC EDGAR XBRL data (cash, burn, shares outstanding, authorized shares, filing history) + market data (ADTV).

Note: Free tier shows current ticker data. Some historical data and API access may require subscription.

---

StockTwits (Live — Internal v2 API)

Social network for traders/investors. The distinctive value vs. every other source here is retail sentiment: an aggregated bullish/bearish poll per symbol returned as a daily historical series. Also exposes a public message stream (social buzz), trending symbols, and per-symbol watchlist counts.

Web base URL: https://stocktwits.com API base URL: https://api.stocktwits.com/api/2/

⚠️ These are StockTwits' internal v2 endpoints (the ones the web app itself calls), NOT the official developer API. The legacy v1 developer API (api.stocktwits.com/api/1/...) is deprecated and now requires an approved token. The v2 endpoints currently need no API key, but:

• You MUST send a browser User-Agent header, or they return 404.
• Stability is not guaranteed — internal endpoints can change without notice and may be rate-limited. Treat as fragile (same risk class as a scraper).

Endpoints

Endpoint	Data
/api/2/symbols/{symbol}/sentiment.json	Social sentiment: daily bullish% / bearish% poll series (most recent first)
/api/2/streams/symbol/{symbol}.json	Message stream for a symbol → body, user, created_at, symbols, prices; cursor for pagination
/api/2/streams/trending.json	Trending symbols stream (messages referencing currently-trending tickers)
/api/2/symbols/{symbol}.json	Symbol metadata (returns 404 for some symbols — use the symbol object inside stream responses instead)

Sentiment — /api/2/symbols/{symbol}/sentiment.json

Returns data[] (newest first), each element:

Field	Type	Description
bullish	float	% of bullish votes (0–100)
bearish	float	% of bearish votes (0–100)
timestamp	string	ISO date (YYYY-MM-DD)

Example (SKYQ, 2026-07-12): bullish: 94.03, bearish: 5.97. The series gives a sentiment trend over time — useful as a contrarian or momentum social signal.

Streams — fields of interest

Within /api/2/streams/symbol/{symbol}.json and /api/2/streams/trending.json:

Field	Use
messages[].body / tokenized_body	Post text (cashtags preserved)
messages[].created_at	Timestamp (ISO 8601, UTC) — count posts per window to derive message volume / social buzz
messages[].user	Author (username, followers, classification, badges)
messages[].symbols[]	Referenced tickers (symbol, title)
messages[].prices[]	Price tagged at post time
symbol.watchlist_count	How many users watch the symbol — social-interest metric
cursor	Opaque pagination token for the next page

Gotchas (verified)

1. Per-message sentiment is null in public streams — users rarely tag bull/bear on individual posts. Do NOT use per-message sentiment. The reliable signal is the aggregated /sentiment poll.
2. Crypto tickers use a .X suffix (e.g. ETH.X, XRP.X) in streams and trending.
3. No auth today, but fragile — internal endpoint, browser UA required, rate limits possible. Wrap in retry/backoff + cache.
4. watchlist_count and message volume are the best proxies for retail attention; the poll is the best proxy for retail direction.

---

DefeatBeta API (Cached — Parquet via DuckDB HTTPFS)

GitHub: https://github.com/defeat-beta/defeatbeta-api PyPI: defeatbeta-api Dataset: https://huggingface.co/datasets/defeatbeta/yahoo-finance-data License: Apache 2.0 Python: 3.11+ Stars: ~683 (Jul 2026)

Overview

El DefeatBeta API es una alternativa open-source a Yahoo Finance con mayor confiabilidad. En lugar de depender de scraping de Yahoo (frágil, rate-limited), hostea todo el dataset de Yahoo Finance transformado en Parquet files en Hugging Face, y usa DuckDB + cache_httpfs para consultarlos con SQL directo sobre HTTP. No necesita API key, no tiene rate limits, y da respuestas sub-segundo.

Arquitectura Interna

┌──────────────┐    ┌────────────────────┐    ┌──────────────────────┐
│   Ticker()   │───→│  HuggingFaceClient │───→│ HF dataset (Parquet) │
│              │    │  (HTTP metadata)    │    │  (read-only, hosted) │
│  price()     │    └────────────────────┘    └────────┬─────────────┘
│  news()      │                                      │
│  info()      │    ┌────────────────────┐            │  HTTP range requests
│  transcripts │───→│   DuckDBClient     │←───────────┘  (cache_httpfs)
│  statements  │    │  (in-memory OLAP)  │
│  ...         │    │  SQL on parquet    │
└──────────────┘    └────────────────────┘

Flujo de datos:

1. HuggingFaceClient obtiene metadatos (update_time desde spec.json) y genera URLs de Parquet vía API REST de HuggingFace.
2. DuckDBClient conecta DuckDB en memoria y ejecuta SQL directamente sobre los archivos .parquet via HTTP range requests.
3. cache_httpfs (extensión DuckDB) cachea los bytes localmente → segundas consultas son instantáneas.
4. Las SQL queries viven en archivos .sql en defeatbeta_api/data/sql/, cargadas por sql_loader.py con templates tipo {ticker}, {url}.

Caché Local

• Ubicación: ~/.cache/httpfs/ (configurable)
• Validación automática: Al iniciar, compara update_time del spec.json remoto vs local. Si cambió, limpia el caché.
• TTL: Determinado por el update_time del dataset (se actualiza cada ~24h)

Datasets Disponibles (15 tablas Parquet)

Cada tabla es un archivo .parquet en Hugging Face datasets (defeatbeta/yahoo-finance-data):

Constante Interna	Dataset (Parquet)	Contenido	Período
stock_prices	stock_prices.parquet	OHLCV diario (open, high, low, close, volume)	~15+ años
stock_profile	stock_profile.parquet	Perfil empresa: sector, industry, employees, descripción, website	Actual
stock_officers	stock_officers.parquet	Ejecutivos clave (CEO, CFO, etc.)	Actual
stock_earning_calendar	stock_earning_calendar.parquet	Calendario de earnings (fecha reporte, estimados)	Próximos quarters
stock_tailing_eps	stock_tailing_eps.parquet	TTM EPS histórico	~15+ años
stock_statement	stock_statement.parquet	Financial statements (income, balance, cash flow) en formato largo con template_type	~10+ años
stock_dividend_events	stock_dividend_events.parquet	Dividendos históricos (monto, frecuencia)	~15+ años
stock_split_events	stock_split_events.parquet	Stock splits (ratio, fecha)	~15+ años
stock_earning_call_transcripts	stock_earning_call_transcripts.parquet	Transcripciones earnings calls (texto completo)	~15+ años
stock_news	stock_news.parquet	Noticias financieras (título, publisher, cuerpo, uuid)	~5+ años
stock_revenue_breakdown	stock_revenue_breakdown.parquet	Revenue por segmento/geografía/producto	~5+ años
stock_shares_outstanding	stock_shares_outstanding.parquet	Shares outstanding histórico	~15+ años
exchange_rate	exchange_rate.parquet	FX rates diarios	~15+ años
daily_treasury_yield	daily_treasury_yield.parquet	Treasury yields diarios	~15+ años
stock_sec_filing	stock_sec_filing.parquet	SEC filings metadata	~15+ años

API Python - Todas las Métricas Disponibles

Categoría 0 - Precio

Método	Returns	Descripción
ticker.price()	DataFrame	OHLCV diario completo
ticker.beta(period, benchmark)	DataFrame	Beta vs benchmark (5y default)

Categoría 1 - Financieros (Statements)

Método	Returns	Descripción
ticker.quarterly_income_statement()	Statement object	Quarterly income statement
ticker.annual_income_statement()	Statement object	Annual income statement
ticker.quarterly_balance_sheet()	Statement object	Quarterly balance sheet
ticker.annual_balance_sheet()	Statement object	Annual balance sheet
ticker.quarterly_cash_flow()	Statement object	Quarterly cash flow
ticker.annual_cash_flow()	Statement object	Annual cash flow
ticker.calendar()	DataFrame	Earnings calendar
ticker.splits()	DataFrame	Split events
ticker.dividends()	DataFrame	Dividend events
ticker.shares()	DataFrame	Shares outstanding
ticker.ttm_eps()	DataFrame	TTM EPS
ticker.ttm_revenue()	DataFrame	TTM Revenue (conversión USD)
ticker.ttm_fcf()	DataFrame	TTM Free Cash Flow (conversión USD)
ticker.ttm_net_income_common_stockholders()	DataFrame	TTM Net Income (conversión USD)
ticker.ttm_ebitda()	DataFrame	TTM EBITDA (conversión USD)
ticker.quarterly_revenue_by_breakdown()	DataFrame	Revenue breakdown (segment/geo/product) largo
ticker.trailing_revenue_by_breakdown()	DataFrame	TTM revenue breakdown largo

Categoría 2 - Valoración (Valuation)

Método	Returns	Descripción
ticker.ttm_pe()	DataFrame	TTM P/E (merge asof price + TTM EPS)
ticker.market_capitalization()	DataFrame	Market cap histórico (price × shares)
ticker.ps_ratio()	DataFrame	P/S ratio histórico
ticker.pb_ratio()	DataFrame	P/B ratio histórico
ticker.peg_ratio()	DataFrame	PEG ratio (P/E ÷ EPS growth)
ticker.enterprise_value()	DataFrame	EV = market cap + debt + minority + preferred - cash
ticker.enterprise_to_revenue()	DataFrame	EV/Revenue
ticker.enterprise_to_ebitda()	DataFrame	EV/EBITDA
ticker.net_debt_ttm()	DataFrame	Net debt TTM
ticker.debt_to_equity()	DataFrame	Debt/Equity
ticker.rope()	DataFrame	ROE calculado
ticker.roa()	DataFrame	ROA calculado
ticker.roic()	DataFrame	ROIC calculado
ticker.roce()	DataFrame	ROCE calculado
ticker.wacc()	DataFrame	WACC (con treasury, beta, S&P500)
ticker.equity_multiplier()	DataFrame	Equity multiplier (ROE/ROA)
ticker.asset_turnover()	DataFrame	Asset turnover (ROA/net margin)

Categoría 3 - Crecimiento (Growth)

Método	Returns	Descripción
ticker.quarterly_revenue_yoy_growth()	DataFrame	Revenue YoY %
ticker.annual_revenue_yoy_growth()	DataFrame	Revenue YoY % anual
ticker.quarterly_operating_income_yoy_growth()	DataFrame	Operating income YoY %
ticker.annual_ebitda_yoy_growth()	DataFrame	EBITDA YoY % anual
ticker.quarterly_net_income_yoy_growth()	DataFrame	Net income YoY %
ticker.annual_fcf_yoy_growth()	DataFrame	FCF YoY % anual
ticker.quarterly_eps_yoy_growth()	DataFrame	EPS YoY %
ticker.quarterly_ttm_eps_yoy_growth()	DataFrame	TTM EPS YoY %

Categoría 4 - Rentabilidad (Profitability)

Método	Returns	Descripción
ticker.quarterly_gross_margin()	DataFrame	Gross margin %
ticker.annual_gross_margin()	DataFrame	Gross margin % anual
ticker.quarterly_operating_margin()	DataFrame	Operating margin %
ticker.annual_net_margin()	DataFrame	Net margin % anual
ticker.quarterly_ebitda_margin()	DataFrame	EBITDA margin %
ticker.annual_fcf_margin()	DataFrame	FCF margin % anual

Categoría 5 - Fundamentales (Info & News)

Método	Returns	Descripción
ticker.info()	DataFrame	Company profile completo
ticker.sec_filing()	DataFrame	SEC filings metadata
ticker.officers()	DataFrame	Key executives
ticker.currency(symbol)	DataFrame	FX rates históricos
ticker.earning_call_transcripts()	Transcripts object	Earnings call transcripts (get_transcripts_list, get_transcript)
ticker.news()	News object	Financial news (get_news_list, get_news by uuid)

Categoría 6 - Comparación por Industria

Método	Descripción
ticker.industry_ttm_pe()	Industry TTM P/E
ticker.industry_ps_ratio()	Industry P/S
ticker.industry_pb_ratio()	Industry P/B
ticker.industry_roe()	Industry ROE
ticker.industry_roic()	Industry ROIC
ticker.industry_quarterly_gross_margin()	Industry gross margin

Batch - Tickers Class

Tickers(['AAPL', 'TSLA', 'NVDA']) ejecuta queries en paralelo vía ThreadPoolExecutor. Cada método retorna o un DataFrame combinado o un dict {symbol: object}.

Uso en LT (Nuestro Proyecto)

Wrapper: core/defeatbeta_provider.py — DefeatBetaProvider como wrapper DI.

Endpoints expuestos:

• GET /api/news/{symbol} → ticker.news().get_news_list() (news headlines)

Uso actual:

• Enriquecimiento de calendario con has_news + headline.
• Enriquecimiento de scan results con headline + publisher.
• Startup pre-warm: fetch_news_batch (WHERE symbol IN (...)) vía DuckDB directo.

Instalación / Dependencias:

# pyproject.toml
[project.dependencies]
defeatbeta-api = ">=0.0.43"

Override de pandas (por conflicto con tradingview-screener):

[tool.uv]
override-dependencies = [
    "pandas>=3.0.1",
]

Ventajas sobre Yahoo Finance Directo

Aspecto	Yahoo Finance (directo)	DefeatBeta API
Rate limits	Sí (~200 req/min)	No (Parquet estáticos)
Confiabilidad	Baja (scraping, caídas frecuentes)	Alta (HF datasets)
Velocidad	200-500ms por request	<100ms (cached) / 2-5s (cold, primer query)
Datos históricos	Limitado a 2-3 años (intraday)	15+ años completo
Earnings transcripts	No disponible nativo	Sí, completos
Revenue breakdown	No disponible	Sí, por segmento/geo/producto
DCF / Excel report	No	Sí (integrados)
LLM analysis	No	Sí (OpenAI integrado)
Windows support	N/A	Sí (v0.0.60+)

Limitaciones Conocidas

1. Cold start: La primera query después de instalar o limpiar caché tarda 2-5s (descarga bytes del parquet via HTTP range).
2. DuckDB in-memory: Solo singleton; no recomendado para workloads multi-process pesados.
3. Datos diarios únicamente: No hay barras intraday (minute, hour). Solo daily OHLCV.
4. Sin streaming: No soporta datos en tiempo real; es un batch histórico actualizado ~24h.
5. Ticker único por instancia: Cada Ticker se instancia con un símbolo; usar Tickers para batch.
6. Windows: Nativo desde v0.0.60; versiones anteriores requerían WSL.