docs/DATA_PIPELINE_AND_CONTRACTS.md

Datasource Pipeline and Contracts

Priority and authority

Datasource 4 → Binance public fallback → Datasource 2

Priority describes fill order, not equal trust.

Datasource 4

Authoritative for:

• Futures contract verification;
• dataState;
• noTradeGuard;
• safety status and rejection reasons;
• primary Futures market fields.

Binance public

• unauthenticated;
• called only for missing, unusable, or stale fields;
• cannot verify a contract or clear DS4 safety;
• HTTP 451 is restricted / Regionally restricted;
• provider timestamps are required to claim freshness.

Datasource 2

• complementary news, sentiment, indicator, order-book, volume, trending, gainers, and correlation context;
• may fill a still-missing legitimate field only after Binance;
• cannot become Futures verification or safety authority.

Datasource 4 request

The DS4 snapshot endpoint is called with:

/api/short-hunter/snapshot/{SYMBOL}

Parameters:

interval: 1m | 5m | 15m | 1h
limit: 1..500 internally; market API exposes 20..500
from: epoch milliseconds
to: epoch milliseconds

normalize_epoch_milliseconds() accepts contemporary epoch seconds or milliseconds and prevents double conversion. build_kucoin_time_range() enforces supported interval, bounded limit, positive ordered timestamps, and millisecond units.

Normalized fields

The merged envelope may contain:

contract
ticker
ohlcv
orderbook
funding
openInterest
indicators
sentiment
atr
market_context

Contract

Normalized contract data should expose symbol, status, type/instrument, and explicit verification evidence when present. The presence of a generic contract object alone does not prove Futures status. Verification requires an explicit DS4 flag or a recognized Futures/perpetual/swap contract type.

Ticker

Accepted aliases are normalized to a bounded ticker object. Consumers should prefer normalized canonical keys where available and tolerate provider-specific supplemental keys.

Common price candidates:

markPrice
lastPrice
last
price
close
indexPrice

OHLCV

Canonical candle shape:

{
  "timestamp": 0,
  "open": 0.0,
  "high": 0.0,
  "low": 0.0,
  "close": 0.0,
  "volume": 0.0
}

A usable OHLCV series requires at least four valid positive close values. The market endpoint never invents missing candles.

Order book

Canonical shape:

{
  "bids": [[60000.0, 0.5]],
  "asks": [[60001.0, 0.4]],
  "timestamp": 0
}

Both sides must have at least one valid level. Prices must be positive; quantities must be non-negative.

Funding

Canonical values may include:

currentFundingRate
fundingRate
lastFundingRate
rate
nextFundingTime

Open Interest

Canonical values may include:

openInterest
sumOpenInterest
oi
changeFraction
change24h
changePercent

Field usability

_is_usable(field, value) performs field-specific validation. Empty values, non-finite values, invalid OHLCV, incomplete order books, and invalid contract/funding/OI shapes are rejected.

Per-field provenance

Every owned field receives metadata:

{
  "value": "bounded or summarized value",
  "source": "datasource4 | binance_public | datasource2 | unavailable",
  "timestamp": "provider timestamp or null",
  "freshness": "fresh | stale | invalid | unknown",
  "validity": "valid | unavailable",
  "observedAt": "server observation time",
  "freshnessBasis": "field_timestamp | datasource4_dataState | missing_provider_timestamp | unavailable",
  "fallbackStatus": "primary | fallback | not_filled"
}

The public API bounds large values:

• OHLCV becomes count plus latest candle summary where appropriate;
• order book becomes level counts and best bid/ask summary;
• diagnostics are sanitized and size-limited.

Freshness

Freshness is based on provider timestamp relative to interval, or on an explicit authoritative DS4 fresh state. Transport success alone is not freshness evidence.

Required fields with stale, invalid, or unknown freshness block readiness.

Critical fields and readiness

Critical fields:

contract
ticker
orderbook
funding
openInterest

The combined context sets:

missingRequiredFields
staleRequiredFields
noTradeGuard
noTradeReasons
mergeStatus
tradingReadiness

Readiness is ready only when no guard remains. DS4 verification failure, DS4 noTradeGuard, missing critical fields, or non-fresh critical fields results in blocked.

Source metadata

Each source returns structured fields:

{
  "name": "Datasource 4",
  "url": "...",
  "status": "ok | degraded | unreachable | unavailable | standby",
  "transportStatus": "healthy | degraded | unavailable | restricted | standby",
  "dataUsability": "usable | degraded | unavailable | not_used",
  "endpoint": "...",
  "httpStatus": 200,
  "latencyMs": 120.4,
  "lastSuccess": "...",
  "freshness": "fresh | stale | unknown",
  "completeness": "complete | partial | unknown",
  "suppliedFields": [],
  "missingFields": [],
  "reason": "concise operator-facing summary"
}

Detailed endpoint/provider errors belong only under technicalDiagnostics, separated by source and sanitized before exposure.

Merge diagnostics

Cross-source problems are not assigned to a datasource card. They appear under:

technicalDiagnostics.merge.status
technicalDiagnostics.merge.missingCriticalFields
technicalDiagnostics.merge.tradingReadiness
technicalDiagnostics.merge.rejectionReasons

Adding a new provider mapping

1. Capture a real redacted payload.
2. Add the narrowest legitimate alias to the relevant normalizer.
3. Preserve provider timestamp and source name.
4. Add field-specific validity checks.
5. Do not infer Futures verification from generic market data.
6. Do not let the provider clear DS4 guard state.
7. Add focused tests for positive, missing, malformed, stale, and ambiguous cases.
8. Verify source-specific diagnostics remain correctly attributed.