docs/DATA_PIPELINE_AND_CONTRACTS.md

# Datasource Pipeline and Contracts

## Priority and authority

```text
Datasource 4 → Binance public fallback → Datasource 2
```

Priority describes fill order, not equal trust.

### Datasource 4

Authoritative for:

- Futures contract verification;
- `dataState`;
- `noTradeGuard`;
- safety status and rejection reasons;
- primary Futures market fields.

### Binance public

- unauthenticated;
- called only for missing, unusable, or stale fields;
- cannot verify a contract or clear DS4 safety;
- HTTP 451 is `restricted` / `Regionally restricted`;
- provider timestamps are required to claim freshness.

### Datasource 2

- complementary news, sentiment, indicator, order-book, volume, trending, gainers, and correlation context;
- may fill a still-missing legitimate field only after Binance;
- cannot become Futures verification or safety authority.

## Datasource 4 request

The DS4 snapshot endpoint is called with:

```text
/api/short-hunter/snapshot/{SYMBOL}
```

Parameters:

```text
interval: 1m | 5m | 15m | 1h
limit: 1..500 internally; market API exposes 20..500
from: epoch milliseconds
to: epoch milliseconds
```

`normalize_epoch_milliseconds()` accepts contemporary epoch seconds or milliseconds and prevents double conversion. `build_kucoin_time_range()` enforces supported interval, bounded limit, positive ordered timestamps, and millisecond units.

## Normalized fields

The merged envelope may contain:

```text
contract
ticker
ohlcv
orderbook
funding
openInterest
indicators
sentiment
atr
market_context
```

### Contract

Normalized contract data should expose symbol, status, type/instrument, and explicit verification evidence when present. The presence of a generic `contract` object alone does not prove Futures status. Verification requires an explicit DS4 flag or a recognized Futures/perpetual/swap contract type.

### Ticker

Accepted aliases are normalized to a bounded ticker object. Consumers should prefer normalized canonical keys where available and tolerate provider-specific supplemental keys.

Common price candidates:

```text
markPrice
lastPrice
last
price
close
indexPrice
```

### OHLCV

Canonical candle shape:

```json
{
  "timestamp": 0,
  "open": 0.0,
  "high": 0.0,
  "low": 0.0,
  "close": 0.0,
  "volume": 0.0
}
```

A usable OHLCV series requires at least four valid positive close values. The market endpoint never invents missing candles.

### Order book

Canonical shape:

```json
{
  "bids": [[60000.0, 0.5]],
  "asks": [[60001.0, 0.4]],
  "timestamp": 0
}
```

Both sides must have at least one valid level. Prices must be positive; quantities must be non-negative.

### Funding

Canonical values may include:

```text
currentFundingRate
fundingRate
lastFundingRate
rate
nextFundingTime
```

### Open Interest

Canonical values may include:

```text
openInterest
sumOpenInterest
oi
changeFraction
change24h
changePercent
```

## Field usability

`_is_usable(field, value)` performs field-specific validation. Empty values, non-finite values, invalid OHLCV, incomplete order books, and invalid contract/funding/OI shapes are rejected.

## Per-field provenance

Every owned field receives metadata:

```json
{
  "value": "bounded or summarized value",
  "source": "datasource4 | binance_public | datasource2 | unavailable",
  "timestamp": "provider timestamp or null",
  "freshness": "fresh | stale | invalid | unknown",
  "validity": "valid | unavailable",
  "observedAt": "server observation time",
  "freshnessBasis": "field_timestamp | datasource4_dataState | missing_provider_timestamp | unavailable",
  "fallbackStatus": "primary | fallback | not_filled"
}
```

The public API bounds large values:

- OHLCV becomes count plus latest candle summary where appropriate;
- order book becomes level counts and best bid/ask summary;
- diagnostics are sanitized and size-limited.

## Freshness

Freshness is based on provider timestamp relative to interval, or on an explicit authoritative DS4 fresh state. Transport success alone is not freshness evidence.

Required fields with `stale`, `invalid`, or `unknown` freshness block readiness.

## Critical fields and readiness

Critical fields:

```text
contract
ticker
orderbook
funding
openInterest
```

The combined context sets:

```text
missingRequiredFields
staleRequiredFields
noTradeGuard
noTradeReasons
mergeStatus
tradingReadiness
```

Readiness is `ready` only when no guard remains. DS4 verification failure, DS4 `noTradeGuard`, missing critical fields, or non-fresh critical fields results in `blocked`.

## Source metadata

Each source returns structured fields:

```json
{
  "name": "Datasource 4",
  "url": "...",
  "status": "ok | degraded | unreachable | unavailable | standby",
  "transportStatus": "healthy | degraded | unavailable | restricted | standby",
  "dataUsability": "usable | degraded | unavailable | not_used",
  "endpoint": "...",
  "httpStatus": 200,
  "latencyMs": 120.4,
  "lastSuccess": "...",
  "freshness": "fresh | stale | unknown",
  "completeness": "complete | partial | unknown",
  "suppliedFields": [],
  "missingFields": [],
  "reason": "concise operator-facing summary"
}
```

Detailed endpoint/provider errors belong only under `technicalDiagnostics`, separated by source and sanitized before exposure.

## Merge diagnostics

Cross-source problems are not assigned to a datasource card. They appear under:

```text
technicalDiagnostics.merge.status
technicalDiagnostics.merge.missingCriticalFields
technicalDiagnostics.merge.tradingReadiness
technicalDiagnostics.merge.rejectionReasons
```

## Adding a new provider mapping

1. Capture a real redacted payload.
2. Add the narrowest legitimate alias to the relevant normalizer.
3. Preserve provider timestamp and source name.
4. Add field-specific validity checks.
5. Do not infer Futures verification from generic market data.
6. Do not let the provider clear DS4 guard state.
7. Add focused tests for positive, missing, malformed, stale, and ambiguous cases.
8. Verify source-specific diagnostics remain correctly attributed.