README.md

---
title: IndicatorsEnv
emoji: 📈
colorFrom: indigo
colorTo: blue
sdk: docker
pinned: true
tags:
  - openenv
  - reinforcement-learning
  - finance
  - stock-market
  - technical-analysis
  - rl-environment
license: mit
---

# IndicatorsEnv v4.1 — Multi-stock Portfolio MDP

> **An OpenEnv-compatible RL environment where agents learn within-sector relative alpha on real NSE (India) equities — with genuine portfolio capital dynamics.**

---

## Environment Description

`IndicatorsEnv` simulates the task a quantitative portfolio manager performs: given 3 stocks from the **same sector**, identify which has the strongest relative momentum signal, declare a direction, and set a conviction level. Reward is the **market-neutral alpha** the agent captures — the chosen stock's return minus the sector average.

### Why Multi-stock + Portfolio State

Single-stock environments collapse to contextual bandits once you remove multi-step structure. IndicatorsEnv v4.1 solves this with two sources of genuine MDP state:

1. **Signal history** — RSI trends, price momentum, and past pick outcomes accumulate across steps. The agent can detect multi-week divergence patterns.
2. **Portfolio capital** — `current_holding`, `capital`, and `drawdown` are in every observation. Switching holdings costs 0.1% of capital. This creates a real exploration-exploitation tradeoff: staying in a position avoids transaction cost; switching to a better signal costs capital.

A random policy earns ≈ 0 expected reward (market-neutral by construction). A skilled policy earns consistently positive alpha.

---

## Action Space

```json
{
  "stock":     "HDFCBANK",
  "direction": "Bullish",
  "conviction": 0.8
}
```

| Field | Type | Description |
|---|---|---|
| `stock` | string | NSE symbol to trade, or `"NONE"` to pass this step |
| `direction` | string (enum) | `Bullish` (long alpha), `Bearish` (short alpha), or `NONE` |
| `conviction` | float [0, 1] | Kelly fraction — fraction of virtual wealth wagered |

Passing (`NONE`) skips the step with zero reward and preserves the current holding at no cost. Switching to a **different** stock from the one held last step costs 0.1% of current capital.

---

## Observation Space

```json
{
  "step": 3,
  "max_steps": 10,
  "term": "MEDIUM",
  "sector": "banking",
  "available_stocks": ["HDFCBANK", "ICICIBANK", "AXISBANK"],
  "stocks": {
    "HDFCBANK":  { "rsi_14": 61.2, "rsi_trend": "up",   "price_momentum_pct": 2.4, "indicators": {...} },
    "ICICIBANK": { "rsi_14": 54.8, "rsi_trend": "flat", "price_momentum_pct": 0.8, "indicators": {...} },
    "AXISBANK":  { "rsi_14": 48.1, "rsi_trend": "down", "price_momentum_pct": -1.1, "indicators": {...} }
  },
  "signal_history": [
    { "step": 1, "picked_stock": "HDFCBANK", "direction": "Bullish", "correct": true,  "alpha_pct": 1.8, "reward": 0.63 },
    { "step": 2, "picked_stock": "NONE",     "direction": "NONE",    "correct": null,  "alpha_pct": 0.0, "reward": 0.0  }
  ],
  "current_holding": "HDFCBANK",
  "capital": 1.0126,
  "drawdown": 0.0,
  "macro": null
}
```

### Per-stock Indicator Suite (25+ signals)

| Category | Indicators |
|---|---|
| **Moving Averages** | SMA(20/50/200), EMA(20/50), Golden/Death Cross |
| **Momentum** | RSI(14) + trend, MACD(12/26/9) + crossover, Stochastic(14/3) |
| **Volatility** | Bollinger Bands (%, width, squeeze), ATR(14), volatility regime |
| **Trend** | ADX(14), +DI/-DI, trend strength |
| **Volume** | OBV, VWAP, MFI(14), CMF(20), A/D Line, volume ratio |
| **Levels** | Pivot Points (R2/R1/P/S1/S2) |

---

## Tasks

### Task 1: Short-term Relative Alpha *(Easy)*
- **Steps**: 5 (1 trading day apart = 1 week)
- **GT**: 1-day forward return, ±0.3% threshold
- **Portfolio**: capital tracked, 5% drawdown terminates episode early
- **Grader**: Directional accuracy on active (non-NONE) steps

### Task 2: Medium-term Relative Alpha *(Medium)*
- **Steps**: 10 (5 trading days apart = 10 weeks)
- **GT**: 5-day forward return, ±1.5% threshold
- **Portfolio**: capital tracked, 10% drawdown terminates episode early
- **Grader**: Weighted accuracy — Bearish/Neutral correct = 1.5× weight; participation bonus

### Task 3: Long-term Risk-Constrained Alpha *(Hard)*
- **Steps**: 15 (20 trading days apart = 15 months)
- **GT**: 20-day forward return, ±2.5% threshold
- **Portfolio**: capital tracked, 15% drawdown terminates episode early; macro context (NIFTY50 trend, market regime) added
- **Grader**: Conviction-calibrated accuracy — correct + conviction ≥ 0.7 → 1.0; overconfident wrong → −0.1

---

## Reward Function

```
alpha          = chosen_stock_period_return − sector_average_return
direction_sign = +1 (Bullish) or −1 (Bearish)
raw_reward     = alpha × direction_sign × conviction × 50
tx_cost        = 0.001 × capital  (if switching from a different held stock, else 0)
reward         = clamp(raw_reward − tx_cost, −1.5, 1.5)
```

**Example (medium task, step 3):**
- HDFCBANK returns +3.5%, sector average +1.5%
- alpha = +2.0%, direction = Bullish, conviction = 0.8
- No switch (was already holding HDFCBANK): tx_cost = 0
- reward = 0.02 × 1 × 0.8 × 50 = **0.80**

**Market-neutral property:** Because reward is relative to the sector average, a random policy earns ≈ 0 in expectation. Broad market moves cancel out. The agent profits only from correctly identifying within-sector outperformers.

**Kelly conviction:** Higher conviction amplifies both gains and losses, incentivizing calibrated confidence rather than uniform maximum bets.

---

## Episode Structure

```
reset()  → sector=banking  stocks=[HDFCBANK, ICICIBANK, AXISBANK]  capital=1.0  holding=NONE

step 1   → pick HDFCBANK Bullish 0.7   reward=+0.63  done=False  capital=1.009  tx_cost=0.0
step 2   → NONE                         reward=0.00   done=False  capital=1.009  tx_cost=0.0
step 3   → pick ICICIBANK Bullish 0.8   reward=+0.41  done=False  capital=1.017  tx_cost=0.001  ← switch cost
step 4   → pick ICICIBANK Bullish 0.6   reward=-0.15  done=False  capital=1.008  tx_cost=0.0   ← same stock
step 5   → pick HDFCBANK Bearish 0.5    reward=+0.22  done=True   capital=1.012  tx_cost=0.001  ← switch cost
```

---

## Baseline Scores (Random Agent, seed=42)

| Task | Steps per run | Expected score |
|---|---|---|
| Short-term Direction | 50 (10 eps × 5 steps) | ~0.33 |
| Medium-term Direction | 100 (10 eps × 10 steps) | ~0.33 |
| Long-term Conviction | ≤150 (10 eps × ≤15 steps) | ~0.10 |

---

## Setup & Usage

### Local
```bash
pip install -r requirements-server.txt
cd env/
python indicators_env.py   # Runs on http://localhost:7860
```

### Docker
```bash
docker build -t indicators-env .
docker run -p 7860:7860 indicators-env
```

### Inference Agent
```bash
export API_BASE_URL=https://router.huggingface.co/v1/
export MODEL_NAME=meta-llama/Llama-3.2-1B-Instruct
export HF_TOKEN=hf_your_token

python inference.py --env_url http://localhost:7860 --n_episodes 5
```

---

## Links

- **HF Space (live env)**: [bawsi99/indicators-env](https://huggingface.co/spaces/bawsi99/indicators-env)
- **Fine-tuned LoRA**: [bawsi99/indicators-grpo-qwen7b](https://huggingface.co/bawsi99/indicators-grpo-qwen7b)
- **OpenEnv**: [meta-pytorch/OpenEnv](https://github.com/meta-pytorch/OpenEnv)