--- title: IndicatorsEnv emoji: 📈 colorFrom: indigo colorTo: blue sdk: docker pinned: true tags: - openenv - reinforcement-learning - finance - stock-market - technical-analysis - rl-environment license: mit --- # IndicatorsEnv v4.1 — Multi-stock Portfolio MDP > **An OpenEnv-compatible RL environment where agents learn within-sector relative alpha on real NSE (India) equities — with genuine portfolio capital dynamics.** --- ## Environment Description `IndicatorsEnv` simulates the task a quantitative portfolio manager performs: given 3 stocks from the **same sector**, identify which has the strongest relative momentum signal, declare a direction, and set a conviction level. Reward is the **market-neutral alpha** the agent captures — the chosen stock's return minus the sector average. ### Why Multi-stock + Portfolio State Single-stock environments collapse to contextual bandits once you remove multi-step structure. IndicatorsEnv v4.1 solves this with two sources of genuine MDP state: 1. **Signal history** — RSI trends, price momentum, and past pick outcomes accumulate across steps. The agent can detect multi-week divergence patterns. 2. **Portfolio capital** — `current_holding`, `capital`, and `drawdown` are in every observation. Switching holdings costs 0.1% of capital. This creates a real exploration-exploitation tradeoff: staying in a position avoids transaction cost; switching to a better signal costs capital. A random policy earns ≈ 0 expected reward (market-neutral by construction). A skilled policy earns consistently positive alpha. --- ## Action Space ```json { "stock": "HDFCBANK", "direction": "Bullish", "conviction": 0.8 } ``` | Field | Type | Description | |---|---|---| | `stock` | string | NSE symbol to trade, or `"NONE"` to pass this step | | `direction` | string (enum) | `Bullish` (long alpha), `Bearish` (short alpha), or `NONE` | | `conviction` | float [0, 1] | Kelly fraction — fraction of virtual wealth wagered | Passing (`NONE`) skips the step with zero reward and preserves the current holding at no cost. Switching to a **different** stock from the one held last step costs 0.1% of current capital. --- ## Observation Space ```json { "step": 3, "max_steps": 10, "term": "MEDIUM", "sector": "banking", "available_stocks": ["HDFCBANK", "ICICIBANK", "AXISBANK"], "stocks": { "HDFCBANK": { "rsi_14": 61.2, "rsi_trend": "up", "price_momentum_pct": 2.4, "indicators": {...} }, "ICICIBANK": { "rsi_14": 54.8, "rsi_trend": "flat", "price_momentum_pct": 0.8, "indicators": {...} }, "AXISBANK": { "rsi_14": 48.1, "rsi_trend": "down", "price_momentum_pct": -1.1, "indicators": {...} } }, "signal_history": [ { "step": 1, "picked_stock": "HDFCBANK", "direction": "Bullish", "correct": true, "alpha_pct": 1.8, "reward": 0.63 }, { "step": 2, "picked_stock": "NONE", "direction": "NONE", "correct": null, "alpha_pct": 0.0, "reward": 0.0 } ], "current_holding": "HDFCBANK", "capital": 1.0126, "drawdown": 0.0, "macro": null } ``` ### Per-stock Indicator Suite (25+ signals) | Category | Indicators | |---|---| | **Moving Averages** | SMA(20/50/200), EMA(20/50), Golden/Death Cross | | **Momentum** | RSI(14) + trend, MACD(12/26/9) + crossover, Stochastic(14/3) | | **Volatility** | Bollinger Bands (%, width, squeeze), ATR(14), volatility regime | | **Trend** | ADX(14), +DI/-DI, trend strength | | **Volume** | OBV, VWAP, MFI(14), CMF(20), A/D Line, volume ratio | | **Levels** | Pivot Points (R2/R1/P/S1/S2) | --- ## Tasks ### Task 1: Short-term Relative Alpha *(Easy)* - **Steps**: 5 (1 trading day apart = 1 week) - **GT**: 1-day forward return, ±0.3% threshold - **Portfolio**: capital tracked, 5% drawdown terminates episode early - **Grader**: Directional accuracy on active (non-NONE) steps ### Task 2: Medium-term Relative Alpha *(Medium)* - **Steps**: 10 (5 trading days apart = 10 weeks) - **GT**: 5-day forward return, ±1.5% threshold - **Portfolio**: capital tracked, 10% drawdown terminates episode early - **Grader**: Weighted accuracy — Bearish/Neutral correct = 1.5× weight; participation bonus ### Task 3: Long-term Risk-Constrained Alpha *(Hard)* - **Steps**: 15 (20 trading days apart = 15 months) - **GT**: 20-day forward return, ±2.5% threshold - **Portfolio**: capital tracked, 15% drawdown terminates episode early; macro context (NIFTY50 trend, market regime) added - **Grader**: Conviction-calibrated accuracy — correct + conviction ≥ 0.7 → 1.0; overconfident wrong → −0.1 --- ## Reward Function ``` alpha = chosen_stock_period_return − sector_average_return direction_sign = +1 (Bullish) or −1 (Bearish) raw_reward = alpha × direction_sign × conviction × 50 tx_cost = 0.001 × capital (if switching from a different held stock, else 0) reward = clamp(raw_reward − tx_cost, −1.5, 1.5) ``` **Example (medium task, step 3):** - HDFCBANK returns +3.5%, sector average +1.5% - alpha = +2.0%, direction = Bullish, conviction = 0.8 - No switch (was already holding HDFCBANK): tx_cost = 0 - reward = 0.02 × 1 × 0.8 × 50 = **0.80** **Market-neutral property:** Because reward is relative to the sector average, a random policy earns ≈ 0 in expectation. Broad market moves cancel out. The agent profits only from correctly identifying within-sector outperformers. **Kelly conviction:** Higher conviction amplifies both gains and losses, incentivizing calibrated confidence rather than uniform maximum bets. --- ## Episode Structure ``` reset() → sector=banking stocks=[HDFCBANK, ICICIBANK, AXISBANK] capital=1.0 holding=NONE step 1 → pick HDFCBANK Bullish 0.7 reward=+0.63 done=False capital=1.009 tx_cost=0.0 step 2 → NONE reward=0.00 done=False capital=1.009 tx_cost=0.0 step 3 → pick ICICIBANK Bullish 0.8 reward=+0.41 done=False capital=1.017 tx_cost=0.001 ← switch cost step 4 → pick ICICIBANK Bullish 0.6 reward=-0.15 done=False capital=1.008 tx_cost=0.0 ← same stock step 5 → pick HDFCBANK Bearish 0.5 reward=+0.22 done=True capital=1.012 tx_cost=0.001 ← switch cost ``` --- ## Baseline Scores (Random Agent, seed=42) | Task | Steps per run | Expected score | |---|---|---| | Short-term Direction | 50 (10 eps × 5 steps) | ~0.33 | | Medium-term Direction | 100 (10 eps × 10 steps) | ~0.33 | | Long-term Conviction | ≤150 (10 eps × ≤15 steps) | ~0.10 | --- ## Setup & Usage ### Local ```bash pip install -r requirements-server.txt cd env/ python indicators_env.py # Runs on http://localhost:7860 ``` ### Docker ```bash docker build -t indicators-env . docker run -p 7860:7860 indicators-env ``` ### Inference Agent ```bash export API_BASE_URL=https://router.huggingface.co/v1/ export MODEL_NAME=meta-llama/Llama-3.2-1B-Instruct export HF_TOKEN=hf_your_token python inference.py --env_url http://localhost:7860 --n_episodes 5 ``` --- ## Links - **HF Space (live env)**: [bawsi99/indicators-env](https://huggingface.co/spaces/bawsi99/indicators-env) - **Fine-tuned LoRA**: [bawsi99/indicators-grpo-qwen7b](https://huggingface.co/bawsi99/indicators-grpo-qwen7b) - **OpenEnv**: [meta-pytorch/OpenEnv](https://github.com/meta-pytorch/OpenEnv)