File size: 3,814 Bytes
79e6483 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | # Alpha Robustness Runner
This runner is intentionally isolated from the stable backtest path.
Use:
- [deploy/v2/jsonl_alpha_robustness.py](/Users/giaphu/Documents/Pylab/AAE-new/deploy/v2/jsonl_alpha_robustness.py)
- [backtest/robust_factor_executor.py](/Users/giaphu/Documents/Pylab/AAE-new/backtest/robust_factor_executor.py)
- [backtest/robust_qlib_backtester.py](/Users/giaphu/Documents/Pylab/AAE-new/backtest/robust_qlib_backtester.py)
Do **not** modify the production backtest flow when running robustness experiments.
## Purpose
Run robustness backtests for a fixed alpha pack, for example the top 10 alphas,
while varying assumptions such as:
- `TOP_K`
- `rebalance_freq`
- `buy/sell fee`
- weighting mode
- score transform
- liquidity-derived universe filter
- execution limits
## Input format
Pass a JSONL file where each row is at least:
```json
{"name":"ALPHA_001","expr":"RANK($close)"}
```
Optional fields such as `seed_name`, `factor_name`, `factor_expr`,
`candidate_scope`, `proposal_rank`, `turn`, and `call_index` are also accepted.
## Minimal example
```bash
python deploy/v2/jsonl_alpha_robustness.py \
--jsonl data/top10_alphas.jsonl \
--period test \
--backtest-engine custom \
--top-k 5 \
--rebalance-freq 5 \
--custom-weight-mode alpha_score \
--max-pos-each-stock 0.2 \
--enforce-cash-limit \
--capture-detail-artifacts \
--output-dir outputs/robust/top10_baseline
```
## Common robustness sweeps
### 1. `TOP_K` sensitivity
Run the same pack with:
- `--top-k 3`
- `--top-k 5`
- `--top-k 10`
### 2. Rebalance sensitivity
Run the same pack with:
- `--rebalance-freq 5`
- `--rebalance-freq 10`
- `--rebalance-freq 20`
### 3. Fee sensitivity
Run the same pack with:
- `--buy-fee 0.0013 --sell-fee 0.0013`
- `--buy-fee 0.0020 --sell-fee 0.0020`
- `--buy-fee 0.0030 --sell-fee 0.0030`
### 4. Weighting robustness
Examples:
- `--custom-weight-mode equal --max-pos-each-stock 0.2`
- `--custom-weight-mode alpha_score --max-pos-each-stock 0.2`
- `--custom-weight-mode alpha_score --max-pos-each-stock 1.0`
### 5. Score-transform robustness
Examples:
- `--score-transform identity`
- `--score-transform rank`
- `--score-transform zscore`
- `--score-transform rank_zscore`
- `--score-transform clip_zscore --score-clip 3.0`
### 6. Liquidity-derived universe robustness
Examples:
- `--universe-filter top_amount --universe-top-n 70 --universe-lookback-days 20`
- `--universe-filter top_amount --universe-top-n 90 --universe-lookback-days 20`
- `--universe-filter top_volume --universe-top-n 70 --universe-lookback-days 20`
This is an internal proxy universe derived from `daily_pv.h5`. It is **not**
the same as an external market-cap or index-membership universe.
## Outputs
The runner writes the same main artifacts as the standalone backtest runner,
plus a robustness manifest:
- `summary.csv`
- `summary_yearly.csv`
- `trials.csv`
- `trials_yearly.csv`
- `aggregate.json`
- `aggregate_yearly.csv`
- `portfolio_daily.csv`
- `trade_log.csv`
- `holdings_daily.csv`
- `signal_selection_daily.csv`
- `rebalance_plan.csv`
- `rebalance_window_returns.csv`
- `data_quality_report.json`
- `robust_manifest.json` (or the custom name from `--manifest-name`)
## Recommended workflow for 10 alphas
1. Prepare a `top10_alphas.jsonl`
2. Run one baseline bundle
3. Run one axis at a time:
- `top_k`
- `rebalance_freq`
- fee
- weighting
- score transform
- universe filter
4. Compare each bundle using:
- `summary.csv`
- `aggregate.json`
- `portfolio_daily.csv`
- `trade_log.csv`
- `signal_selection_daily.csv`
- `rebalance_plan.csv`
## Important note
This path is separate on purpose. If a robustness experiment fails or needs
extra knobs, patch the robust files above instead of the stable production
backtest path.
|