spikes/005-integrated-trainer-skeleton/verl_path/README.md

# verl_path/ — STATUS: design-only

> **Status (Wave 18, 2026-05-26):** This directory contains a *design
> sketch* of the VeRL adapter, not a tested implementation. Validate
> via Spike 002b (PRIME-RL/VeRL run on real GPU) before relying on
> any code in here.
>
> Wave 7's vision-validation audit explicitly called out: *"the framework
> claims integration with TRL + VeRL but the only tested path is TRL.
> verl_path/ should be marked design-only until validated end-to-end."*
> This README closes that audit finding.

## What this is

`composer_adv.py` and `composer_config.yaml` are reference implementations
of the same `compose_loss` composition contract as `trl_path/`, written
against VeRL's algorithm-library surface (advantage estimator + custom
loss hook). They were authored from primary-source reading of
`volcengine/verl` at the time of Wave 6 and are kept as a design
target — they document HOW we'd wire a 3-channel composer loss into
VeRL's actor/critic update — but they have NOT been run end-to-end
against a real model under VeRL's runtime.

## What this is NOT

- Not pip-tested (VeRL has heavy transitive deps including Ray; we
  haven't paid the cost to install them in the test venv).
- Not import-tested (no `from spikes.005-integrated-trainer-skeleton.verl_path
  import ...` test exists).
- Not a parity oracle. The `trl_path/` adapter is the production-tested
  path; the `verl_path/` files document the proposed VeRL equivalent
  but their numerical equivalence is unverified.

## What to do before relying on it

1. Install VeRL + Ray in a fresh venv: `pip install volcengine-verl ray`
2. Stand up a VeRL trainer using `composer_config.yaml` against a small
   model (Qwen2.5-0.5B-Instruct works for CPU smoke).
3. Verify `composer_adv.py:compose_advantage_with_loss` produces the
   same (lm_ce, sdpo_jsd, trace_replay_dpo) decomposition as
   `trl_path/composer_trainer.py:_compute_loss` on identical inputs.
4. If parity holds: promote this README to `STATUS: tested` and add
   the parity test to `composer_replication/recipes/verl/tests/`.
5. If parity fails: file a follow-up wave to fix the mismatched code
   path before any user-facing claim that VeRL is supported.

## Cross-references

- `docs/research/RL_FRAMEWORKS_LANDSCAPE.md` — landscape audit that
  fed the design (see "Realised in v0.1" callout for the post-Wave-14b
  PRIME-RL recipe; the VeRL design here is older).
- `docs/VISION_VALIDATION.md` § 7, Objection 3 — Wave 7 audit identifying
  this gap.
- `composer_replication/recipes/prime_rl/` — the actually-tested
  third-RL-framework recipe (PRIME-RL, with shadow-parity tests against
  upstream `default_loss_fn`). VeRL would land alongside it once
  validated.