spikes/005-integrated-trainer-skeleton/verl_path/README.md

verl_path/ — STATUS: design-only

Status (Wave 18, 2026-05-26): This directory contains a design sketch of the VeRL adapter, not a tested implementation. Validate via Spike 002b (PRIME-RL/VeRL run on real GPU) before relying on any code in here.

Wave 7's vision-validation audit explicitly called out: "the framework claims integration with TRL + VeRL but the only tested path is TRL. verl_path/ should be marked design-only until validated end-to-end." This README closes that audit finding.

What this is

composer_adv.py and composer_config.yaml are reference implementations of the same compose_loss composition contract as trl_path/, written against VeRL's algorithm-library surface (advantage estimator + custom loss hook). They were authored from primary-source reading of volcengine/verl at the time of Wave 6 and are kept as a design target — they document HOW we'd wire a 3-channel composer loss into VeRL's actor/critic update — but they have NOT been run end-to-end against a real model under VeRL's runtime.

What this is NOT

• Not pip-tested (VeRL has heavy transitive deps including Ray; we haven't paid the cost to install them in the test venv).
• Not import-tested (no from spikes.005-integrated-trainer-skeleton.verl_path import ... test exists).
• Not a parity oracle. The trl_path/ adapter is the production-tested path; the verl_path/ files document the proposed VeRL equivalent but their numerical equivalence is unverified.

What to do before relying on it

1. Install VeRL + Ray in a fresh venv: pip install volcengine-verl ray
2. Stand up a VeRL trainer using composer_config.yaml against a small model (Qwen2.5-0.5B-Instruct works for CPU smoke).
3. Verify composer_adv.py:compose_advantage_with_loss produces the same (lm_ce, sdpo_jsd, trace_replay_dpo) decomposition as trl_path/composer_trainer.py:_compute_loss on identical inputs.
4. If parity holds: promote this README to STATUS: tested and add the parity test to composer_replication/recipes/verl/tests/.
5. If parity fails: file a follow-up wave to fix the mismatched code path before any user-facing claim that VeRL is supported.

Cross-references

• docs/research/RL_FRAMEWORKS_LANDSCAPE.md — landscape audit that fed the design (see "Realised in v0.1" callout for the post-Wave-14b PRIME-RL recipe; the VeRL design here is older).
• docs/VISION_VALIDATION.md § 7, Objection 3 — Wave 7 audit identifying this gap.
• composer_replication/recipes/prime_rl/ — the actually-tested third-RL-framework recipe (PRIME-RL, with shadow-parity tests against upstream default_loss_fn). VeRL would land alongside it once validated.