HansBug/qwen3-8b-seta-agent57-i271

qwen3-8b seta-agent57 iter 0000271

PyTorch Distributed Checkpoint (DCP) of a Qwen3-8B RL training run with Agent57-lite intrinsic exploration on terminal/software-engineering tasks (the seta dataset).

For downstream eval that must align with training: see § Inference / Eval reproduction. Machine-readable configs live in config/; the most useful single file is config/inference_config.json. All sglang values below are cross-checked against the actual training log (config/sglang_runtime_observed.log).

Source

• Run id: terminal-rl_qwen3-8b_8gpu_seta_dapo_nodynamic_exploration_simhash_life_fp_ucb_v0615_envtolerant_fastwarm_dualadv_think_2026-06-16_075145
• Run started: 2026-06-16 07:51:50 UTC (Beijing 15:51:50)
• Iteration: 271 (last checkpoint of the run; the same number is in latest_checkpointed_iteration.txt)
• Internal training step counter: _monotonic_train_step = 418
• Save interval: every 8 iterations (full sequence: 199, 207, 215, 223, 231, 239, 247, 255, 263, 271)
• Training host: gpu-lg-cmc-h-h200-0806.host.h.pjlab.org.cn (single 8×H200 node)

Code

• Repo: https://github.com/Gen-Verse/OpenClaw-RL
• Branch: dev-agenticrl-safety-exploration-harness
• Commit: 14e52fa5c3b4a10b482bb8a8467e5b354a563300
• Entrypoint: slime/train_async.py (Megatron-LM async-RL on top of slime, harness via terminal-rl/)
• Python: 3.12.12, custom env lightrft_py312

Base model & topology

Field	Value
Base model	Qwen3-8B (Qwen/Qwen3-8B HF ref)
Layers	36
Hidden	4096
FFN hidden	12288
Heads / KV groups	32 / 8 (GQA)
KV channels	128
Vocab	151,936
Normalization	RMSNorm (ε=1e-6)
RoPE base	1,000,000
Activation	SwiGLU
qk_layernorm	yes
Untie embed / output	yes
Precision (train)	bf16; attention softmax fp32; grad accumulate fp32
Precision (sglang load)	torch.bfloat16 (confirmed across all 4 TP ranks, train.log L2324-2327)
TP / PP / CP / EP	4 / 1 / 1 / 1
Actor GPUs	4
Rollout GPUs (sglang)	4

RL algorithm

• Estimator: GRPO (advantage_estimator='grpo')
• Loss: DAPO clip — eps_clip=0.2, eps_clip_high=0.28 (asymmetric clip)
• Per-token loss, no KL loss (use_kl_loss=0, kl_loss_coef=0)
• Dynamic sampling: OFF (dapo_dynamic_sampling=0, hence nodynamic in the run name)
• n_samples_per_prompt=8, group_size=2, grpo_iterations=2
• rollout_batch_size=8, global_batch_size=32, num_rollout=2000
• Optimizer: Adam (β1=0.9, β2=0.98), lr=1e-6 constant, weight_decay=0.1, clip_grad=1.0
• Rollout max context / response: 16,384 / 8,192 tokens
• Overlong-buffer enabled: 4096-token buffer, factor 1.0

Exploration — Agent57-lite (spear_lite profile)

This run's defining feature. Agent57-style intrinsic exploration plumbed into the GRPO advantage:

• Episodic novelty memory: SimHash-KNN, 4096-entry capacity, k=5, cosine distance, 256-dim vectors, 64-bit signatures
• Lifelong novelty memory: SQLite-backed (agent57_lite.sqlite3, ~8 MB), 200k capacity, count_decay=0.995, hierarchical with dataset/skill/global tiers (weights 0.5 / 0.35 / 0.15)
• 8 explore arms, betas [0, 0.002, 0.004, 0.006, 0.008, 0.01, 0.015, 0.02], temperatures lightly varied, top_p=1
• UCB arm controller: c=0.5, window=256, ε=0.02, dataset-aware, seed 20260605
• Intrinsic reward: cosine schedule, coef=0.015, decay over 120 steps, NGU lite combine mode
• Dual-stream advantage: λ=0.08, intrinsic key explore_agent57_intrinsic_signal, clip=0.5
• Soft trust gate: completed=1.0, truncated/failed=0.05, warmup=0.3
• Think mode enabled (CoT)
• Truncation penalty: −0.03

Dataset & harness

• Dataset tag: seta (terminal-bench / software-engineering tasks, "none-c0" safety variant)
• Prompt data: terminal-rl/dataset/seta_env_convert/train.filtered.jsonl
• Reward key: score, semantics = terminal-task unit-test pass rate (0..1)
• Harness: camel-agent, max_iteration=10, tool_call_parser=qwen25 (on the slime side; sglang's own tool_call_parser is None), non_think_mode=false

Training trajectory (excerpts from metrics.jsonl, 351 rollouts logged)

rollout	unit-test pass_rate	task_reward
0	0.106	−0.789
40	0.519	+0.038
120	0.245	−0.509
200	0.447	−0.105
220	0.406	−0.187
260	0.540	+0.080
≥280	(rollouts started failing with completed=0; run effectively ended)

iter_0000271 was saved right after the 0.54 pass-rate peak at rollout 260, and is the de-facto final checkpoint of the run.

---

Inference / Eval reproduction

Every sglang/router value in this section is cross-referenced against the actual training log (config/sglang_runtime_observed.log / .json), not just the launcher script. Where slime's command-line value and sglang's runtime value differ, both are shown.

Files in config/

File	Purpose
config/inference_config.json	Start here. Distilled training-aligned eval config (sglang slime cmdline + sglang observed runtime + router observed + generation + harness + dataset + exploration).
config/sglang_runtime_observed.log	Verbatim log lines from train.log capturing the slime/sglang router args, the full ServerArgs(...) print, the post-init KV-cache banner, and the weight-load dtype=torch.bfloat16 records. Single source of truth for the values below.
config/sglang_runtime_observed.json	Same data, parsed into structured JSON.
config/training_args.json	Full argparse.Namespace dump from common.pt — all 1235 args slime/Megatron saw at train time.
config/train_command.sh	The exact train_async.py command line captured at run start (copy-paste reproducible).
config/run_config.json	High-level run config snapshot the launcher wrote next to the checkpoint.
config/rollout_config.yaml	Verbatim copy of the YAML slime read at runtime.
config/meta.json	Git commit + host + start time + path layout.
config/launcher_v0623.sh	The wrapper launcher script the user pointed us at (...ucb_pu_v0623.sh). See note on v0615 → v0623 below.
config/launcher_exploration.sh	Middle-layer launcher (...mixed_dapo_nodynamic_exploration_pu.sh).
config/launcher_inner.sh	Bottom-layer launcher with the actual python -u train_async.py ... invocation.

sglang engine — what slime asked for vs what sglang chose

Knob	slime cmdline (training_args.json)	sglang ServerArgs at runtime (train.log L2249)
TP size	--rollout-num-gpus-per-engine 4	tp_size=4
PP / DP / EP	(none)	pp_size=1, dp_size=1, ep_size=1
KV mem	--sglang-mem-fraction-static 0.6	mem_fraction_static=0.6
Attention backend	--attention-backend flash (this is Megatron's actor backend)	attention_backend='fa3' (sglang chose FlashAttn 3 by default; --sglang-attention-backend was unset)
Sampling backend	(default)	sampling_backend='flashinfer'
Grammar backend	(default)	grammar_backend='xgrammar'
Schedule policy	(default)	schedule_policy='fcfs'
Chunked prefill	--sglang-chunked-prefill-size unset	chunked_prefill_size=8192 (sglang derived)
Max prefill tokens	(default)	max_prefill_tokens=16384
Page size	(default)	page_size=1
Radix cache	(default)	disable_radix_cache=False (enabled)
CUDA graph	(default)	disable_cuda_graph=False, cuda_graph_max_bs=512, 52 captured batch sizes
Dtype	(default)	dtype='auto' → resolved to torch.bfloat16
KV cache dtype	(default)	kv_cache_dtype='auto'
Trust remote code	(default)	trust_remote_code=True
Random seed (sglang internal)	(default)	random_seed=1234 (distinct from slime rollout_seed=42)
chat_template	unset	None
tool_call_parser	unset	None (slime harness handles qwen25 parsing outside sglang)
reasoning_parser	unset	None
speculative_algorithm	unset	None
Served name	(default)	served_model_name='/mnt/shared-storage-user/puyuan/code/slime/Qwen3-8B/'
Watchdog	(default)	watchdog_timeout=300

sglang post-init runtime banner (train.log L2473)

max_total_num_tokens=2301930   chunked_prefill_size=8192
max_prefill_tokens=16384       max_running_requests=4096
context_len=40960              available_gpu_mem=53.25 GB  (per H200, after init)

context_len=40960 is the model's full RoPE range that sglang allocates KV for. The slime rollout layer caps individual requests at rollout_max_context_len=16384, so the engine has headroom but never receives a request longer than 16K during this run.

slime/sglang router (train.log L2207)

RouterArgs(host='10.102.223.16', port=4578, policy='cache_aware', backend='sglang',
           cache_threshold=0.3, max_payload_size=536870912,
           request_timeout_secs=14400,        # ← runtime value, slime cmdline was 1800
           max_concurrent_requests=-1, queue_size=100, queue_timeout_secs=60,
           retry_max_retries=5, retry_initial_backoff_ms=50, retry_max_backoff_ms=30000,
           cb_failure_threshold=10, cb_success_threshold=3, cb_window_duration_secs=120,
           health_check_endpoint='/health', health_check_interval_secs=60,
           prometheus_host='0.0.0.0', prometheus_port=4811, log_level='warn',
           backend='sglang', history_backend='memory', ...)

Equivalent external sglang launch for eval

Run a stand-alone sglang server mirroring the engine state above:

python -m sglang.launch_server \
  --model-path        <consolidated_qwen3_8b_dir> \
  --tokenizer-path    <consolidated_qwen3_8b_dir> \
  --served-model-name qwen3-8b-seta-agent57-i271 \
  --tp-size           4 \
  --dp-size           1 \
  --mem-fraction-static 0.6 \
  --attention-backend fa3 \
  --sampling-backend  flashinfer \
  --grammar-backend   xgrammar \
  --chunked-prefill-size 8192 \
  --max-prefill-tokens   16384 \
  --schedule-policy   fcfs \
  --page-size         1 \
  --dtype             auto \
  --kv-cache-dtype    auto \
  --context-length    40960 \
  --random-seed       1234 \
  --trust-remote-code \
  --port              30000

Generation knobs (rollout + eval)

Use the same sampling as training-time rollouts to keep policy distribution aligned:

Knob	Training-time value	Notes
rollout_temperature	1.0
rollout_top_p	1.0
rollout_top_k	−1 (disabled)
rollout_max_response_len	8192
rollout_max_context_len	16384	(engine's context_len=40960 is upper bound, not used)
rollout_max_prompt_len	16383	derived
apply_chat_template	false	slime applies it inside generate.generate
rollout_skip_special_tokens	false
n_samples_per_prompt	8	for training-style rollouts
n_samples_per_eval_prompt	16	for slime's --eval-* path
eval_max_response_len	16384	eval lets responses go to full context
eval_top_p	1.0
eval_temperature	null → inherits 1.0
rollout_seed	42	slime-side reproducible sampling
sglang_random_seed	1234	sglang's internal seed (different from rollout_seed)
num_steps_per_rollout	2	DAPO 2-step partial-trajectory rollout

Harness (verbatim from rollout_config.yaml)

harness_option: camel-agent
max_iteration: 10
model_type: Qwen3
non_think_mode: false
tool_call_parser: qwen25          # qwen25 parsing lives in slime harness, NOT in sglang
trajectory_save_interval: 10
trajectory_save_interval_seta: 10

Note on v0615 vs v0623 launcher

271 was trained from the v0615 launcher (the script name carries the same tag). The user-supplied config/launcher_v0623.sh is a newer revision of the same family that tweaks three exploration knobs:

Knob	v0615 (used by 271)	v0623
EXPLORE_AGENT57_TRUST_TRUNCATED	0.05	0.2
EXPLORE_ADVANTAGE_LAMBDA	0.08	0.05
EXPLORE_TRUNCATION_PENALTY	−0.03	0

For exact 271 reproduction, take values from training_args.json / inference_config.json (which encode v0615). For ongoing training continuation the maintainers prefer v0623.

Quickstart for downstream eval

1. Consolidate the DCP shards:

   python -m torch.distributed.checkpoint.format_utils \
       dcp_to_torch_save <local_clone_of_this_repo> consolidated.pt
   

2. Convert to HF format with the slime/Megatron torch_dist → HF exporter, using the architecture in inference_config.json:base_model.architecture (Qwen3-8B). The tokenizer comes from Qwen/Qwen3-8B.
3. Launch sglang with the command above (single 4-GPU TP engine, attention_backend=fa3, mem-fraction-static=0.6, chunked_prefill_size=8192, context_length=40960).
4. Sample with the aligned knobs in the Generation knobs table; wrap the model in the camel-agent harness with max_iteration=10, parse tool calls with qwen25 parser on the harness side.
5. For policy-only eval, disable all exploration: set the EXPLORE_* env vars to 0 (or call generate.generate without the exploration bonus post-process). For exploration-bonus eval, use the values in inference_config.json:exploration verbatim.

Files (in this repo)

11 weight files, 107 GiB total, plus config/ (245 KB):

__{rank}_{shard}.distcp   for rank in 0..3, shard in 0..1   (8 files, 13.35 GiB each)
.metadata                  (562 KiB, DCP shard index)
common.pt                  (128 KiB, optimizer state + args.Namespace)
metadata.json              (119 B, backend identifiers)
config/                    (training + inference config payload — see § Inference / Eval reproduction)
README.md

Loading the DCP shards natively

The native loader expects the same tensor-parallel topology (TP=4, PP=1, CP=1, EP=1):

import torch.distributed.checkpoint as dcp
state_dict = {...}  # template with empty tensors of expected shapes
dcp.load(state_dict, checkpoint_id="path/to/checkpoint_dir")

Verification

All 11 weight files were verified against the source by size + sha256 after upload (LFS sha256 for the 10 LFS files, content download + sha256 for metadata.json). All matched.

Companion runs

• HansBug/qwen3-8b-spear-i375 — earlier mixed-DAPO run with spear_lite exploration but without Agent57 layer
• HansBug/qwen3-8b-mt10-i343 — earlier mt10 baseline, no exploration bonus