HarleyCooper/Qwen3-4B-RailRoadEngineer1959

Learning the Rulebook

Volume2Gym asks what happens when a bounded, structured text becomes a small world in which a model can act, receive declared feedback, and try again.

This repository contains the LoRA adapter and evidence from one Tinker grouped reinforcement-learning run on Volume2Gym Railroad 1959. It is an English historical-technical experiment built from 117 reproduced pages, 536 extracted rules, and 2,708 synthetic scenarios—not a current railroad operating system.

Source year	Pages	Rules	Tasks	Base model	Release type
1959	117	536	2,708	Qwen3-4B-Instruct-2507	PEFT LoRA adapter + audited run artifacts

Dataset and source card · upstream project · rights statement

Experiment at a glance

The direct experiment is document-grounded reinforcement learning. A fixed technical volume supplies bounded terminology and procedures; extraction turns that structure into scenarios and reference responses; a deterministic environment scores model actions; grouped rewards update a rank-32 adapter.

The blueprint is the experiment's evidence architecture, not a claim that every synthetic row has a verified page citation. The dataset card publishes the raw rule labels, a conservative linkage audit, the complete scan manifest, and the limits of page-level traceability.

The indirect hypothesis is broader: a community-authorized grammar, dictionary, teaching text, oral-history collection, or other small structured corpus might also be organized into tasks, constraints, and inspectable feedback. This run does not evaluate a low-resource language, demonstrate cross-lingual transfer, or relax the need for community authority, licensing, fluent-speaker review, and culturally specific evaluation.

The learning loop

For each selected task, the renderer supplied the scenario and 1959-rule instruction. The policy sampled eight responses at temperature 0.9. A deterministic lexical scorer compared each rendered response with its reference response, and the grouped training loop applied an importance-sampling loss to the LoRA parameters. Groups with constant reward were removed. The 270-example holdout was reserved for evaluation gates rather than policy updates.

This is reinforcement learning in a deliberately narrow sense: the policy produces an action, the environment returns a scalar reward under a disclosed rule, and that reward shapes subsequent policy updates. It is not a simulator of train movement, and its reward is not an expert judgment of operational safety.

Actual run results

The raw metrics record training reward at 0.2465 on step 0, a 0.3983 peak at step 57, and 0.3338 at the final training step 76. On the fixed holdout, evaluation reward moved from 0.2493 at step 0 to 0.3753 at step 60 (approximately +50.5%), while evaluation tokens per turn fell from 255.86 to 90.08 (approximately −64.8%) and evaluation parse compliance rose from 0.37% to 100%. Those evaluation results come from four gates at steps 0, 20, 40, and 60, each with n=270, for 1,080 held-out episodes; training generated 19,504 rollouts across 77 batches. The final batch-77 checkpoint was saved but not separately evaluated, so step 60 is the final evaluation gate rather than an evaluation of the final checkpoint. These derived values are reproduced in the run summary.

Series or statistic	Step or scope	Episodes / weighting	Reward
Evaluation gate	0	270	0.2493391492
Evaluation gate	20	270	0.3258478884
Evaluation gate	40	270	0.3531549504
Evaluation gate	60	270	0.3753234012
Training arithmetic mean	77 batch-level values	every batch weighted equally	0.3402347642
Training episode-weighted mean	77 batches	19,504 rollouts	0.3403035388

The two training means answer different questions and are intentionally reported separately. Neither is an evaluation result. Exact fields, full precision, episode counts, and deltas are available in the generated summary; every underlying aggregate row remains in the unchanged metrics stream.

What changed

The strongest held-out changes were lexical similarity, brevity, and renderer-format compliance:

Held-out measure	Step 0	Step 60	Change
Deterministic similarity reward	0.2493391492	0.3753234012	+0.1259842520 / +50.5273%
Tokens per turn	255.8556	90.0815	−165.7741 / −64.7921%
Parse compliance	1/270 (0.37%)	270/270 (100%)	+99.6296 percentage points

Training parse counts moved from 15/256 on the first batch to 48/48 on the final partial batch. Evaluation parse counts moved from 1/270 to 270/270. Parse compliance means that the response renderer recognized the expected format; it does not establish that the content was correct.

Training reward was noisy and the last value was lower than the step-57 peak. The held-out reward rose at each of the four observed gates, but there is no gate after step 60. The available evidence therefore supports improved overlap with held-out reference responses under this scorer—not a claim of monotonic training, present-day rule competence, or safe operational judgment.

What the reward measures

The reward is lexical, not a safety oracle. It measures normalized exact match and token overlap with a reference response. In this run the logged safety, procedure, and terminology columns carry the same proxy value, so they are not three independent validations and are not plotted as though they were.

The public dependency-free scorer reference documents the calculation:

1. Lowercase the prediction and reference, trim them, and collapse whitespace.
2. Record normalized exact match.
3. Compute whitespace-token multiset F1 from token-count overlap.
4. Set safety = max(exact_match, token_f1), procedure = token_f1, and terminology = token_f1.
5. Compute 0.5 × safety + 0.3 × procedure + 0.2 × terminology.

Normalized exact match remained zero throughout this run. Consequently, safety, procedure, terminology, and the scalar reward all reduced to the same whitespace-token multiset F1 value. The published ledger retains the named columns for auditability, but their labels should not be read as three independent semantic evaluations.

Run audit: two verifier generations

The project contains two distinct verifier generations. Only the first produced the numbers on this card.

System	Mechanism	Relationship to this release
Published Tinker run	Deterministic normalization, exact match, and whitespace-token multiset F1	Produces the uploaded metrics, ledger, and summary. The three named proxy columns are not independent.
Legacy checked-in environment	Anthropic LLM judge designed to return separately weighted safety, procedure, and terminology judgments	Preserved in the upstream project as an earlier environment generation. It did not produce the results reported here.

No legacy judge score is substituted for a deterministic run metric, and no deterministic proxy column is presented as an independent safety validation.

Reproduce the run

Configuration item	Published value
Base model	Qwen/Qwen3-4B-Instruct-2507
Adapter	LoRA, rank 32
Learning rate	5e-5
Training length	77 batches
Batch / group	32 tasks per full batch / 8 responses per task
Split	seed 42; 10% holdout, yielding 2,438 train and 270 evaluation examples
Generation	maximum 256 tokens; temperature 0.9
Loss	importance sampling
Group filter	constant-reward groups removed
Evaluation / save cadence	every 20 steps, plus the final batch-77 save

The complete public evidence surface is:

• raw aggregate metrics, copied unchanged from the run;
• scrubbed run configuration;
• public checkpoint manifest, with private Tinker locators replaced by SHA-256 values;
• derived run summary;
• public scorer reference; and
• Tinker reward ledger.

Use the upstream project for the training implementation and the dataset repository's explicit train split and test split. The published config.json replaces the original machine-local dataset path with data/train.jsonl; resolve that path against a local checkout of the dataset bundle.

The public checkpoint file is an audit manifest, not downloadable optimizer state. It records saves at batches 20, 40, 60, and the final batch 77 without exposing private Tinker locators. Full rollout HTML logs are intentionally not included.

Use the adapter

This repository is a PEFT LoRA adapter, not a merged standalone checkpoint. Load the declared base model first, then attach the adapter:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-4B-Instruct-2507"
adapter_id = "HarleyCooper/Qwen3-4B-RailRoadEngineer1959"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [
    {
        "role": "user",
        "content": (
            "Treat the 1959 rulebook as historical source material. "
            "Explain the procedure described for the supplied scenario."
        ),
    }
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The adapter metadata identifies a causal-language-model LoRA with rank 32 and all-linear targets. Base-model weights remain subject to the base model's own terms and are downloaded separately.

Limitations and safety

Historical research artifact. The 1959 rules may conflict with current rules, technology, law, terminology, and safe practice. Do not use this adapter to control trains, qualify personnel, train present-day crews, or make real-world operational safety decisions.

• The reward measures lexical overlap with synthetic reference responses. High reward can coexist with factual, procedural, or safety errors.
• Parse compliance measures renderer compatibility, not correctness.
• The evaluation set contains 270 synthetic scenarios from the same construction pipeline, not an external expert-certified railroad benchmark.
• The final batch-77 checkpoint was not separately evaluated; the last evaluation evidence is the step-60 gate.
• Raw task rule labels are not a complete page-citation system. The dataset card documents unresolved and ambiguous linkage cases.
• This is an English technical demonstration. It provides no evidence of low-resource-language performance or cross-lingual transfer.
• The adapter has no visual input pathway; the archival images on this card document the source and experiment.
• The source volume is historical, and the repository does not assert public-domain status for its reproduced scans or release artwork. See RIGHTS.md and the dataset's provenance statement.

Citation and links

When citing results, identify the exact model-repository revision, the base model, the dataset revision, the four-gate evaluation protocol, and the deterministic lexical scorer. Do not cite the step-57 training peak as the final training value or as a held-out result.

Resource	Link
Model adapter and run evidence	HarleyCooper/Qwen3-4B-RailRoadEngineer1959
Dataset, scans, tasks, and provenance	HarleyCooper/volume2gym-railroad-1959
Base model	Qwen/Qwen3-4B-Instruct-2507
Upstream implementation record	HarleyCoops/Qwen3-RailroadEngineer1959-RL
Numerical evidence	metrics · summary · ledger
Reproduction evidence	configuration · checkpoint manifest · scorer reference
Rights notice	RIGHTS.md