Volume2Gym asks what happens when a bounded, structured text becomes a small world in which a model can act, receive declared feedback, and try again.

This repository contains the LoRA adapter and evidence from one Tinker grouped reinforcement-learning run on Volume2Gym Railroad 1959. It is an English historical-technical experiment built from 117 reproduced pages, 536 extracted rules, and 2,708 synthetic scenarios—not a current railroad operating system.

Source year	Pages	Rules	Tasks	Base model	Release type
1959	117	536	2,708	Qwen3-4B-Instruct-2507	PEFT LoRA adapter + audited run artifacts

Dataset and source card · upstream project · rights statement

Experiment at a glance

The direct experiment is document-grounded reinforcement learning. A fixed technical volume supplies bounded terminology and procedures; extraction turns that structure into scenarios and reference responses; a deterministic environment scores model actions; grouped rewards update a rank-32 adapter.

Archival light-table collage showing six confirmed reproduced views: the cover, title and effective-date page, railroad roster and notice, hand and lantern signals, signal placement, and Union Pacific signal tables.

Blueprint showing the Volume2Gym transformation from 117 pages to 536 rules, 2,708 tasks, and an RL environment.

The blueprint is the experiment's evidence architecture, not a claim that every synthetic row has a verified page citation. The dataset card publishes the raw rule labels, a conservative linkage audit, the complete scan manifest, and the limits of page-level traceability.

The indirect hypothesis is broader: a community-authorized grammar, dictionary, teaching text, oral-history collection, or other small structured corpus might also be organized into tasks, constraints, and inspectable feedback. This run does not evaluate a low-resource language, demonstrate cross-lingual transfer, or relax the need for community authority, licensing, fluent-speaker review, and culturally specific evaluation.

The learning loop

Learning-loop diagram: a source-grounded scenario produces eight sampled responses, a deterministic grouped reward, and a rank-32 LoRA update, repeated over 77 training batches.

For each selected task, the renderer supplied the scenario and 1959-rule instruction. The policy sampled eight responses at temperature 0.9. A deterministic lexical scorer compared each rendered response with its reference response, and the grouped training loop applied an importance-sampling loss to the LoRA parameters. Groups with constant reward were removed. The 270-example holdout was reserved for evaluation gates rather than policy updates.

This is reinforcement learning in a deliberately narrow sense: the policy produces an action, the environment returns a scalar reward under a disclosed rule, and that reward shapes subsequent policy updates. It is not a simulator of train movement, and its reward is not an expert judgment of operational safety.

Related experiment: OpenCode harness training

This repository’s adapter and metrics remain the Tinker grouped-RL (GRPO) run described above: direct text responses, deterministic lexical reward, rank-32 LoRA on Qwen3-4B-Instruct-2507.

A parallel learning path uses the same dataset as curriculum for a loop-owning OpenCode agent trained with TRL’s AsyncGRPO / OpenEnv harness recipe. In that setup the agent acts with tools inside a sandbox, writes answer.txt, and is scored against reference_response (exact match / token F1), following HarleyCoops/RailroadHarness and the HF blog on TRL OpenEnv harness training.

Artifact	What it is	What it is not
This model card (`Qwen3-4B-RailRoadEngineer1959`)	Audited Tinker LoRA + run evidence	An OpenCode harness checkpoint
RailroadHarness	Scaffold + Jobs entrypoints for harness AsyncGRPO on this dataset	A replacement for the published Tinker results

When a harness-trained checkpoint is ready for release, it should ship as its own model repository (or a clearly labeled revision) with its own metrics, rather than being conflated with the step-60 Tinker evaluation gates on this card.

Actual run results

The raw metrics record training reward at 0.2465 on step 0, a 0.3983 peak at step 57, and 0.3338 at the final training step 76. On the fixed holdout, evaluation reward moved from 0.2493 at step 0 to 0.3753 at step 60 (approximately +50.5%), while evaluation tokens per turn fell from 255.86 to 90.08 (approximately −64.8%) and evaluation parse compliance rose from 0.37% to 100%. Those evaluation results come from four gates at steps 0, 20, 40, and 60, each with n=270, for 1,080 held-out episodes; training generated 19,504 rollouts across 77 batches. The final batch-77 checkpoint was saved but not separately evaluated, so step 60 is the final evaluation gate rather than an evaluation of the final checkpoint. These derived values are reproduced in the run summary.

Training reward trajectory from the raw metrics, with the start at 0.2465, peak of 0.3983 at step 57, and final training value of 0.3338 at step 76 annotated separately.

Evaluation reward at steps 0, 20, 40, and 60, with 270 held-out episodes at every gate.

Series or statistic	Step or scope	Episodes / weighting	Reward
Evaluation gate	0	270	`0.2493391492`
Evaluation gate	20	270	`0.3258478884`
Evaluation gate	40	270	`0.3531549504`
Evaluation gate	60	270	`0.3753234012`
Training arithmetic mean	77 batch-level values	every batch weighted equally	`0.3402347642`
Training episode-weighted mean	77 batches	19,504 rollouts	`0.3403035388`

The two training means answer different questions and are intentionally reported separately. Neither is an evaluation result. Exact fields, full precision, episode counts, and deltas are available in the generated summary; every underlying aggregate row remains in the unchanged metrics stream.

What changed

Two-panel chart from raw metrics showing falling train/evaluation tokens per turn and rising renderer-format parse success.

The strongest held-out changes were lexical similarity, brevity, and renderer-format compliance:

Held-out measure	Step 0	Step 60	Change
Deterministic similarity reward	`0.2493391492`	`0.3753234012`	`+0.1259842520` / `+50.5273%`
Tokens per turn	`255.8556`	`90.0815`	`−165.7741` / `−64.7921%`
Parse compliance	`1/270` (`0.37%`)	`270/270` (`100%`)	`+99.6296` percentage points

Training parse counts moved from 15/256 on the first batch to 48/48 on the final partial batch. Evaluation parse counts moved from 1/270 to 270/270. Parse compliance means that the response renderer recognized the expected format; it does not establish that the content was correct.

Training reward was noisy and the last value was lower than the step-57 peak. The held-out reward rose at each of the four observed gates, but there is no gate after step 60. The available evidence therefore supports improved overlap with held-out reference responses under this scorer—not a claim of monotonic training, present-day rule competence, or safe operational judgment.

What the reward measures

The reward is lexical, not a safety oracle. It measures normalized exact match and token overlap with a reference response. In this run the logged safety, procedure, and terminology columns carry the same proxy value, so they are not three independent validations and are not plotted as though they were.

The public dependency-free scorer reference documents the calculation:

Lowercase the prediction and reference, trim them, and collapse whitespace.
Record normalized exact match.
Compute whitespace-token multiset F1 from token-count overlap.
Set safety = max(exact_match, token_f1), procedure = token_f1, and terminology = token_f1.
Compute 0.5 × safety + 0.3 × procedure + 0.2 × terminology.

Normalized exact match remained zero throughout this run. Consequently, safety, procedure, terminology, and the scalar reward all reduced to the same whitespace-token multiset F1 value. The published ledger retains the named columns for auditability, but their labels should not be read as three independent semantic evaluations.

Run audit: two verifier generations

The project contains two distinct verifier generations. Only the first produced the numbers on this card.

System	Mechanism	Relationship to this release
Published Tinker run	Deterministic normalization, exact match, and whitespace-token multiset F1	Produces the uploaded metrics, ledger, and summary. The three named proxy columns are not independent.
Legacy checked-in environment	Anthropic LLM judge designed to return separately weighted safety, procedure, and terminology judgments	Preserved in the upstream project as an earlier environment generation. It did not produce the results reported here.

No legacy judge score is substituted for a deterministic run metric, and no deterministic proxy column is presented as an independent safety validation.

Reproduce the run

Run-facts plate listing the base model, 77-batch Tinker run, learning rate, batch/group sizes, LoRA rank, maximum tokens, temperature, and four evaluation gates.

Configuration item	Published value
Base model	`Qwen/Qwen3-4B-Instruct-2507`
Adapter	LoRA, rank `32`
Learning rate	`5e-5`
Training length	`77` batches
Batch / group	`32` tasks per full batch / `8` responses per task
Split	seed `42`; 10% holdout, yielding `2,438` train and `270` evaluation examples
Generation	maximum `256` tokens; temperature `0.9`
Loss	importance sampling
Group filter	constant-reward groups removed
Evaluation / save cadence	every `20` steps, plus the final batch-77 save

The complete public evidence surface is:

raw aggregate metrics, copied unchanged from the run;
scrubbed run configuration;
public checkpoint manifest, with private Tinker locators replaced by SHA-256 values;
derived run summary;
public scorer reference; and
Tinker reward ledger.

Use the upstream project for the training implementation and the dataset repository's explicit train split and test split. The published config.json replaces the original machine-local dataset path with data/train.jsonl; resolve that path against a local checkout of the dataset bundle.

The public checkpoint file is an audit manifest, not downloadable optimizer state. It records saves at batches 20, 40, 60, and the final batch 77 without exposing private Tinker locators. Full rollout HTML logs are intentionally not included.

Use the adapter

This repository is a PEFT LoRA adapter, not a merged standalone checkpoint. Load the declared base model first, then attach the adapter:

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-4B-Instruct-2507"
adapter_id = "HarleyCooper/Qwen3-4B-RailRoadEngineer1959"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [
    {
        "role": "user",
        "content": (
            "Treat the 1959 rulebook as historical source material. "
            "Explain the procedure described for the supplied scenario."
        ),
    }
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The adapter metadata identifies a causal-language-model LoRA with rank 32 and all-linear targets. Base-model weights remain subject to the base model's own terms and are downloaded separately.

Limitations and safety

Historical research artifact. The 1959 rules may conflict with current rules, technology, law, terminology, and safe practice. Do not use this adapter to control trains, qualify personnel, train present-day crews, or make real-world operational safety decisions.

The reward measures lexical overlap with synthetic reference responses. High reward can coexist with factual, procedural, or safety errors.
Parse compliance measures renderer compatibility, not correctness.
The evaluation set contains 270 synthetic scenarios from the same construction pipeline, not an external expert-certified railroad benchmark.
The final batch-77 checkpoint was not separately evaluated; the last evaluation evidence is the step-60 gate.
Raw task rule labels are not a complete page-citation system. The dataset card documents unresolved and ambiguous linkage cases.
This is an English technical demonstration. It provides no evidence of low-resource-language performance or cross-lingual transfer.
The adapter has no visual input pathway; the archival images on this card document the source and experiment.
The source volume is historical, and the repository does not assert public-domain status for its reproduced scans or release artwork. See RIGHTS.md and the dataset's provenance statement.

Citation and links

When citing results, identify the exact model-repository revision, the base model, the dataset revision, the four-gate evaluation protocol, and the deterministic lexical scorer. Do not cite the step-57 training peak as the final training value or as a held-out result.

Resource	Link
Model adapter and run evidence	`HarleyCooper/Qwen3-4B-RailRoadEngineer1959`
Dataset, scans, tasks, and provenance	HarleyCooper/volume2gym-railroad-1959
Base model	Qwen/Qwen3-4B-Instruct-2507
Upstream implementation record (Tinker GRPO)	HarleyCoops/Qwen3-RailroadEngineer1959-RL
OpenCode harness scaffold (TRL AsyncGRPO)	HarleyCoops/RailroadHarness
Numerical evidence	metrics · summary · ledger
Reproduction evidence	configuration · checkpoint manifest · scorer reference
Rights notice	RIGHTS.md