# Scholarly/code subword pilot `modal_scholarly_pilot.py` connects the merged modern 100M-family model to the frozen 32K tokenizer and `TokenMixture`. It preserves the merged 14×768 shape (124,389,632 unique parameters), with context increased to 1,024. This is a bounded execution/learning pilot, not the planned 20B-token run. From `training/`, first prepare a fresh snapshot with `modal_prepare_scholarly.py`. Then: ```sh modal run --detach modal_scholarly_pilot.py \ --run-id scholarly-124m-pilot-20261016-01 \ --snapshot scholarly-124m-20261016/snapshots/pilot-001 ``` One L40S worker, four CPUs, 16 GiB host RAM, no retries, and a 30-minute hard worker timeout. Training stops at 600 optimizer updates, 5M loss-bearing tokens, 20 minutes including intermediate checkpoint overhead, or source exhaustion. The existing loader enforces at most two passes per source. No full pretraining or automatic continuation is launched. The pilot deliberately permits snapshots with pending corpus audits; this override must not be used for substantial runs. Optimization: FP32 parameters, BF16 autocast, AdamW (betas 0.9/0.95, weight decay 0.1), gradient norm clipping at 1, linear 30-update warmup to 3e-4 then constant learning rate. Batch size 2 with four accumulated microbatches; gradients are normalized by actual nonpadding targets across the complete optimizer batch. This short pilot does not select a production learning-rate schedule. The CPU qualification runs modern-model, tied-embedding and corpus-loader tests. The GPU worker verifies tiny-set fitting, then reloads the actual step-10 checkpoint and checks identical next-batch IDs, labels and logits. Checkpoints include optimizer/RNG/data state, source hashes, model settings, token counts and manifest/tokenizer identities. Checkpoints and logs use fresh paths on `lm-exploration-training//`; the final report is also copied to `results/`. This pilot checkpoint format is separate from the character-only `lm-generate` format. Cross-job resume and generation CLI integration remain future work. Validation uses one fixed document-local window per source, reporting each source and the token-mixture-weighted mean. This small provisional validation sample is for detecting learning/failures, not capability claims or production checkpoint selection. Samples use three fixed prompts, seed 42, full-softmax temperature 1, and up to 64 new tokens. PAD/BOS/UNK are excluded from sampled outputs. The full corpus still needs split reconciliation, global near deduplication, provenance/eligibility and contamination audits, and capacity evidence. Measure throughput before selecting the substantial-run hardware and budget. The current model retains explicit attention; a supported SDPA kernel is a possible separately verified optimization. In the managed cloud environment, Modal's client needs the system CA trust: install `modal[api-proxy-support]==1.6.2` and `truststore`, call `truststore.inject_into_ssl()` before running the Modal CLI module. Keep inherited proxy settings and TLS verification enabled. ## First measured run [The 2026-10-16 report](../results/scholarly-124m-pilot-20261016-01.md) records 600 updates / 4,078,340 tokens, weighted pilot validation loss 10.5811 → 6.0272, 23 passing qualification tests, and a successful checkpoint/data reload. The final checkpoint is `lm-exploration-training/scholarly-124m-pilot-20261016-01/step-000600.pt`. Samples remain incoherent; full corpus audits and substantial training are pending.