drrockso72/rockso1.85b_checkpoints

Rockso 1.85B — current training checkpoint

Public development snapshot from the fresh final_32_seed42_20261030 run, saved on November 4, 2026 at 12:00:37 PM America/Chicago (18:00:37 UTC): step 20,206, 6,621,102,080 training input tokens. model.pt is an exact, byte-preserving copy of the local Ctrl+C export.

This is the latest verified save from the current run. No benchmark scores are claimed for this checkpoint.

Acknowledgements — Sapient Intelligence and HRM-Text

Rockso builds on the open research and resources released by Sapient Intelligence, particularly HRM-Text, its sampled RSI pretraining dataset, and its original 65,536-token BPE tokenizer. Their dataset, tokenizer, and PrefixLM instruction/response conventions form a substantial part of Rockso’s foundation.

The tokenizer distributed with Rockso is preserved byte-for-byte from Sapient’s dataset release. We gratefully acknowledge the Sapient team and the HRM-Text authors for making these resources publicly available.

Rockso is an independently trained model with its own architectural and training modifications. It is not an official Sapient release, and this acknowledgement does not imply affiliation or endorsement.

Upstream resources

• Dataset and original tokenizer: sapientinc/hrm-text-rsi-sampled-20260923
• HRM-Text codebase: sapientinc/HRM-Text
• Research paper: HRM-Text: Efficient Pretraining Beyond Scaling

Upstream research citation

The HRM-Text authors provide the following citation for their research:

@misc{wang2026hrmtextefficientpretrainingscaling,
      title={HRM-Text: Efficient Pretraining Beyond Scaling},
      author={Guan Wang and Changling Liu and Chenyu Wang and Cai Zhou and Yuhao Sun and Yifei Wu and Shuai Zhen and Luca Scimeca and Yasin Abbasi Yadkori},
      year={2026},
      eprint={2605.20613},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.20613},
}

Checkpoint and source

The model contains 1,846,878,764 unique parameters, 32 layers, width 2048, 32 query heads / 8 KV heads, SwiGLU width 6832, a 65,536-token vocabulary and a 2,048-token total context. The exact native architecture is in config.json and in the checkpoint itself. This JSON is not a Transformers PretrainedConfig.

All 565 exported state-dict tensors and their stored dtypes match the corresponding full local training checkpoint exactly. That companion records step 20,206, 6,621,102,080 tokens; the run summary independently agrees. The export itself does not contain these progress counters.

model.pt is 3,694,628,377 bytes, with SHA-256:

3ae65e9d3ad586738caf7375f29cbb9c011c35ab0fb25fe17027e7dd1ced8bb2

Nothing was stripped or rewritten, including the original tokenizer-path metadata. The supplied portable loader permits a relocated tokenizer only after verifying its exact bytes and special IDs. Tied embeddings and the shared MUDD projection are preserved.

This is a model-only export, not a resumable training checkpoint. It contains no optimizer, master weights, RNG state or consumed data cursor. The full 9,072,748,875-byte raw training checkpoint remains local and is not substituted for this download.

File	Purpose
model.pt	Exact current native model-only export.
model.provenance.json	Identity, fresh-run counters, companion comparison, source hashes and resume limitations.
config.json	Exact resolved native model configuration.
tokenizer.json, tokenizer_config.json, special_tokens_map.json	Original tokenizer files, preserved byte-for-byte.
tokenizer_source_config.json	Original tokenizer-directory config.json, renamed to avoid overwriting the model config. Its qwen3 label is tokenizer-side metadata, not Rockso's architecture.
chat_template.jinja, inference_contract.json	PrefixLM prompt format, attention boundary, answer-stop ID and current checkpoint identity.
rockso_current_source_2026-11-02.zip	Previously published source/tokenizer bundle. Its native source and tokenizer match the current checkpoint; its bundled checkpoint-specific audit records describe the November 2 save.
CPU_VALIDATION.json	Completed strict native CPU-load and structural validation receipt.
SHA256SUMS	SHA-256 identities for the current files, except this checksum file itself.

The source archive includes the live trainer/model, launcher, vendored NorMuon kernels, Adam grouping and writeback, runtime versions, parameter assignments, data/ledger contracts and source hashes. It excludes credentials and the full training corpus. No new source archive or operational run-data bundle is published in this refresh. Use the root provenance and checksums for the current checkpoint; the older archive retains its original historical audit records.

The verified recipe uses NorMuon LR 0.007 and Adam LR 0.0005. NorMuon momentum ramps from 0.85 to 0.95 over 1,000 updates; the extra shape LR multiplier has been removed. NorMuon matrices use stochastic BF16 writeback without persistent FP32 matrix masters. Adam embedding masters and moments are FP32; Adam writeback uses ordinary dtype conversion. Detailed assignments and code are in the source archive. FP8 MLP GEMM padding to 6912 does not change the logical SwiGLU width of 6832.

Public download and CPU loading

This repository is public. The model, source archive and metadata can be downloaded without a Hugging Face access token. Pin the repository revision when reproducing a test.

hf download drrockso72/rockso1.85b_checkpoints --local-dir ./rockso1.85b_checkpoints
cd rockso1.85b_checkpoints
sha256sum -c SHA256SUMS
unzip rockso_current_source_2026-11-02.zip
cd source_bundle
CUDA_VISIBLE_DEVICES= PYTHONDONTWRITEBYTECODE=1 python - <<'PY'
from portable_cpu_check import load_native_cpu
model, state, pad_id, aliases, result = load_native_cpu('../model.pt')
print(result)
PY

For source, configuration and tokenizer files without the large checkpoint:

hf download drrockso72/rockso1.85b_checkpoints --local-dir ./rockso1.85b_checkpoints --exclude '*.pt'

Pin the repository commit when reproducing an experiment. The archive records Python 3.10, torch 2.12.0.dev20260316+cu130 and the full package inventory; exact nightly wheel availability on another platform is not guaranteed. The loader command performs strict native CPU loading, value/dtype and alias checks, and tokenizer verification. It does not run model inference, backpropagation, optimizer updates or CUDA. The full structural check was rerun locally for the new export, and CPU_VALIDATION.json records the passing checks. Do not run the older archive's checkpoint-hash main verifier unchanged: its bundled expected checksum belongs to November 2. Its fixed fixture is a structural example, not an unseen quality benchmark.

For native loading, the archive provides load_native_cpu in portable_cpu_check.py. The preserved source and runtime instructions support further testing with the exact architecture. This release does not claim plug-and-play Transformers AutoModel, vLLM or llama.cpp support. No new GPU generation test or quality evaluation was performed for this update, and no training was restarted. Safetensors conversion and a tested standalone answer-generation command are not included.

Native prompt contract

For short, direct answers:

<|im_start|><|object_ref_start|>What is 2 + 2?<|im_end|>

For worked reasoning, use synth,cot in that order:

<|im_start|><|quad_end|><|object_ref_end|>Solve for x: x + sqrt(x - 2) = 4<|im_end|>

Direct begins with IDs [2, 4]; synth,cot begins with [2, 9, 5]. Both prompts end with <|im_end|> (ID 3). Start generation immediately afterward; stop on <|box_end|> (ID 7). The tokenizer's original EOS token is <|endoftext|> (ID 1); do not confuse it with Rockso's answer-end token.

The entire initial prompt, including ID 3, is a bidirectional PrefixLM prefix. Freeze that boundary throughout generation. Answer tokens are causal and can attend to the complete prefix. Ordinary causal prefill, generic Qwen role formatting, adding <|box_start|>, or extending the bidirectional prefix with generated tokens changes the model's inference contract.

The Jinja template accepts condition="direct" or condition="synth,cot", one user message, an optional preceding system message, and an optional completed assistant answer. It rejects unsupported multi-turn/tool-role serialization. It formats text only; it cannot enforce the required attention mask.

No license has been selected for this release; none is inferred from tokenizer metadata.

History

The repository retains only the latest checkpoint weights. Superseded weight objects and their historical references are removed after verifying each replacement. Source and documentation history may remain. Local production and checkpoint files are preserved.

Architecture overview

Token IDs + condition tokens
          │
          ├── Token embeddings → residual stream
          │
          └── Shared value embeddings → selected attention layers

                32 transformer blocks
          ┌────────────────────────────────────────┐
          │ MUDD depth mixing at two sites          │
          │                 ↓                      │
          │ RMSNorm → Q/K/V projections             │
          │                 ↓                      │
          │ QK normalization + partial RoPE         │
          │                 ↓                      │
          │ Gated value-embedding injection         │
          │                 ↓                      │
          │ Grouped-query PrefixLM attention        │
          │                 ↓                      │
          │ XSA → attention output gates            │
          │                 ↓                      │
          │ Output projection + scaled residual     │
          │                 ↓                      │
          │ RMSNorm → SwiGLU + scaled residual      │
          └────────────────────────────────────────┘
                           │
                      Final RMSNorm
                           │
                    Tied vocabulary head
                           │
                     Next-token logits