Residual
Collection
4 items โข Updated
Every sublayer adds its output to one unscaled stream. The control for HC, mHC and Block AttnRes.
This is the final Standard Residual checkpoint in the NextTransformer 49.152B-token architecture comparison. It has 1,355,810,568 parameters and a 2,048-token context. The model was pretrained on ClimbMix for 46,875 updates. The 49B in the repository name refers to training tokens.
| Final evaluation | Result |
|---|---|
| Validation BPB | 0.6740 |
| CORE-22 (ร100) | 31.95 |
Install the tested NextTransformer implementation and the safetensors reader in a Python 3.10โ3.13 environment.
pip install "nexttransformer @ git+https://github.com/Interactive-Active-AI/NextTransformer.git@7aa267a88b45e4a19a9739ccb4ba3f196d7aacc0" safetensors
The example encodes a prompt with the checkpoint's tokenizer and prints its most likely next token.
import base64
import json
from pathlib import Path
import tiktoken
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from nexttransformer.models import build_model, config_from_dict
root = Path(snapshot_download("NextTransformer/Standard-Residual-49B-Tokens"))
config = config_from_dict(json.loads((root / "model_config.json").read_text()))
model = build_model(config, device="cpu")
model.load_state_dict(load_file(root / "model.safetensors"), strict=True)
if hasattr(model, "set_step"):
model.set_step(46875)
model.eval()
spec = json.loads((root / "tokenizer.tiktoken.json").read_text())
encoding = tiktoken.Encoding(
name=spec["name"],
pat_str=spec["pat_str"],
mergeable_ranks={base64.b64decode(b): rank for b, rank in spec["mergeable_ranks"]},
special_tokens=spec["special_tokens"],
)
prompt = [encoding.encode_single_token("<|bos|>")] + encoding.encode_ordinary("The sky is")
with torch.inference_mode():
logits = model(torch.tensor([prompt]))
print(encoding.decode([int(logits[0, -1].argmax())]))