ESM-ClimbMix-1B

OpenESM d26 model trained on ClimbMix. This repository contains the pretraining checkpoint without optimizer state conversion.

Files

• model.ckpt: Lightning checkpoint.
• modeling_esm.py: standalone model and checkpoint loader.
• tokenizer/tokenizer.pkl: serialized ESM tokenizer.
• tokenizer/token_bytes.pt: token byte table for BPB evaluation.

The 1B label describes the model parameter count. Training token counts are intentionally omitted from the repository name and file names.

Load

import torch
from modeling_esm import load_checkpoint

model, tokenizer, hparams, device = load_checkpoint(
    "model.ckpt",
    tokenizer_path="tokenizer",
)
tokens = tokenizer.encode("Hello world", append=tokenizer.get_bos_token_id())
input_ids = torch.tensor([tokens], device=device)
logits = model(input_ids)
print(logits.shape)

The code and checkpoint format are maintained in the OpenESM repository.