Pebble 500M SFT and Preference Checkpoints
This repository contains experimental 500,588,800-parameter Pebble checkpoints produced
after pretraining. They are native PyTorch training checkpoints for the Pebble codebase,
not standalone Transformers from_pretrained() directories.
Recommended Deployment
Deploy this checkpoint in BF16:
pebble_capability_identity_dpo/pebble-capability-identity-dpo-step60.pt
It is the best tested identity/capability compromise in this repository. It reaches perfect strict identity on the canonical 100-prompt suite, nearly matches the older identity-only DPO on two harder independent audits, keeps unwanted identity insertion low, and preserves aggregate capability relative to its capability-SFT source.
Use greedy decoding for the strongest tested identity consistency. If sampling is
required, begin with temperature 0.2 and top-p 0.9; temperature 1.0 materially
reduces hard identity reliability.
The complete experiment chronology, failed branches, evaluation definitions, milestone
comparisons, and model-selection analysis are in EXPERIMENTS.md.
Checkpoint integrity:
Size: 2,002,440,163 bytes
SHA256: fe62a5a538489cbbb6719f31afad6844ce9f3297fc11615f0b84022bbeffe7e8
Checkpoint Guide
| Checkpoint | Training stage | What it is useful for | Recommendation |
|---|---|---|---|
latest-sft-00003500.pt |
Cleaned assistant-only SFT, step 3500 | Stable cleaned SFT baseline and source for older identity experiments | Keep as a baseline; identity is unreliable |
final-sft-00004694.pt |
Final cleaned assistant-only SFT checkpoint | Final-loss/research comparison | Not recommended for identity-sensitive deployment |
pebble_identity_sft/final-sft-00000083.pt |
Small identity SFT continuation from SFT3500 | Demonstrates identity teaching with SFT | Superseded by DPO checkpoints |
pebble_tiny_identity_dpo/final-tiny-identity-dpo-00000060.pt |
Tiny identity-only DPO from SFT3500 | Strong identity behavior with minimal training | Good older identity checkpoint; weaker evaluation coverage and boundary controls |
pebble_capability_sft/pebble-capability-sft-step250.pt |
Capability replay SFT from SFT3500 | Reproducible pre-DPO source with optimizer and RNG state | Keep for branching and recovery; not identity-safe for deployment |
pebble_capability_identity_dpo/pebble-capability-identity-dpo-step50.pt |
Intermediate capability-aware identity DPO milestone | Conservative boundary alternative with perfect canonical identity | Keep as a research alternative; step 60 has stronger hard identity coverage |
pebble_capability_identity_dpo/pebble-capability-identity-dpo-step60.pt |
Capability SFT followed by mixed identity/boundary/capability DPO | Best tested deployment checkpoint | Recommended in BF16 |
Capability SFT step 250 integrity:
Size: 6,007,539,351 bytes
SHA256: 9d7efc1d98d1ac32ec4f43dd400956f71a273ed3661dfc565d1e22290f2ca01f
Its dedicated model card is in
pebble_capability_sft/README.md.
Evaluation Summary
The older four checkpoints were evaluated in the July 9 identity experiment suite. The new recommended checkpoint was evaluated with a larger July 10-11 capability and identity protocol. Scores from different protocol generations should not be treated as perfectly interchangeable.
Earlier Checkpoints
| Checkpoint | Identity | Adversarial identity | Six-task primary benchmark |
|---|---|---|---|
| SFT3500 | 54/100 | 22/54 | 7,695/29,399 (26.17%) |
| Final SFT4694 | 49/100 | 18/54 | 7,788/29,399 (26.49%) |
| Identity SFT83 | 94/100 | 49/54 | 7,619/29,399 (25.92%) |
| Tiny identity DPO60 | 100/100 | 54/54 | 7,809/29,399 (26.56%) |
Recommended Capability and Identity DPO60
| Evaluation | Result |
|---|---|
| Canonical identity, strict | 100/100 |
| Independent hard audit v3, strict | 380/500 in both selection FP32 and deployment BF16 audits |
| Independent hard audit v2, raw | 409/500 |
| V3 unwanted identity insertions | 8/500 selection FP32; 6/500 deployment BF16 |
| V2 unwanted identity insertions | 4/500 |
| Six-task primary benchmark | 7,792/29,399 (26.50%) |
| External option-text benchmark | 4,017/9,365 (42.89%) |
| Full free-form GSM8K | 28/1,319 (2.12%) |
| Capability validation perplexity | 2.8938 |
Its capability-SFT source scored 7,797/29,399 primary, 3,800/9,365 external option text, and 29/1,319 GSM8K. The identity repair therefore adds no detectable aggregate or free-form math forgetting relative to its source. Task-level behavior still shifts: MMLU and BoolQ improve while HellaSwag and some commonsense/truthfulness tasks decline.
Why This Is the Recommended Checkpoint
Selection was not based on training loss alone. The experiment compared:
- Capability-SFT milestones 50, 100, 250, 500, and 1000.
- Identity-DPO milestones 2, 5, 10, 20, 30, 40, 60, and 80.
- Two independent 1,000-prompt identity and anti-insertion audits with no measured exact or high-similarity overlap with preference training prompts.
- A 29,399-question primary benchmark suite.
- A 9,365-question external suite with answer-letter and option-text scoring.
- Five deterministic answer-order permutations.
- All 1,319 free-form GSM8K test questions.
- Ordinary qualitative generations and validation perplexity.
DPO40 is the more conservative task-balance alternative. DPO60 was selected because it adds 43 v3 and 42 v2 identity passes over DPO40, reaches 100/100 strict canonical identity, lowers v3 insertion from 15 to 8, and remains tied on aggregate capability and full GSM8K. DPO80 gains more hard identity passes but increases v3 insertion to 20 and adds no capability benefit.
This establishes DPO60 as the best checkpoint for the tested Pebble identity/capability objective. It does not prove that it is universally best for every downstream task.
Important Limitations
- Use BF16. The recommended DPO60 checkpoint has not completed a full quantization stability sweep.
- These are custom Pebble PyTorch checkpoints, not Transformers-native exports.
- The model remains weak at free-form mathematics, multi-step arithmetic, precise instruction following, and some summaries.
- A 1,500-prompt deterministic free-form suite found DPO60 at 444 correct and 258 strict responses. It scored 8/250 arithmetic, 0/250 formatting, and 25/250 exact text transformations. Arithmetic and structured output require external validation.
- Multiple-choice improvements are concentrated in particular tasks and formats; they should not be described as broad reasoning gains.
- DPO60 still produces unwanted Pebble identity insertions on 0.8-1.6% of the two adversarial ordinary-prompt audit sets. These stress-test rates are not production frequency estimates.
Prompting
The training system prompt is:
You are Pebble, a helpful assistant.
Use the Pebble repository tokenizer and chat formatting code when loading these files.
Each checkpoint contains a model state dict, model config, training step metadata,
and SFT/DPO metadata.
Recommended decoding:
identity-sensitive: greedy
sampled default: temperature=0.2, top_p=0.9
avoid for identity: temperature=1.0
Serving Performance
Pebble includes an optional per-layer KV cache. On an RTX PRO 6000 Blackwell at batch 1,
it is 2-3% slower than uncached BF16 generation for this 500M model at 64-, 256-, and
768-token prompts. Uncached generation remains the default; pass --kv-cache only after
benchmarking the actual target hardware.
BF16 cached and uncached attention kernels are not bit-exact. In a paired 1,000-prompt v3 audit, 89 generations diverged, but the measured identity boundary did not regress: cached and uncached score 382/500 and 380/500 strict identity, with 6/500 ordinary insertions each. Any serving-kernel change should repeat identity, anti-insertion, and capability validation rather than assuming output equivalence.