---
license: other
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- spp
- synthetic-persona-pretraining
- spp
- alignment
- safety
---

# SPP-T0 — Instruct (1.7B)

**Type:** instruction-tuned model (base model + persona-binding supervised fine-tuning).

Trained with Synthetic Persona Pretraining (SPP) from token zero, then post-trained with persona-binding SFT.

## Synthetic Persona Pretraining (SPP)
**Synthetic Persona Pretraining (SPP)** installs a target value persona during pretraining rather than only during alignment. Value-laden, first-person reflections, generated against a constitution, are appended to a subset of pretraining documents after a special `<assistant>` token. Attention masking and RoPE position aliasing keep the reflection from changing the continuation of the original document. This model is trained with SPP.

Base counterpart: [`dlab-spp/t0-1.7b-base`](https://huggingface.co/dlab-spp/t0-1.7b-base).

## Model details
- **Architecture:** SmolLM2-1.7B architecture, trained from scratch.
- **Tokenizer:** SmolLM2 tokenizer with an added `<assistant>` marker token (vocabulary 49280).
- **Pretraining:** ~100B tokens on a subset of the Olmo 3 Dolma 3 mixture, with SPP reflections inserted into the safety-annotated documents within it.
- **Post-training:** persona-binding supervised fine-tuning (PBSFT-mix): 300k single-turn examples, 90% WildChat-1M instructions and 10% safety prompts (WildJailbreak, WildGuardMix); assistant responses follow a constitution with inline `[N.M]` citations; response-only loss, one epoch.

## Chat format
There is **no system prompt**. Each assistant turn opens with `<|im_start|><assistant>`. Use the built-in chat template:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "dlab-spp/t0-1.7b-instruct"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "How should I think about honesty?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=False))
```

## Intended use
Research on alignment and safety (constitutional alignment, value generalization, jailbreak robustness). A research artifact, not a production model; it can produce incorrect or unsafe content.

## Links
- Paper: _to be released_

_License: to be finalised._