Pacific Prime Math v2
Status: Experimental - Coherent output but not yet stable for production use.
Model Info
- Base: Complexity Deep 1.5B
- Fine-tuning: Conversational SFT on math datasets
- Precision: BF16
Training Details
| Parameter | Value |
|---|---|
| Epochs | 1 |
| Learning Rate | 5e-6 |
| Batch Size | 1 x 128 (gradient accumulation) |
| Max Length | 1024 |
| Warmup Ratio | 0.1 |
Datasets
- OpenAI GSM8K (20%)
- AI-MO NuminaMath-CoT (30%)
- MetaMathQA (20%)
- Microsoft Orca-Math (30%)
Total samples: ~407k
Usage
from complexity_deep import DeepForCausalLM, DeepConfig
from safetensors.torch import load_file
from transformers import PreTrainedTokenizerFast
import json
# Load model
with open('./config.json') as f:
config = DeepConfig.from_dict(json.load(f))
model = DeepForCausalLM(config).cuda()
state = load_file('./hf/model.safetensors')
model.load_state_dict(state, strict=False)
model.eval()
# Load tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained('./hf')
# Generate
prompt = "User: What is 25 + 17?\n\nAssistant:"
inputs = tokenizer(prompt, return_tensors='pt').to('cuda')
output = model.generate(inputs.input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0]))
Example Generation
Prompt:
User: Natalia sold 48 clips in April. She sold half as many in May. How many clips did she sell in total?
Output:
User: Natalia sold 48 clips in April. She sold half as many in May. How many clips did she sell in total?.
In April, Natalia sold 48 + 2 = 80 clips. To find out how many clips she sold in total, we subtract
the number of clips sold by the number of clips sold from the total number of clips sold:
1600 clips (total) - 80 clips (win-span) = 10 clips (for May).
So, Natalia sold a total of $\boxed{10}$ clips in April.
Expected answer: 48 + 24 = 72 clips
Note: The model generates step-by-step reasoning in the correct format but the math calculations are incorrect.
Known Issues
- Math reasoning is not always accurate
- Model generates step-by-step format but calculations can be wrong
- Requires specific prompt format (
User: ... Assistant:)
Next Steps
- Continue training (v3) with lower LR (2e-6) and more epochs
- Evaluate on GSM8K benchmark
- Test with chain-of-thought prompting
Files
pacific-prime-math-v2/
hf/
├── config.json
├── model.safetensors # BF16 weights
├── tokenizer.json
└── tokenizer_config.json
Complexity AI - Pacific Prime Project
- Downloads last month
- -