File size: 5,731 Bytes
df675c0 716408c df675c0 bc60ae4 df675c0 64444a2 df675c0 4302f37 716408c 4302f37 29d0d7b 6883aa8 29d0d7b 6883aa8 29d0d7b 4302f37 29d0d7b 97bae5f acb89ef 6883aa8 97bae5f f43cebf 29d0d7b f43cebf 29d0d7b f43cebf 29d0d7b f43cebf 4302f37 f43cebf b188a8d f43cebf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 | ---
tags:
- Coder
- Math
- qwen2
- thinking
- reasoning
model-index:
- name: Palmyra-mini-thinking-a
results: []
license: apache-2.0
pipeline_tag: text-generation
language:
- en
library_name: transformers
---
<div align="center">
<h1>Palmyra-mini-thinking-a</h1>
</div>
### Model Description
- **Language(s) (NLP):** English
- **License:** Apache-2.0
- **Finetuned from model:** Qwen/Qwen2.5-1.5B
- **Context window:** 131,072 tokens
- **Parameters:** 1.7 billion
## Model Details
The palmyra-mini-thinking-a model demonstrates exceptional performance in advanced mathematical reasoning and competitive programming. Its capabilities are highlighted by an outstanding score of 0.886 on the 'MATH500' benchmark, showcasing a robust ability to solve complex mathematical problems. The strength of the model in quantitative challenges is further confirmed by its score of 0.8287 on 'gsm8k (strict-match)', which demonstrates proficiency in multi-step arithmetic reasoning. Additionally, the model proves its aptitude for high-level problem-solving with a score of 0.8 on 'AMC23'. The model also shows strong potential in the coding domain, achieving a score of 0.5631 on 'Codeforces (pass_rate)' and 0.5481 on 'Olympiadbench (extractive_match)', indicating competence in generating correct solutions for programming challenges.
## Benchmark Performance
This section provides a detailed breakdown of the palmyra-mini-thinking-a model's performance across a standardized set of industry benchmarks. The data is presented in its original order from the source evaluation.
| Benchmark | Score |
|:-----------------------------------------------------------------|---------:|
| gsm8k (strict-match) | 0.8287 |
| minerva_math(exact_match) | 0.3842 |
| mmlu_pro(exact_match) | 0.2748 |
| hendrycks_math | 0.0054 |
| ifeval (inst_level_loose_acc) | 0.3657 |
| mathqa (acc) | 0.4171 |
| humaneval (pass@1) | 0.2378 |
| BBH (get-answer)(exact_match) | 0.462 |
| mbpp | 0.304 |
| leadboard_musr (acc_norm) | 0.3413 |
| gpqa lighteval gpqa diamond_pass@1:8_samples | 0.3826 |
| AIME24(pass@1)(avg-of-1) | 0.4333 |
| AIME25(pass@1)(avg-of-1) | 0.3667 |
| Livecodebench-codegen (livecodebench/code_generation_lite v4_v5) | 0.1784 |
| AMC23 | 0.8 |
| MATH500 | 0.886 |
| Minerva | 0.3493 |
| Olympiadbench (extractive_match) | 0.5481 |
| Codecontests (pass_rate) | 0.1778 |
| Codeforces (pass_rate) | 0.5631 |
| Taco (pass_rate) | 0.3083 |
| APPS (all_levels) | 0.0447 |
| HMMT23 (extractive_match) | 0.1 |
| Average | 0.380839 |
### Use with transformers
You can run conversational inference using the Transformers Auto classes with the `generate()` function. Here's an example:
```py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "Writer/palmyra-mini-thinking-a"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto",
attn_implementation="flash_attention_2",
)
messages = [
{
"role": "user",
"content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?"
}
],
input_ids = tokenizer.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
)
gen_conf = {
"max_new_tokens": 256,
"eos_token_id": tokenizer.eos_token_id,
"temperature": 0.3,
"top_p": 0.9,
}
with torch.inference_mode():
output_id = model.generate(input_ids, **gen_conf)
output_text = tokenizer.decode(output_id[0][input_ids.shape[1] :])
print(output_text)
```
## Running with vLLM
```py
vllm serve Writer/palmyra-mini-thinking-a
```
```py
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Writer/palmyra-mini-thinking-a",
"messages": [
{
"role": "user",
"content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?"
}
],
"max_tokens": 8000,
"temperature": 0.2
}'
```
## Ethical Considerations
As with any language model, there is a potential for generating biased or inaccurate information. Users should be aware of these limitations and use the model responsibly.
### Citation and Related Information
To cite this model:
```
@misc{Palmyra-mini-thinking-a,
author = {Writer Engineering team},
title = {{Palmyra-mini: A powerful LLM designed for math and coding}},
howpublished = {\url{https://dev.writer.com}},
year = 2025,
month = Sep
}
```
Contact Hello@writer.com |