README.md

---
license: apache-2.0
base_model:
- Writer/palmyra-mini-thinking-b
tags:
- mlx
- qwen2
- palmyra
- thinking
- reasoning
---

# Palmyra Mini Thinking B - MLX BF16

## Model Description

This is a bfloat16 precision version of the [palmyra-mini-thinking-b model](https://huggingface.co/Writer/palmyra-mini-thinking-b), optimized for Apple Silicon using the MLX framework. This model is based on the Qwen2 architecture and represents an advanced iteration of the thinking model series, featuring improved reasoning capabilities and a refined chat template using the ChatML format.

## Quick Start

### Installation

```bash
pip install mlx-lm
```

### Usage

```python
from mlx_lm import load, generate

# Load the model
model, tokenizer = load("/Users/thomas/Documents/Model Weights/SPW2 Mini Launch/palmyra-mini-thinking-b/MLX/palmyra-mini-thinking-b-MLX-bf16")

# Generate text
prompt = "<|im_start|>user\nSolve this step by step: What is 20% of 350?<|im_end|>\n<|im_start|>assistant\n"
response = generate(model, tokenizer, prompt=prompt, verbose=True, max_tokens=512)
print(response)
```

## Technical Specifications

### Model Architecture
- **Model Type**: `qwen2` (Qwen2 Architecture)
- **Architecture**: `Qwen2ForCausalLM`
- **Parameters**: ~1.7 billion parameters
- **Precision**: bfloat16
- **Specialization**: Advanced reasoning and thinking tasks

### Core Parameters
| Parameter | Value |
|-----------|-------|
| Hidden Size | 1,536 |
| Intermediate Size | 8,960 |
| Number of Layers | 28 |
| Attention Heads | 12 |
| Key-Value Heads | 2 |
| Head Dimension | 128 |
| Vocabulary Size | 151,936 |

### Attention Mechanism
- **Attention Type**: Full attention across all 28 layers
- **Max Position Embeddings**: 131,072 tokens
- **Attention Dropout**: 0.0
- **Sliding Window**: Not used
- **Max Window Layers**: 28

### RoPE (Rotary Position Embedding) Configuration
- **RoPE Theta**: 1,000,000.0 (enhanced for longer contexts)
- **RoPE Scaling**: None

### Advanced Features
- **Extended Context**: Enhanced RoPE theta for better long-context performance
- **Improved Tokenizer**: Qwen2Tokenizer with 151,936 vocabulary size
- **ChatML Format**: Uses standard ChatML conversation format
- **Word Embeddings**: Tied embeddings for efficiency

### File Structure
```
palmyra-mini-thinking-b/MLX/palmyra-mini-thinking-b-MLX-bf16/
├── config.json                    # Model configuration
├── model.safetensors              # Model weights (2.9GB)
├── model.safetensors.index.json   # Model sharding index
├── tokenizer.json                 # Tokenizer configuration
├── tokenizer_config.json          # Tokenizer settings
├── special_tokens_map.json        # Special tokens mapping
├── chat_template.jinja            # ChatML template
├── generation_config.json         # Generation parameters
├── added_tokens.json              # Additional tokens
├── merges.txt                     # BPE merges
└── vocab.json                     # Vocabulary mapping
```

## Performance Characteristics

### Hardware Requirements
- **Platform**: Apple Silicon (M1, M2, M3, M4 series)
- **Memory**: ~2.9GB for model weights
- **Recommended RAM**: 10GB+ for optimal performance
- **Precision**: Full bfloat16 precision

### Layer Configuration
All 28 layers use full attention mechanism as specified in the `layer_types` configuration, providing consistent attention patterns optimized for reasoning tasks.

## Training Details

### Tokenizer
- **Type**: Qwen2Tokenizer with 151,936 vocabulary size
- **Special Tokens**:
  - EOS Token ID: 151643 (`<|endoftext|>`)
  - Pad Token ID: 151643 (`<|endoftext|>`)
  - IM Start: 151644 (`<|im_start|>`)
  - IM End: 151645 (`<|im_end|>`)

### Model Configuration
- **Hidden Activation**: SiLU (Swish)
- **Normalization**: RMSNorm (ε = 1e-06)
- **Initializer Range**: 0.02
- **Attention Dropout**: 0.0
- **Word Embeddings**: Tied (for efficiency)
- **Use Cache**: True (optimized for inference)

### Chat Template
The model uses the standard ChatML format for conversations:
```
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
{assistant_response}<|im_end|>
```

### Generation Configuration
- **BOS Token ID**: 151643
- **EOS Token ID**: 151643
- **Transformers Version**: 4.53.0

## Usage Examples

### Reasoning Task
```python
prompt = """<|im_start|>system
<|im_end|>
<|im_start|>user
A train travels 180 miles in 3 hours. If it maintains the same speed, how far will it travel in 7 hours?<|im_end|>
<|im_start|>assistant
"""

response = generate(model, tokenizer, prompt=prompt, max_tokens=300)
```

### Problem Solving
```python
prompt = """<|im_start|>system
You are a helpful assistant that thinks step by step.<|im_end|>
<|im_start|>user
Explain the process of photosynthesis in plants.<|im_end|>
<|im_start|>assistant
"""

response = generate(model, tokenizer, prompt=prompt, max_tokens=400)
```

### Multi-turn Conversation
```python
prompt = """<|im_start|>system
<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
The capital of France is Paris.<|im_end|>
<|im_start|>user
What is the population of that city?<|im_end|>
<|im_start|>assistant
"""

response = generate(model, tokenizer, prompt=prompt, max_tokens=200)
```

## Known Limitations

1. **Platform Dependency**: Optimized specifically for Apple Silicon; may not run on other platforms
2. **Memory Requirements**: Requires significant memory due to full precision weights
3. **Context Management**: While supporting long contexts, performance may vary with very long sequences
4. **Format Dependency**: Optimized for ChatML format; other formats may not work as well

## Compatibility

- **MLX-LM**: Requires recent version with Qwen2 support
- **Apple Silicon**: M1, M2, M3, M4 series processors
- **macOS**: Compatible with recent macOS versions supporting MLX
- **Transformers**: Version 4.53.0+

## License

Apache 2.0

#### Original model card below:

---


<div align="center">
  <h1>Palmyra-mini-thinking-b</h1>

</div>

<p align="center">  
  <img src="https://huggingface.co/Writer/palmyra-mini-thinking-b/resolve/main/logo-mini-b%20benchmark-performance.png?download=true" width="800"/>
</p>

### Model Description

- **Language(s) (NLP):** English
- **License:** Apache-2.0
- **Finetuned from model:** Qwen/Qwen2.5-1.5B
- **Context window:** 131,072 tokens
- **Parameters:** 1.7 billion

## Introduction

Palmyra-mini-thinking-b represents a significant step forward in generative AI, demonstrating exceptional capabilities in complex reasoning and problem-solving domains. This model excels in mathematical and programming challenges, showcasing a robust understanding of abstract concepts and logical structures. Its performance is not just a measure of its power but a testament to its specialized training, which has honed its ability to tackle tasks that demand deep, multi-step thinking.

## Mathematical Prowess

The model's mathematical abilities are particularly noteworthy. It achieves an impressive score of 0.925 on the AMC23 benchmark, indicating a strong grasp of advanced high school mathematics. This is further complemented by its performance on MATH500, where it scores 0.882, proving its proficiency across a wide range of mathematical problems. The model also shows its strength in competitive mathematics, scoring 0.6 on AIME24(pass@1)(avg-of-1) and 0.5733 on Olympiadbench (extractive_match). These scores highlight the model's capacity for sophisticated mathematical reasoning, making it a powerful tool for both educational and research applications.

## Excellence in Competitive Programming

Beyond mathematics, Palmyra-mini-thinking-b demonstrates strong performance in the competitive programming arena. Its score of 0.6343 on the Codeforces (pass_rate) benchmark underscores its ability to understand complex algorithmic problems and generate correct, efficient code. This capability suggests the model is well-suited for tasks involving code generation, debugging, and algorithmic design, making it a valuable asset for software developers and computer science researchers.

## Benchmark Scores (sampling params: temperature:0.6, top_p:0.95)

Pass@1(avg-of-64)

| Benchmark | Pass@1 (avg-of-64)   | Majority@64  |
| :-------- | :------------------- | :----------- |
| AIME24    | 59.43%               | 71.67%       |
| AIME25    | 49.69%               | 60.00%       |
| GPQA      | 42.01%               | 47.22%       |
| HMMT25    | 27.86%               | 30.00%       |
| HLE       | 5.22%                | N/A          |
| MMLU-PRO  | 55.49%               | 60.60%       |
| MATH500   | 93.80%               | 95.40%       |
| LCB       | 34.51%               | N/A          |

LCB here is version v6_2408_2505


Pass@1(avg-of-1)

| Benchmark                                                        |   Score (%) |
|:-----------------------------------------------------------------|------------:|
| GSM8K (strict-match)                                             | 42.68%      |
| Minerva Math (exact match)                                       | 7.08%       |
| MMLU-PRO (exact match)                                           | 29.26%      |
| MATH (Hendrycks)                                                 | 0.16%       |
| IFEval (inst_level_loose_acc)                                    | 32.97%      |
| MathQA (acc)                                                     | 30.45%      |
| HumanEval (pass@1)                                               | 7.32%       |
| BBH (get-answer)(exact match)                                    | 28.80%      |
| MBPP                                                             | 16.80%      |
| GPQA (diamond, pass@1: 8 samples)                                | 39.58%      |
| AIME24 (pass@1)(avg-of-1)                                        | 60.00%      |
| AIME25 (pass@1)(avg-of-1)                                        | 50.00%      |
| Livecodebench-codegen (livecodebench/code_generation_lite v4_v5) | 28.73%      |
| AMC23                                                            | 92.50%      |
| MATH500                                                          | 88.20%      |
| Minerva                                                          | 29.41%      |
| Olympiadbench (extractive_match)                                 | 57.33%      |
| Codecontests (pass_rate)                                         | 20.18%      |
| Codeforces (pass_rate)                                           | 63.43%      |
| Taco (pass_rate)                                                 | 34.56%      |
| APPS (all_levels)                                                | 5.84%       |
| HMMT (Feb 2025) (extractive_match)                               | 23.33%      |
| Average                                                          | 35.94%      |

### Use with transformers

You can run conversational inference using the Transformers Auto classes with the `generate()` function. Here's an example:

```py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Writer/palmyra-mini-thinking-b"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    attn_implementation="flash_attention_2",
)

messages = [
      {
        "role": "user",
        "content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?"
      }
    ],

input_ids = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
)

gen_conf = {
    "max_new_tokens": 256,
    "eos_token_id": tokenizer.eos_token_id,
    "temperature": 0.3,
    "top_p": 0.9,
}

with torch.inference_mode():
    output_id = model.generate(input_ids, **gen_conf)

output_text = tokenizer.decode(output_id[0][input_ids.shape[1] :])

print(output_text)
```

## Running with vLLM
```py
vllm serve Writer/palmyra-mini-thinking-b
```
```py
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Writer/palmyra-mini-thinking-b",
    "messages": [
      {
        "role": "user",
        "content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?"
      }
    ],
    "max_tokens": 8000,
    "temperature": 0.2
  }'
```

## Ethical Considerations

As with any language model, there is a potential for generating biased or inaccurate information. Users should be aware of these limitations and use the model responsibly.


### Footnotes

- Base model: This model builds on NVIDIA's OpenReasoning-Nemotron-1.5B (`https://huggingface.co/nvidia/OpenReasoning-Nemotron-1.5B`).
- Evaluation methodology:
  - Pass@1 (avg-of-1): computed using `lm_eval` and `lighteval`.
  - Pass@1 (avg-of-64) and Majority@64: computed using `nemoskills`.

### Citation and Related Information


To cite this model:
```
@misc{Palmyra-mini-thinking-b,
  author = {Writer Engineering team},
  title = {{Palmyra-mini: A powerful LLM designed for math and coding}},
  howpublished = {\url{https://dev.writer.com}},
  year = 2025,
  month = Sep 
}
```
Contact Hello@writer.com