| | --- |
| | tags: |
| | - Coder |
| | - Math |
| | - qwen2 |
| | - thinking |
| | - reasoning |
| | model-index: |
| | - name: Palmyra-mini-thinking-a |
| | results: [] |
| | license: apache-2.0 |
| | pipeline_tag: text-generation |
| | language: |
| | - en |
| | library_name: transformers |
| | --- |
| | |
| | <div align="center"> |
| | <h1>Palmyra-mini-thinking-a</h1> |
| |
|
| | </div> |
| |
|
| | ### Model Description |
| |
|
| | - **Language(s) (NLP):** English |
| | - **License:** Apache-2.0 |
| | - **Finetuned from model:** Qwen/Qwen2.5-1.5B |
| | - **Context window:** 131,072 tokens |
| | - **Parameters:** 1.7 billion |
| |
|
| |
|
| | ## Model Details |
| |
|
| | The palmyra-mini-thinking-a model demonstrates exceptional performance in advanced mathematical reasoning and competitive programming. Its capabilities are highlighted by an outstanding score of 0.886 on the 'MATH500' benchmark, showcasing a robust ability to solve complex mathematical problems. The strength of the model in quantitative challenges is further confirmed by its score of 0.8287 on 'gsm8k (strict-match)', which demonstrates proficiency in multi-step arithmetic reasoning. Additionally, the model proves its aptitude for high-level problem-solving with a score of 0.8 on 'AMC23'. The model also shows strong potential in the coding domain, achieving a score of 0.5631 on 'Codeforces (pass_rate)' and 0.5481 on 'Olympiadbench (extractive_match)', indicating competence in generating correct solutions for programming challenges. |
| |
|
| | ## Benchmark Performance |
| |
|
| | This section provides a detailed breakdown of the palmyra-mini-thinking-a model's performance across a standardized set of industry benchmarks. The data is presented in its original order from the source evaluation. |
| |
|
| | | Benchmark | Score | |
| | |:-----------------------------------------------------------------|---------:| |
| | | gsm8k (strict-match) | 0.8287 | |
| | | minerva_math(exact_match) | 0.3842 | |
| | | mmlu_pro(exact_match) | 0.2748 | |
| | | hendrycks_math | 0.0054 | |
| | | ifeval (inst_level_loose_acc) | 0.3657 | |
| | | mathqa (acc) | 0.4171 | |
| | | humaneval (pass@1) | 0.2378 | |
| | | BBH (get-answer)(exact_match) | 0.462 | |
| | | mbpp | 0.304 | |
| | | leadboard_musr (acc_norm) | 0.3413 | |
| | | gpqa lighteval gpqa diamond_pass@1:8_samples | 0.3826 | |
| | | AIME24(pass@1)(avg-of-1) | 0.4333 | |
| | | AIME25(pass@1)(avg-of-1) | 0.3667 | |
| | | Livecodebench-codegen (livecodebench/code_generation_lite v4_v5) | 0.1784 | |
| | | AMC23 | 0.8 | |
| | | MATH500 | 0.886 | |
| | | Minerva | 0.3493 | |
| | | Olympiadbench (extractive_match) | 0.5481 | |
| | | Codecontests (pass_rate) | 0.1778 | |
| | | Codeforces (pass_rate) | 0.5631 | |
| | | Taco (pass_rate) | 0.3083 | |
| | | APPS (all_levels) | 0.0447 | |
| | | HMMT23 (extractive_match) | 0.1 | |
| | | Average | 0.380839 | |
| |
|
| |
|
| |
|
| | ### Use with transformers |
| |
|
| | You can run conversational inference using the Transformers Auto classes with the `generate()` function. Here's an example: |
| |
|
| | ```py |
| | import torch |
| | from transformers import AutoTokenizer, AutoModelForCausalLM |
| | |
| | model_id = "Writer/palmyra-mini-thinking-a" |
| | |
| | tokenizer = AutoTokenizer.from_pretrained(model_id) |
| | |
| | model = AutoModelForCausalLM.from_pretrained( |
| | model_id, |
| | torch_dtype=torch.float16, |
| | device_map="auto", |
| | attn_implementation="flash_attention_2", |
| | ) |
| | |
| | messages = [ |
| | { |
| | "role": "user", |
| | "content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?" |
| | } |
| | ], |
| | |
| | input_ids = tokenizer.apply_chat_template( |
| | messages, tokenize=True, add_generation_prompt=True, return_tensors="pt" |
| | ) |
| | |
| | gen_conf = { |
| | "max_new_tokens": 256, |
| | "eos_token_id": tokenizer.eos_token_id, |
| | "temperature": 0.3, |
| | "top_p": 0.9, |
| | } |
| | |
| | with torch.inference_mode(): |
| | output_id = model.generate(input_ids, **gen_conf) |
| | |
| | output_text = tokenizer.decode(output_id[0][input_ids.shape[1] :]) |
| | |
| | print(output_text) |
| | ``` |
| |
|
| | ## Running with vLLM |
| | ```py |
| | vllm serve Writer/palmyra-mini-thinking-a |
| | ``` |
| | ```py |
| | curl -X POST http://localhost:8000/v1/chat/completions \ |
| | -H "Content-Type: application/json" \ |
| | -d '{ |
| | "model": "Writer/palmyra-mini-thinking-a", |
| | "messages": [ |
| | { |
| | "role": "user", |
| | "content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?" |
| | } |
| | ], |
| | "max_tokens": 8000, |
| | "temperature": 0.2 |
| | }' |
| | ``` |
| |
|
| |
|
| | ## Ethical Considerations |
| |
|
| | As with any language model, there is a potential for generating biased or inaccurate information. Users should be aware of these limitations and use the model responsibly. |
| |
|
| | ### Citation and Related Information |
| |
|
| | To cite this model: |
| | ``` |
| | @misc{Palmyra-mini-thinking-a, |
| | author = {Writer Engineering team}, |
| | title = {{Palmyra-mini: A powerful LLM designed for math and coding}}, |
| | howpublished = {\url{https://dev.writer.com}}, |
| | year = 2025, |
| | month = Sep |
| | } |
| | ``` |
| | Contact Hello@writer.com |
