| --- |
| library_name: transformers |
| pipeline_tag: text-generation |
| gated: manual |
| extra_gated_prompt: "Access to ProKope-7B weights is provided for academic evaluation, benchmarking, and commercial licensing inquiries. Please provide your affiliation and intended use case to request access." |
| extra_gated_fields: |
| Company or Institution: text |
| Corporate or Academic Email: text |
| Intended Evaluation Use Case: text |
| datasets: |
| - openai/gsm8k |
| - TIGER-Lab/MMLU-Pro |
| - Idavidrein/gpqa |
| - TAUR-Lab/MuSR |
| - lukaemon/bbh |
| - google/IFEval |
| - lighteval/MATH-Hard |
| tags: |
| - text-generation |
| - reasoning |
| - eval-results |
| - gsm8k |
| - mmlu-pro |
| - gpqa |
| - musr |
| - bbh |
| - ifeval |
| - qwen2 |
| - prokope |
| license: apache-2.0 |
| language: |
| - en |
| model_name: ProKope-7B |
| author: Alan Canto |
| creator: Alan Canto |
| --- |
| |
| <p align="center"> |
| <img src="banner.jpg" alt="ProKope-7B" width="100%" style="border-radius: 8px; margin-bottom: 16px;"> |
| </p> |
|
|
| # ProKope-7B |
|
|
| ## Introduction |
|
|
| **ProKope-7B** is a 7.6B-parameter autoregressive language model developed by [ProKope AI](https://huggingface.co/ProKope-AI) for multi-step analytical reasoning, mathematical problem solving, and strict instruction following. |
|
|
| Built on the `Qwen2ForCausalLM` architecture with a 128K native context window (extensible to 512K via YaRN), ProKope-7B is compatible out of the box with `transformers`, `vLLM`, `SGLang`, and `llama.cpp`. |
|
|
| --- |
|
|
| ## Evaluation |
|
|
| Evaluated via EleutherAI's `lm-evaluation-harness` across standard mathematical reasoning and Open LLM Leaderboard v2 benchmarks: |
|
|
| | Benchmark | Dataset | Setting / Metric | ProKope-7B | Chocolatine-Fusion-14B (8.4B) | Qwen2.5-Dyanka-7B | Qwen2.5-7B-Instruct | Llama-3.1-8B-Instruct | |
| | :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: | |
| | **GSM8K** | [`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k) | 5-shot (`strict-match`) | **81.50%** | — | — | **82.71%** | 84.53% | |
| | **IFEval** | [`google/IFEval`](https://huggingface.co/datasets/google/IFEval) | 0-shot (`prompt_strict_acc`) | 70.61% | **79.90%** | 76.40% | 74.80% | 76.10% | |
| | **BBH** | [`lukaemon/bbh`](https://huggingface.co/datasets/lukaemon/bbh) | 3-shot CoT (`acc_norm`) | **53.31%** | 49.50% | 36.62% | 42.10% | 40.80% | |
| | **MuSR** | [`TAUR-Lab/MuSR`](https://huggingface.co/datasets/TAUR-Lab/MuSR) | 0-shot CoT (`acc_norm`) | **44.84%** | 24.10% | 15.51% | 12.80% | 11.20% | |
| | **MMLU-Pro** | [`TIGER-Lab/MMLU-Pro`](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) | 5-shot CoT (`acc`) | **44.03%** | 38.80% | 37.51% | 32.20% | 31.90% | |
| | **GPQA Diamond** | [`Idavidrein/gpqa`](https://huggingface.co/datasets/Idavidrein/gpqa) | 0-shot CoT (`acc_norm`) | **36.87%** | 15.30% | 8.95% | 10.71% | 13.52% | |
| | **MATH Level 5** | [`lighteval/MATH-Hard`](https://huggingface.co/datasets/lighteval/MATH-Hard) | 4-shot CoT (`exact_match`) | 18.86% | 34.56% | **48.79%** | 35.05% | 29.40% | |
| | **Leaderboard v2 Avg** | 6-Task Macro Average | — | **44.75%** | 40.36% | 37.30% | 34.61% | 33.82% | |
|
|
| --- |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| | :--- | :--- | |
| | **Architecture** | `Qwen2ForCausalLM` | |
| | **Total Parameters** | 7.61B (`6.53B` non-embedding parameters) | |
| | **Layers** | 28 Transformer decoder blocks | |
| | **Attention Heads** | 28 Query / 4 Key-Value (GQA) | |
| | **Hidden / Intermediate Size** | 3,584 / 18,944 (SwiGLU) | |
| | **Vocabulary Size** | 152,064 | |
| | **Context Length** | 131,072 tokens native (up to 524,288 with YaRN) | |
| | **Recommended VRAM** | 16 GB+ for `bfloat16` inference | |
| | **Precision** | `torch.bfloat16` (4 `.safetensors` shards, `15.2 GB`) | |
| | **License** | Apache 2.0 (Weights gated under manual review) | |
|
|
| --- |
|
|
| ## Quickstart |
|
|
| ### 1. Standard Inference (`transformers`) |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "ProKope-AI/ProKope-7B" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| ) |
| |
| messages = [ |
| {"role": "system", "content": "You are a helpful and precise analytical assistant."}, |
| {"role": "user", "content": "Explain why grouped-query attention reduces KV-cache memory overhead during long-context decoding."}, |
| ] |
| |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| with torch.no_grad(): |
| outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False) |
| |
| print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ### 2. Serving with `vLLM` |
|
|
| ```bash |
| vllm serve ProKope-AI/ProKope-7B \ |
| --dtype bfloat16 \ |
| --max-model-len 32768 |
| ``` |
|
|
| ### 3. Long-Context Extension (YaRN up to 512K) |
|
|
| ```python |
| from transformers import AutoConfig, AutoModelForCausalLM |
| import torch |
| |
| model_id = "ProKope-AI/ProKope-7B" |
| config = AutoConfig.from_pretrained(model_id) |
| config.rope_scaling = { |
| "type": "yarn", |
| "factor": 4.0, |
| "original_max_position_embeddings": 131072, |
| } |
| config.max_position_embeddings = 524288 |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| config=config, |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| ) |
| ``` |
|
|
| --- |
|
|
| ## Limitations & Intended Use |
|
|
| - **Intended Use**: Analytical reasoning, scientific and technical question answering, code generation, and structured instruction following. |
| - **Limitations**: As an autoregressive language model, outputs may contain factual inaccuracies or mathematical errors. Verify critical outputs independently. |
|
|
| --- |
|
|
| ## Contact & Citation |
|
|
| - **Organization**: [ProKope AI](https://huggingface.co/ProKope-AI) |
| - **Authors**: Alan Canto, Wesley Foreman |
| - **Contact**: [prokope.model@gmail.com](mailto:prokope.model@gmail.com) |
|
|
| ```bibtex |
| @misc{canto2026prokope7b, |
| author = {Alan Canto and Wesley Foreman}, |
| title = {ProKope-7B: Reasoning and Instruction-Tuned Language Model}, |
| howpublished = {Hugging Face}, |
| year = {2026}, |
| url = {https://huggingface.co/ProKope-AI/ProKope-7B} |
| } |
| ``` |
|
|
