README.md

---
base_model: unsloth/qwen2.5-vl-7b-instruct-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- qwen2_5_vl
license: apache-2.0
language:
- en
datasets:
- AI4Math/MathVista
- unsloth/LaTeX_OCR
- mychen76/invoices-and-receipts_ocr_v1
- corto-ai/handwritten-text
---

# Cernis-Thinking: Multi-Task Vision Language Model for Document Understanding

**Cernis-Thinking** is a reasoning-capable vision language model fine-tuned with reinforcement learning (GRPO/GSPO) for document understanding tasks. Built on Qwen2.5-VL-7B, it excels at mathematical reasoning, LaTeX OCR, invoice extraction, and handwriting transcription.

## Model Details

- **Base Model**: [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)
- **Training Method**: Group Relative Policy Optimization (GRPO) with GSPO extensions
- **Training Data**: ~2,000 samples across 4 document understanding tasks
- **Model Size**: 7B parameters
- **License**: Apache 2.0

## Capabilities

Cernis-Thinking is trained on four distinct document understanding tasks:

1. **Mathematical Reasoning** - Solves math problems from images with step-by-step reasoning
2. **LaTeX OCR** - Converts mathematical notation images to LaTeX code
3. **Invoice Extraction** - Extracts structured information from invoices and receipts
4. **Handwriting Transcription** - Transcribes handwritten text from images

## Training Details

### Datasets

- [AI4Math/MathVista](https://huggingface.co/datasets/AI4Math/MathVista) - Mathematical reasoning (filtered for numeric answers)
- [unsloth/LaTeX_OCR](https://huggingface.co/datasets/unsloth/LaTeX_OCR) - LaTeX formula recognition
- [mychen76/invoices-and-receipts_ocr_v1](https://huggingface.co/datasets/mychen76/invoices-and-receipts_ocr_v1) - Invoice extraction
- [corto-ai/handwritten-text](https://huggingface.co/datasets/corto-ai/handwritten-text) - Handwriting transcription

### Reinforcement Learning Approach

The model was trained using GRPO (Group Relative Policy Optimization) with custom reward functions:

**1. Formatting Reward Function**
- Rewards proper use of `<REASONING>` and `<SOLUTION>` tags
- Penalizes malformed outputs (e.g., excessive "addCriterion" artifacts)
- Encourages structured, parseable responses

**2. Task-Specific Correctness Reward**
- **Math**: Exact numeric matching (2.0 points)
- **LaTeX/Handwriting**: String similarity with word overlap scoring (0.75-2.0 points)
- **Invoices**: Partial credit for extracting key information (1.5 points)

**3. ROUGE-like Word Overlap**
- For text-heavy tasks, rewards based on word overlap ratio:
  - >50% overlap: 1.5 points
  - >30% overlap: 0.75 points
  - Prevents wasted training on completely wrong outputs

### Training Configuration

```python
training_args = GRPOConfig(
    learning_rate = 5e-6,
    num_train_epochs = 0.5,
    per_device_train_batch_size = 1,
    gradient_accumulation_steps = 2,
    num_generations = 4,
    max_prompt_length = 1024,
    max_completion_length = 1024,
    
    # GSPO settings
    importance_sampling_level = "sequence",
    loss_type = "dr_grpo",
)
```

## Usage

### With Transformers

```python
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from PIL import Image

# Load model and processor
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "coolAI/cernis-thinking",
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("coolAI/cernis-thinking")

# Prepare image and prompt
image = Image.open("document.jpg")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Extract the key information from this invoice. First provide your reasoning between <REASONING> and </REASONING>, then your answer between <SOLUTION> and </SOLUTION>"}
        ]
    }
]

# Prepare inputs
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True).to(model.device)

# Generate
output_ids = model.generate(**inputs, max_new_tokens=1024)
generated_text = processor.batch_decode(output_ids, skip_special_tokens=True)
print(generated_text[0])
```

### With vLLM (Recommended for Production)

```python
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset

# Initialize vLLM
llm = LLM(
    model="coolAI/cernis-thinking",
    max_model_len=16384,
    gpu_memory_utilization=0.8
)

# Prepare prompt
prompt = "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>What is the LaTeX code shown in this image? Provide your answer between <SOLUTION> and </SOLUTION><|im_end|>\n<|im_start|>assistant\n"

# Sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_k=50,
    max_tokens=1024
)

# Generate
outputs = llm.generate(
    {
        "prompt": prompt,
        "multi_modal_data": {"image": ImageAsset("formula.png").pil_image}
    },
    sampling_params=sampling_params
)

print(outputs[0].outputs[0].text)
```

## Example Outputs

### Mathematical Reasoning
**Input**: Image of geometry problem  
**Output**:
```
<REASONING>
To solve this parallelogram problem, I need to use the properties:
1. Opposite sides are equal in a parallelogram
2. Angle bisectors create specific relationships...
</REASONING>

<SOLUTION>
42
</SOLUTION>
```

### LaTeX OCR
**Input**: Image of mathematical formula  
**Output**:
```
<SOLUTION>
\frac{2}{3} < a^{2} \alpha^{2} \leq 1
</SOLUTION>
```

### Invoice Extraction
**Input**: Invoice image  
**Output**:
```
<SOLUTION>
Invoice No: 53553822
Date: 07/24/2012
Vendor: Leo Brown
Seller Address: 082 Christopher Club Apt. 771 Thomasberg, OH 42949
Seller Tax ID: 926-74-9803
Total: $247.50
</SOLUTION>
```

## Citation

```bibtex
@misc{cernis-thinking-2025,
  title={Cernis-Thinking: Multi-Task Vision Language Model for Document Understanding},
  author={Your Name},
  year={2025},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/coolAI/cernis-thinking}}
}
```

## Acknowledgments

- Built with [Unsloth](https://github.com/unslothai/unsloth) for efficient VLM training
- Base model: [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)
- Training datasets: AI4Math, Unsloth, mychen76, corto-ai

## License

Apache 2.0 - Free for commercial and research use