---
license: apache-2.0
language:
- es
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- markdown
- text-to-markdown
- faithful-generation
- llama
- from-scratch
- spanish
- english
- ocr-postprocessing
- rag
- document-understanding
- tiny-llm
model-index:
- name: PASITA
  results:
  - task:
      type: text-generation
    dataset:
      name: PASITA-bench-1000 (held-out, private)
      type: custom
    metrics:
    - name: gfm_validity
      type: accuracy
      value: 0.956
    - name: faithfulness
      type: faithfulness
      value: 0.927
    - name: semantic_faithfulness
      type: faithfulness
      value: 0.888
    - name: table_fidelity
      type: accuracy
      value: 1.0
---

# PASITA v1 — plain text to Markdown (faithful)

A **decoder-only language model trained 100% from scratch** (no base model), specialized in a single task: converting **plain text into valid Markdown while preserving information**.

> Compiler behavior, not chatbot behavior: it adds structure, it does not invent content.

## Architecture (lab notes)

| Parameter | Value |
|---|---|
| Class | `LlamaForCausalLM` (decoder-only, dense, no MoE) |
| Parameters | **88,099,584** (~88M) in `bfloat16` |
| Layers / hidden / FFN | 12 / 768 / 2048 (SwiGLU) |
| Attention | GQA 12Q/4KV, head_dim 64, no bias |
| Positions | RoPE theta=100000, ctx 2048, RMSNorm eps=1e-5 |
| Embeddings | tied (saves ~12.6M params) |
| File | `model.safetensors` (176 MB, 110 tensors, sha `664665e7…`) |
| Tokenizer | Custom 16k byte-level BPE, verified ByteLevel decoder (~4.0 chars/token ES/EN) |
| Special tokens | `<pad> <s> </s> <unk> <think> </think>` |

## Training

1. **SFT** 58M tokens x2 epochs — final loss 0.09, token accuracy 98.4%
2. **DPO** beta=0.1 — preference margin 4.2
3. **GRPO** 550+150 steps, G=4, verifiable rewards (format + numeric fidelity + anti-overformatting)
4. Data: 80M human markdown-derived tokens (ES/EN Wikipedia, StackExchange, WikiHow) scaled to 625M in v4 corpus

## Benchmark (held-out n=1000, greedy)

| Global | GFM 0.956 - faith 0.927 - sem 0.888 - tables 1.0 |
|---|---|
| code / ocr / docs / math / tables / html | 0.93 - 1.00 |
| control (strict instructions) | 0.19 (known limitation) |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("OpceanAI/PASITA")
model = AutoModelForCausalLM.from_pretrained("OpceanAI/PASITA", dtype="auto")
prompt = "CONVIERTE A MARKDOWN:\n" + text + "\n\n### Markdown:\n"
ids = tok(prompt, return_tensors="pt", truncation=True, max_length=1024)
out = model.generate(**ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
```

Valid regime: medium/long documents (OCR output, pasted HTML, meeting notes, tutorials). Fragile on 1-3 line inputs.

## Limitations

May truncate digits, drop secondary data, emit echo H1s, or continue past completion. Recommended: adaptive `max_new_tokens` + beam search + fidelity rerank.