README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: translation
language:
- en
tags:
- code
- code-translation
- sas
- r
- python
- lora
- axolotl
---

# Euclid-2.6

Bidirectional SAS ↔ R ↔ Python program translation. A 24B dense code model, LoRA-tuned and merged to standalone BF16 weights.

## Model Summary

Euclid-2.6 translates complete statistical and data-processing programs across all six directed pairs over {SAS, Python, R}. The training objective is **behavioural equivalence**: given identical inputs, the translated program must compute identical values and bind them to identically-named results.

| | |
|---|---|
| Developer | ProCogia |
| Method | LoRA SFT (r=128), adapter merged into the weights |
| Checkpoint | step 368 (1.0 epoch) |
| Parameters | 24B dense |
| Hidden / layers / intermediate | 5120 / 40 / 32768 |
| Attention heads / KV heads / head dim | 32 / 8 / 128 |
| Tokenizer | Tekken, 131,072 vocab |
| Context | 256k architectural; trained at 8,192 |
| Precision | BF16, ~48 GB |
| License | Apache 2.0 |

**Multimodality.** A vision tower is present in the architecture and untouched by fine-tuning. The model is text-only in practice; the tower carries ~1–2 GB of inert VRAM and dictates the loader class (see [How to Use](#how-to-use)).

## Intended Use

- Single-turn translation of complete programs across the six directed pairs over {SAS, Python, R}.
- Target environments matching the training distribution: **R** — base R plus `dplyr`; **Python** — `pandas`, `numpy`, `scipy`, `statsmodels`.
- Output contract: the translated program only, no prose, no markdown fences.

## How to Use

### Loader class

`AutoModelForCausalLM` raises `Unrecognized configuration class`.

```python
import torch
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained(
    "ProCogia/Euclid-2.6",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
```

Tokenization must route through `mistral-common` using the bundled `tekken.json`.

### vLLM

```bash
vllm serve ProCogia/Euclid-2.6 \
  --tokenizer-mode mistral \
  --max-model-len 16384
```

`--tokenizer-mode mistral` is required. Routing through a jinja template instead of `mistral-common` produces systematically degraded output that closely resembles a genuine accuracy result.

### Prompt format

The model was trained against a single system prompt across every row. Deviating from its contract is off-distribution. The exact prompt (MD5 prefix `dc99ebd18483`) ships with the proprietary training data; the block below reproduces its contract:

```text
You are a code translation engine for statistical and data-processing programs written in SAS, Python, and R.

You are given one complete program in a source language and produce the equivalent program in the target language. Behavioural equivalence is the only criterion: given the same inputs, your program must compute the same values and place them in results carrying the same names.

- Preserve the source program's structure, step order, and intent. Carry its comments across as comments in the target language, and add a brief comment where the target expresses a source construct non-obviously.
- Name every result exactly as the source names it, so results can be compared name for name.
- SAS semantics decide the answer even when SAS is not the target language. A SAS date is whole days since 1960-01-01 and a datetime is seconds since 1960-01-01, both stored as plain numbers. A SAS FORMAT changes only how a value is displayed, never what is stored. Missing (.) sorts below every number, so `x < 5` is true when x is missing. Character values are blank-padded to a declared length and compare ignoring trailing blanks.
- A step that only prints or plots produces no data and has no translation outside SAS; leave it out rather than inventing an equivalent.
- Respond with the translated program and nothing else: no prose, no explanation, no markdown code fences, no placeholders.
```

The user turn is:

1. `Translate the following {SOURCE} program to {TARGET}.`
2. A target-language instruction block (one of three: R, Python, SAS) specifying the permitted library environment.
3. The source program, fenced with the **source** language tag.

The model emits a bare program. Absence of a ```` ```sas ```` fence is the direct signal that the fine-tuned weights are active — the untuned weights emit markdown fences, these do not.

## Training Data

**Proprietary. Not released.** Composition is disclosed below for reproducibility of method, not of data.

| | |
|---|---|
| Training rows |**12,952**|
| In-loop validation | 120|
| Held-out execution set | 424|
| Format | JSONL, single-turn `system` + `user` + `assistant` |
| Length (chars) | mean 6,226 / p50 4,682 / p90 11,690 / p99 21,205 / max 68,627 |

Direction balance:

| Direction | Rows | Direction | Rows |
|---|---|---|---|
| Python→R | 2,070 | R→SAS | 2,213 |
| Python→SAS | 2,199 | SAS→Python | 2,199 |
| R→Python | 2,070 | SAS→R | 2,213 |

## Training Procedure

### Weight preparation

The starting checkpoint ships in native FP8 (`float8_e4m3fn`, block-quantized) with no official BF16 weights, and FP8 does not support training. Casting via `model.to(torch.bfloat16)` **silently no-ops** on quantized linear layers, writing FP8 bytes labelled BF16.

Weights were dequantized by streaming safetensors shards directly:

```
W_bf16 = W_fp8.to(float32) × expand_blocks(weight_scale_inv)
```

280 tensors (40 layers × 7 projections) were converted — exactly the modules LoRA attaches to. `activation_scale`, `input_scale`, and `kv_scale` are FP8-runtime only and were discarded. Verification: 280/280 converted, on-disk dtype scan `Counter({'BF16': 585})` with zero `F8_E4M3`, finiteness assertion passed on every parameter, 48.0 GB output, and a live generation coherence check.

### LoRA configuration

| Parameter | Value |
|---|---|
| Rank `r` | 128 |
| `alpha` | 256 (α = 2r → rank-independent scaling factor of 2) |
| Dropout | 0.05 |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
| Scope | Language model only; vision tower and projector excluded |
| Embeddings / `lm_head` | Frozen |
| Trainable | 739M (≈3.0%) |

Vision exclusion is structural rather than enumerated: the targeting regex keys on `self_attn|mlp` parent names, which exist only in the language model, while the vision tower uses `attention`/`feed_forward`.

MLP targeting accounts for 78.7% of available per-layer LoRA capacity at hidden 5120 / intermediate 32768; attention-only targeting would forfeit four-fifths of it.

### Optimization

| Parameter | Value |
|---|---|
| Learning rate | 1e-4, cosine to 10% of peak |
| Warmup | 30 steps (fixed count, ~4% of 736) |
| Optimizer | `adamw_torch_fused`, β = (0.9, 0.95), wd 0.01 |
| Gradient clipping | 1.0 |
| Micro-batch × accumulation | 1 × 8 (≈36 examples/step) |
| Sequence length | 8,192, sample packing on, cross-sample masked |
| Loss | Completion-only |
| Precision | BF16 + gradient checkpointing, FlashAttention-2 |
| Epochs | 2, checkpointed every 0.5 |
| Seed | 42 |
| Total steps | 736 |

At α=2r, LR 1e-4 is equivalent in effective update magnitude to LR 2e-4 at α=r. Over-length rows were **dropped, never truncated** — 12 rows exceeded 8,192 tokens (0.09%), measured with exact `mistral-common` counts. Truncating an assistant target teaches premature EOS, which is directly harmful on a task whose contract is "the complete program and nothing else."

Six pre-flight gates ran before training: exact token lengths, loss-mask decoding (asserting supervised positions contain only the assistant program), adapter scope and parameter count, packing confirmation, leakage checks, and system-prompt integrity. An adapter weight scan for NaN and residual all-zero `lora_B` tensors was added mid-project and is a required gate for any rerun of this recipe.

### Infrastructure

| | |
|---|---|
| GPU | 1× H100 80GB SXM |
| Framework | Axolotl 0.17.0.dev0, torch 2.10.0+cu128, transformers v5 |
| Peak VRAM | 71.1 GB training / 60.3 GB eval |
| Throughput | ~1,150–1,360 tok/s, ~22 s/step |

## Evaluation

**Pending.**

The planned protocol:

| Element | Specification |
|---|---|
| Set | 424 examples, repo-disjoint, offline |
| Method | Execute source and translation on identical inputs; compare at dataframe level |
| Metric | pass@1, pooled |
| Prompt | Training system prompt, greedy decoding |


## Hardware Requirements

| | |
|---|---|
| Weights on disk | ~48 GB (BF16) |
| KV cache @ 16k context | ~2.6 GB per sequence (40 layers × 8 KV heads × 128 dim) |
| Inert vision tower | ~1–2 GB |
| Practical single-GPU floor | 80 GB (H100 / H200 / A100 80GB) |
| Multi-GPU | 2× 48 GB (L40S, A6000) with tensor parallelism |

## Citation

```bibtex
@misc{euclid_2_5,
  title  = {Euclid-2.6: Bidirectional SAS/R/Python Program Translation},
  author = {ProCogia},
  year   = {2026},
  url    = {https://huggingface.co/ProCogia/Euclid-2.6}
}
```