README.md

---
license: other
tags:
- code-translation
- sas
- r
- python
- lora-merged
language:
- en
---

# Euclid-1.0

Bidirectional SAS ↔ R ↔ Python program translation. A 31B hybrid Mamba-2 / MoE / attention model, LoRA-tuned and merged to 16-bit weights.

## Model Summary

Euclid-1.0 translates complete statistical and data-processing programs across all six directed pairs over {SAS, Python, R}. The training objective is **behavioural equivalence**: given identical inputs, the translated program must compute identical values and bind them to identically-named results.

| | |
|---|---|
| Developer | ProCogia |
| Method | LoRA SFT (r=64), response-only supervision, adapter merged to 16-bit |
| Parameters | 31.58B total / 45.4M trained (0.14%) |
| Layers | 52 — 23 Mamba-2 mixer, 23 MoE, 6 attention (indices 5, 12, 19, 26, 33, 42) |
| `hidden_size` | 2688 |
| Attention | 32 Q heads / 2 KV heads (GQA), `head_dim` 128 |
| Mamba-2 | 64 heads × 64 head_dim, `n_groups` 8, `ssm_state_size` 128, `conv_kernel` 4 |
| MoE | 128 routed experts, top-6, plus 1 shared expert; `relu2` ungated |
| Vocab | 131,072, untied embeddings |
| Context | 262,144 architectural; trained at 8,192 |
| Precision | BF16, ~60 GiB across 14 shards |
| License | See `LICENSE` in this repository |

**Reasoning is disabled by design.** Every training row was rendered with `<think></think>` already closed, so labels begin exactly where generation begins under `enable_thinking=False`. Serving with `enable_thinking=True` under the training system prompt produces the trained reflex — an immediate `</think>` — or degenerate filler, which under greedy decoding can loop until the context window fills. This is the specification working as written. Reasoning-before-translating is a data change, not a serving flag.

## Intended Use

- Single-turn translation of complete programs across the six directed pairs over {SAS, Python, R}.
- Deterministic serving under the verbatim inference contract below: greedy decoding, `enable_thinking=False`, exact system prompt and user format.
- Output contract: the translated program only, no prose, no markdown fences, result names preserved exactly.

## How to Use

### Loader class

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ProCogia/Euclid-1.0",
    torch_dtype=torch.bfloat16,
    device_map={"": 0},
)
tokenizer = AutoTokenizer.from_pretrained("ProCogia/Euclid-1.0")
```

`mamba_ssm` and `causal_conv1d` must be installed. Without them the 23 Mamba layers fall back to `torch_forward`, which materializes a single 32 GiB contraction tensor at 8,192 tokens and OOMs regardless of batch size. No prebuilt wheels exist for this stack; the kernels compile from source in ~25 minutes and should be cached.

The saved tokenizer carries a `pad_token` substitution (`<SPECIAL_999>`). The original `pad_token` collided with EOS, which trains padded positions as real EOS.

### vLLM

```bash
vllm serve ProCogia/Euclid-1.0 \
    --max-model-len 8192 \
    --mamba-ssm-cache-dtype float16
```

An R→SAS probe through the vLLM API returned byte-identical output to the same probe through transformers, confirming merge fidelity and deterministic greedy decoding across serving stacks.

Context can be raised well beyond the trained 8,192 at negligible memory cost — only 6 of 52 layers are attention, so KV cache is ~6 KB/token and Mamba state is length-independent. Quality beyond 8,192 total tokens is outside the trained regime and unvalidated.

Note that `prompt_tokens + max_tokens ≤ max_model_len` is enforced per request before generation; `max_tokens` is a hard reservation, not a preference.

### Inference contract

Non-negotiable — the model is conditioned on exactly this:

| Setting | Value |
|---|---|
| System prompt | `system_prompt.txt`, verbatim (1,394 chars) |
| User message | `Translate the following {SAS\|Python\|R} program to {SAS\|Python\|R}.\n\n{program}` |
| Chat template | `enable_thinking=False` |
| Decoding | greedy (`temperature=0`) |
| Stop tokens | model EOS **and** `<\|im_end\|>` (id 11) |
| `max_tokens` | ≥ 6000 for full programs, or omit to auto-size |
| Output | bare program — no fences, no prose |

### Prompt format

The model was trained against a single system prompt across every row, and a drift guard asserted the notebook prompt matched the dataset prompt at eval time. Deviating from its contract is off-distribution. The verbatim prompt ships as `system_prompt.txt`; the block below reproduces its contract:

```text
You are a code translation engine for statistical and data-processing programs written in SAS, Python, and R.

You are given one complete program in a source language and produce the equivalent program in the target language. Behavioural equivalence is the only criterion: given the same inputs, your program must compute the same values and place them in results carrying the same names.

- Preserve the source program's structure, step order, and intent. Carry its comments across as comments in the target language, and add a brief comment where the target expresses a source construct non-obviously.
- Name every result exactly as the source names it, so results can be compared name for name.
- SAS semantics decide the answer even when SAS is not the target language. A SAS date is whole days since 1960-01-01 and a datetime is seconds since 1960-01-01, both stored as plain numbers. A SAS FORMAT changes only how a value is displayed, never what is stored. Missing (.) sorts below every number, so `x < 5` is true when x is missing. Character values are blank-padded to a declared length and compare ignoring trailing blanks.
- A step that only prints or plots produces no data and has no translation outside SAS; leave it out rather than inventing an equivalent.
- Respond with the translated program and nothing else: no prose, no explanation, no markdown code fences, no placeholders.
```

The user turn is a single line followed by the bare source program:

```
Translate the following {SOURCE} program to {TARGET}.

{program}
```

The source program is **not** fenced. The model emits a bare program, likewise unfenced.

## Training Data

**Proprietary. Not released.** Composition is disclosed below for reproducibility of method, not of data.

| | |
|---|---|
| Training rows | **12,893** after filtering (from 12,964) |
| In-training eval subset | 120 (stratified, 20 per direction) |
| Held-out eval set | 544 |
| Format | JSONL, single-turn `system` + `user` + `assistant` |
| Length (tokens, rendered template) | p50 1,369 / p95 4,754 / p99 7,142 / max 43,654 |

## Training Procedure

### Adaptable surface

93% of this model was frozen by necessity rather than by choice, and the reasons are architectural.

**Routed experts are unreachable.** They account for 29.375B parameters (93.0%) and are stored as fused 3-D `nn.Parameter` tensors. Attaching LoRA to them routes every expert forward through PEFT's `ParamWrapper.get_delta_weight`, which materializes the full delta per fused tensor — `128 × 2688 × 1856` in fp32, roughly 2.3 GiB per MoE layer per forward, across 23 layers. This OOMs regardless of sequence length or batch size. An `exclude_modules` regex cannot prevent it, because the fused experts are parameters rather than modules and there is nothing to exclude; dotted target names are what keep suffix-matching confined to the shared expert's `nn.Linear`.

**Quantization is unavailable.** bitsandbytes quantizes `nn.Linear` only, so `load_in_4bit=True` against fused expert parameters either errors or silently leaves 93% of the model in bf16 while appearing to be QLoRA. This sets a hard 80 GB floor with no memory escape hatch.

**Only `in_proj` is adaptable on the Mamba side.** PEFT's `_check_lora_target_modules_mamba` raises on `out_proj` and `conv1d` as a guard against selective-scan gradient instability. The cost is negligible — about 4M parameters at r=64.

**Packing is forbidden.** It is incompatible with response-only masking, and more fundamentally this is a hybrid Mamba model: the SSM recurrence carries state along the sequence, so packed documents leak into each other. Attention can be masked; a recurrence cannot.

Capacity was therefore redirected to the dense pathway.

### LoRA configuration

| Parameter | Value |
|---|---|
| Rank `r` | 64 |
| `alpha` | 128 |
| Dropout | 0 |
| Bias | none |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `in_proj`, `shared_experts.up_proj`, `shared_experts.down_proj` |
| Scope | Attention (6 layers), Mamba input projection (23 layers), shared expert (23 MoE layers) |
| Routed experts, router `.gate`, MTP head | Never targeted; enforced by three runtime assertions |
| Trainable | 45.4M (0.14%) |

The shared expert is the highest-value target available: it is dense and runs on every token of all 23 MoE layers, unlike routed experts which each see roughly 6 of every 128 tokens. `r = 64` compensates for the small module count.

### Optimization

| Parameter | Value |
|---|---|
| Learning rate | 2e-4, cosine |
| Warmup | 48 steps (3%) |
| Optimizer | `adamw_8bit`, wd 0.01 |
| Gradient clipping | 0.5 |
| Micro-batch × accumulation | 1 × 16 (effective batch 16) |
| Sequence length | 8,192, packing off |
| Loss | Response-only |
| Precision | BF16, `sdpa` attention, unsloth gradient checkpointing |
| Epochs | 2, eval and save every 100 steps |
| Seed | 3407 |
| Total steps | 1,611 |

Response-only supervision masks the system prompt, instruction, and source program to `-100`; measured supervised fraction is mean 0.344, min 0.083. The response marker includes `<think></think>`, so training labels begin exactly where generation begins at serve time. Masking at the shorter `assistant\n` marker would train the model to emit a second `<think></think>` that the inference prompt already supplies. Assertions guard both failure directions — all-masked labels produce a permanent loss of 0.0, and unmasked labels train on prompts.

Gradient norms stayed well under the 0.5 clip (0.20–0.30 at full learning rate) and the clip monitor never fired, retroactively validating the 2e-4 rate.

### Infrastructure

| | |
|---|---|
| GPU | 1× H100 80GB SXM5, single GPU, no sharding |
| Framework | unsloth 2026.8.18, torch 2.11.0+cu128, transformers 5.5.4, trl 1.10.0 |
| Kernels | `mamba_ssm` 2.3.2.post1, `causal_conv1d` 1.6.2.post1, compiled from source |
| Loaded footprint | 58.8 GiB bf16 |
| Peak VRAM | 65.1 GiB steady across the full run, no leak |
| Throughput | ~10.4 s/step average |
| Wall clock | ~10 h |

The torch `+cu128` build is load-bearing: the container toolkit is CUDA 12.8, and torch refuses to compile extensions across a CUDA major-version gap, which makes the Mamba kernels unbuildable on the `+cu130` default.

## Evaluation

**Pending.**
The planned protocol:

| Element | Specification |
|---|---|
| Set | 544 examples, repo-disjoint, per-direction |
| Method | Execute source and translation on identical inputs; diff results name-for-name |
| Metric | pass@1, pooled and per-direction |
| Prompt | Training system prompt, greedy decoding, `enable_thinking=False` |

Any comparison against an untuned baseline must run each model under its own contract. Measuring this model with reasoning forced on and `temperature=0.6` measures nothing about training quality.

Spot-checks surfaced two error classes that token-level loss cannot see: invented I/O (a `proc means data=sales` source with an unbound dataset reference produced a Python translation that added `pd.read_csv("sales.csv")`, dropped the `n` statistic, and added a `print`), and statistical-option semantics (an R `aggregate(...)` → SAS translation added `NOMISS` to `PROC MEANS`, which affects class-variable missing handling rather than the statistic).

## Hardware Requirements

| | |
|---|---|
| Weights on disk | ~60 GiB (BF16, 14 shards) |
| Loaded footprint | 58.8 GiB |
| KV cache | ~6 KB/token (only 6 of 52 layers are attention) |
| Mamba state | Length-independent |
| Practical single-GPU floor | 80 GB (H100 / H200 / A100 80GB) |
| Quantization | Unavailable — see below |

4-bit and 8-bit loading do not work on this architecture. bitsandbytes quantizes `nn.Linear` only, while the routed experts holding 93% of the parameters are fused 3-D `nn.Parameter` tensors. The 80 GB floor is hard.

For reference, training peaked at 65.1 GiB on a single H100 80GB at batch size 1, sequence length 8,192, with gradient checkpointing on and packing off.

## Citation

```bibtex
@misc{euclid_1_0,
  title  = {Euclid-1.0: Bidirectional SAS/R/Python Program Translation},
  author = {ProCogia},
  year   = {2026},
  url    = {https://huggingface.co/ProCogia/Euclid-1.0}
}
```