---
license: cc-by-nc-4.0
base_model: JetBrains/Mellum2-12B-A2.5B-Thinking
tags:
  - cobol
  - mainframe
  - gnucobol
  - code-generation
  - moe
  - gguf
language:
  - en
pipeline_tag: text-generation
---

# FL-12B-MoE-2

A COBOL-specialised Mixture-of-Experts model, adapted from
[JetBrains/Mellum2-12B-A2.5B-Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking).

12B total parameters, 2.5B active per token. Built for writing and repairing
fixed-format COBOL that compiles under GnuCOBOL 3.2.

## Results

Evaluated on [COBOLEval](https://github.com/BloopAI/COBOLEval): 146 problems
translated from HumanEval into COBOL, 821 test cases, scored by compiling and
executing every generated program with GnuCOBOL 3.2. A solution counts only if
it passes every test for its problem.

| | pass@1 | compile rate | test pass rate |
|---|---:|---:|---:|
| Base model (Mellum2-12B-A2.5B-Thinking) | 0.68% | 12.18% | 3.05% |
| **FL-12B-MoE-2** | **13.70%** | **48.36%** | **23.51%** |
| | 20x | 4.0x | 7.7x |

Specialisation takes the base model from 1 solved problem out of 146 to 20, and
from 100 compiling test programs to 397.

For reference, the COBOLEval authors report 10.27% pass@1 and 47.94% compile
rate for GPT-4 on the same benchmark.

All figures use identical decoding: greedy (temperature 0), one sample per
problem, 2048 maximum new tokens, 8192 context, chat prompt format. The same
column-normalisation post-processing is applied to every model measured,
including the base.

## What the model is good at

**Writing fixed-format COBOL that compiles.** Column discipline is respected:
sequence area in columns 1 to 6, indicator in column 7, code from column 8.

**Callable subprograms.** Recognises a `LINKAGE SECTION` with
`PROCEDURE DIVISION USING` and returns control with `GOBACK` rather than
`STOP RUN`, which would terminate the entire run unit and leave the caller
without control.

**Reading existing COBOL.** Record layouts, byte counts, control flow, and
"what does this program do" questions, from roughly 24M tokens of COBOL
comprehension data in the training mix.

**Repairing compiler errors.** Trained on paired defect and fix examples
covering column violations, missing `FUNCTION` keywords, `DIVIDE` syntax,
`INSPECT` operands, invented intrinsics, figurative constants and subprogram
termination.

## Intended use

Assisting developers working with COBOL: drafting subprograms, completing data
divisions, diagnosing compiler errors, and explaining legacy code. Generated
code should be compiled, reviewed and tested before use. Not suitable for
unsupervised modification of production mainframe systems.

## Training

Adapted with LoRA on the attention projections only.

| | |
|---|---|
| Base | JetBrains/Mellum2-12B-A2.5B-Thinking |
| Architecture | MoE, 64 experts, 8 active per token, 28 layers, d_model 2304 |
| Attention | GQA 32Q/4KV, sliding window 1024, 131,072 context |
| Adapter | LoRA r=32, alpha 64, dropout 0, on `q/k/v/o_proj` (112 modules) |
| Trainable | 16,515,072 parameters, 0.136% of the model |
| Never adapted | MoE router, expert MLPs, `lm_head`, embeddings |
| Precision | bfloat16, no quantisation during training |
| Sequence | 8192 with packing, loss on assistant turns only |
| Optimiser | AdamW 8-bit, cosine schedule, 3% warmup, weight decay 0.01 |
| Total tokens | 187M |
| Total steps | 1,428 |
| Hardware | 1x RTX 5090 (32 GB), roughly 22 GPU-hours |

The MoE router is deliberately left untouched. Adapting it changes which
experts fire and degrades the model in ways that surface as diffuse quality
loss rather than as an error.

### Data

The bulk of the corpus is COBOL, around 70% of tokens, with the remainder
covering JCL, C#, x86 assembly and step-by-step reasoning traces. Tasks include
source continuation, full subprogram generation, and instruction data on
reading and analysing existing COBOL.

On top of that sit small, targeted sets built by mining the same corpus for
constructs the model was getting wrong: subprogram termination, column
discipline, the `FUNCTION` keyword, `DIVIDE ... GIVING`, `INSPECT` operands,
reference modification `X(pos:len)` in place of a non-existent `SUBSTRING`, and
figurative constants. Repair examples were produced by injecting a known defect
into working code, so the reference answer is the original source rather than a
generated one.

The list of valid intrinsic functions was derived by counting `FUNCTION`
occurrences across 4,293 real programs rather than written from memory.

## Usage

GGUF quantisations are provided for llama.cpp and compatible runtimes.

| Quantisation | Notes |
|---|---|
| `Q8_0` | Highest fidelity |
| `Q6_K` | Close to Q8_0, smaller |
| `Q4_K_M` | Recommended balance |
| `Q2_K` | Smallest, expect noticeable quality loss |

For a Mixture-of-Experts model with 2.5B active parameters, quantisation error
concentrates in a small number of active experts, so `Q2_K` degrades more
sharply than it would on a dense model of similar size. `Q4_K_M` is the
practical floor.

```bash
llama-cli -hf DarkKnighToS223/FL-12B-MoE-2:Q4_K_M
```

```bash
llama-server -hf DarkKnighToS223/FL-12B-MoE-2:Q4_K_M --port 8080
```

Recommended sampling: temperature 0 for code generation, or 0.3 with top_p 0.9
when you want variety.

## License and attribution

Released under CC-BY-NC-4.0.

Derived from `JetBrains/Mellum2-12B-A2.5B-Thinking`, licensed under Apache 2.0.
The Apache 2.0 notice and attribution for the base model are retained. The
non-commercial condition applies to this derivative work and to the adaptation
contributed here. If you intend to use this model commercially, review the base
model's terms and seek your own legal advice.

Training data was drawn from public COBOL and mainframe source corpora.

## Citation

```bibtex
@misc{fl12bmoe2,
  title  = {FL-12B-MoE-2: a COBOL-specialised Mixture-of-Experts model},
  author = {DarkKnighToS223},
  year   = {2026},
  url    = {https://huggingface.co/DarkKnighToS223/FL-12B-MoE-2}
}
```