README.md

---
license: cc-by-nc-4.0
base_model: Qwen/Qwen2.5-Coder-7B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
tags:
  - code
  - cobol
  - legacy
  - mainframe
  - gguf
  - lora
language:
  - en
---

# FL-7B-3: COBOL Code Generation

**A 7B model that writes COBOL that actually compiles.**

Ask any general coding model for COBOL and you get confident nonsense. The base model here,
[Qwen/Qwen2.5-Coder-7B](https://huggingface.co/Qwen/Qwen2.5-Coder-7B), scores a **flat zero**
on COBOLEval. Not "low", zero. Only 3.65% of its test programs compile at all.

FL-7B-3 is a supervised finetune of that same base on a curated COBOL instruction corpus.
It solves **15.75% of COBOLEval**, matching published GPT-4 results with a model roughly
two orders of magnitude smaller, and more than doubling GPT-4's compilation rate.

## Benchmarks

COBOLEval ([zorse-project/COBOLEval](https://github.com/zorse-project/COBOLEval)), 146 problems /
821 test cases. Every sample is **compiled and executed** with GnuCOBOL 3.2.0. No self-reported
or LLM-judged scores. Greedy decoding, `repetition_penalty=1.0`, `max_new_tokens=1536`.

| Model | pass@1 | Compile rate | Tests passed |
|---|---|---|---|
| Qwen2.5-Coder-7B (base) | 0.00% | 3.65% | 4 / 821 |
| **FL-7B-3** | **15.75%** | **51.16%** | 204 / 821 |

Published reference numbers on the same benchmark:

| Model | pass@1 | Compile rate |
|---|---|---|
| GPT-4 | 15.75% | 24.12% |
| GPT-4o | 16.40% | 41.80% |
| **FL-7B-3 (7B)** | **15.75%** | **51.16%** |

The headline: **COBOL goes from unusable to useful.** Compilation rate is where the gap is
widest: FL-7B-3 produces syntactically valid COBOL roughly twice as often as GPT-4 and
25% more often than GPT-4o.

## ⚠️ Required inference setting

**Set `repetition_penalty` to exactly `1.0`.** This is not a stylistic preference. It costs
you real accuracy:

| repetition_penalty | pass@1 |
|---|---|
| **1.0** | **15.75%** |
| 1.05 (common default) | 13.01% |
| 1.15 | 2.74% |

COBOL mandates repetition. `PROGRAM-ID` must match `END PROGRAM` character for character,
data names recur constantly, division headers are fixed boilerplate. Any repetition penalty
pushes the model away from re-emitting tokens the language *requires* it to re-emit, and the
program stops compiling. At 1.15 the model produces identifiers like `TESTTRUNCATENUMBER`
where `TRUNCATE-NUMBER` was required.

## Usage

### Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("FLs-AI/FL-7B-3-safetensors")
model = AutoModelForCausalLM.from_pretrained(
    "FLs-AI/FL-7B-3-safetensors", device_map="auto", torch_dtype="bfloat16")

messages = [{"role": "user", "content": "Write a COBOL program that ..."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

out = model.generate(
    **tok(prompt, return_tensors="pt").to(model.device),
    max_new_tokens=1536,
    do_sample=False,
    repetition_penalty=1.0,   # <- required, see above
)
print(tok.decode(out[0], skip_special_tokens=True))
```

### GGUF / llama.cpp

```bash
llama-cli -hf FLs-AI/FL-7B-3:Q4_K_M --repeat-penalty 1.0 -n 1536
```

| Quant | Size | Notes |
|---|---|---|
| `Q8_0` | ~8 GB | Effectively lossless |
| `Q6_K` | ~6 GB | Recommended if you have the RAM |
| `Q4_K_M` | ~4.5 GB | **Recommended default**, realistic quality floor |
| `Q2_K` | ~3 GB | Experimental. 2-bit on a 7B degrades badly; not benchmarked |

Quantized variants were **not** re-benchmarked. The reported 15.75% is bf16.

## Training

| | |
|---|---|
| Method | LoRA SFT, assistant-only loss masking |
| Base | `Qwen/Qwen2.5-Coder-7B` @ `0396a761` |
| LoRA | r=32, α=64, dropout=0.0, 196 modules (q/k/v/o + gate/up/down_proj) |
| Trainable | 80.7M / 7.70B (1.05%) |
| Precision | bf16, gradient checkpointing |
| Data | 20,332 instruction pairs → 21.2M tokens, packed into 2,586 × 8192-token blocks |
| Schedule | 2 epochs, 324 steps, 131,072 tokens/step |
| Optimizer | adamw_8bit, LR 1e-4 cosine → 0, warmup 3%, wd 0.01, clip 1.0 |
| Hardware | 1× H100 80GB SXM, 2h03m |
| Eval loss | 0.557 → 0.4183 |

**On epoch count:** epoch 1 moved eval loss by −0.126, epoch 2 by only −0.013. One epoch
captures the overwhelming majority of the gain on a corpus this size. Validation loss decreased
monotonically across all 64 evaluations with no overfitting inflection.


## Limitations

**Honest failure analysis.** 65 of 146 COBOLEval solutions fail to compile standalone. The
errors are a long tail, not one systematic bug:

- **Degenerate generation on hard problems.** The model can enter repetition loops, appending
  `-TEMP` to an identifier until it exceeds COBOL's 63-character limit, or enumerating
  `IF WS-NUMBER = 58 ... IF WS-NUMBER = 59 ...` instead of writing a loop. Correct solutions
  are consistently short (median 1,368 characters, max 3,248); output much longer than that is
  a strong signal the generation has derailed.
- **Enumeration over algorithm.** On problems requiring real logic (primality, parsing), it
  sometimes hardcodes cases rather than implementing the algorithm.
- **Occasional undefined identifiers**, e.g. using `I` or a linkage record without declaring it.
- **Structural breakage** on long outputs: missing `PROCEDURE DIVISION` header, unbalanced
  parentheses, mismatched `END PROGRAM`.

**Scope:** trained on COBOL only. General coding ability inherited from the base was not
re-measured and may have regressed. Not evaluated on real mainframe dialects (IBM Enterprise
COBOL, CICS, JCL, DB2 embedded SQL). GnuCOBOL 3.2 only.

**Do not deploy generated COBOL to production systems without review.** A 15.75% pass@1 means
roughly five out of six generated programs are wrong.

## License

The finetune is released under **CC BY-NC 4.0** (non-commercial, attribution required).

The base model, `Qwen/Qwen2.5-Coder-7B`, is Apache-2.0 and remains so. The non-commercial
restriction applies to this finetune's contribution, and cannot and does not revoke any rights
you already hold in the base weights under Apache-2.0. If your use case is commercial, you can
still use the base model freely; you just cannot use these weights.

> 💡 **Note on FL-3.1:** This model version (FL-7B-3) is designed specifically for COBOL code generation. We are actively training **FL-3.1**, which expands the architecture into a full-fledged **Mainframe Assistant** (adding JCL, CICS, DB2, and general mainframe architectural guidance).
## Citation

```bibtex
@misc{fl7b3,
  title  = {FL-7B-3: COBOL Code Generation},
  author = {FLs-AI},
  year   = {2026},
  url    = {https://huggingface.co/FLs-AI/FL-7B-3}
}
```