| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-9B-Base |
| tags: |
| - cobol |
| - mainframe |
| - code |
| - legacy-modernization |
| - qwen3_5 |
| - lora |
| language: |
| - en |
| pipeline_tag: text-generation |
| --- |
| |
| # FL-9B-4 |
|
|
| FL-9B-4 is a COBOL / mainframe code model fine-tuned from **Qwen/Qwen3.5-9B-Base** |
| via supervised fine-tuning (SFT) on a curated COBOL instruction dataset. It targets |
| legacy-code understanding, COBOL generation, and COBOL-to-Java translation. |
|
|
| - **Base model:** Qwen/Qwen3.5-9B-Base (dense 9B, hybrid linear + full attention) |
| - **Method:** LoRA SFT (assistant-only masking), ~3 epochs, bf16 |
| - **Domain:** COBOL, GnuCOBOL, mainframe knowledge, COBOL and Java |
|
|
| ## Benchmark results |
|
|
| All code benchmarks compile and execute generated programs against reference |
| tests. Evaluated greedy (temperature 0), single sample per task, via vLLM. |
| "Base" = Qwen/Qwen3.5-9B-Base (no fine-tuning), evaluated with the same harness |
| and an injected ChatML template, so the delta reflects the SFT alone. |
|
|
| | Benchmark | Metric | Base | **FL-9B-4** | |
| |---|---|---|---| |
| | **COBOLEval** | pass@1 | 0.68% | **36.99%** | |
| | | compile rate | 8.65% | 82.10% | |
| | | test pass rate | 1.46% | 52.98% | |
| | **COBOL-JavaTrans (C2J)** | pass@1 | 42.66% | **80.42%** | |
| | | compile success rate (CSR) | 46.85% | 96.50% | |
| | **MainframeBench** | MCQ accuracy | 66.23% | **71.26%** | |
| | | QA - Token F1 | 11.68% | 12.75% | |
| | | QA - ROUGE-L | 9.33% | 10.29% | |
| | | Summarization - Token F1 | 23.62% | 27.64% | |
| | | Summarization - ROUGE-L | 16.38% | 20.25% | |
| | **CobolCodeBench** | INSTRUCT compile rate | 2.17% | **47.83%** | |
| | | COMPLETE compile rate | 0.00% | 32.61% | |
|
|
| The fine-tuning produces very large gains on COBOL generation and understanding: |
| COBOLEval pass@1 rises from ~1% to 37%, COBOL compile rate from 9% to 82%, and |
| CobolCodeBench COMPLETE from 0% to 33%. COBOL-to-Java translation nearly doubles in |
| pass@1 (from 43% to 80%). MainframeBench MCQ moves less (from 66% to 71%), since factual |
| mainframe knowledge is largely already present in the base model. |
|
|
| ### Notes on evaluation |
|
|
| The MainframeBench MCQ, CobolCodeBench INSTRUCT and COMPLETE numbers were produced |
| after fixing harness-side generation limits (the default 16-token MCQ budget and |
| 2048-token code budget truncated answers, and single-format `cobc` invocation |
| rejected valid programs written in a different column format). Fixed evaluation |
| uses a larger generation budget and tries `variable`, `free` and `fixed` COBOL |
| formats when compiling. Reported numbers reflect the model's actual capability, |
| not the truncated defaults. |
|
|
| The strongest results - COBOL-to-Java translation (80% pass@1) and COBOLEval |
| (82% compile) - show the model reliably produces valid, working COBOL and |
| translates legacy code into working Java. |
|
|
| ## Intended use |
|
|
| - Translating legacy COBOL programs to Java |
| - Completing and generating GnuCOBOL programs |
| - Answering mainframe / COBOL knowledge questions |
| - Assisting with legacy-code modernization workflows |
|
|
| ## Limitations |
|
|
| - Open-ended QA and summarization scores (Token F1 / ROUGE-L) are modest; the |
| model is stronger at code generation and translation than at free-form prose. |
| - COBOL generation quality varies with column-format conventions; generated code |
| may mix fixed and free formats. |
| - Not evaluated for safety-critical or production mainframe deployment without |
| human review. |
|
|
| ## Training |
|
|
| | Setting | Value | |
| |---|---| |
| | Base | Qwen/Qwen3.5-9B-Base | |
| | Method | LoRA (r=32, alpha=64), assistant-only SFT | |
| | Precision | bf16 | |
| | Epochs | ~3 | |
| | Sequence length | 8192 (packed) | |
| | Hardware | 1x NVIDIA RTX PRO 6000 Blackwell (96 GB) | |
| | Frameworks | Unsloth + Transformers | |
|
|
| LoRA adapters were applied to attention projections, MLP projections, and the |
| linear-attention (`in_proj_*`/`out_proj`) modules of the hybrid Qwen3.5 |
| architecture; the vision tower, MTP head, and router/embedding/LM-head were |
| excluded. |
|
|
| ## How to use |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "FLs-AI/FL-9B-4" # adjust to your repo |
| tok = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto") |
| |
| messages = [{"role": "user", "content": "Translate this COBOL program to Java:\n\n<COBOL here>"}] |
| inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device) |
| out = model.generate(inputs, max_new_tokens=2048, temperature=0.0) |
| print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True)) |
| ``` |
