Instructions to use FLs-AI/FL-7B-3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FLs-AI/FL-7B-3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="FLs-AI/FL-7B-3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("FLs-AI/FL-7B-3")
model = AutoModelForCausalLM.from_pretrained("FLs-AI/FL-7B-3", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use FLs-AI/FL-7B-3 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FLs-AI/FL-7B-3:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf FLs-AI/FL-7B-3:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FLs-AI/FL-7B-3:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf FLs-AI/FL-7B-3:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FLs-AI/FL-7B-3:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf FLs-AI/FL-7B-3:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FLs-AI/FL-7B-3:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FLs-AI/FL-7B-3:Q4_K_M

Use Docker

docker model run hf.co/FLs-AI/FL-7B-3:Q4_K_M

LM Studio
Jan

vLLM

How to use FLs-AI/FL-7B-3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FLs-AI/FL-7B-3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FLs-AI/FL-7B-3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FLs-AI/FL-7B-3:Q4_K_M

SGLang

How to use FLs-AI/FL-7B-3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FLs-AI/FL-7B-3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FLs-AI/FL-7B-3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FLs-AI/FL-7B-3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FLs-AI/FL-7B-3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use FLs-AI/FL-7B-3 with Ollama:
```
ollama run hf.co/FLs-AI/FL-7B-3:Q4_K_M
```

Unsloth Studio

How to use FLs-AI/FL-7B-3 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FLs-AI/FL-7B-3 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FLs-AI/FL-7B-3 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for FLs-AI/FL-7B-3 to start chatting

How to use FLs-AI/FL-7B-3 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FLs-AI/FL-7B-3:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FLs-AI/FL-7B-3:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use FLs-AI/FL-7B-3 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FLs-AI/FL-7B-3:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "FLs-AI/FL-7B-3:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use FLs-AI/FL-7B-3 with Docker Model Runner:
```
docker model run hf.co/FLs-AI/FL-7B-3:Q4_K_M
```

Lemonade

How to use FLs-AI/FL-7B-3 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FLs-AI/FL-7B-3:Q4_K_M

Run and chat with the model

lemonade run user.FL-7B-3-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use FLs-AI/FL-7B-3 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FLs-AI/FL-7B-3:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FLs-AI/FL-7B-3:Q4_K_M

Run Hermes

hermes

Atomic Chat

FL-7B-3 / README.md

FLs-AI

Update README.md

edc3fb7 verified 6 days ago

preview code

Raw

History Blame Contribute Delete

6.7 kB

	---
	license: cc-by-nc-4.0
	base_model: Qwen/Qwen2.5-Coder-7B
	base_model_relation: finetune
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- code
	- cobol
	- legacy
	- mainframe
	- gguf
	- lora
	language:
	- en
	---

	# FL-7B-3: COBOL Code Generation

	A 7B model that writes COBOL that actually compiles.

	Ask any general coding model for COBOL and you get confident nonsense. The base model here,
	[Qwen/Qwen2.5-Coder-7B](https://huggingface.co/Qwen/Qwen2.5-Coder-7B), scores a flat zero
	on COBOLEval. Not "low", zero. Only 3.65% of its test programs compile at all.

	FL-7B-3 is a supervised finetune of that same base on a curated COBOL instruction corpus.
	It solves 15.75% of COBOLEval, matching published GPT-4 results with a model roughly
	two orders of magnitude smaller, and more than doubling GPT-4's compilation rate.

	## Benchmarks

	COBOLEval ([zorse-project/COBOLEval](https://github.com/zorse-project/COBOLEval)), 146 problems /
	821 test cases. Every sample is compiled and executed with GnuCOBOL 3.2.0. No self-reported
	or LLM-judged scores. Greedy decoding, `repetition_penalty=1.0`, `max_new_tokens=1536`.

	\| Model \| pass@1 \| Compile rate \| Tests passed \|
	\|---\|---\|---\|---\|
	\| Qwen2.5-Coder-7B (base) \| 0.00% \| 3.65% \| 4 / 821 \|
	\| FL-7B-3 \| 15.75% \| 51.16% \| 204 / 821 \|

	Published reference numbers on the same benchmark:

	\| Model \| pass@1 \| Compile rate \|
	\|---\|---\|---\|
	\| GPT-4 \| 15.75% \| 24.12% \|
	\| GPT-4o \| 16.40% \| 41.80% \|
	\| FL-7B-3 (7B) \| 15.75% \| 51.16% \|

	The headline: COBOL goes from unusable to useful. Compilation rate is where the gap is
	widest: FL-7B-3 produces syntactically valid COBOL roughly twice as often as GPT-4 and
	25% more often than GPT-4o.

	## ⚠️ Required inference setting

	Set `repetition_penalty` to exactly `1.0`. This is not a stylistic preference. It costs
	you real accuracy:

	\| repetition_penalty \| pass@1 \|
	\|---\|---\|
	\| 1.0 \| 15.75% \|
	\| 1.05 (common default) \| 13.01% \|
	\| 1.15 \| 2.74% \|

	COBOL mandates repetition. `PROGRAM-ID` must match `END PROGRAM` character for character,
	data names recur constantly, division headers are fixed boilerplate. Any repetition penalty
	pushes the model away from re-emitting tokens the language requires it to re-emit, and the
	program stops compiling. At 1.15 the model produces identifiers like `TESTTRUNCATENUMBER`
	where `TRUNCATE-NUMBER` was required.

	## Usage

	### Transformers

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	tok = AutoTokenizer.from_pretrained("FLs-AI/FL-7B-3-safetensors")
	model = AutoModelForCausalLM.from_pretrained(
	"FLs-AI/FL-7B-3-safetensors", device_map="auto", torch_dtype="bfloat16")

	messages = [{"role": "user", "content": "Write a COBOL program that ..."}]
	prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

	out = model.generate(
	**tok(prompt, return_tensors="pt").to(model.device),
	max_new_tokens=1536,
	do_sample=False,
	repetition_penalty=1.0, # <- required, see above
	)
	print(tok.decode(out[0], skip_special_tokens=True))
	```

	### GGUF / llama.cpp

	```bash
	llama-cli -hf FLs-AI/FL-7B-3:Q4_K_M --repeat-penalty 1.0 -n 1536
	```

	\| Quant \| Size \| Notes \|
	\|---\|---\|---\|
	\| `Q8_0` \| ~8 GB \| Effectively lossless \|
	\| `Q6_K` \| ~6 GB \| Recommended if you have the RAM \|
	\| `Q4_K_M` \| ~4.5 GB \| Recommended default, realistic quality floor \|
	\| `Q2_K` \| ~3 GB \| Experimental. 2-bit on a 7B degrades badly; not benchmarked \|

	Quantized variants were not re-benchmarked. The reported 15.75% is bf16.

	## Training

	\| \| \|
	\|---\|---\|
	\| Method \| LoRA SFT, assistant-only loss masking \|
	\| Base \| `Qwen/Qwen2.5-Coder-7B` @ `0396a761` \|
	\| LoRA \| r=32, α=64, dropout=0.0, 196 modules (q/k/v/o + gate/up/down_proj) \|
	\| Trainable \| 80.7M / 7.70B (1.05%) \|
	\| Precision \| bf16, gradient checkpointing \|
	\| Data \| 20,332 instruction pairs → 21.2M tokens, packed into 2,586 × 8192-token blocks \|
	\| Schedule \| 2 epochs, 324 steps, 131,072 tokens/step \|
	\| Optimizer \| adamw_8bit, LR 1e-4 cosine → 0, warmup 3%, wd 0.01, clip 1.0 \|
	\| Hardware \| 1× H100 80GB SXM, 2h03m \|
	\| Eval loss \| 0.557 → 0.4183 \|

	On epoch count: epoch 1 moved eval loss by −0.126, epoch 2 by only −0.013. One epoch
	captures the overwhelming majority of the gain on a corpus this size. Validation loss decreased
	monotonically across all 64 evaluations with no overfitting inflection.


	## Limitations

	Honest failure analysis. 65 of 146 COBOLEval solutions fail to compile standalone. The
	errors are a long tail, not one systematic bug:

	- Degenerate generation on hard problems. The model can enter repetition loops, appending
	`-TEMP` to an identifier until it exceeds COBOL's 63-character limit, or enumerating
	`IF WS-NUMBER = 58 ... IF WS-NUMBER = 59 ...` instead of writing a loop. Correct solutions
	are consistently short (median 1,368 characters, max 3,248); output much longer than that is
	a strong signal the generation has derailed.
	- Enumeration over algorithm. On problems requiring real logic (primality, parsing), it
	sometimes hardcodes cases rather than implementing the algorithm.
	- Occasional undefined identifiers, e.g. using `I` or a linkage record without declaring it.
	- Structural breakage on long outputs: missing `PROCEDURE DIVISION` header, unbalanced
	parentheses, mismatched `END PROGRAM`.

	Scope: trained on COBOL only. General coding ability inherited from the base was not
	re-measured and may have regressed. Not evaluated on real mainframe dialects (IBM Enterprise
	COBOL, CICS, JCL, DB2 embedded SQL). GnuCOBOL 3.2 only.

	Do not deploy generated COBOL to production systems without review. A 15.75% pass@1 means
	roughly five out of six generated programs are wrong.

	## License

	The finetune is released under CC BY-NC 4.0 (non-commercial, attribution required).

	The base model, `Qwen/Qwen2.5-Coder-7B`, is Apache-2.0 and remains so. The non-commercial
	restriction applies to this finetune's contribution, and cannot and does not revoke any rights
	you already hold in the base weights under Apache-2.0. If your use case is commercial, you can
	still use the base model freely; you just cannot use these weights.

	> 💡 Note on FL-3.1: This model version (FL-7B-3) is designed specifically for COBOL code generation. We are actively training FL-3.1, which expands the architecture into a full-fledged Mainframe Assistant (adding JCL, CICS, DB2, and general mainframe architectural guidance).
	## Citation

	```bibtex
	@misc{fl7b3,
	title = {FL-7B-3: COBOL Code Generation},
	author = {FLs-AI},
	year = {2026},
	url = {https://huggingface.co/FLs-AI/FL-7B-3}
	}
	```