README.md

---
base_model: LiquidAI/LFM2.5-2.6B
datasets:
- saidutta69/fable-5-premium
library_name: transformers
pipeline_tag: text-generation
tags:
- lfm2
- full-parameter-fine-tuning
- supervised-fine-tuning
- assistant-only-loss
- tool-use
- coding-agent
- conversational
---

# LFM2.5-2.6B Fable-5 Coding Agent

`AyoubChLin/lfm2.5-2.6b-fable5-coding-agent` is a **full-parameter supervised fine-tune** of [`LiquidAI/LFM2.5-2.6B`](https://huggingface.co/LiquidAI/LFM2.5-2.6B) on [`saidutta69/fable-5-premium`](https://huggingface.co/datasets/saidutta69/fable-5-premium).

The run optimized assistant responses in multi-turn conversations, including reasoning-style text and tool-call patterns. All **2,697,198,592 parameters** were trainable. This repository contains a complete BF16 model checkpoint—not a LoRA, QLoRA, PEFT adapter, or quantized-weight checkpoint. The 8-bit optimizer affected optimizer-state storage only.

## Model details

| Field | Value |
|---|---|
| Base model | `LiquidAI/LFM2.5-2.6B` |
| Architecture | Causal language model |
| Fine-tuning method | Full-parameter supervised fine-tuning |
| Parameters | 2,697,198,592 total; 100% trainable |
| Training precision | BF16, with TF32 enabled |
| Maximum sequence length used for SFT | 32,000 tokens |
| Training objective | Assistant-only next-token loss |
| Chat formatting | Base model's native chat template |
| Tool-call preprocessing | JSON argument strings converted to mappings for the native LFM2.5 tool-call format |
| Reasoning data | Preserved during training (`PRESERVE_THINKING=True`) |

## Intended use

This checkpoint is intended for research and evaluation involving:

- multi-turn assistant behavior;
- code generation and explanation;
- structured tool-call generation in a controlled agent harness; and
- further evaluation or domain adaptation.

It should not be treated as production-ready based on the evidence currently available. The recorded run did not measure code correctness, tool-call validity, factuality, security, safety, bias, multilingual performance, instruction following, or agent-task completion.

## Training data

The run loaded the `openai_chat` Parquet files explicitly so each published split was included once. It used the **first 5,000 rows** of the training split and the complete validation and test splits.

Only assistant tokens contributed to the loss. System, user, tool-result, and padding tokens were masked with label `-100`; assistant tool calls remained supervised. No row was removed by the post-tokenization assistant-label check.

### Tokenized split statistics

| Split | Rows | Mean tokens | P95 tokens | Rows truncated at 32,000 | Mean supervised assistant tokens |
|---|---:|---:|---:|---:|---:|
| Train | 5,000 | 23,167.7 | 32,000 | 2,609 (52.18%) | 6,335.1 |
| Validation | 318 | 23,010.3 | 32,000 | 165 (51.89%) | 6,331.9 |
| Test | 319 | 22,812.0 | 32,000 | 154 (48.28%) | 6,370.4 |

Before truncation, the 5,000 selected training rows had the following length distribution:

| Statistic | Tokens |
|---|---:|
| P50 | 33,351 |
| P90 | 65,820 |
| P95 | 76,467 |
| P99 | 92,063 |
| Maximum | 104,776 |

Because more than half of the selected training rows exceeded the 32,000-token training cap, long conversations were frequently truncated.

## Training procedure

| Hyperparameter | Recorded value |
|---|---|
| Epochs | 1 |
| Micro-batch size | 2 |
| Gradient accumulation | 4 |
| Effective batch size | 8 sequences per optimizer step |
| Evaluation batch size | 1 |
| Learning rate | `2e-5` |
| Weight decay | `0.1` |
| Scheduler | Cosine |
| Warm-up argument | `0.03` supplied to `warmup_steps` |
| Optimizer | 8-bit AdamW (`adamw_bnb_8bit`) |
| Gradient clipping | `1.0` |
| Gradient checkpointing | Enabled, non-reentrant |
| Seed / data seed | 42 / 42 |
| Evaluation cadence | Every 100 optimizer steps |
| Checkpoint strategy | Once per epoch, model weights only |
| Hardware | 1× NVIDIA B200, 178.4 GiB VRAM |
| Software observed | PyTorch 2.8.0+cu129; CUDA 12.9; Transformers 5.15.0 |

Checkpoints were saved with `save_only_model=True`. They are suitable for evaluation or deployment, but they do not contain optimizer and scheduler states for an exact training resume.

## Results

| Split / metric | Value | Derived perplexity |
|---|---:|---:|
| Training loss | 0.1316 | 1.1406 |
| Validation loss | 0.3445 | 1.4113 |
| Held-out test loss | 0.3458 | 1.4131 |

Training completed in **15,553.2 seconds** (approximately **4 h 19 min 13 s**) at 0.321 samples/second and 0.040 optimizer steps/second. The run reported approximately `2.134e18` floating-point operations.

Perplexity is calculated as `exp(loss)`. All losses cover only the assistant tokens selected by the masking procedure, so they are not directly comparable with full-sequence language-model losses. Training loss is averaged over the optimization trajectory, whereas validation and test losses were measured after training.

The held-out test split was not used for optimization or periodic validation. No pre-fine-tuning baseline, external benchmark, confidence interval, or repeated-seed result was recorded. These results establish held-out assistant-token loss for this run; they do not by themselves demonstrate improvement over the base model or general coding-agent quality.

## Qualitative observation

For one interval-merging prompt, the checkpoint produced a structured plan and emitted a native `write(...)` tool call without an explicit tool schema in the prompt. The generation reached the configured `max_new_tokens=768` limit before completing the program, and the resulting code was not executed or scored.

This is an illustration, not an evaluation. In deployment:

1. Provide explicit tool definitions through the serving or agent layer.
2. Parse, authorize, and validate every generated tool call before execution.
3. Run generated code in a sandbox and verify it with independent tests.
4. Do not expose preserved reasoning traces when the product requires private internal reasoning.

## Inference with Transformers

Install a recent Transformers release:

```bash
pip install -U "transformers>=5.2.0,<6" torch
```

Then apply the checkpoint's native chat template:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AyoubChLin/lfm2.5-2.6b-fable5-coding-agent"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

messages = [
    {
        "role": "system",
        "content": "You are a careful coding assistant. Make focused changes and verify the result.",
    },
    {
        "role": "user",
        "content": "Write a tested Python function that merges overlapping integer intervals.",
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=768,
        do_sample=True,
        temperature=0.2,
        top_k=50,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=False))
```

`skip_special_tokens=False` preserves native reasoning and tool-call delimiters for inspection by a compatible parser. Do not send raw reasoning or unvalidated tool syntax directly to end users or executors.

## Reproducibility notes

- The source run used a single NVIDIA B200 with native BF16 support.
- The model remained in BF16 and all parameters were updated; `adamw_bnb_8bit` reduced optimizer-state memory only.
- OpenAI-style tool-call argument strings were normalized into mappings before the native chat template was applied.
- `PRESERVE_THINKING=True` retained supplied thinking content.
- The variable named `WARMUP_RATIO` was passed to `warmup_steps`, not `warmup_ratio`; this card reports the executed configuration rather than reinterpreting it.
- The bitsandbytes runtime reported that no CUDA 12.9 binary was available and loaded its CUDA 12.8 build instead.
- The environment reported Linux kernel 4.19.0, below the Trainer warning's recommended minimum of 5.5.0.

## Limitations and responsible use

- Generated code and tool calls may be incomplete, incorrect, unsafe, or incompatible with the target environment.
- Reasoning-style text may be exposed because the training data preserved it.
- The training set was a deterministic 5,000-row prefix rather than the complete published training split.
- Heavy 32K truncation may weaken behavior that depends on information appearing late in long conversations.
- Tool-call patterns were learned without complete tool schemas; applications must supply schemas and enforce permissions externally.
- The checkpoint inherits limitations from the base model and the fine-tuning dataset.

Review and comply with the licenses and terms of both the [base model](https://huggingface.co/LiquidAI/LFM2.5-2.6B) and the [training dataset](https://huggingface.co/datasets/saidutta69/fable-5-premium) before use or redistribution. This model card does not grant additional rights.

## Acknowledgements

- Base model: [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B)
- Training dataset: [saidutta69/fable-5-premium](https://huggingface.co/datasets/saidutta69/fable-5-premium)