| --- |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| language: en |
| tags: |
| - lfm2 |
| - agent |
| - fine-tune |
| - tool-use |
| - liquid-ai |
| base_model: LiquidAI/LFM2.5-1.2B-Thinking |
| datasets: |
| - TeichAI/Ox-Alpha-Pi-Traces |
| metrics: |
| - loss |
| --- |
| |
| # LiquidAgent-1.2B |
|
|
| A supervised fine-tune of [LiquidAI/LFM2.5-1.2B-Thinking](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking) on agent traces from [TeichAI/Ox-Alpha-Pi-Traces](https://huggingface.co/datasets/TeichAI/Ox-Alpha-Pi-Traces), with tool results properly included in the training data. |
|
|
| ## What this is |
|
|
| The base LFM2.5-1.2B-Thinking model is a 1.17B-param hybrid architecture (10 LIV-conv + 6 GQA blocks, 32K context, 65536 vocab) with built-in thinking. This fine-tune adapts it for agentic tool-use: the model learns to emit tool calls, see the tool results, and reason about them in subsequent turns. |
|
|
| **Key fix over prior attempts:** The original parser for the TeichAI data silently dropped all `toolResult` messages (the `type` field is always `"message"` in the raw data, and tool results arrive as `message.role == "toolResult"` - a branch that did not exist). This fine-tune uses a fixed parser that recovers 100% of tool turns (verified: 562/562 on a 50-file sample). |
|
|
| ## Training details |
|
|
| - **Base:** LiquidAI/LFM2.5-1.2B-Thinking (bfloat16) |
| - **Data:** 379 examples, ~1.34M assistant tokens (from Ox-Alpha-Pi-Traces, 400 session files, 397 usable) |
| - **Optimizer:** AdamW 8-bit, lr 2e-6 to 8e-6 (cosine warmup over 47 steps) |
| - **Hardware:** NVIDIA RTX 5090 (32 GB), gradient checkpointing, batch size 1 |
| - **Duration:** 81 seconds (47 steps) |
| - **Final loss:** 0.5965 (step 40) - training completed at step 47 |
|
|
| ## Architecture |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Params | 1,170,340,608 (tied embedding) | |
| | Layers | 16 (10 conv + 6 full attention) | |
| | Hidden size | 2048 | |
| | FF dim | 12288 | |
| | Attention heads | 32 Q / 8 KV | |
| | Vocab | 65,536 | |
| | Context | 32,768 (max_position_embeddings: 128,000) | |
| | Dtype | bfloat16 | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "Compactbot/liquidagent-1.2b", |
| torch_dtype="bfloat16", |
| device_map="auto", |
| ) |
| tokenizer = AutoTokenizer.from_pretrained("Compactbot/liquidagent-1.2b") |
| |
| messages = [{"role": "user", "content": "Write a Python function to parse a JSON config file."}] |
| text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(text, return_tensors="pt").to(model.device) |
| out = model.generate(**inputs, max_new_tokens=512, temperature=0.7, top_p=0.9) |
| print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ## Honest limitations |
|
|
| - Trained on only 379 examples (~1.3M tokens) - this is a very light adaptation, not a full agent training run. |
| - The model retains the base's thinking pattern (emits thinking blocks before responding). |
| - No independent benchmark evaluation was run; quality was verified by generation samples (coherent, on-topic, no degenerate repetition). |
| - The 32k sequence length requested by the requester was not used in this run (data was packed at shorter lengths to fit the 32GB GPU with 8-bit AdamW). |
