| --- |
| library_name: mlx |
| license: apache-2.0 |
| base_model: LiquidAI/LFM2.5-230M |
| tags: |
| - mlx |
| - lora |
| - dialogue |
| - conversational |
| language: en |
| pipeline_tag: text-generation |
| --- |
| |
| # questions-lfm2-4bit |
|
|
| LFM2.5-230M fine-tuned with **MLX LoRA** to extract the most recent question from a multi-turn dialogue. Input is a transcript tagged with `[S]` and `[M]` line prefixes. The model **prioritizes** questions appearing under the `[S]` tag; if no question is present there, it falls back to the latest `[M]` block. |
|
|
| End-to-end latency on M-series Mac: **100–150 ms** per call (greedy decoding, max 80 tokens, 4-bit quantized). |
|
|
| ## Pipeline |
|
|
| 1. LoRA fine-tune in MLX (rank 32, 200 iters, 6.19M trainable params / 2.7%) |
| 2. Fuse adapter into base model → fp16 |
| 3. Quantize to 4-bit (group size 64) |
|
|
| ## Input format |
|
|
| Transcript with `[S]` and `[M]` line prefixes: |
|
|
| ``` |
| [S] what would you like to discuss today |
| [M] i was thinking about the architecture of the new service |
| [S] ok |
| [M] could you walk me through the current approach |
| ``` |
|
|
| ## Output |
|
|
| A single extracted question (no question mark, no quotes). Priority order: |
|
|
| 1. Latest question found under `[S]` |
| 2. Otherwise, latest question found under `[M]` |
| 3. Otherwise, the most recent `[M]` content |
|
|
| ## Usage (Python) |
|
|
| ```python |
| from mlx_lm import load, generate |
| |
| model, tokenizer = load("GameGC/questions-lfm2-4bit") |
| |
| messages = [ |
| {"role": "system", "content": "Extract the most recent question from the dialogue."}, |
| {"role": "user", "content": "[S] what would you like to discuss\n[M] i was thinking about the architecture\n[S] ok\n[M] could you walk me through the current approach"}, |
| ] |
| prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True) |
| out = generate(model, tokenizer, prompt=prompt, max_tokens=80) |
| print(out) |
| ``` |
|
|
| ## Usage (Swift) |
|
|
| ```swift |
| import MLXLMCommon |
| |
| let config = ModelConfiguration(id: "GameGC/questions-lfm2-4bit") |
| ``` |
|
|
| ## Performance |
|
|
| | Metric | Value | |
| |---|---| |
| | Latency (M-series Mac, 4-bit) | 100–150 ms per call | |
| | Model size on disk | ~134 MB | |
| | Max context | 1024 tokens | |
| | Quantization | 4-bit, group size 64 | |
|
|
| ## Limitations |
|
|
| - Declarative statements under `[M]` containing question-sounding words ("how", rhetorical "right") may trigger false-positive extraction. Will be addressed in v2. |
| - 4-bit quantization introduces a minor quality regression vs the fp16 fused version. |
| - Maximum context length: 1024 tokens. Longer transcripts are truncated. |
|
|
| ## Training |
|
|
| | Hyperparameter | Value | |
| |---|---| |
| | Base | LiquidAI/LFM2.5-230M | |
| | Method | MLX LoRA | |
| | Rank | 32 | |
| | Alpha | 64 | |
| | LoRA keys | `self_attn.{q,k,v,out}_proj`, `feed_forward.{w1,w2,w3}` | |
| | Trainable params | 6.19M (2.696%) | |
| | Iters | 200 | |
| | Learning rate | 2e-4 (cosine decay, warmup 10) | |
| | Max seq length | 1024 | |
| | Batch size | 4 × 4 grad accum | |
| | Val loss | 3.264 → 0.360 | |
| | Train loss | 3.144 → 0.368 | |
| | Duration | 110s on M-series Mac | |
