Buckets:
external-sharing-models

Thox-ai/models / README.md
tommytracx's picture
|
download
raw
7.31 kB
---
license: other
license_name: lfm1.0
license_link: LICENSE
base_model: LiquidAI/LFM3-Omni-2B
tags:
- gguf
- lfm3
- lfm3-omni
- liquid
- omni
- vision
- audio
- speech
- quantized
- mixed-precision
- uncensored
- abliterated
- llama.cpp
- xolotl
language:
- en
pipeline_tag: any-to-any
---
# LFM3-Omni-2B · XQ, abliterated
Liquid AI's [LFM3-Omni-2B](https://huggingface.co/LiquidAI/LFM3-Omni-2B) (text, images, video and audio in; text
and speech out) with the refusal direction edited out of the weights, then quantised with Indra's Mirror's
mixed-precision **XQ** pipeline at five sizes.
**XQ-6.0 (1.29 GiB) agrees with the abliterated 8-bit model on 95.3% of next tokens** (KL 0.021) and decodes at
**432 tok/s** on an RTX 4090. XQ-4.5 is the same size as Q4_K_M with about a third less KL divergence.
---
## Files
| file | size | what |
|---|---|---|
| `LFM3-Omni-2B-XQ-6.0-Uncensored.gguf` | 1.29 GiB | **recommended**: best quality per GB, still small enough to sit beside a big model |
| `LFM3-Omni-2B-XQ-5.0-Uncensored.gguf` | 1.12 GiB | |
| `LFM3-Omni-2B-XQ-4.5-Uncensored.gguf` | 1.04 GiB | same size as Q4_K_M, ~34% lower KL |
| `LFM3-Omni-2B-XQ-4.0-Uncensored.gguf` | 0.95 GiB | |
| `LFM3-Omni-2B-XQ-3.5-Uncensored.gguf` | 0.87 GiB | phones and small GPUs |
| `LFM3-Omni-2B-Uncensored-Q8_0.gguf` | 1.72 GiB | the abliterated Q8_0 every XQ file is measured against |
| `mmproj-LFM3-Omni-2B-Q8_0.gguf` | 0.47 GiB | Liquid AI's official vision/audio projector, unchanged; needed for images, video and audio |
The text model is the only thing changed. The projector is Liquid AI's own file, copied as-is so one download
has everything.
---
## How it was made
1. **Abliteration** on the full-precision weights: a rank-2 refusal-direction edit, fitted on Liquid AI's BF16
checkpoint. Checked on 100 held-out harmful prompts and 100 harmless ones it never saw:
| | harmful prompts refused | harmless prompts refused |
|---|---|---|
| original LFM3-Omni-2B | 92 / 100 | 5 / 100 |
| **abliterated** | **10 / 100** | **2 / 100** |
2. **Quantisation** with XQ: every tensor gets its own format (IQ3_S, IQ4_XS, IQ4_NL, Q3_K to Q6_K, Q8_0, BF16),
picked by measured sensitivity on a coding/agent/OCR/long-context calibration set for a given bits-per-weight
budget. The XQ pipeline itself is private; the files are standard GGUF and need no special loader.
LFM3 is a non-reasoning model: the chat template opens every answer with an empty `<think></think>`.
---
## Quality
Teacher-forced on a held-out text set (~256 KB, never used for calibration), against the **abliterated Q8_0**
(perplexity 6.379). KL = mean KL divergence of the next-token distribution; top-1 = same most likely token.
`llama-perplexity --kl-divergence`, llama.cpp f3586e6.
| file | size | perplexity | KL (lower is better) | 99th pct KL | same top-1 |
|---|---|---|---|---|---|
| **XQ-6.0** | 1.29 GiB | 6.466 | **0.0207** | 0.225 | **95.29%** |
| XQ-5.0 | 1.11 GiB | 6.442 | 0.0428 | | 93.18% |
| XQ-4.5 | 1.03 GiB | 6.447 | 0.0543 | | 92.32% |
| XQ-4.0 | 967 MiB | 6.549 | 0.0773 | | 90.97% |
| XQ-3.5 | 879 MiB | 6.828 | 0.1311 | | 88.31% |
| official Q4_K_M (not abliterated) | 1.02 GiB | 6.346 | 0.0822 | | 91.17% |
| official Q8_0 (not abliterated) | 1.71 GiB | 6.464 | 0.0059 | 0.068 | 97.44% |
The official files are not abliterated, so their KL includes the abliteration difference as well as their
quantisation (Q8_0's 0.0059 is roughly that difference alone). Past 6.0 the curve flattens: going from XQ-6.0 to
the Q8_0 costs another 0.42 GiB for 0.021 KL.
**Multimodal and tool smoke test** (XQ-6.0 + the Q8_0 projector, llama-server, 32K context): tool call correct,
image described correctly (shapes, colours and the number), a speech clip transcribed exactly. Speech *output*
(XQ-6.0, through Xolotl below): the model's own voice transcribed back by the model at WER 0.00 / 0.00 / 0.11 on
three sentences (the one miss is the word "Xolotl").
---
## Speed
RTX 4090 24 GB, Ryzen 7 5700X, Linux. `llama-bench` tg128, all layers on the GPU:
| file | tg128 |
|---|---|
| XQ-3.5 | 514 tok/s |
| XQ-4.0 | 501 tok/s |
| XQ-4.5 | 481 tok/s |
| XQ-5.0 | 466 tok/s |
| **XQ-6.0** | **432 tok/s** |
| official Q4_K_M | 497 tok/s |
| official Q8_0 | 355 tok/s |
Served (llama-server, XQ-6.0 + projector, 32K context): **3.5 GB of VRAM**, 389 tok/s decode on the GPU, 23 tok/s
decode CPU-only (8 threads).
---
## Quick start
Needs a llama.cpp with LFM3 support (upstream from 31 Oct 2026, f3586e6 or later).
```sh
hf download IndrasMirror/LFM3-Omni-2B-XQ-Uncensored \
LFM3-Omni-2B-XQ-6.0-Uncensored.gguf mmproj-LFM3-Omni-2B-Q8_0.gguf --local-dir lfm3
llama-server -m lfm3/LFM3-Omni-2B-XQ-6.0-Uncensored.gguf --mmproj lfm3/mmproj-LFM3-Omni-2B-Q8_0.gguf \
-ngl 99 -c 32768 --jinja -fa auto
```
Images, audio (`input_audio`) and video (`input_video`, with its soundtrack) go through the usual
`/v1/chat/completions`; speech comes out of `/tts` and the `/v1/realtime` websocket (see llama.cpp's server
README). Liquid AI's recommended sampling and system prompts are on the
[original model card](https://huggingface.co/LiquidAI/LFM3-Omni-2B).
---
## Serve with Xolotl
[Xolotl](https://github.com/Indras-Mirror/Xolotl) runs these files natively in its own LFM3 engine (`xolotl-lfm3`,
[docs/LFM3.md](https://github.com/Indras-Mirror/Xolotl/blob/main/docs/LFM3.md)), behind its OpenAI / Anthropic
server: chat, tool calls, pictures, audio, video, `/v1/audio/speech` and `/v1/audio/transcriptions` from the model
itself, and park/unpark so it can share a card with a big model. Measured on the RTX 4090 with XQ-6.0, 32K context:
| | |
|---|---|
| prefill (fresh prompt) | 32,796 tok/s at 5.3K · 26,049 tok/s at 14K |
| decode | ~300 tok/s |
| follow-up turn on a 7K conversation | 0.03 s |
| HumanEval (21) / tool calls (10) / agentic loops (3) | 13 / 9 / 0 - a 2B model, thinking off |
| speech out | 12x real time, first audio 0.02 s once loaded |
| park / unpark | 51 ms / ~75 ms, ~1.4 GB of VRAM handed back |
Its logits match llama.cpp's on the same file (mean KL 1.3e-7 with the same memory layout).
```sh
git clone https://github.com/Indras-Mirror/Xolotl && cd Xolotl # build as in its README, then:
ninja -C build-xolotl xolotl-lfm3
tools/lfm3/build-helpers.sh ~/llama.cpp # the picture/audio and speech helpers, on llama.cpp f3586e6+
cp tools/lfm3/xol-lfm3.example.json xol-lfm3.json # point it at the GGUF, the mmproj and the tokenizer
.venv/bin/python -m serve.server --engine strata --config xol-lfm3.json --port 8132
```
---
## Licence
[LFM Open License v1.0](LICENSE), the same as the original model. These files are **modified** versions of
LiquidAI/LFM3-Omni-2B (abliterated and quantised by Indra's Mirror); the projector is unmodified. Commercial use is
licensed only below Liquid AI's revenue threshold; read the licence.
## Credits
* **Liquid AI** for [LFM3-Omni-2B](https://huggingface.co/LiquidAI/LFM3-Omni-2B) and its official GGUF projector.
* **ggml / llama.cpp** for LFM3 support and the evaluation tools.
* Abliteration, XQ quantisation and measurements: Indra's Mirror.
## Support
If this has been useful and you want to help keep it going, you can
[buy me a coffee](https://ko-fi.com/indrasmirror).

Xet Storage Details

Size:
7.31 kB
·
Xet hash:
bb433b22cc0c618b1b64914321a1d136ced63a84177402b814e680d3effa07c8

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.