README.md

---
base_model: LiquidAI/LFM2.5-1.2B-Instruct
library_name: transformers
license: other
license_name: lfm1.0
license_link: https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct/blob/868df74dd56ff8a0c2ac5dbf281690c2dbebe4c9/LICENSE
pipeline_tag: text-generation
tags:
  - lfm2
  - lfm2.5
  - liquid
  - int8
  - w8a8
  - compressed-tensors
  - vllm
  - lora
---

# LFM2.5-1.2B-Instruct INT8 W8A8

This repository contains a post-training-quantized checkpoint of
[`LiquidAI/LFM2.5-1.2B-Instruct`](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct),
prepared for efficient vLLM inference and per-request LoRA serving.

## Base model

- Model: `LiquidAI/LFM2.5-1.2B-Instruct`
- Model revision: `868df74dd56ff8a0c2ac5dbf281690c2dbebe4c9`
- Tokenizer revision: `868df74dd56ff8a0c2ac5dbf281690c2dbebe4c9`
- Upstream license: Liquid AI Model License 1.0

## Quantization

- Scheme: W8A8 INT8
- Weight quantization: symmetric, per-channel INT8
- Activation quantization: symmetric, dynamic per-token INT8
- Format: `compressed-tensors`
- Pipeline: data-free dynamic-activation quantization
- Quantized modules: 72 attention and feed-forward projections
- Kept unquantized: embeddings, LM head, normalization layers, and 20 Liquid convolution
  projections

This is therefore a mixed-precision checkpoint: the targeted attention and feed-forward
projections use W8A8 INT8, while unsupported Liquid convolution projections remain BF16.
They were intentionally excluded because Transformers names them `conv`, while vLLM 0.24.0
uses `short_conv` and does not remap the compressed scale names.

The full target-module list, exclusions, source revision, and compatibility rationale are
recorded in `recipe.yaml` and `quantization_manifest.json`.

## Integrity

SHA-256 for `model.safetensors`:

```text
1071099fafdda7400ac9624b916ff45c56cb276eb061b22ea6c0ba4343f34edc
```

## vLLM

Verified with vLLM 0.24.0 using the native compressed-tensors W8A8 INT8 kernels.

```bash
vllm serve PointGuardAI/LFM2.5-1.2B-Instruct-INT8 \
  --served-model-name lfm25-1.2b-int8 \
  --max-model-len 5120 \
  --gpu-memory-utilization 0.90 \
  --performance-mode throughput \
  --max-num-batched-tokens 8192
```

For per-request LoRA serving:

```bash
vllm serve PointGuardAI/LFM2.5-1.2B-Instruct-INT8 \
  --enable-lora \
  --max-loras 4 \
  --max-lora-rank 16 \
  --lora-modules application-policy-match=/path/to/adapter
```

Chat-completion example:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lfm25-1.2b-int8",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Explain dynamic INT8 quantization in one sentence."}
    ],
    "max_tokens": 64,
    "temperature": 0
  }'
```

## Guardrail validation

When paired with the separately trained PointGuard application-policy LoRA
(not included in this repository), this checkpoint achieved the following on a balanced
558-row held-out development-validation set:

- Accuracy: 96.42%
- Macro F1: 96.41%
- AUPRC: 99.44%
- MATCH recall: 98.57%
- False negatives: 4 of 279 MATCH examples

Using vLLM generative scoring with one prompt per HTTP request:

| Concurrency | Median throughput | Median p95 latency |
|---:|---:|---:|
| 16 | 62.4 requests/second | 303 ms |
| 32 | 67.7 requests/second | 543 ms |

These measurements are specific to the evaluated prompt distribution, LoRA adapter, NVIDIA
A10G GPU, and vLLM configuration; they are not general language-model benchmarks.

## Intended use and limitations

This repository contains the quantized general-purpose base checkpoint, not a standalone
guardrail. It does not include a task-specific LoRA adapter, policy engine, calibrated decision
thresholds, or safety guarantees.

Quantization can change model quality and probability calibration. Evaluate the exact production
adapter, prompt format, serving runtime, thresholds, and deployment data before use.