File size: 5,032 Bytes

a85ba20

---
license: apache-2.0
language:
- en
- zh
- ru
- es
- fr
- de
- ar
- nl
- vi
- hi
- ko
- ja
- it
- id
- pt
- pl
- tr
- da
- th
- sv
- fa
- uk
- cs
- 'no'
- el
- ca
- ro
- fi
- bg
- tl
- gl
- my
- hy
- km
- ne
- hu
- eu
- he
- lo
- sw
- az
- lv
- si
- sk
- tg
- et
- lt
- ms
- hr
- is
- sl
- sr
- ur
- bn
- af
- ta
- ka
- te
- ml
- mn
- nn
- kk
- cy
- mr
- sq
- nb
- mk
- jv
- kn
- eo
- la
- gu
- uz
- am
- oc
- be
- mg
- vo
- pa
- lb
- ht
- br
- ga
- xh
- tt
- bs
- yo
base_model:
- codefuse-ai/F2LLM-v2-4B
base_model_relation: quantized
pipeline_tag: feature-extraction
library_name: mlx
tags:
- mlx
- mlx-embeddings
- sentence-transformers
- feature-extraction
- qwen3
datasets:
- codefuse-ai/F2LLM-v2
---
# fcmeyer/F2LLM-v2-4B-mlx-8bit

[codefuse-ai/F2LLM-v2-4B](https://huggingface.co/codefuse-ai/F2LLM-v2-4B) converted to native MLX format —
**8-bit (affine, group size 64)**, 4.0 GB on disk.

F2LLM-v2-4B is a 4B-parameter multilingual text embedding model from CodeFuse (Qwen3
trunk, 2560-dimensional embeddings, last-token pooling with L2 normalization). All credit
for the model belongs to the original authors; this repository only changes the weight
format so the model runs natively on Apple Silicon through
[mlx-embeddings](https://github.com/Blaizzy/mlx-embeddings).

## Usage

```bash
pip install mlx-embeddings
```

```python
from mlx_embeddings import load, generate
import mlx.core as mx

model, tokenizer = load("fcmeyer/F2LLM-v2-4B-mlx-8bit")

# Queries get the instruction prompt; documents do not.
query_prompt = "Instruct: Given a question, retrieve passages that can help answer the question.\nQuery: "
query = "What is F2LLM used for?"
documents = [
    "We present F2LLM, a family of fully open embedding LLMs that achieve a strong balance between model size, training data, and embedding performance.",
    "F2LLM is a model for computing text embeddings that can be used for various NLP tasks such as information retrieval, semantic search, and text classification.",
    "F2LLM 是 CodeFuse 开源的系列嵌入模型。",
    "F2LLM — это модель вычисления встраивания текста, которую можно использовать для различных задач НЛП, таких как поиск информации, семантический поиск и классификация текста."
]

# Pass max_length explicitly — generate() defaults to max_length=512 with
# truncation enabled, which silently clips longer inputs.
query_embedding = generate(model, tokenizer, texts=[query_prompt + query],
                           max_length=8192).text_embeds
document_embeddings = generate(model, tokenizer, texts=documents,
                               max_length=8192).text_embeds

similarity = query_embedding @ document_embeddings.T
print(similarity)
# [0.6342, 0.8541, 0.7194, 0.8373]
```

### Prompt format

Custom instructions follow the base model's format:

```text
Instruct: your_instruction
Query: your_query
```

For retrieval and reranking, prompt the queries and leave documents unprompted. For
symmetric tasks (STS, clustering, bitext mining) the model works with or without prompts
on both sides.

## Conversion

```bash
pip install "mlx-embeddings @ git+https://github.com/Blaizzy/mlx-embeddings@9b28270be81211f2b8daed0041aec65ea5dc4b28"
python -m mlx_embeddings.convert --hf-path codefuse-ai/F2LLM-v2-4B \
    --mlx-path F2LLM-v2-4B-mlx-8bit --dtype bfloat16 -q --q-bits 8 --q-group-size 64
```

Converted with `mlx` 0.32.2 and `mlx-embeddings` 0.1.1
(commit [`9b28270be812`](https://github.com/Blaizzy/mlx-embeddings/commit/9b28270be81211f2b8daed0041aec65ea5dc4b28)).

All linear layers **and** the token embedding table are quantized to 8 bits with MLX affine quantization at group size 64. RMSNorm weights and the quantization scales/biases stay in bfloat16.

## Accuracy check

Each build was compared against a PyTorch bfloat16 reference run of the original
checkpoint on the 5-string fixture from the base model card (one prompted English query
plus four documents in English, Chinese, and Russian). Embeddings were re-normalized in
float32 before comparison, since bfloat16 output leaves vectors slightly off unit norm.

| Build | Size | Min cosine vs PyTorch | Max Δ on query→document similarity | Ranking preserved |
|---|---|---|---|---|
| `F2LLM-v2-4B-mlx-bf16` | 7.5 GB | 0.99984 | 0.0017 | yes |
| `F2LLM-v2-4B-mlx-8bit` **(this repo)** | 4.0 GB | 0.99954 | 0.0020 | yes |
| `F2LLM-v2-4B-mlx-6bit` | 3.1 GB | 0.99696 | 0.0078 | yes |

The PyTorch reference itself reproduces the similarity row published on the base model
card to within 0.005 (bfloat16 on CPU versus the card's bfloat16 on CUDA).

This is a small smoke-test fixture, not a benchmark. No MTEB or retrieval evaluation was
run on the quantized builds — if quantization loss matters for your task, measure it on
your own data.

## License

Apache 2.0, inherited from [codefuse-ai/F2LLM-v2-4B](https://huggingface.co/codefuse-ai/F2LLM-v2-4B).