fcmeyer/F2LLM-v2-4B-mlx-bf16

codefuse-ai/F2LLM-v2-4B converted to native MLX format — bfloat16 (unquantized), 7.5 GB on disk.

F2LLM-v2-4B is a 4B-parameter multilingual text embedding model from CodeFuse (Qwen3 trunk, 2560-dimensional embeddings, last-token pooling with L2 normalization). All credit for the model belongs to the original authors; this repository only changes the weight format so the model runs natively on Apple Silicon through mlx-embeddings.

Usage

pip install mlx-embeddings

from mlx_embeddings import load, generate
import mlx.core as mx

model, tokenizer = load("fcmeyer/F2LLM-v2-4B-mlx-bf16")

# Queries get the instruction prompt; documents do not.
query_prompt = "Instruct: Given a question, retrieve passages that can help answer the question.\nQuery: "
query = "What is F2LLM used for?"
documents = [
    "We present F2LLM, a family of fully open embedding LLMs that achieve a strong balance between model size, training data, and embedding performance.",
    "F2LLM is a model for computing text embeddings that can be used for various NLP tasks such as information retrieval, semantic search, and text classification.",
    "F2LLM 是 CodeFuse 开源的系列嵌入模型。",
    "F2LLM — это модель вычисления встраивания текста, которую можно использовать для различных задач НЛП, таких как поиск информации, семантический поиск и классификация текста."
]

# Pass max_length explicitly — generate() defaults to max_length=512 with
# truncation enabled, which silently clips longer inputs.
query_embedding = generate(model, tokenizer, texts=[query_prompt + query],
                           max_length=8192).text_embeds
document_embeddings = generate(model, tokenizer, texts=documents,
                               max_length=8192).text_embeds

similarity = query_embedding @ document_embeddings.T
print(similarity)
# [0.6364, 0.8533, 0.7156, 0.8351]

Prompt format

Custom instructions follow the base model's format:

Instruct: your_instruction
Query: your_query

For retrieval and reranking, prompt the queries and leave documents unprompted. For symmetric tasks (STS, clustering, bitext mining) the model works with or without prompts on both sides.

Conversion

pip install "mlx-embeddings @ git+https://github.com/Blaizzy/mlx-embeddings@9b28270be81211f2b8daed0041aec65ea5dc4b28"
python -m mlx_embeddings.convert --hf-path codefuse-ai/F2LLM-v2-4B \
    --mlx-path F2LLM-v2-4B-mlx-bf16 --dtype bfloat16

Converted with mlx 0.32.2 and mlx-embeddings 0.1.1 (commit 9b28270be812).

This is the unquantized reference conversion: every weight is stored in bfloat16, the same precision as the original checkpoint.

Accuracy check

Each build was compared against a PyTorch bfloat16 reference run of the original checkpoint on the 5-string fixture from the base model card (one prompted English query plus four documents in English, Chinese, and Russian). Embeddings were re-normalized in float32 before comparison, since bfloat16 output leaves vectors slightly off unit norm.

Build	Size	Min cosine vs PyTorch	Max Δ on query→document similarity	Ranking preserved
F2LLM-v2-4B-mlx-bf16 (this repo)	7.5 GB	0.99984	0.0017	yes
F2LLM-v2-4B-mlx-8bit	4.0 GB	0.99954	0.0020	yes
F2LLM-v2-4B-mlx-6bit	3.1 GB	0.99696	0.0078	yes

The PyTorch reference itself reproduces the similarity row published on the base model card to within 0.005 (bfloat16 on CPU versus the card's bfloat16 on CUDA).

This is a small smoke-test fixture, not a benchmark. No MTEB or retrieval evaluation was run on the quantized builds — if quantization loss matters for your task, measure it on your own data.

License

Apache 2.0, inherited from codefuse-ai/F2LLM-v2-4B.