README.md

---
language:
- en
- zh
- multilingual
license: apache-2.0
library_name: mlx
pipeline_tag: feature-extraction
base_model: Octen/Octen-Embedding-4B
tags:
- mlx
- mlx-lm
- embeddings
- text-embedding
- retrieval
- multilingual
- qwen3
- 4-bit
---

# Octen-Embedding-4B-MLX-4bit

Community MLX 4-bit conversion of
[Octen/Octen-Embedding-4B](https://huggingface.co/Octen/Octen-Embedding-4B)
for Apple silicon.

This repository is not an official Octen release. The original model was
developed by Octen and is distributed under the Apache License 2.0.

## Model details

| Property | Value |
|---|---|
| Upstream model | `Octen/Octen-Embedding-4B` |
| Upstream revision | `fea468fae3f0caffbae8a12ba792d1c394b6277d` |
| Runtime architecture | Standard MLX-LM `qwen3.Model` |
| Parameters | 4B |
| Embedding dimension | 2560 |
| Context length | 32,768 tokens |
| Quantization | MLX affine 4-bit |
| Quantization group size | 64 |
| Pooling | Last token |
| Normalization | L2 normalization in float32 |

The upstream checkpoint contains encoder-only `Qwen3Model` weights without an
LM head. Its weight keys were normalized from `embed_tokens.*` / `layers.*` to
the standard MLX-LM `model.embed_tokens.*` / `model.layers.*` layout before
running the official `mlx_lm.convert` command. The final repository uses the
built-in `mlx_lm.models.qwen3.Model` and contains no custom Python model code.

For embeddings, use the built-in model's `model` encoder submodule, which
returns hidden states shaped `[batch, sequence, 2560]`.

## Prompt format

Use the same prompt contract as the upstream Sentence Transformers model:

```text
Query:
Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:{query}

Document:
 {document}
```

The document prefix is one space. Query and document embeddings must use the
same checkpoint and processing contract.

## Usage

```python
import mlx.core as mx
from mlx_lm import load

repo = "davied-he/Octen-Embedding-4B-MLX-4bit"
model, tokenizer = load(repo)

QUERY_PREFIX = (
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\n"
    "Query:"
)


def encode(text: str, is_query: bool) -> mx.array:
    prefix = QUERY_PREFIX if is_query else " "
    input_ids = mx.array([tokenizer.encode(prefix + text)])
    hidden_states = model.model(input_ids)
    embedding = hidden_states[:, -1, :].squeeze().astype(mx.float32)
    norm = mx.sqrt(mx.sum(mx.square(embedding)))
    return embedding / mx.maximum(norm, mx.array(1e-12))


query = encode("What is semantic retrieval?", is_query=True)
document = encode(
    "Semantic retrieval finds documents by meaning rather than exact keywords.",
    is_query=False,
)

similarity = float(mx.sum(query * document))
print(query.shape)  # (2560,)
print(similarity)
```

`estha_embedding_config.json` contains the same runtime contract in a
machine-readable form.

## Conversion

The upstream model uses bare encoder weight names such as `embed_tokens.*` and
`layers.*`. Before conversion, those keys were deterministically prefixed with
`model.` so they match the official MLX-LM Qwen3 model layout. No tensor values
were changed during this normalization step.

```bash
mlx_lm.convert \
  --hf-path Octen/Octen-Embedding-4B \
  --mlx-path Octen-Embedding-4B-MLX-4bit \
  --quantize \
  --q-group-size 64 \
  --q-bits 4
```

Conversion environment:

- `mlx-lm==0.31.3`
- official `mlx_lm.convert`
- affine 4-bit quantization
- 4.501 effective bits per weight reported by MLX-LM
- no custom `model_file` or remote code

## Validation

The published artifact was validated on Apple silicon:

- MLX-LM loads the repository directly with built-in `qwen3.Model`.
- The repository contains no custom Python model files.
- Hidden-state output shape is `[batch, sequence, 2560]`.
- Query and document embeddings have shape `[2560]`.
- Float32 L2-normalized vector norms are `1.0`.
- On three English and Traditional Chinese samples, cosine similarity between
  upstream BF16 and MLX 4-bit embeddings was `0.9748–0.9782`.
- A two-query bilingual retrieval smoke test returned the expected Top-1
  documents.

Artifact checksum:

```text
model.safetensors
423d1c857da56367a84bffa182a4a875c5844a54a979c0866da38072525f6aed
```

## Limitations

- Quantization introduces numerical differences from the upstream BF16 model.
- This conversion exposes embeddings, not text generation.
- Do not mix vectors from this model with vectors generated by another model,
  even when both models use 2560 dimensions.
- Retrieval quality should be evaluated on your own language and domain data.

## Attribution and license

The upstream `Octen/Octen-Embedding-4B` model and its base
`Qwen/Qwen3-Embedding-4B` are licensed under Apache License 2.0. This
conversion preserves that license and attribution. See `LICENSE`.

Please cite and credit the upstream Octen model when using this conversion.