Instructions to use lostium/privacy-filter-multilingual-mixed-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lostium/privacy-filter-multilingual-mixed-gguf with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="lostium/privacy-filter-multilingual-mixed-gguf",
	filename="privacy-filter-multilingual-mixed.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use lostium/privacy-filter-multilingual-mixed-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf lostium/privacy-filter-multilingual-mixed-gguf
# Run inference directly in the terminal:
llama cli -hf lostium/privacy-filter-multilingual-mixed-gguf

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf lostium/privacy-filter-multilingual-mixed-gguf
# Run inference directly in the terminal:
llama cli -hf lostium/privacy-filter-multilingual-mixed-gguf

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf lostium/privacy-filter-multilingual-mixed-gguf
# Run inference directly in the terminal:
./llama-cli -hf lostium/privacy-filter-multilingual-mixed-gguf

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf lostium/privacy-filter-multilingual-mixed-gguf
# Run inference directly in the terminal:
./build/bin/llama-cli -hf lostium/privacy-filter-multilingual-mixed-gguf

Use Docker

docker model run hf.co/lostium/privacy-filter-multilingual-mixed-gguf

LM Studio
Jan
Ollama
How to use lostium/privacy-filter-multilingual-mixed-gguf with Ollama:
```
ollama run hf.co/lostium/privacy-filter-multilingual-mixed-gguf
```

Unsloth Studio

How to use lostium/privacy-filter-multilingual-mixed-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for lostium/privacy-filter-multilingual-mixed-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for lostium/privacy-filter-multilingual-mixed-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for lostium/privacy-filter-multilingual-mixed-gguf to start chatting

Atomic Chat new
Docker Model Runner
How to use lostium/privacy-filter-multilingual-mixed-gguf with Docker Model Runner:
```
docker model run hf.co/lostium/privacy-filter-multilingual-mixed-gguf
```

Lemonade

How to use lostium/privacy-filter-multilingual-mixed-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull lostium/privacy-filter-multilingual-mixed-gguf

Run and chat with the model

lemonade run user.privacy-filter-multilingual-mixed-gguf-{{QUANT_TAG}}

List all available models

lemonade list

Privacy Filter — Multilingual, Mixed-Precision GGUF (Q4_0 / Q8_0)

A mixed-precision GGUF quantization of the multilingual OpenAI Privacy Filter PII/NER model, built for in-browser, CPU-only inference via WebAssembly. It is ≈2.8× smaller than the f16 GGUF (≈1.0 GB vs ≈2.82 GB) with no measurable loss in detection quality.

⚠️ This GGUF requires privacy-filter.cpp, not standard llama.cpp. The model uses the custom openai-privacy-filter architecture (gpt-oss-style MoE encoder with a token-classification head). It will not load in generic GGUF runtimes.

Attribution & provenance

This is a derivative work. Full credit to the original authors:

Original model: openai/privacy-filter — © OpenAI, licensed under Apache 2.0.
Source GGUF (f16) we quantized from: LocalAI-io/privacy-filter-multilingual-GGUF (privacy-filter-multilingual-f16.gguf), which is itself a GGUF conversion of the OpenAI model.

Changes made (Apache 2.0 §4): mixed-precision quantization of selected weight tensors (see below). No architecture, vocabulary, label taxonomy, or training change. This model is distributed under the same Apache 2.0 license as the original (see LICENSE and NOTICE in this repository).

What this is

The model is a bidirectional token classifier for PII / Named Entity Recognition: gpt-oss-style pre-norm encoder, grouped-query attention with RoPE/YaRN, a sparse mixture-of-experts feed-forward (128 experts), and a token-classification head. It detects person names, organizations, addresses/locations, dates, and structured identifiers across multiple languages (validated on Spanish, Catalan, and English).

Quantization recipe

Mixed precision, applied per tensor (block size 32 — note Q4_K/Q5_K are not applicable to this architecture because its inner tensor dimensions, e.g. 128/640/896/217, are not divisible by 256):

Tensor group	Type	Count
FFN MoE experts (`ffn_gate_exps.weight`, `ffn_up_exps.weight`, `ffn_down_exps.weight`)	Q4_0	24
Attention weights (`attn_q/k/v/output.weight`)	Q8_0	32
Token embeddings, all norms, all biases, classifier head (`cls.output.*`)	kept verbatim (F16/F32)	100

Why mixed and not uniform Q4_0: uniform Q4_0 (0.99 GB) degrades short-token detections (it dropped a first name and a city in testing). Keeping attention at Q8_0 recovers full quality at essentially the same size (the MoE experts dominate the file), while the sensitive classifier head and embeddings stay verbatim.

Evaluation (f16 reference vs this mixed model)

Measured with privacy-filter.cpp (pf-cli, threshold 0.5) over a diverse Spanish/Catalan/English PII corpus including real documents (contracts, an insurance policy, a court ruling, an administrative resignation, a flight itinerary) plus detector fixtures:

Metric	Result
Reference spans (f16)	328
Recall of mixed vs f16	98.5% (323/328)
Threshold-crossing regressions (f16 ≥ 0.5 → mixed < 0.5)	0
Recall on real-document content	100%
Per-type recall	DNI/SSN 100%, IBAN 100%, dates 100%, person 99.0%, address 98.2%, phone 96.3%, email 96.2%

Efficiency vs f16 (native, CPU): size ≈1.0 GB (−65%), peak RSS ≈1.06 GB (−63%), latency ≈0.2 s/inference (−45%). In browser (Chrome, WebAssembly): loads with a plain WASM32 build (no memory64, no OPFS needed because the file is < 2 GiB), ≈1.04 GiB WASM heap, ≈1.4 s total (stream + load + classify) for a short input.

Intended use

On-device / in-browser PII detection where the full f16 model (2.82 GB) is too heavy and a GPU (WebGPU) is unavailable. Built for Ocultia's client-side anonymization pipeline, running entirely on CPU via WebAssembly. Also suitable for air-gapped / on-premise deployments.

How to run

# Build privacy-filter.cpp (CPU), then:
cat input.txt | pf-cli --classify privacy-filter-multilingual-mixed.gguf 0.5 cpu

Output is a JSON array of entity spans: {"entity_group","start","end","score","text"} (byte offsets). For WebAssembly usage, compile privacy-filter.cpp with Emscripten and load the GGUF into MEMFS (the file fits under the browser's ≈2 GiB single-ArrayBuffer limit).

Limitations

Evaluated against the f16 model as reference over a diverse but not canonically labeled corpus; absolute precision/recall against gold labels is not measured here.
The quantization recipe is specific to this architecture; do not assume it transfers to other GGUF models.
Language coverage is inherited from the original multilingual model. This quantization is language-agnostic and adds no per-language tuning; we only validated detection parity (vs f16) on Spanish, Catalan, and English. Other languages supported by the base model are expected to work but were not measured here.
Inherits the capabilities and biases of the original OpenAI Privacy Filter model.