Hugging Face's logo

wenshutang
/
ternlight

Sentence Similarity
English
sentence-embeddings
semantic-search
on-device
wasm
webassembly
bitnet
ternary
quantization
distillation
edge-deployment
Model card Files
xet
 Community
ternlight / README.md
wenshutang's picture
wenshutang
Upload README.md with huggingface_hub
8e4cb58 verified
preview code
|
Raw
History Blame Contribute Delete
10.1 kB
metadata 
license: mit
language: en
base_model: sentence-transformers/all-MiniLM-L6-v2
base_model_relation: quantized
pipeline_tag: sentence-similarity
tags:
  - sentence-embeddings
  - sentence-similarity
  - semantic-search
  - on-device
  - wasm
  - webassembly
  - bitnet
  - ternary
  - quantization
  - distillation
  - edge-deployment

ternlight

A 1.58-bit BitNet-style sentence embedding model distilled from sentence-transformers/all-MiniLM-L6-v2 via quantization-aware training, with post-training int4 quantization at the embedding layer. Weights are ternary {-1, 0, +1}, so inference is adds and subtracts rather than float matmuls, and the whole model — engine, tokenizer, and weights — ships as a single WASM bundle that runs on CPU with no API calls, no GPU, and no runtime download.

ternlight ships in two tiers, same API and same 384-dim output — pick by the size/quality trade. It is designed for short-string semantic similarity (search queries, intent classification, FAQ matching, product cards) deployed on-device (browser, Node, edge runtimes, ARM single-board computers). It is not a frontier model; it trades absolute quality for size and on-device deployability.

Tiers

Tier Architecture Params Wire (gzip) Latency (p50, CPU) Throughput Spearman vs teacher SciFact NDCG@10
base ⭐ 2-layer · d_model 384 · 6 heads ~15.4M 7.2 MB 5.1 ms ~195 emb/s 0.844 0.465
mini 2-layer · d_model 256 · 4 heads ~9.5M 5.0 MB 2.5 ms ~400 emb/s 0.820 0.439
teacher (MiniLM-L6) 6-layer · d_model 384 ~22.7M ~90 MB (fp32) — — 1.000 (ref) —
  • base — the quality tier. ~12× smaller on the wire than the fp32 teacher while retaining 0.84 rank-correlation. Use when quality matters more than the last 2 MB.
  • mini — the small/fast tier. ~1.6× the throughput at 5 MB, a modest quality step down.

Both tiers: ternary linear weights + int4 embedding table, 384-dim L2-normalized output, 128-token max input, BERT WordPiece vocabulary (30,522, identical to the teacher). Numbers measured on the shipped int4 builds (M-series Mac, Node single-threaded).

How to use

ternlight runs via a custom Rust→WASM inference engine, not via the transformers library. Two paths:

Path 1 — via npm (recommended) 

npm install @ternlight/base    # quality tier — 7 MB wire, ~5 ms/embed
npm install @ternlight/mini    # small tier   — 5 MB wire, ~2.5 ms/embed

import { embed, cosineSim, similar } from '@ternlight/base';   // or '@ternlight/mini'

const v1 = embed("arctic terns migrate from pole to pole");
const v2 = embed("longest migration in the animal kingdom");
cosineSim(v1, v2);   // ~0.71 — semantically related, different wording

// Nearest-neighbor search over a corpus
similar("which seabird travels farthest", corpus, { topK: 5 });

The model and tokenizer are bundled into each npm package — no separate download. Works in Node ≥ 18, browsers (via any bundler), Cloudflare Workers, Vercel Edge, Deno, and Bun.

Path 2 — direct download 

from huggingface_hub import hf_hub_download

# pick a tier (adjust the paths to match this repo's file layout)
model_bin = hf_hub_download(repo_id="wenshutang/ternlight", filename="base/model-int4.bin")
tokenizer = hf_hub_download(repo_id="wenshutang/ternlight", filename="tokenizer.json")

The .bin files are a custom BitNet b1.58 format. See the engine source for the binary layout and reference forward pass if you want to implement a custom loader in another language or runtime — there is no transformers.AutoModel.from_pretrained() path.

Model details

Property base mini
Layers 2-layer Transformer encoder 2-layer Transformer encoder
d_model 384 256
Attention heads 6 4
Parameters ~15.4M ~9.5M
Output dimension 384 (L2-normalized) 384 (L2-normalized)
Max input 128 tokens (~95 English words; longer inputs are silently truncated) same
Vocabulary 30,522 (BERT WordPiece, identical to teacher) same
Linear weights Ternary {-1, 0, +1} + per-matrix fp32 scale same
Embedding table 4-bit per-row PTQ + per-row fp32 scale same

Training

Distilled from sentence-transformers/all-MiniLM-L6-v2:

  1. Distillation objective — cosine/MSE loss between student and teacher 384-dim embeddings over ~1M sentence pairs (search queries, paraphrases, statements).
  2. BitNet b1.58 quantization-aware training — all linear layers use ternary weights trained end-to-end with the straight-through estimator. Training under the quantization constraint from the start (rather than quantizing post-hoc) is what preserves fidelity: a naive post-training ternary quant of the same encoder drops sharply, so nearly all of the retained quality comes from the QAT, not the format.
  3. Post-training int4 quantization — applied to the token embedding table after QAT, chosen via an ablation over int8/int4/ternary on the table. The embedding table dominates parameter count, so compressing it gives the largest size win for the smallest quality cost.

Training data (base): mix_v3_1M — ~1M pairs from MS MARCO, Quora duplicates, AllNLI, GooAQ, and StackExchange duplicates (English-only, seed=42). See configs/mix-v3-robust.yaml.

Provenance

base (model-int4.bin)

Training run robust-d384-mixv3-ep40
Source checkpoint checkpoint_ep40.pt
Source code commit 178e227
Data manifest mix_v3_1M (seed 42)
Packed at 2026-07-05
Bin size 7,492,312 bytes
SHA-256 68cc2c43…d2ee1db8
Gate (2026-07-04) test/spearman 0.851, ndcg@10 0.4665 (int8 form)

mini (model-int4.bin) — ⚠️ verify against the currently shipped @ternlight/mini build; this sidecar predates the 2026-07-05 package rebuild.

Training run qat-resume-ep10-ep40
Source checkpoint checkpoint_ep40.pt
Source code commit dff16b1
Packed at 2026-06-03
Bin size 4,839,512 bytes
SHA-256 07d8cfdb…f2e5b6c98

Each .bin ships with a .bin.json sidecar containing full provenance for reproducibility.

Evaluation

Fidelity — Spearman rank correlation vs teacher. Held-out queries, 1,000 deterministic random pairs, seed=42. Spearman of 1.0 = ranks pair similarities identically to the teacher.

Retrieval — SciFact NDCG@10. Absolute retrieval quality on the BEIR SciFact task.

Model Spearman vs teacher SciFact NDCG@10
MiniLM-L6 (teacher) 1.000 (ref) —
ternlight base ⭐ 0.844 0.465
ternlight mini 0.820 0.439

Full methodology and reproduction scripts: eval/quality/RESULTS.md.

Intended use

Designed for:

  • Short-string semantic similarity (queries, intents, FAQs, product titles, tags)
  • On-device deployment — browsers, Node services, Cloudflare Workers, Deno Deploy, Vercel Edge, Raspberry Pi-class ARM single-board computers
  • Cost-free embedding at any scale (no per-call API charges)
  • Privacy-sensitive workloads where queries cannot leave the user's device

Not designed for:

  • Long-document understanding (max input is 128 tokens — silently truncated above)
  • Multilingual workloads (English-only, inherited from MiniLM-L6)
  • Maximum absolute quality (use a frontier model like text-embedding-3-large or voyage-3 if quality dominates over size and deployability)

Limitations

  • English-only: teacher, tokenizer (bert-base-uncased, no CJK vocab), and training data are English. Non-English text will not tokenize or embed sensibly. Multilingual is the most-requested feature and the pipeline is language-agnostic, but it is not done yet.
  • 128-token cap: text longer than 128 BERT WordPiece tokens is silently truncated. Embed at sentence or short-paragraph granularity, not full document.
  • Custom runtime required: no transformers path. Use the npm packages or implement a custom loader from the binary format.
  • Inherited biases: distilled from all-MiniLM-L6-v2; the same demographic and topical bias caveats from the sentence-transformers corpus apply.
  • v0.1: the binary format and JS API may change before v1.0.

License

MIT, matching the teacher model and the ternlight project. See LICENSE.

Citation 

@software{ternlight2026,
  title  = {ternlight: a 1.58-bit BitNet sentence embedder in a few MB of WASM},
  author = {Tang, Wen Shu},
  year   = {2026},
  url    = {https://github.com/soycaporal/ternlight}
}

ternlight builds on:

Links