lipika-fontclip — natural-language search over Indic fonts

Describe a font in plain English, get back ranked font families across 14 scripts (Devanagari, Bengali, Tamil, Telugu, Kannada, Malayalam, Gurmukhi, Gujarati, Odia, Sinhala, Tibetan, Ol Chiki, Meetei Mayek, Latin). Part of the Lipika Indic font-intelligence family.

"a very heavy Devanagari font for posters"  → Gajraj One, Tillana, Modak…
"a calligraphic Bengali font"               → Ekushey Punarbhaba, Sharifa…
"a friendly rounded Gurmukhi font"          → Baloo Paaji 2, Mukta Mahee…

This is, to our knowledge, the first open font↔language embedding model for Indic scripts (prior art: FontCLIP, Latin-centric).

🖥️ Try it live

→ Gradio demo: anilpai/lipika-demo — the Search by description tab runs this model. Results render live in each font (hosted WOFF2), with attribute chips, similarity bars, match counts, a script filter, and a toggle for legacy 90s DTP fonts. The Recognize tab is the companion image→font recognizer.

What's in this repo

file	description
adapter/	LoRA adapter (r16, q/v proj, last 8 blocks of both towers) for openai/clip-vit-large-patch14
index.npz	Precomputed embeddings for 595 font families × up to 3 faces (regular / heaviest / lightest) × per-script specimens (1250 entries), plus per-family metadata: attribute scores, scripts, legacy flag, legacy encoding
attributes.json	46 visual-attribute scores (0–100) per family (VLM-labeled, two-pass fused)
vocab.json	The attribute vocabulary
specimens/	596 rendered specimen banners (1024×220 PNG), one per family — preview images used by the demo when no hosted WOFF2 exists

Quickstart (lipika package — recommended)

pip install "lipika[search] @ git+https://github.com/Loopdesk-AI/lipika.git"

from pathlib import Path
from huggingface_hub import snapshot_download
from fontrecog.fontclip.search import FontClipSearcher

fc = Path(snapshot_download("loopdesk-ai/lipika-fontclip",
                            allow_patterns=["index.npz", "adapter/*"]))
searcher = FontClipSearcher(fc / "index.npz", ckpt=str(fc / "adapter"))

hits = searcher.search("a thin delicate Devanagari font",
                       k=5, script="devanagari")
# [{'family': 'Khula', 'score': 0.21, 'matched_script': 'devanagari',
#   'matched_face': 'light', 'scripts': ['devanagari', 'latin']}, ...]

stats = searcher.search_with_stats("a round playful Tamil font",
                                   k=10, script="tamil")
# {'results': [...], 'n_candidates': 31, 'n_shown': 10}

# Legacy (non-Unicode 90s DTP) families — 326 of 595, e.g. Kruti Dev,
# DevLys — are excluded by default; opt in explicitly:
searcher.search("a retro 90s desktop-publishing Hindi font",
                script="devanagari", include_legacy=True)

Quickstart (raw transformers + peft)

import json, numpy as np, torch, torch.nn.functional as F
from huggingface_hub import hf_hub_download
from transformers import CLIPModel, CLIPTokenizerFast
from peft import PeftModel

repo = "loopdesk-ai/lipika-fontclip"
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
model = PeftModel.from_pretrained(model, repo,
                                  subfolder="adapter").merge_and_unload().eval()
tok = CLIPTokenizerFast.from_pretrained("openai/clip-vit-large-patch14")

z = np.load(hf_hub_download(repo, "index.npz"))
E = torch.from_numpy(z["embeddings"])
fams = [str(x) for x in z["families"]]
scripts = [str(x) for x in z["scripts"]]
meta = json.loads(z["meta"].tobytes().decode())["families"]

q = "a thin delicate Devanagari font"
with torch.no_grad():
    t = F.normalize(model.get_text_features(
        **tok([q], return_tensors="pt")), dim=-1)[0]
sims = E @ t
best = {}
for i in sims.argsort(descending=True).tolist():
    f = fams[i]
    if scripts[i] != "devanagari" or meta[f].get("legacy"):
        continue
    best.setdefault(f, sims[i].item())
    if len(best) == 5:
        break
print(list(best))

Tested with transformers==4.49.0, peft>=0.14. (transformers>=5 changed the get_text_features return type — use <5 or the lipika package.)

Index metadata

index.npz → meta (JSON) → families[family]:

key	description
attrs	46 attribute scores 0–100 (family-fused)
scripts	scripts covered by the family
files	font file per indexed face (rep/heavy/light)
legacy	true for non-Unicode 90s DTP families
encoding	legacy cmap encoding (e.g. krutidev010) or null — text must be converted before rendering in these fonts

Each family is indexed with up to 3 static faces so weight-specific queries ("very heavy…") can match the right face; matched_face in results tells you which one won.

Training

• Data: ~13k (font file × script × specimen text) render combos from 595 open-license families (Google Fonts + SMC, Ekushey, RIT, Lohit, legacy DTP conversions), prompts sampled from 46 VLM-scored visual attributes with inverse-frequency boosting; per-file weight/width/italic overrides from OS/2 metrics.
• Recipe: LoRA (r16, α16) on q/v projections of the last 8 blocks of both towers; symmetric InfoNCE with same-family negatives masked; hinged cross-script pair loss (same family, different script → cos ≥ 0.9); hinged attribute-ranking loss against frozen (detached) text probes; batch 256, 4k steps on 1×H100, best checkpoint @2000.
• Trainable probes for the ranking loss were falsified (the text tower games the hinge and attribute directions collapse); detaching them is what made the loss help.

Evaluation

Frozen 50-query benchmark + held-out families (30 families never seen in training), multi-face index:

metric	zero-shot CLIP	lipika-fontclip
Text→font P@5 (norm., answerable queries)	0.150	0.450 (3.0×)
Attribute ranking (mean Spearman, held-out)	−0.075	0.365
Cross-script consistency (same family, cos)	0.618	0.884
Bold-above-light ordering	50.7%	100%

Zero-shot CLIP is near-chance on Indic typography (weight ordering is a coin flip; attribute correlations are negative) — fine-tuning is what makes this usable. 8/8 retrieval regression spot-checks pass.

Limitations

• Attribute labels are VLM-generated (Qwen2.5-VL-72B, two-pass fused), not human-annotated; treat scores as soft signals.
• "Serif/high-contrast" concepts are weakly represented in the corpus and retrieval quality reflects that.
• 326/595 families are legacy (non-Unicode) 90s DTP fonts (Kruti Dev, DevLys, …); they flood stylistic queries unless filtered — excluded by default in the lipika searcher, include_legacy=True restores them, and the legacy/encoding metadata supports custom handling.
• English queries only.

Related

• Demo Space: anilpai/lipika-demo
• Recognizer model (image→font): loopdesk-ai/lipika
• Code: Loopdesk-AI/lipika (fontrecog.fontclip — training, eval gates, index builder, searcher)

License

Apache-2.0 (adapter, index, labels, specimen images). Fonts referenced are under their own licenses (mostly OFL); no font binaries are redistributed here.