Upload ThingsLID multi-task model (epoch 3, avg acc 88.0%)

Browse files

Files changed (5) hide show

README.md +133 -0
label_maps.json +395 -0
model.pt +3 -0
model.py +86 -0
task_configs.json +6 -0

README.md ADDED Viewed

	@@ -0,0 +1,133 @@

+---
+language: multilingual
+license: apache-2.0
+tags:
+- language-identification
+- text-classification
+- programming-language-detection
+- spam-detection
+- toxicity-detection
+- byte-level
+- multi-task
+- lightweight
+pipeline_tag: text-classification
+---
+# ThingsLID — Multi-Task Byte-Level Text Classifier
+**4M parameters** · **5 tasks** · **No tokenizer needed** · **Runs on CPU**
+ThingsLID is an ultra-compact multi-task text classification model that operates directly on raw UTF-8 bytes. A single shared backbone serves 5 classification heads simultaneously.
+## Tasks & Performance
+| Task | Labels | Val Accuracy | Description |
+|------|--------|-------------|-------------|
+| `lang_id` | 351 | 94.3% | Language identification |
+| `prog_lang` | 30 | 94.4% | Programming language detection |
+| `spam` | 2 | 96.1% | Spam vs ham classification |
+| `toxic` | 2 | 67.3% | Toxicity detection (multilingual) |
+**Average validation accuracy: 88.0%**
+## Architecture
+ThingsLID uses a byte-level hybrid architecture inspired by [CommonLingua](https://huggingface.co/PleIAs/CommonLingua):
+- **Input**: Raw UTF-8 bytes (no tokenizer), padded to 512 bytes
+- **Trigram hash embedding**: Polynomial rolling hash of byte 3-grams → 8192-bucket embedding table
+- **Byte unigram embedding**: Standard embedding for individual bytes
+- **4× Conv1D blocks**: Causal convolutions + BatchNorm + GELU + residual
+- **2× Bidirectional attention**: Multi-head self-attention with RoPE
+- **Global average pooling** → per-task classification heads
+Total: ~4M shared parameters + small per-task linear heads.
+## Usage
+```python
+import torch
+from huggingface_hub import hf_hub_download
+# Download and load
+ckpt_path = hf_hub_download("ThingAI/ThingsLID", "model.pt")
+ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
+# Reconstruct model (see model.py in repo)
+from model import MultiTaskLID
+model = MultiTaskLID(ckpt["task_configs"]).eval()
+model.load_state_dict(ckpt["model"])
+# Predict
+def predict(text, task="lang_id", top_k=3):
+    raw = text.encode("utf-8")[:512]
+    byte_ids = list(raw) + [0] * (512 - len(raw))
+    inp = torch.tensor([byte_ids], dtype=torch.long)
+    with torch.no_grad():
+        logits = model(inp, task)["logits"][0]
+    probs = torch.softmax(logits, dim=-1)
+    idx2label = {i: l for l, i in ckpt["label_maps"][task].items()}
+    topk = probs.topk(top_k)
+    return [(idx2label[topk.indices[i].item()], topk.values[i].item()) for i in range(top_k)]
+# Examples
+print(predict("La pizza napoletana è patrimonio UNESCO", task="lang_id"))
+# [('ita', 0.98), ('cos', 0.003), ...]
+print(predict("def foo(x): return x + 1", task="prog_lang"))
+# [('Python', 0.95), ...]
+print(predict("You won a FREE iPhone!!!", task="spam"))
+# [('spam', 0.99), ...]
+```
+## Quick predict script
+```bash
+python predict.py --text "Ciao, come stai?"
+# lang_id: ita (98.2%)
+python predict.py --task prog_lang --text "fn main() { println!("hello"); }"
+# prog_lang: Rust (97.1%)
+python predict.py --task spam --text "URGENT: Click here to win!"
+# spam: spam (99.5%)
+```
+## Training Data
+| Task | Dataset | Samples |
+|------|---------|---------|
+| Language ID | [PleIAs/CommonLingua-Train](https://huggingface.co/datasets/PleIAs/CommonLingua-Train) | 200K (capped) |
+| Programming Language | [cakiki/rosetta-code](https://huggingface.co/datasets/cakiki/rosetta-code) | ~26K |
+| Spam Detection | [ucirvine/sms_spam](https://huggingface.co/datasets/ucirvine/sms_spam) | 5.5K |
+| Toxicity | [textdetox/multilingual_toxicity_dataset](https://huggingface.co/datasets/textdetox/multilingual_toxicity_dataset) | ~45K |
+## Key Design Decisions
+- **Byte-level input**: No tokenizer means it works on any language, script, or encoding without preprocessing
+- **Trigram hashing**: +1.2 F1 over unigram-only baseline, acts as regularization via hash collisions
+- **Multi-task learning**: Shared backbone learns universal text representations; per-task heads are tiny (~130K params each)
+- **No attention masking**: Bidirectional attention for classification (not causal)
+- **OneCycleLR**: Fast convergence in few epochs
+## Limitations
+- Designed for paragraph-level classification (50+ bytes). Short texts (<20 bytes) may be unreliable
+- Toxicity detection accuracy is lower than specialized models (trained on limited multilingual data)
+- Programming language detection trained on Rosetta Code samples which have a specific style
+## Citation
+```bibtex
+@misc{thingslid2026,
+  author = {ThingAI},
+  title  = {ThingsLID: Multi-Task Byte-Level Text Classifier},
+  year   = {2026},
+  url    = {https://huggingface.co/ThingAI/ThingsLID}
+}
+```
+## License
+Apache 2.0

label_maps.json ADDED Viewed

	@@ -0,0 +1,395 @@

+{
+  "lang_id": {
+    "abk": 0,
+    "ace": 1,
+    "ach": 2,
+    "ady": 3,
+    "afr": 4,
+    "aka": 5,
+    "akk": 6,
+    "alt": 7,
+    "amh": 8,
+    "ami": 9,
+    "ang": 10,
+    "anp": 11,
+    "arb": 12,
+    "arc": 13,
+    "arg": 14,
+    "arq": 15,
+    "ary": 16,
+    "arz": 17,
+    "asm": 18,
+    "ast": 19,
+    "atj": 20,
+    "ava": 21,
+    "avk": 22,
+    "awa": 23,
+    "aym": 24,
+    "azb": 25,
+    "azj": 26,
+    "bak": 27,
+    "bam": 28,
+    "ban": 29,
+    "bar": 30,
+    "bbc": 31,
+    "bel": 32,
+    "bem": 33,
+    "ben": 34,
+    "bho": 35,
+    "bik": 36,
+    "bis": 37,
+    "bjn": 38,
+    "blk": 39,
+    "bod": 40,
+    "bos": 41,
+    "bpy": 42,
+    "bre": 43,
+    "bug": 44,
+    "bul": 45,
+    "bxr": 46,
+    "bzj": 47,
+    "cat": 48,
+    "cbk": 49,
+    "cdo": 50,
+    "ceb": 51,
+    "ces": 52,
+    "che": 53,
+    "chr": 54,
+    "chu": 55,
+    "chv": 56,
+    "chy": 57,
+    "ckb": 58,
+    "cor": 59,
+    "cos": 60,
+    "crh": 61,
+    "csb": 62,
+    "cym": 63,
+    "dag": 64,
+    "dan": 65,
+    "dcr": 66,
+    "deu": 67,
+    "dga": 68,
+    "din": 69,
+    "dip": 70,
+    "diq": 71,
+    "div": 72,
+    "dsb": 73,
+    "dty": 74,
+    "dzo": 75,
+    "egx": 76,
+    "egy": 77,
+    "ell": 78,
+    "elx": 79,
+    "eml": 80,
+    "eng": 81,
+    "epo": 82,
+    "est": 83,
+    "ett": 84,
+    "eus": 85,
+    "ewe": 86,
+    "ext": 87,
+    "fao": 88,
+    "fas": 89,
+    "fat": 90,
+    "fij": 91,
+    "fil": 92,
+    "fin": 93,
+    "fon": 94,
+    "fra": 95,
+    "fro": 96,
+    "frp": 97,
+    "frr": 98,
+    "fry": 99,
+    "ful": 100,
+    "fur": 101,
+    "gag": 102,
+    "gan": 103,
+    "gaz": 104,
+    "gcr": 105,
+    "gez": 106,
+    "gla": 107,
+    "gle": 108,
+    "glg": 109,
+    "glk": 110,
+    "glv": 111,
+    "gom": 112,
+    "gor": 113,
+    "got": 114,
+    "gpe": 115,
+    "grc": 116,
+    "gsw": 117,
+    "guc": 118,
+    "gug": 119,
+    "guj": 120,
+    "gur": 121,
+    "guw": 122,
+    "hak": 123,
+    "hat": 124,
+    "hau": 125,
+    "haw": 126,
+    "hbo": 127,
+    "hbs": 128,
+    "heb": 129,
+    "hif": 130,
+    "hin": 131,
+    "hit": 132,
+    "hrv": 133,
+    "hsb": 134,
+    "hun": 135,
+    "hye": 136,
+    "hyw": 137,
+    "ibo": 138,
+    "ido": 139,
+    "iku": 140,
+    "ile": 141,
+    "ilo": 142,
+    "ina": 143,
+    "ind": 144,
+    "inh": 145,
+    "ipk": 146,
+    "isl": 147,
+    "ita": 148,
+    "jam": 149,
+    "jav": 150,
+    "jbo": 151,
+    "jpn": 152,
+    "kaa": 153,
+    "kab": 154,
+    "kal": 155,
+    "kan": 156,
+    "kas": 157,
+    "kat": 158,
+    "kaz": 159,
+    "kbd": 160,
+    "kbp": 161,
+    "kcg": 162,
+    "khk": 163,
+    "khm": 164,
+    "kik": 165,
+    "kin": 166,
+    "kir": 167,
+    "kmr": 168,
+    "koi": 169,
+    "kok": 170,
+    "kom": 171,
+    "kon": 172,
+    "kor": 173,
+    "kpe": 174,
+    "krc": 175,
+    "ksh": 176,
+    "lad": 177,
+    "lao": 178,
+    "lat": 179,
+    "latex": 180,
+    "lav": 181,
+    "lbe": 182,
+    "lez": 183,
+    "lfn": 184,
+    "lij": 185,
+    "lim": 186,
+    "lin": 187,
+    "lit": 188,
+    "lld": 189,
+    "lmo": 190,
+    "lrc": 191,
+    "ltg": 192,
+    "ltz": 193,
+    "lug": 194,
+    "luo": 195,
+    "lzh": 196,
+    "mad": 197,
+    "mai": 198,
+    "mal": 199,
+    "mar": 200,
+    "mdf": 201,
+    "mhr": 202,
+    "min": 203,
+    "mkc": 204,
+    "mkd": 205,
+    "mlg": 206,
+    "mlt": 207,
+    "mni": 208,
+    "mnw": 209,
+    "mri": 210,
+    "mrj": 211,
+    "msa": 212,
+    "mwl": 213,
+    "mya": 214,
+    "myv": 215,
+    "mzn": 216,
+    "nah": 217,
+    "nan": 218,
+    "nap": 219,
+    "nav": 220,
+    "nds": 221,
+    "nep": 222,
+    "new": 223,
+    "nia": 224,
+    "nld": 225,
+    "nno": 226,
+    "nor": 227,
+    "nov": 228,
+    "nqo": 229,
+    "nrf": 230,
+    "nso": 231,
+    "nya": 232,
+    "nyn": 233,
+    "oci": 234,
+    "olo": 235,
+    "orm": 236,
+    "ory": 237,
+    "oss": 238,
+    "pag": 239,
+    "pam": 240,
+    "pan": 241,
+    "pap": 242,
+    "pcd": 243,
+    "pcm": 244,
+    "pdc": 245,
+    "peo": 246,
+    "pfl": 247,
+    "pih": 248,
+    "pli": 249,
+    "pms": 250,
+    "pnb": 251,
+    "pnt": 252,
+    "pol": 253,
+    "por": 254,
+    "pus": 255,
+    "pwn": 256,
+    "quy": 257,
+    "rcf": 258,
+    "rmy": 259,
+    "roh": 260,
+    "ron": 261,
+    "rue": 262,
+    "run": 263,
+    "rup": 264,
+    "rus": 265,
+    "sah": 266,
+    "san": 267,
+    "sas": 268,
+    "sat": 269,
+    "scn": 270,
+    "sgs": 271,
+    "shi": 272,
+    "shn": 273,
+    "sin": 274,
+    "skr": 275,
+    "slk": 276,
+    "slv": 277,
+    "sme": 278,
+    "smn": 279,
+    "smo": 280,
+    "sna": 281,
+    "snd": 282,
+    "som": 283,
+    "sot": 284,
+    "spa": 285,
+    "sqi": 286,
+    "srd": 287,
+    "srn": 288,
+    "srp": 289,
+    "ssw": 290,
+    "stq": 291,
+    "sun": 292,
+    "sux": 293,
+    "swe": 294,
+    "swh": 295,
+    "szl": 296,
+    "szy": 297,
+    "tah": 298,
+    "tam": 299,
+    "tat": 300,
+    "tay": 301,
+    "tcy": 302,
+    "tel": 303,
+    "tet": 304,
+    "tgk": 305,
+    "tgl": 306,
+    "tha": 307,
+    "tir": 308,
+    "tly": 309,
+    "ton": 310,
+    "tpi": 311,
+    "trv": 312,
+    "tsn": 313,
+    "tso": 314,
+    "tuk": 315,
+    "tum": 316,
+    "tur": 317,
+    "txb": 318,
+    "tyv": 319,
+    "udm": 320,
+    "uig": 321,
+    "ukr": 322,
+    "urd": 323,
+    "uzb": 324,
+    "uzn": 325,
+    "vec": 326,
+    "ven": 327,
+    "vep": 328,
+    "vie": 329,
+    "vls": 330,
+    "vol": 331,
+    "vro": 332,
+    "war": 333,
+    "wln": 334,
+    "wol": 335,
+    "wuu": 336,
+    "xal": 337,
+    "xcl": 338,
+    "xho": 339,
+    "xmf": 340,
+    "xog": 341,
+    "xto": 342,
+    "ydd": 343,
+    "yor": 344,
+    "yue": 345,
+    "zea": 346,
+    "zgh": 347,
+    "zha": 348,
+    "zho": 349,
+    "zul": 350
+  },
+  "prog_lang": {
+    "11l": 0,
+    "Ada": 1,
+    "C": 2,
+    "C#": 3,
+    "C++": 4,
+    "D": 5,
+    "Factor": 6,
+    "FreeBASIC": 7,
+    "Go": 8,
+    "Haskell": 9,
+    "J": 10,
+    "Java": 11,
+    "Julia": 12,
+    "Kotlin": 13,
+    "Lua": 14,
+    "Nim": 15,
+    "Perl": 16,
+    "Phix": 17,
+    "PicoLisp": 18,
+    "Python": 19,
+    "REXX": 20,
+    "Racket": 21,
+    "Raku": 22,
+    "Ruby": 23,
+    "Rust": 24,
+    "Scala": 25,
+    "Sidef": 26,
+    "Tcl": 27,
+    "Wren": 28,
+    "zkl": 29
+  },
+  "spam": {
+    "ham": 0,
+    "spam": 1
+  },
+  "toxic": {
+    "non-toxic": 0,
+    "toxic": 1
+  }
+}

model.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2f5b2455a714c855abb4cda333f3e12223bafefb6297610ce0a63a23daa6340b
+size 14823241

model.py ADDED Viewed

	@@ -0,0 +1,86 @@

+"""ThingsLID model definition — standalone, no dependencies beyond PyTorch."""
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+class TrigramHashEmbedding(nn.Module):
+    def __init__(self, n_buckets=8192, d_embed=64, prime=31):
+        super().__init__()
+        self.n_buckets, self.prime = n_buckets, prime
+        self.embed = nn.Embedding(n_buckets, d_embed)
+    def forward(self, x):
+        xp = F.pad(x.long(), (2, 0), value=0)
+        h = (xp[:, :-2] * self.prime * self.prime + xp[:, 1:-1] * self.prime + xp[:, 2:]) % self.n_buckets
+        return self.embed(h)
+class RoPE(nn.Module):
+    def __init__(self, head_dim, max_len=1024, theta=10000.0):
+        super().__init__()
+        inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self._build(max_len)
+    def _build(self, seq_len):
+        t = torch.arange(seq_len, device=self.inv_freq.device).float()
+        freqs = torch.outer(t, self.inv_freq); emb = torch.cat([freqs, freqs], dim=-1)
+        self.register_buffer("cos", emb.cos()[None, None], persistent=False)
+        self.register_buffer("sin", emb.sin()[None, None], persistent=False); self._max = seq_len
+    @staticmethod
+    def _rotate(x):
+        x1, x2 = x.chunk(2, dim=-1); return torch.cat([-x2, x1], dim=-1)
+    def forward(self, q, k):
+        T = q.size(2)
+        if T > self._max: self._build(T)
+        c, s = self.cos[:,:,:T], self.sin[:,:,:T]
+        return q*c + self._rotate(q)*s, k*c + self._rotate(k)*s
+class Attention(nn.Module):
+    def __init__(self, d_model, n_heads):
+        super().__init__()
+        self.n_heads, self.head_dim = n_heads, d_model // n_heads
+        self.qkv = nn.Linear(d_model, 3 * d_model); self.out = nn.Linear(d_model, d_model)
+        self.rope = RoPE(self.head_dim); self.norm = nn.LayerNorm(d_model)
+    def forward(self, x):
+        res = x; x = self.norm(x); B, T, C = x.shape
+        qkv = self.qkv(x).view(B, T, 3, self.n_heads, self.head_dim)
+        q, k, v = qkv.permute(2, 0, 3, 1, 4); q, k = self.rope(q, k)
+        out = F.scaled_dot_product_attention(q, k, v, is_causal=False)
+        return res + self.out(out.transpose(1, 2).contiguous().view(B, T, C))
+class ConvBlock(nn.Module):
+    def __init__(self, d_in, d_out, kernel=3):
+        super().__init__()
+        self.conv = nn.Conv1d(d_in, d_out, kernel, padding=kernel // 2)
+        self.bn = nn.BatchNorm1d(d_out)
+        self.residual = nn.Conv1d(d_in, d_out, 1) if d_in != d_out else nn.Identity()
+    def forward(self, x): return F.gelu(self.bn(self.conv(x))) + self.residual(x)
+class MultiTaskLID(nn.Module):
+    """
+    ThingsLID: Multi-task byte-level text classifier.
+    ~4M shared parameters + per-task classification heads.
+    """
+    def __init__(self, task_configs, max_len=512, d_byte=64, d_tri=64,
+                 n_buckets=8192, d_model=384, n_conv=4, n_attn=2, n_heads=6, dropout=0.0):
+        super().__init__()
+        self.max_len = max_len; self.d_model = d_model
+        self.byte_embed = nn.Embedding(256, d_byte)
+        self.tri_embed = TrigramHashEmbedding(n_buckets, d_tri)
+        self.proj = nn.Linear(d_byte + d_tri, d_model)
+        self.convs = nn.ModuleList([ConvBlock(d_model, d_model, 3) for _ in range(n_conv)])
+        self.attns = nn.ModuleList([Attention(d_model, n_heads) for _ in range(n_attn)])
+        self.drop = nn.Dropout(dropout); self.norm = nn.LayerNorm(d_model)
+        self.heads = nn.ModuleDict({t: nn.Linear(d_model, n) for t, n in task_configs.items()})
+    def forward(self, x, task):
+        h = self.proj(torch.cat([self.byte_embed(x), self.tri_embed(x)], dim=-1))
+        h = h.transpose(1, 2)
+        for conv in self.convs: h = conv(h)
+        h = h.transpose(1, 2)
+        for attn in self.attns: h = attn(h)
+        return {"logits": self.heads[task](self.drop(self.norm(h).mean(dim=1)))}

task_configs.json ADDED Viewed

	@@ -0,0 +1,6 @@

+{
+  "lang_id": 351,
+  "prog_lang": 30,
+  "spam": 2,
+  "toxic": 2
+}