2b: data -> private repo; pointer in data/README + root banner

4f8fc41 verified 22 days ago

4.41 kB

---
base_model: Qwen/Qwen3-Embedding-0.6B
base_model_relation: adapter
library_name: peft
pipeline_tag: feature-extraction
license: apache-2.0
pretty_name: Jev topic head — Qwen3-Embedding-0.6B (LoRA + prototype head)
tags:
  - embeddings
  - text-classification
  - distillation
  - knowledge-distillation
  - generated-code
  - status:scaffold
---
# Jev topic classifier — local replacement

> 📄 **Review & open questions live here:** https://huggingface.co/spaces/Kodep/jev-review — our read of the spike report, the numbers that matter, and the questions to confirm in the code.

> 🔒 **Data is kept private & separate:** https://huggingface.co/datasets/Kodep/jev-topic-data — public code/weights, but the raw item text (school-email-derived) is gated. The `data/` folder in this repo is only a pointer.

Replaces the hosted **Jev** step of the school-email digest pipeline: classify each item into **one of 56 topics** so the fixed rules table can decide show/hide.

**Chosen approach** (per the baseline report): `Qwen/Qwen3-Embedding-0.6B` + **LoRA (rank 16)** + a **prototype head** (56 learned vectors; score = cosine similarity × learned scale), running on **CPU** — ~0.2 s/item, ~2.2 GiB RAM, 0 GPU.

> **STATUS: SCAFFOLD.** This repo is ready to receive the code, data, and weights from the owner. Nothing owner-side has been pushed yet. The card metadata above is provisional until the real LoRA adapter + head weights land. The numbers in `docs/report_claude_v1.md` are **Claude-written and unverified** — treat them as claims to confirm against the code and a re-run of `evaluate.py`, not as established facts.

## Intended layout
```
jev-topic-head/
├── README.md
├── docs/report_claude_v1.md     # baseline claims (Claude-written, unverified)
├── config.yaml                  # all hyperparameters          ← to push
├── data/                        # → pointer to private repo Kodep/jev-topic-data (not stored here)
├── src/                         # train_head.py, evaluate.py, rules_table.json ← to push
├── checkpoints/                 # LoRA adapter + head prototypes (seed 0)   ← to push
└── eval/                        # per-item test dump + confusion matrix    ← to push
```

Data, the topic taxonomy, and the human ("gold") labels live in the **private** dataset repo [`Kodep/jev-topic-data`](https://huggingface.co/datasets/Kodep/jev-topic-data). To pull them for a run (collaborators only): `hf download Kodep/jev-topic-data --repo-type dataset --local-dir data`.

## What unblocks verification (priority order)
1. **`src/train_head.py` + `config.yaml`** — confirm the actual training matches the report's claims (checklist below).
2. **`eval/per_item_test.jsonl`** — recompute top-1 accuracy, macro-F1, and show/hide without re-running the model.
3. **`checkpoints/` + data (private repo)** — reproduce end to end and re-confirm the CPU latency/RAM.
## Claims to verify against the code
- [ ] LoRA on attention + feed-forward layers, **rank 16 / alpha 32 / dropout 0.05**; rest frozen
- [ ] Head = **56 prototype vectors**, score = cosine(item, proto) × learned scale (scale init **20**)
- [ ] Prototypes initialized from the **embedding of each topic description** (14 topics have no training item)
- [ ] Loss = softmax cross-entropy, **label smoothing 0.05**, single-label (not multi-label)
- [ ] AdamW: LoRA lr **3e-4** (wd 0.01), head lr **2e-3** (no wd); warmup 10% → linear decay to 5%
- [ ] Batch **16**, ≤ **12 epochs**, early stop on **validation macro-F1**, **bf16**
- [ ] Class-balanced sampling (weight **1/√count**), hard topic pairs ×2, **no augmentation / no synthetic items**
- [ ] **3 seeds**; macro-F1 spanned **0.49–0.59** across seeds; shipped run = **seed 0**
- [ ] "87.6%" = **top-1 topic accuracy vs Jev's fresh answer** on test (92/105); **macro-F1 0.572**; show/hide 95/105 (90.5%)
## Headline caveat
All scores are measured **against Jev**, not ground truth. Jev matches itself only 97.1% of the time and matched a human label 11/20 on one 20-item set. Only **31 items** have independent human labels, and the test split was used for decisions (treated as dev data). A fresh, unopened 100-item set is the intended final benchmark.
## Reproduction
_TBD once `src/` lands — will document exact commands (env, base-model path, seed, config) to reproduce train + eval._

	---
	base_model: Qwen/Qwen3-Embedding-0.6B
	base_model_relation: adapter
	library_name: peft
	pipeline_tag: feature-extraction
	license: apache-2.0
	pretty_name: Jev topic head — Qwen3-Embedding-0.6B (LoRA + prototype head)
	tags:
	- embeddings
	- text-classification
	- distillation
	- knowledge-distillation
	- generated-code
	- status:scaffold
	---

	# Jev topic classifier — local replacement

	> 📄 Review & open questions live here: https://huggingface.co/spaces/Kodep/jev-review — our read of the spike report, the numbers that matter, and the questions to confirm in the code.

	> 🔒 Data is kept private & separate: https://huggingface.co/datasets/Kodep/jev-topic-data — public code/weights, but the raw item text (school-email-derived) is gated. The `data/` folder in this repo is only a pointer.

	Replaces the hosted Jev step of the school-email digest pipeline: classify each item into one of 56 topics so the fixed rules table can decide show/hide.

	Chosen approach (per the baseline report): `Qwen/Qwen3-Embedding-0.6B` + LoRA (rank 16) + a prototype head (56 learned vectors; score = cosine similarity × learned scale), running on CPU — ~0.2 s/item, ~2.2 GiB RAM, 0 GPU.

	> STATUS: SCAFFOLD. This repo is ready to receive the code, data, and weights from the owner. Nothing owner-side has been pushed yet. The card metadata above is provisional until the real LoRA adapter + head weights land. The numbers in `docs/report_claude_v1.md` are Claude-written and unverified — treat them as claims to confirm against the code and a re-run of `evaluate.py`, not as established facts.

	## Intended layout
	```
	jev-topic-head/
	├── README.md
	├── docs/report_claude_v1.md # baseline claims (Claude-written, unverified)
	├── config.yaml # all hyperparameters ← to push
	├── data/ # → pointer to private repo Kodep/jev-topic-data (not stored here)
	├── src/ # train_head.py, evaluate.py, rules_table.json ← to push
	├── checkpoints/ # LoRA adapter + head prototypes (seed 0) ← to push
	└── eval/ # per-item test dump + confusion matrix ← to push
	```

	Data, the topic taxonomy, and the human ("gold") labels live in the private dataset repo [`Kodep/jev-topic-data`](https://huggingface.co/datasets/Kodep/jev-topic-data). To pull them for a run (collaborators only): `hf download Kodep/jev-topic-data --repo-type dataset --local-dir data`.

	## What unblocks verification (priority order)
	1. `src/train_head.py` + `config.yaml` — confirm the actual training matches the report's claims (checklist below).
	2. `eval/per_item_test.jsonl` — recompute top-1 accuracy, macro-F1, and show/hide without re-running the model.
	3. `checkpoints/` + data (private repo) — reproduce end to end and re-confirm the CPU latency/RAM.

	## Claims to verify against the code
	- [ ] LoRA on attention + feed-forward layers, rank 16 / alpha 32 / dropout 0.05; rest frozen
	- [ ] Head = 56 prototype vectors, score = cosine(item, proto) × learned scale (scale init 20)
	- [ ] Prototypes initialized from the embedding of each topic description (14 topics have no training item)
	- [ ] Loss = softmax cross-entropy, label smoothing 0.05, single-label (not multi-label)
	- [ ] AdamW: LoRA lr 3e-4 (wd 0.01), head lr 2e-3 (no wd); warmup 10% → linear decay to 5%
	- [ ] Batch 16, ≤ 12 epochs, early stop on validation macro-F1, bf16
	- [ ] Class-balanced sampling (weight 1/√count), hard topic pairs ×2, no augmentation / no synthetic items
	- [ ] 3 seeds; macro-F1 spanned 0.49–0.59 across seeds; shipped run = seed 0
	- [ ] "87.6%" = top-1 topic accuracy vs Jev's fresh answer on test (92/105); macro-F1 0.572; show/hide 95/105 (90.5%)

	## Headline caveat
	All scores are measured against Jev, not ground truth. Jev matches itself only 97.1% of the time and matched a human label 11/20 on one 20-item set. Only 31 items have independent human labels, and the test split was used for decisions (treated as dev data). A fresh, unopened 100-item set is the intended final benchmark.

	## Reproduction
	_TBD once `src/` lands — will document exact commands (env, base-model path, seed, config) to reproduce train + eval._