Upload folder using huggingface_hub

README.md

@@ -25,7 +25,7 @@ Multi-task code classification model for filtering large-scale code datasets. Bu
 
 - **Base model:** microsoft/unixcoder-base (125M params)
 - **Architecture:** Shared encoder + three task-specific linear heads
-- **Training data:** 191,776 code samples from bigcode/starcoderdata, labeled by GPT-5-nano (Batch API)
+- **Training data:** 191,776 code samples from bigcode/starcoderdata, labeled by GPT-5-nano (~$80 Batch API)
 - **Languages:** Python, JavaScript, TypeScript, Java, Go, Rust, SQL, Shell (25K per language)
 - **Training:** 3 epochs, batch size 16, lr 2e-5, AMP (bf16), torch.compile
 
@@ -111,44 +111,6 @@ For this filtering use case, what matters is **rank ordering**, not exact classi
 
 The primary bottleneck is **training data volume and class balance**, not model capacity:
 
-1. **Scale up the GPT-5-nano labeling set.** The current model was trained on 192K labeled samples (~$20 via Batch API). Doubling to 400K samples (~$40) would particularly help quality levels 2 and 5, where the model struggles most. Level 5 (excellent code) had only 2,345 training examples — far too few for the model to learn the pattern.
-
-2. **Oversample rare classes.** Content types like tutorial (2,197 samples), data (1,413), and generated (2,975) are underrepresented. A targeted labeling run that specifically seeks out these types — e.g., filtering by filename patterns like `*_test.*`, `*.generated.*`, `tutorial*` — would improve recall on rare types without relabeling the entire dataset.
+1. **Scale up the GPT-5-nano labeling set.** The current model was trained on 192K labeled samples. Doubling to 400K samples (~$80) would particularly help quality levels 2 and 5, where the model struggles most. Level 5 (excellent code) had only 2,345 training examples — far too few for the model to learn the pattern.
 
 3. **Increase max token length.** The current model uses 512 tokens, but code files often need more context to assess quality. Increasing to 1024 or 2048 tokens (UniXcoder supports up to 1024) would give the model more signal, particularly for quality assessment where style and documentation patterns emerge over longer spans.
-
-4. **Add a second training round with hard examples.** After running inference on the full StarCoderData, sample files where the model is least confident (prediction near the decision boundary, e.g., quality between 2.5 and 3.5) and send those to GPT-5-nano for labeling. Training on these hard cases would sharpen the model's performance exactly where filtering decisions are made.
-
-## Usage
-
-```python
-from train_starcoderdata import CodeClassifierModel, load_model
-from transformers import AutoTokenizer
-import torch
-
-model_dir = "models/starcoderdata-classifier"
-tokenizer = AutoTokenizer.from_pretrained(model_dir)
-model = load_model(model_dir)
-model.eval()
-
-code = "def hello(): print('world')"
-inputs = tokenizer(code, return_tensors="pt", truncation=True, max_length=512)
-with torch.no_grad():
-    quality, structured_data, content_type_logits = model(
-        inputs["input_ids"], inputs["attention_mask"]
-    )
-
-print(f"Quality: {quality.item():.1f}")
-print(f"Structured Data: {structured_data.item():.1f}")
-print(f"Content Type: {['library','application','test','config','tutorial','data','generated','script','other'][content_type_logits.argmax()]}")
-```
-
-## Files
-
-- `config.json`, `model.safetensors` — UniXcoder encoder weights (HuggingFace format)
-- `classifier_heads.pt` — Quality, structured data, and content type head weights
-- `tokenizer.json`, `tokenizer_config.json` — Tokenizer
-- `label_config.json` — Label definitions and task types
-- `test_metrics.json` — Full test set metrics
-- `training_history.csv` — Per-epoch training/validation metrics
-- `checkpoint.pt` — Full training checkpoint (for resume)