Sync card and scripts from the monorepo

README.md

@@ -11,16 +11,10 @@ metrics:
   - accuracy
   - f1
 pipeline_tag: text-classification
+datasets:
+  - custom-synthetic
 language:
   - en
-library_name: transformers
-widget:
-  - text: "B.S. in Computer Science, Stanford University, 2021. GPA: 3.9. Relevant coursework: Machine Learning, Distributed Systems."
-    example_title: Education
-  - text: "Senior Software Engineer at Stripe, 2019-Present. Led migration of payment infrastructure to Kubernetes, reducing deploy times by 60%."
-    example_title: Experience
-  - text: "Python, TypeScript, React, PostgreSQL, Docker, AWS, Terraform"
-    example_title: Skills
 ---
 
 # Resume Section Classifier
@@ -76,23 +70,39 @@ Default configuration produces **1,920 examples** (80 base examples x 3 variants
 
 ## Metrics
 
-Evaluated on the held-out synthetic test set (192 examples, stratified, seed 42; run of 2026-07-12 on transformers 5.13):
+> **These figures are INDICATIVE only.** They are measured on a held-out split of the model's own **synthetic, self-generated** data (192 examples, stratified) — not a real-world benchmark. Because the train and test sets are produced by the same template-based generator, they share vocabulary and structure, so these scores **overstate** how the model will perform on real resumes.
 
-| Metric | Score |
-|--------|-------|
+Measured on the published checkpoint (4 epochs, seed 42, 192-example
+stratified synthetic test split):
+
+| Metric | Score (synthetic test set) |
+|--------|----------------------------|
 | Accuracy | 1.000 |
 | F1 (macro) | 1.000 |
 | F1 (weighted) | 1.000 |
-| Test loss | 0.195 |
-
-> **Read these numbers with care.** The test split is drawn from the same
-> template-based generator as the training data, so perfect scores mainly
-> reflect that the synthetic distribution is easy to separate — not
-> real-world performance. On a small spot-check of hand-written,
-> out-of-distribution snippets the model resolved 3/5 correctly, with
-> misclassifications skewing toward `summary` at low confidence. Expect
-> materially lower accuracy on real resumes; fine-tune on labeled
-> real-world sections for production use.
+| Precision (weighted) | 1.000 |
+| Recall (weighted) | 1.000 |
+| Eval loss | 0.195 |
+
+A perfect score here is evidence of the leak described above, not of quality:
+the test split is generated by the same templates as the training split.
+
+### Out-of-distribution spot-check
+
+To give the synthetic score some context, the checkpoint was run against eight
+hand-written snippets in the style of real resumes, one per section type, none
+drawn from the generator:
+
+| Result | Count |
+|--------|-------|
+| Correct | 6 / 8 |
+
+Both misses were low-confidence and collapsed toward the broader categories —
+an `experience` bullet read as `summary` (0.39), and a `projects` bullet read
+as `skills` (0.48). Treat **~0.75 on real text** as the more realistic
+expectation, and treat predictions under ~0.5 confidence as unreliable.
+
+> Note: exact metrics depend on the random seed and training run.
 
 ## Usage
 

data_generator.py

@@ -10,9 +10,7 @@ Author: Lorenzo Scaturchio (gr8monk3ys)
 
 import csv
 import random
-import itertools
 from pathlib import Path
-from typing import Optional
 
 # ---------------------------------------------------------------------------
 # Entity pools – used to fill templates with realistic variation
@@ -854,33 +852,20 @@ def save_to_csv(dataset: list[dict], path: str) -> None:
 
 def load_as_hf_dataset(dataset: list[dict]):
     """Convert to HuggingFace Dataset with train/val/test splits."""
-    from datasets import ClassLabel, Dataset, DatasetDict, Features, Value
+    from datasets import Dataset, DatasetDict
 
     ds = Dataset.from_list(dataset)
 
-    # Stratified splitting requires a ClassLabel column
-    class_label = ClassLabel(names=sorted(set(d["label"] for d in dataset)))
-    ds = ds.cast_column("label", class_label)
-
     # 80/10/10 split
     train_test = ds.train_test_split(test_size=0.2, seed=42, stratify_by_column="label")
     val_test = train_test["test"].train_test_split(test_size=0.5, seed=42, stratify_by_column="label")
 
-    splits = DatasetDict({
+    return DatasetDict({
         "train": train_test["train"],
         "validation": val_test["train"],
         "test": val_test["test"],
     })
 
-    # Downstream tokenization maps label names via label2id, so restore strings.
-    # Explicit output features are required: mapping into a ClassLabel column
-    # would silently re-encode the names back to ints.
-    return splits.map(
-        lambda batch: {"label": class_label.int2str(batch["label"])},
-        batched=True,
-        features=Features({"text": Value("string"), "label": Value("string")}),
-    )
-
 
 def get_label_mapping(dataset: list[dict]) -> tuple[dict, dict]:
     """Create label <-> id mappings."""

requirements.txt

@@ -1,8 +1,7 @@
-transformers>=4.36.0
-datasets>=2.16.0
-torch>=2.1.0
-scikit-learn>=1.3.0
-accelerate>=0.25.0
-evaluate>=0.4.0
-pandas>=2.0.0
-huggingface_hub>=0.20.0
+transformers>=4.36.0,<5.0.0
+datasets>=2.16.0,<4.0.0
+torch>=2.1.0,<3.0.0
+scikit-learn>=1.3.0,<2.0.0
+accelerate>=0.25.0,<2.0.0
+evaluate>=0.4.0,<1.0.0
+huggingface_hub>=0.20.0,<1.0.0

train.py

@@ -16,7 +16,6 @@ Usage:
 
 import json
 import logging
-import os
 import sys
 from pathlib import Path
 
@@ -219,6 +218,7 @@ def train(
 
     training_args = TrainingArguments(
         output_dir=output_dir,
+        overwrite_output_dir=True,
         # Training hyperparameters
         num_train_epochs=epochs,
         per_device_train_batch_size=batch_size,
@@ -261,7 +261,7 @@ def train(
         args=training_args,
         train_dataset=tokenized_dataset["train"],
         eval_dataset=tokenized_dataset["validation"],
-        processing_class=tokenizer,
+        tokenizer=tokenizer,
         data_collator=data_collator,
         compute_metrics=build_compute_metrics(id2label),
         callbacks=callbacks,