hsmashiana/distilled

SetFit with nreimers/albert-small-v2

This is a SetFit model trained on the ag_news dataset that can be used for Text Classification. This SetFit model uses nreimers/albert-small-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: nreimers/albert-small-v2
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 4 classes
• Training Dataset: ag_news

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	'Bangladesh paralysed by strikes Opposition activists have brought many towns and cities in Bangladesh to a halt, the day after 18 people died in explosions at a political rally.' 'Will Putin #39;s Power Play Make Russia Safer? Outwardly, Russia has not changed since the barrage of terrorist attacks that culminated in the school massacre in Beslan on Sept.' 'S African TV in beheading blunder Public broadcaster SABC apologises after news bulletin shows footage of American beheaded in Iraq.'
1	'Desiring Stability Redskins coach Joe Gibbs expects few major personnel changes in the offseason and wants to instill a culture of stability in Washington.' 'Mutombo says he #39;s being traded to Rockets; will back up, mentor <b>...</b> Dikembe Mutombo, 38, has agreed to a sign-and-trade deal that will send him from the Chicago Bulls to Houston in exchange for Eric Piatkowski, Adrian Griffin and Mike Wilks, the Houston Chronicle reports.' 'They #146;re in the wrong ATHENS -- Matt Emmons was focusing on staying calm. He should have been focusing on the right target.'
3	'U2 pitches for Apple New iTunes ads airing during baseball games Tuesday will feature the advertising-shy Irish rockers.' 'A Cosmic Storm: When Galaxy Clusters Collide Astronomers have found what they are calling the perfect cosmic storm, a galaxy cluster pile-up so powerful its energy output is second only to the Big Bang.' 'Computer Assoc. Cuts 800 Jobs Worldwide (AP) AP - Computer Associates International Inc. announced a restructuring plan Wednesday that would reduce its work force by 800 people worldwide, saving the business software maker #36;70 million annually once the plan is fully implemented.'
2	'Economy builds steam in KC Fed district The economy continued to strengthen in September and early October in the Great Plains and Rocky Mountain regions covered by the Tenth Federal Reserve District, the Federal Reserve Bank of Kansas City said Wednesday.' 'RBC Centura CEO steps down RALEIGH, NC - The head of RBC Centura Bank has stepped down, and his successor will run the bank out of Raleigh rather than Rocky Mount, where the bank is based.' 'Oracle acquisition of PeopleSoft leads flurry of deals NEW YORK (CBS.MW) -- US stocks closed higher Monday, with the Dow Jones Industrial Average ending at its best level in more than nine months amid better-than-expected economic data and merger-related optimism.'

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("hsmashiana/distilled")
# Run inference
preds = model("President Blasts Firing Notre Dame's outgoing president criticized the decision to fire Tyrone Willingham after just three seasons, saying he was surprised the coach was not given more time to try to succeed.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	14	38.204	143

Label	Training Sample Count
0	244
1	243
2	242
3	271

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0008	1	0.1226	-
0.04	50	0.0023	-
0.08	100	0.0001	-
0.12	150	0.0001	-
0.16	200	0.0	-
0.2	250	0.0	-
0.24	300	0.0	-
0.28	350	0.0	-
0.32	400	0.0	-
0.36	450	0.0	-
0.4	500	0.0	-
0.44	550	0.0	-
0.48	600	0.0	-
0.52	650	0.0	-
0.56	700	0.0	-
0.6	750	0.0	-
0.64	800	0.0	-
0.68	850	0.0	-
0.72	900	0.0	-
0.76	950	0.0	-
0.8	1000	0.0	-
0.84	1050	0.0	-
0.88	1100	0.0	-
0.92	1150	0.0	-
0.96	1200	0.0	-
1.0	1250	0.0	-

Framework Versions

• Python: 3.10.13
• SetFit: 1.0.3
• Sentence Transformers: 2.6.1
• Transformers: 4.39.2
• PyTorch: 2.2.2+cu118
• Datasets: 2.18.0
• Tokenizers: 0.15.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}