SetFit with codefuse-ai/F2LLM-v2-80M

This is a SetFit model that can be used for Text Classification. This SetFit model uses codefuse-ai/F2LLM-v2-80M as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: codefuse-ai/F2LLM-v2-80M
Classification head: a LogisticRegression instance
Maximum Sequence Length: 40960 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
negative	'His first comment was simply a reflex allusion to the fact that French policy is to stay apart from the common military arrangements of the rest of Western Europe.' "The President and Chancellor agreed during their private meeting on the importance of modernizing the West's nuclear defenses and on the threat of growing Soviet nuclear power, French officials said." 'Senate ratification hearings are scheduled to start on January 25.'
positive	'And you have an escalation in an enormously tense relationship between two countries that have fought three wars, which means you have an enormous amount of destructive capability on a fairly short trigger.' "It was August 1991, the days and nights of the failed coup d'etat which was meant to restore Stalinism but which brought Boris Yeltsin to power." 'A crisis of international security emerges.'

Label

Examples

negative

'His first comment was simply a reflex allusion to the fact that French policy is to stay apart from the common military arrangements of the rest of Western Europe.'
"The President and Chancellor agreed during their private meeting on the importance of modernizing the West's nuclear defenses and on the threat of growing Soviet nuclear power, French officials said."
'Senate ratification hearings are scheduled to start on January 25.'

positive

'And you have an escalation in an enormously tense relationship between two countries that have fought three wars, which means you have an enormous amount of destructive capability on a fairly short trigger.'
"It was August 1991, the days and nights of the failed coup d'etat which was meant to restore Stalinism but which brought Boris Yeltsin to power."
'A crisis of international security emerges.'

Evaluation

Metrics

Label	F1_Macro	F1_Binary
all	0.8303	0.8105

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("fefofico/crisis_trained_f2llm_selection")
# Run inference
preds = model("There is talk of five years of austerity.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	22.4121	74

Label	Training Sample Count
negative	499
positive	360

Training Hyperparameters

batch_size: (256, 256)
num_epochs: (5, 5)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (1e-06, 1e-06)
head_learning_rate: 0.003
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0074	1	0.4596	-
0.1481	20	0.4139	-
0.2963	40	0.3995	-
0.4444	60	0.369	-
0.5926	80	0.3209	-
0.7407	100	0.2825	-
0.8889	120	0.2615	-
1.0	135	-	0.2649
1.0370	140	0.2548	-
1.1852	160	0.2496	-
1.3333	180	0.245	-
1.4815	200	0.2373	-
1.6296	220	0.2326	-
1.7778	240	0.228	-
1.9259	260	0.2179	-
2.0	270	-	0.2277
2.0741	280	0.2057	-
2.2222	300	0.1982	-
2.3704	320	0.1884	-
2.5185	340	0.1752	-
2.6667	360	0.1639	-
2.8148	380	0.1526	-
2.9630	400	0.1425	-
3.0	405	-	0.1906
3.1111	420	0.1334	-
3.2593	440	0.1157	-
3.4074	460	0.1075	-
3.5556	480	0.0966	-
3.7037	500	0.0866	-
3.8519	520	0.0746	-
4.0	540	0.0704	0.1889
4.1481	560	0.0666	-
4.2963	580	0.0603	-
4.4444	600	0.0533	-
4.5926	620	0.0514	-
4.7407	640	0.0519	-
4.8889	660	0.0506	-
5.0	675	-	0.1930

Framework Versions

Python: 3.12.13
SetFit: 1.1.3
Sentence Transformers: 3.4.1
Transformers: 4.57.6
PyTorch: 2.11.0+cu128
Datasets: 5.0.0
Tokenizers: 0.22.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}