README.md

metadata

tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
widget:
  - text: There is talk of five years of austerity.
  - text: >-
      Vadym Boychenko, mayor of Mariupol, said that Russian forces have killed
      twice as many of the city's residents in the two months of the war as Nazi
      Germany did in its two years of occupation.
  - text: >-
      But by allowing Kosovo to separate relatively peacefully from Serbia, it
      caused little lasting damage.
  - text: >-
      Dubbed Satan 2 by Western analysts, the Sarmat missile is formidable,
      purportedly designed to deploy numerous nuclear warheads or other weapons
      from its main 100-tonne missile at hypersonic speed.
  - text: >-
      Hagel said that the "military prowess" of the Islamic State, coupled with
      its deep sources of financing, poses an unprecedented threat to the United
      States.
metrics:
  - f1_macro
  - f1_binary
pipeline_tag: text-classification
library_name: setfit
inference: true
base_model: codefuse-ai/F2LLM-v2-80M
model-index:
  - name: SetFit with codefuse-ai/F2LLM-v2-80M
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: f1_macro
            value: 0.8302631578947368
            name: F1_Macro
          - type: f1_binary
            value: 0.8105263157894737
            name: F1_Binary

SetFit with codefuse-ai/F2LLM-v2-80M

This is a SetFit model that can be used for Text Classification. This SetFit model uses codefuse-ai/F2LLM-v2-80M as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: codefuse-ai/F2LLM-v2-80M
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 40960 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
negative	'His first comment was simply a reflex allusion to the fact that French policy is to stay apart from the common military arrangements of the rest of Western Europe.' "The President and Chancellor agreed during their private meeting on the importance of modernizing the West's nuclear defenses and on the threat of growing Soviet nuclear power, French officials said." 'Senate ratification hearings are scheduled to start on January 25.'
positive	'And you have an escalation in an enormously tense relationship between two countries that have fought three wars, which means you have an enormous amount of destructive capability on a fairly short trigger.' "It was August 1991, the days and nights of the failed coup d'etat which was meant to restore Stalinism but which brought Boris Yeltsin to power." 'A crisis of international security emerges.'

Evaluation

Metrics

Label	F1_Macro	F1_Binary
all	0.8303	0.8105

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("fefofico/crisis_trained_f2llm_selection")
# Run inference
preds = model("There is talk of five years of austerity.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	22.4121	74

Label	Training Sample Count
negative	499
positive	360

Training Hyperparameters

• batch_size: (256, 256)
• num_epochs: (5, 5)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (1e-06, 1e-06)
• head_learning_rate: 0.003
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0074	1	0.4596	-
0.1481	20	0.4139	-
0.2963	40	0.3995	-
0.4444	60	0.369	-
0.5926	80	0.3209	-
0.7407	100	0.2825	-
0.8889	120	0.2615	-
1.0	135	-	0.2649
1.0370	140	0.2548	-
1.1852	160	0.2496	-
1.3333	180	0.245	-
1.4815	200	0.2373	-
1.6296	220	0.2326	-
1.7778	240	0.228	-
1.9259	260	0.2179	-
2.0	270	-	0.2277
2.0741	280	0.2057	-
2.2222	300	0.1982	-
2.3704	320	0.1884	-
2.5185	340	0.1752	-
2.6667	360	0.1639	-
2.8148	380	0.1526	-
2.9630	400	0.1425	-
3.0	405	-	0.1906
3.1111	420	0.1334	-
3.2593	440	0.1157	-
3.4074	460	0.1075	-
3.5556	480	0.0966	-
3.7037	500	0.0866	-
3.8519	520	0.0746	-
4.0	540	0.0704	0.1889
4.1481	560	0.0666	-
4.2963	580	0.0603	-
4.4444	600	0.0533	-
4.5926	620	0.0514	-
4.7407	640	0.0519	-
4.8889	660	0.0506	-
5.0	675	-	0.1930

Framework Versions

• Python: 3.12.13
• SetFit: 1.1.3
• Sentence Transformers: 3.4.1
• Transformers: 4.57.6
• PyTorch: 2.11.0+cu128
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}