SetFit with codefuse-ai/F2LLM-v2-80M

This is a SetFit model that can be used for Text Classification. This SetFit model uses codefuse-ai/F2LLM-v2-80M as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: codefuse-ai/F2LLM-v2-80M
Classification head: a LogisticRegression instance
Maximum Sequence Length: 40960 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
negative	'The more and the sooner we get the financial support we have requested, the sooner there will be peace.' 'The recent upgrade to the national power grid has significantly improved reliability, reducing the number of blackouts experienced by customers.' 'This is not an economic crisis.'
positive	'This problem is more complex than we initially understood.' 'Climate change is altering ocean currents, with unpredictable consequences for marine ecosystems.' 'The systematic underfunding of renewable energy research has left nations dangerously dependent on fossil fuels during global supply chain disruptions.'

Evaluation

Metrics

Label	F1_Macro	F1_Binary
all	0.8968	0.8756

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("fefofico/crisis_trained_f2llm_temp")
# Run inference
preds = model("We managed to prevent a possible crisis.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	18.3023	65

Label	Training Sample Count
negative	1307
positive	876

Training Hyperparameters

batch_size: (128, 128)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (1e-07, 1e-07)
head_learning_rate: 0.0001
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.3
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0015	1	0.4239	-
0.0293	20	0.432	-
0.0586	40	0.4262	-
0.0878	60	0.4188	-
0.1171	80	0.4131	-
0.1464	100	0.4274	-
0.1757	120	0.4332	-
0.2050	140	0.4136	-
0.2343	160	0.428	-
0.2635	180	0.4066	-
0.2928	200	0.4064	-
0.3221	220	0.4144	-
0.3514	240	0.4072	-
0.3807	260	0.4008	-
0.4100	280	0.3918	-
0.4392	300	0.4047	-
0.4685	320	0.3979	-
0.4978	340	0.4005	-
0.5271	360	0.3825	-
0.5564	380	0.3728	-
0.5857	400	0.3666	-
0.6149	420	0.3737	-
0.6442	440	0.3626	-
0.6735	460	0.3456	-
0.7028	480	0.3565	-
0.7321	500	0.3476	-
0.7613	520	0.3409	-
0.7906	540	0.3481	-
0.8199	560	0.3298	-
0.8492	580	0.3303	-
0.8785	600	0.3257	-
0.9078	620	0.328	-
0.9370	640	0.3195	-
0.9663	660	0.3183	-
0.9956	680	0.3067	-
1.0	683	-	0.3051
1.0249	700	0.3067	-
1.0542	720	0.3009	-
1.0835	740	0.2928	-
1.1127	760	0.2993	-
1.1420	780	0.288	-
1.1713	800	0.2892	-
1.2006	820	0.2934	-
1.2299	840	0.2817	-
1.2592	860	0.2818	-
1.2884	880	0.2857	-
1.3177	900	0.2807	-
1.3470	920	0.28	-
1.3763	940	0.2792	-
1.4056	960	0.277	-
1.4348	980	0.2783	-
1.4641	1000	0.2743	-
1.4934	1020	0.2748	-
1.5227	1040	0.2731	-
1.5520	1060	0.2744	-
1.5813	1080	0.2643	-
1.6105	1100	0.2742	-
1.6398	1120	0.2698	-
1.6691	1140	0.2681	-
1.6984	1160	0.2698	-
1.7277	1180	0.27	-
1.7570	1200	0.2642	-
1.7862	1220	0.2668	-
1.8155	1240	0.2641	-
1.8448	1260	0.2645	-
1.8741	1280	0.2642	-
1.9034	1300	0.2625	-
1.9327	1320	0.265	-
1.9619	1340	0.2619	-
1.9912	1360	0.2643	-
2.0	1366	-	0.2608
2.0205	1380	0.2661	-
2.0498	1400	0.2638	-
2.0791	1420	0.2637	-
2.1083	1440	0.2597	-
2.1376	1460	0.2639	-
2.1669	1480	0.2637	-
2.1962	1500	0.262	-
2.2255	1520	0.2595	-
2.2548	1540	0.2564	-
2.2840	1560	0.2618	-
2.3133	1580	0.2601	-
2.3426	1600	0.2585	-
2.3719	1620	0.2598	-
2.4012	1640	0.2614	-
2.4305	1660	0.2543	-
2.4597	1680	0.2595	-
2.4890	1700	0.2552	-
2.5183	1720	0.2565	-
2.5476	1740	0.2569	-
2.5769	1760	0.2605	-
2.6061	1780	0.2581	-
2.6354	1800	0.2579	-
2.6647	1820	0.2567	-
2.6940	1840	0.2516	-
2.7233	1860	0.2536	-
2.7526	1880	0.2545	-
2.7818	1900	0.2548	-
2.8111	1920	0.2585	-
2.8404	1940	0.2547	-
2.8697	1960	0.2495	-
2.8990	1980	0.2519	-
2.9283	2000	0.2547	-
2.9575	2020	0.2561	-
2.9868	2040	0.2535	-
3.0	2049	-	0.2526
3.0161	2060	0.2554	-
3.0454	2080	0.2495	-
3.0747	2100	0.2537	-
3.1040	2120	0.2513	-
3.1332	2140	0.2548	-
3.1625	2160	0.2562	-
3.1918	2180	0.258	-
3.2211	2200	0.2547	-
3.2504	2220	0.2521	-
3.2796	2240	0.2531	-
3.3089	2260	0.2532	-
3.3382	2280	0.2502	-
3.3675	2300	0.2486	-
3.3968	2320	0.2498	-
3.4261	2340	0.2529	-
3.4553	2360	0.2529	-
3.4846	2380	0.2469	-
3.5139	2400	0.2517	-
3.5432	2420	0.2506	-
3.5725	2440	0.2468	-
3.6018	2460	0.2517	-
3.6310	2480	0.2491	-
3.6603	2500	0.251	-
3.6896	2520	0.2547	-
3.7189	2540	0.2488	-
3.7482	2560	0.2492	-
3.7775	2580	0.2498	-
3.8067	2600	0.2521	-
3.8360	2620	0.2473	-
3.8653	2640	0.2504	-
3.8946	2660	0.2466	-
3.9239	2680	0.2486	-
3.9531	2700	0.249	-
3.9824	2720	0.2485	-
4.0	2732	-	0.2477
4.0117	2740	0.2494	-
4.0410	2760	0.2496	-
4.0703	2780	0.2487	-
4.0996	2800	0.2484	-
4.1288	2820	0.2453	-
4.1581	2840	0.2444	-
4.1874	2860	0.2486	-
4.2167	2880	0.2482	-
4.2460	2900	0.2491	-
4.2753	2920	0.2483	-
4.3045	2940	0.2498	-
4.3338	2960	0.2462	-
4.3631	2980	0.2451	-
4.3924	3000	0.2511	-
4.4217	3020	0.2464	-
4.4510	3040	0.2452	-
4.4802	3060	0.2472	-
4.5095	3080	0.2474	-
4.5388	3100	0.2482	-
4.5681	3120	0.2468	-
4.5974	3140	0.2511	-
4.6266	3160	0.2499	-
4.6559	3180	0.2498	-
4.6852	3200	0.2476	-
4.7145	3220	0.2471	-
4.7438	3240	0.2472	-
4.7731	3260	0.2464	-
4.8023	3280	0.245	-
4.8316	3300	0.2475	-
4.8609	3320	0.2473	-
4.8902	3340	0.2446	-
4.9195	3360	0.2436	-
4.9488	3380	0.2478	-
4.9780	3400	0.2453	-
5.0	3415	-	0.2459
0.0015	1	0.2562	-
0.0293	20	0.2496	-
0.0586	40	0.2473	-
0.0878	60	0.2431	-
0.1171	80	0.2425	-
0.1464	100	0.2458	-
0.1757	120	0.2435	-
0.2050	140	0.2381	-
0.2343	160	0.2391	-
0.2635	180	0.2353	-
0.2928	200	0.2353	-
0.3221	220	0.2351	-
0.3514	240	0.2302	-
0.3807	260	0.2299	-
0.4100	280	0.2227	-
0.4392	300	0.2264	-
0.4685	320	0.2243	-
0.4978	340	0.2247	-
0.5271	360	0.2195	-
0.5564	380	0.2177	-
0.5857	400	0.2127	-
0.6149	420	0.2164	-
0.6442	440	0.2152	-
0.6735	460	0.2102	-
0.7028	480	0.2102	-
0.7321	500	0.2104	-
0.7613	520	0.2104	-
0.7906	540	0.2121	-
0.8199	560	0.2068	-
0.8492	580	0.2039	-
0.8785	600	0.1995	-
0.9078	620	0.2029	-
0.9370	640	0.2051	-
0.9663	660	0.2049	-
0.9956	680	0.2062	-
1.0	683	-	0.2034
0.0015	1	0.2147	-
0.0293	20	0.2061	-
0.0586	40	0.2027	-
0.0878	60	0.1997	-
0.1171	80	0.1948	-
0.1464	100	0.1966	-
0.1757	120	0.1945	-
0.2050	140	0.1834	-
0.2343	160	0.1838	-
0.2635	180	0.1796	-
0.2928	200	0.1761	-
0.3221	220	0.1754	-
0.3514	240	0.1715	-
0.3807	260	0.1691	-
0.4100	280	0.1635	-
0.4392	300	0.1667	-
0.4685	320	0.164	-
0.4978	340	0.1639	-
0.5271	360	0.1522	-
0.5564	380	0.1515	-
0.5857	400	0.1535	-
0.6149	420	0.1534	-
0.6442	440	0.1546	-
0.6735	460	0.1523	-
0.7028	480	0.1477	-
0.7321	500	0.1504	-
0.7613	520	0.1485	-
0.7906	540	0.1521	-
0.8199	560	0.147	-
0.8492	580	0.1425	-
0.8785	600	0.138	-
0.9078	620	0.1414	-
0.9370	640	0.1462	-
0.9663	660	0.1435	-
0.9956	680	0.1462	-
1.0	683	-	0.1555
0.0015	1	0.1516	-
0.0293	20	0.1424	-
0.0586	40	0.1432	-
0.0878	60	0.1412	-
0.1171	80	0.1377	-
0.1464	100	0.1396	-
0.1757	120	0.1384	-
0.2050	140	0.1298	-
0.2343	160	0.13	-
0.2635	180	0.1312	-
0.2928	200	0.1277	-
0.3221	220	0.1277	-
0.3514	240	0.1278	-
0.3807	260	0.1269	-
0.4100	280	0.1228	-
0.4392	300	0.1257	-
0.4685	320	0.1253	-
0.4978	340	0.1238	-
0.5271	360	0.1121	-
0.5564	380	0.1131	-
0.5857	400	0.1184	-
0.6149	420	0.1176	-
0.6442	440	0.1183	-
0.6735	460	0.1189	-
0.7028	480	0.1136	-
0.7321	500	0.1177	-
0.7613	520	0.1145	-
0.7906	540	0.1175	-
0.8199	560	0.1162	-
0.8492	580	0.1101	-
0.8785	600	0.1065	-
0.9078	620	0.1098	-
0.9370	640	0.1136	-
0.9663	660	0.1119	-
0.9956	680	0.1161	-
1.0	683	-	0.1434
0.0015	1	0.1178	-
0.0293	20	0.1088	-
0.0586	40	0.112	-
0.0878	60	0.1098	-
0.1171	80	0.1086	-
0.1464	100	0.1097	-
0.1757	120	0.1099	-
0.2050	140	0.1034	-
0.2343	160	0.1047	-
0.2635	180	0.1067	-
0.2928	200	0.1037	-
0.3221	220	0.1031	-
0.3514	240	0.1061	-
0.3807	260	0.1047	-
0.4100	280	0.1024	-
0.4392	300	0.1039	-
0.4685	320	0.1057	-
0.4978	340	0.1031	-
0.5271	360	0.0931	-
0.5564	380	0.0948	-
0.5857	400	0.1006	-
0.6149	420	0.1003	-
0.6442	440	0.1004	-
0.6735	460	0.1018	-
0.7028	480	0.0976	-
0.7321	500	0.1017	-
0.7613	520	0.0981	-
0.7906	540	0.1011	-
0.8199	560	0.1006	-
0.8492	580	0.0949	-
0.8785	600	0.092	-
0.9078	620	0.095	-
0.9370	640	0.0982	-
0.9663	660	0.0974	-
0.9956	680	0.1023	-
1.0	683	-	0.1398
0.0015	1	0.1006	-
0.0293	20	0.0933	-
0.0586	40	0.0973	-
0.0878	60	0.0947	-
0.1171	80	0.0942	-
0.1464	100	0.0945	-
0.1757	120	0.0949	-
0.2050	140	0.089	-
0.2343	160	0.091	-
0.2635	180	0.092	-
0.2928	200	0.0893	-
0.3221	220	0.0883	-
0.3514	240	0.0921	-
0.3807	260	0.0899	-
0.4100	280	0.0887	-
0.4392	300	0.0884	-
0.4685	320	0.0913	-
0.4978	340	0.0881	-
0.5271	360	0.0789	-
0.5564	380	0.0809	-
0.5857	400	0.0864	-
0.6149	420	0.0864	-
0.6442	440	0.0855	-
0.6735	460	0.0869	-
0.7028	480	0.0836	-
0.7321	500	0.0874	-
0.7613	520	0.0834	-
0.7906	540	0.086	-
0.8199	560	0.0859	-
0.8492	580	0.0804	-
0.8785	600	0.0786	-
0.9078	620	0.0809	-
0.9370	640	0.0831	-
0.9663	660	0.0831	-
0.9956	680	0.0883	-
1.0	683	-	0.1367

Framework Versions

Python: 3.12.13
SetFit: 1.1.3
Sentence Transformers: 3.4.1
Transformers: 4.57.6
PyTorch: 2.11.0+cu128
Datasets: 5.0.0
Tokenizers: 0.22.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}