Upload 13 files

7be0c1e verified 12 months ago

38 kB

base_model: intfloat/multilingual-e5-large
library_name: setfit
metrics:
  - accuracy
  - precision
  - recall
  - f1
pipeline_tag: text-classification
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
widget:
  - text: |+
      men det kan så åbne nogle nye

  - text: |-
      som jeg siger, der jo en grund til at jeg har fået et
      handicapskilt og sådan
  - text: |+
      meget, ellers har jeg overholdt alt.

  - text: |
      sige det er 15 timer, du får betalte timer, jamen det er også en start,
  - text: og jo det er nok rigtigt, det er sådan, jeg skal gøre det
inference: true
model-index:
  - name: SetFit with intfloat/multilingual-e5-large
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.9724770642201835
            name: Accuracy
          - type: precision
            value: 0.9557522123893806
            name: Precision
          - type: recall
            value: 0.9908256880733946
            name: Recall
          - type: f1
            value: 0.972972972972973
            name: F1

SetFit with intfloat/multilingual-e5-large

This is a SetFit model that can be used for Text Classification. This SetFit model uses intfloat/multilingual-e5-large as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: intfloat/multilingual-e5-large
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
reported speech	'der fortalte jeg dig alt det der om ret og pligt ' 'Så havde vi de der snakke om, at du ligesom selv fik lov at styre dit\nNem-ID.' 'Amina sagde at min\ndumme mor havde ringet'
not reported speech	'sag. ' 'du klage over ik Lykke?\n\n' 'S: nej. '

Evaluation

Metrics

Label	Accuracy	Precision	Recall	F1
all	0.9725	0.9558	0.9908	0.9730

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("men det kan så åbne nogle nye

")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	19.1755	196

Label	Training Sample Count
not reported speech	265
reported speech	265

Training Hyperparameters

batch_size: (32, 32)
num_epochs: (6, 6)
max_steps: -1
sampling_strategy: oversampling
body_learning_rate: (1.0770502781075495e-06, 1.0770502781075495e-06)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0002	1	0.3495	-
0.0113	50	0.3524	-
0.0227	100	0.3496	-
0.0340	150	0.3464	-
0.0454	200	0.3419	-
0.0567	250	0.328	-
0.0681	300	0.3166	-
0.0794	350	0.3012	-
0.0908	400	0.277	-
0.1021	450	0.259	-
0.1135	500	0.2568	-
0.1248	550	0.2483	-
0.1362	600	0.2457	-
0.1475	650	0.2263	-
0.1589	700	0.2361	-
0.1702	750	0.2108	-
0.1816	800	0.2025	-
0.1929	850	0.1881	-
0.2043	900	0.1559	-
0.2156	950	0.1055	-
0.2270	1000	0.0693	-
0.2383	1050	0.0332	-
0.2497	1100	0.0287	-
0.2610	1150	0.0185	-
0.2724	1200	0.0421	-
0.2837	1250	0.0087	-
0.2951	1300	0.0233	-
0.3064	1350	0.0083	-
0.3177	1400	0.0043	-
0.3291	1450	0.0037	-
0.3404	1500	0.0033	-
0.3518	1550	0.0019	-
0.3631	1600	0.0016	-
0.3745	1650	0.0012	-
0.3858	1700	0.002	-
0.3972	1750	0.0014	-
0.4085	1800	0.0012	-
0.4199	1850	0.001	-
0.4312	1900	0.001	-
0.4426	1950	0.0037	-
0.4539	2000	0.0006	-
0.4653	2050	0.0009	-
0.4766	2100	0.001	-
0.4880	2150	0.0006	-
0.4993	2200	0.0007	-
0.5107	2250	0.0005	-
0.5220	2300	0.001	-
0.5334	2350	0.0006	-
0.5447	2400	0.0004	-
0.5561	2450	0.0003	-
0.5674	2500	0.0004	-
0.5788	2550	0.0005	-
0.5901	2600	0.0003	-
0.6015	2650	0.0003	-
0.6128	2700	0.0003	-
0.6241	2750	0.0003	-
0.6355	2800	0.0004	-
0.6468	2850	0.0003	-
0.6582	2900	0.0002	-
0.6695	2950	0.0003	-
0.6809	3000	0.0003	-
0.6922	3050	0.0003	-
0.7036	3100	0.0003	-
0.7149	3150	0.0002	-
0.7263	3200	0.0003	-
0.7376	3250	0.0001	-
0.7490	3300	0.0002	-
0.7603	3350	0.0002	-
0.7717	3400	0.0002	-
0.7830	3450	0.0002	-
0.7944	3500	0.0002	-
0.8057	3550	0.0002	-
0.8171	3600	0.0002	-
0.8284	3650	0.0002	-
0.8398	3700	0.0003	-
0.8511	3750	0.0002	-
0.8625	3800	0.0002	-
0.8738	3850	0.0002	-
0.8852	3900	0.0002	-
0.8965	3950	0.0002	-
0.9079	4000	0.0001	-
0.9192	4050	0.0002	-
0.9305	4100	0.0002	-
0.9419	4150	0.0001	-
0.9532	4200	0.0001	-
0.9646	4250	0.0001	-
0.9759	4300	0.0001	-
0.9873	4350	0.0002	-
0.9986	4400	0.0002	-
1.0	4406	-	0.0601
1.0100	4450	0.0001	-
1.0213	4500	0.0001	-
1.0327	4550	0.0001	-
1.0440	4600	0.0001	-
1.0554	4650	0.0001	-
1.0667	4700	0.0001	-
1.0781	4750	0.0001	-
1.0894	4800	0.0001	-
1.1008	4850	0.0001	-
1.1121	4900	0.0001	-
1.1235	4950	0.0001	-
1.1348	5000	0.0001	-
1.1462	5050	0.0002	-
1.1575	5100	0.0001	-
1.1689	5150	0.0001	-
1.1802	5200	0.0001	-
1.1916	5250	0.0001	-
1.2029	5300	0.0001	-
1.2143	5350	0.0001	-
1.2256	5400	0.0001	-
1.2369	5450	0.0001	-
1.2483	5500	0.0001	-
1.2596	5550	0.0001	-
1.2710	5600	0.0001	-
1.2823	5650	0.0001	-
1.2937	5700	0.0001	-
1.3050	5750	0.0001	-
1.3164	5800	0.0001	-
1.3277	5850	0.0001	-
1.3391	5900	0.0001	-
1.3504	5950	0.0001	-
1.3618	6000	0.0001	-
1.3731	6050	0.0001	-
1.3845	6100	0.0001	-
1.3958	6150	0.0001	-
1.4072	6200	0.0001	-
1.4185	6250	0.0001	-
1.4299	6300	0.0001	-
1.4412	6350	0.0001	-
1.4526	6400	0.0001	-
1.4639	6450	0.0	-
1.4753	6500	0.0001	-
1.4866	6550	0.0011	-
1.4980	6600	0.0001	-
1.5093	6650	0.0001	-
1.5207	6700	0.0	-
1.5320	6750	0.0	-
1.5433	6800	0.0001	-
1.5547	6850	0.0001	-
1.5660	6900	0.0001	-
1.5774	6950	0.0001	-
1.5887	7000	0.0001	-
1.6001	7050	0.0001	-
1.6114	7100	0.0001	-
1.6228	7150	0.0	-
1.6341	7200	0.0	-
1.6455	7250	0.0001	-
1.6568	7300	0.0001	-
1.6682	7350	0.0001	-
1.6795	7400	0.0001	-
1.6909	7450	0.0	-
1.7022	7500	0.0001	-
1.7136	7550	0.0001	-
1.7249	7600	0.0001	-
1.7363	7650	0.0	-
1.7476	7700	0.0	-
1.7590	7750	0.0	-
1.7703	7800	0.0001	-
1.7817	7850	0.0	-
1.7930	7900	0.0	-
1.8044	7950	0.0	-
1.8157	8000	0.0001	-
1.8271	8050	0.0	-
1.8384	8100	0.0	-
1.8498	8150	0.0001	-
1.8611	8200	0.0	-
1.8724	8250	0.0001	-
1.8838	8300	0.0	-
1.8951	8350	0.0001	-
1.9065	8400	0.0001	-
1.9178	8450	0.0001	-
1.9292	8500	0.0	-
1.9405	8550	0.0	-
1.9519	8600	0.0	-
1.9632	8650	0.0	-
1.9746	8700	0.0	-
1.9859	8750	0.0	-
1.9973	8800	0.0	-
2.0	8812	-	0.0549
2.0086	8850	0.0	-
2.0200	8900	0.0	-
2.0313	8950	0.0	-
2.0427	9000	0.0	-
2.0540	9050	0.0	-
2.0654	9100	0.0	-
2.0767	9150	0.0	-
2.0881	9200	0.0	-
2.0994	9250	0.0	-
2.1108	9300	0.0	-
2.1221	9350	0.0	-
2.1335	9400	0.0001	-
2.1448	9450	0.0	-
2.1562	9500	0.0	-
2.1675	9550	0.0	-
2.1788	9600	0.0	-
2.1902	9650	0.0	-
2.2015	9700	0.0	-
2.2129	9750	0.0	-
2.2242	9800	0.0	-
2.2356	9850	0.0	-
2.2469	9900	0.0	-
2.2583	9950	0.0	-
2.2696	10000	0.0	-
2.2810	10050	0.0	-
2.2923	10100	0.0	-
2.3037	10150	0.0	-
2.3150	10200	0.0	-
2.3264	10250	0.0	-
2.3377	10300	0.0	-
2.3491	10350	0.0	-
2.3604	10400	0.0	-
2.3718	10450	0.0001	-
2.3831	10500	0.0	-
2.3945	10550	0.0	-
2.4058	10600	0.0	-
2.4172	10650	0.0	-
2.4285	10700	0.0	-
2.4399	10750	0.0	-
2.4512	10800	0.0	-
2.4626	10850	0.0	-
2.4739	10900	0.0	-
2.4852	10950	0.0	-
2.4966	11000	0.0	-
2.5079	11050	0.0	-
2.5193	11100	0.0	-
2.5306	11150	0.0	-
2.5420	11200	0.0	-
2.5533	11250	0.0	-
2.5647	11300	0.0	-
2.5760	11350	0.0	-
2.5874	11400	0.0	-
2.5987	11450	0.0	-
2.6101	11500	0.0	-
2.6214	11550	0.0	-
2.6328	11600	0.0	-
2.6441	11650	0.0	-
2.6555	11700	0.0	-
2.6668	11750	0.0	-
2.6782	11800	0.0	-
2.6895	11850	0.0	-
2.7009	11900	0.0	-
2.7122	11950	0.0	-
2.7236	12000	0.0	-
2.7349	12050	0.0	-
2.7463	12100	0.0	-
2.7576	12150	0.0	-
2.7690	12200	0.0	-
2.7803	12250	0.0	-
2.7916	12300	0.0	-
2.8030	12350	0.0	-
2.8143	12400	0.0	-
2.8257	12450	0.0	-
2.8370	12500	0.0	-
2.8484	12550	0.0	-
2.8597	12600	0.0	-
2.8711	12650	0.0	-
2.8824	12700	0.0	-
2.8938	12750	0.0	-
2.9051	12800	0.0	-
2.9165	12850	0.0	-
2.9278	12900	0.0	-
2.9392	12950	0.0	-
2.9505	13000	0.0	-
2.9619	13050	0.0	-
2.9732	13100	0.0	-
2.9846	13150	0.0	-
2.9959	13200	0.0	-
3.0	13218	-	0.0469
3.0073	13250	0.0	-
3.0186	13300	0.0	-
3.0300	13350	0.0	-
3.0413	13400	0.0	-
3.0527	13450	0.0	-
3.0640	13500	0.0	-
3.0754	13550	0.0	-
3.0867	13600	0.0	-
3.0980	13650	0.0	-
3.1094	13700	0.0	-
3.1207	13750	0.0	-
3.1321	13800	0.0	-
3.1434	13850	0.0	-
3.1548	13900	0.0	-
3.1661	13950	0.0	-
3.1775	14000	0.0	-
3.1888	14050	0.0	-
3.2002	14100	0.0	-
3.2115	14150	0.0	-
3.2229	14200	0.0	-
3.2342	14250	0.0	-
3.2456	14300	0.0	-
3.2569	14350	0.0	-
3.2683	14400	0.0	-
3.2796	14450	0.0	-
3.2910	14500	0.0	-
3.3023	14550	0.0	-
3.3137	14600	0.0	-
3.3250	14650	0.0	-
3.3364	14700	0.0	-
3.3477	14750	0.0	-
3.3591	14800	0.0	-
3.3704	14850	0.0	-
3.3818	14900	0.0	-
3.3931	14950	0.0	-
3.4044	15000	0.0	-
3.4158	15050	0.0	-
3.4271	15100	0.0	-
3.4385	15150	0.0	-
3.4498	15200	0.0	-
3.4612	15250	0.0	-
3.4725	15300	0.0	-
3.4839	15350	0.0	-
3.4952	15400	0.0	-
3.5066	15450	0.0	-
3.5179	15500	0.0	-
3.5293	15550	0.0	-
3.5406	15600	0.0	-
3.5520	15650	0.0	-
3.5633	15700	0.0	-
3.5747	15750	0.0	-
3.5860	15800	0.0	-
3.5974	15850	0.0	-
3.6087	15900	0.0	-
3.6201	15950	0.0	-
3.6314	16000	0.0	-
3.6428	16050	0.0	-
3.6541	16100	0.0	-
3.6655	16150	0.0	-
3.6768	16200	0.0	-
3.6882	16250	0.0	-
3.6995	16300	0.0	-
3.7108	16350	0.0	-
3.7222	16400	0.0	-
3.7335	16450	0.0	-
3.7449	16500	0.0	-
3.7562	16550	0.0	-
3.7676	16600	0.0	-
3.7789	16650	0.0	-
3.7903	16700	0.0	-
3.8016	16750	0.0	-
3.8130	16800	0.0	-
3.8243	16850	0.0	-
3.8357	16900	0.0	-
3.8470	16950	0.0	-
3.8584	17000	0.0	-
3.8697	17050	0.0	-
3.8811	17100	0.0	-
3.8924	17150	0.0	-
3.9038	17200	0.0	-
3.9151	17250	0.0	-
3.9265	17300	0.0	-
3.9378	17350	0.0	-
3.9492	17400	0.0	-
3.9605	17450	0.0	-
3.9719	17500	0.0	-
3.9832	17550	0.0	-
3.9946	17600	0.0	-
4.0	17624	-	0.0404
4.0059	17650	0.0	-
4.0172	17700	0.0	-
4.0286	17750	0.0	-
4.0399	17800	0.0	-
4.0513	17850	0.0	-
4.0626	17900	0.0	-
4.0740	17950	0.0	-
4.0853	18000	0.0	-
4.0967	18050	0.0	-
4.1080	18100	0.0	-
4.1194	18150	0.0	-
4.1307	18200	0.0	-
4.1421	18250	0.0	-
4.1534	18300	0.0	-
4.1648	18350	0.0	-
4.1761	18400	0.0	-
4.1875	18450	0.0	-
4.1988	18500	0.0	-
4.2102	18550	0.0	-
4.2215	18600	0.0	-
4.2329	18650	0.0	-
4.2442	18700	0.0	-
4.2556	18750	0.0	-
4.2669	18800	0.0	-
4.2783	18850	0.0	-
4.2896	18900	0.0	-
4.3010	18950	0.0	-
4.3123	19000	0.0	-
4.3236	19050	0.0	-
4.3350	19100	0.0	-
4.3463	19150	0.0	-
4.3577	19200	0.0	-
4.3690	19250	0.0	-
4.3804	19300	0.0	-
4.3917	19350	0.0	-
4.4031	19400	0.0	-
4.4144	19450	0.0	-
4.4258	19500	0.0	-
4.4371	19550	0.0	-
4.4485	19600	0.0	-
4.4598	19650	0.0	-
4.4712	19700	0.0	-
4.4825	19750	0.0	-
4.4939	19800	0.0	-
4.5052	19850	0.0	-
4.5166	19900	0.0	-
4.5279	19950	0.0	-
4.5393	20000	0.0	-
4.5506	20050	0.0	-
4.5620	20100	0.0	-
4.5733	20150	0.0	-
4.5847	20200	0.0	-
4.5960	20250	0.0	-
4.6074	20300	0.0	-
4.6187	20350	0.0	-
4.6300	20400	0.0	-
4.6414	20450	0.0	-
4.6527	20500	0.0	-
4.6641	20550	0.0	-
4.6754	20600	0.0	-
4.6868	20650	0.0	-
4.6981	20700	0.0	-
4.7095	20750	0.0	-
4.7208	20800	0.0	-
4.7322	20850	0.0	-
4.7435	20900	0.0	-
4.7549	20950	0.0	-
4.7662	21000	0.0	-
4.7776	21050	0.0	-
4.7889	21100	0.0	-
4.8003	21150	0.0	-
4.8116	21200	0.0	-
4.8230	21250	0.0	-
4.8343	21300	0.0	-
4.8457	21350	0.0	-
4.8570	21400	0.0	-
4.8684	21450	0.0	-
4.8797	21500	0.0	-
4.8911	21550	0.0	-
4.9024	21600	0.0	-
4.9138	21650	0.0	-
4.9251	21700	0.0	-
4.9365	21750	0.0	-
4.9478	21800	0.0	-
4.9591	21850	0.0	-
4.9705	21900	0.0	-
4.9818	21950	0.0	-
4.9932	22000	0.0	-
5.0	22030	-	0.038
5.0045	22050	0.0	-
5.0159	22100	0.0	-
5.0272	22150	0.0	-
5.0386	22200	0.0	-
5.0499	22250	0.0	-
5.0613	22300	0.0	-
5.0726	22350	0.0	-
5.0840	22400	0.0	-
5.0953	22450	0.0	-
5.1067	22500	0.0	-
5.1180	22550	0.0	-
5.1294	22600	0.0	-
5.1407	22650	0.0	-
5.1521	22700	0.0	-
5.1634	22750	0.0	-
5.1748	22800	0.0	-
5.1861	22850	0.0	-
5.1975	22900	0.0	-
5.2088	22950	0.0	-
5.2202	23000	0.0	-
5.2315	23050	0.0	-
5.2429	23100	0.0	-
5.2542	23150	0.0	-
5.2655	23200	0.0	-
5.2769	23250	0.0	-
5.2882	23300	0.0	-
5.2996	23350	0.0	-
5.3109	23400	0.0	-
5.3223	23450	0.0	-
5.3336	23500	0.0	-
5.3450	23550	0.0	-
5.3563	23600	0.0	-
5.3677	23650	0.0	-
5.3790	23700	0.0	-
5.3904	23750	0.0	-
5.4017	23800	0.0	-
5.4131	23850	0.0	-
5.4244	23900	0.0	-
5.4358	23950	0.0	-
5.4471	24000	0.0	-
5.4585	24050	0.0	-
5.4698	24100	0.0	-
5.4812	24150	0.0	-
5.4925	24200	0.0	-
5.5039	24250	0.0	-
5.5152	24300	0.0	-
5.5266	24350	0.0	-
5.5379	24400	0.0	-
5.5493	24450	0.0	-
5.5606	24500	0.0	-
5.5719	24550	0.0	-
5.5833	24600	0.0	-
5.5946	24650	0.0	-
5.6060	24700	0.0	-
5.6173	24750	0.0	-
5.6287	24800	0.0	-
5.6400	24850	0.0	-
5.6514	24900	0.0	-
5.6627	24950	0.0	-
5.6741	25000	0.0	-
5.6854	25050	0.0	-
5.6968	25100	0.0	-
5.7081	25150	0.0	-
5.7195	25200	0.0	-
5.7308	25250	0.0	-
5.7422	25300	0.0	-
5.7535	25350	0.0	-
5.7649	25400	0.0	-
5.7762	25450	0.0	-
5.7876	25500	0.0	-
5.7989	25550	0.0	-
5.8103	25600	0.0	-
5.8216	25650	0.0	-
5.8330	25700	0.0	-
5.8443	25750	0.0	-
5.8557	25800	0.0	-
5.8670	25850	0.0	-
5.8783	25900	0.0	-
5.8897	25950	0.0	-
5.9010	26000	0.0	-
5.9124	26050	0.0	-
5.9237	26100	0.0	-
5.9351	26150	0.0	-
5.9464	26200	0.0	-
5.9578	26250	0.0	-
5.9691	26300	0.0	-
5.9805	26350	0.0	-
5.9918	26400	0.0	-
6.0	26436	-	0.0382

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.3
SetFit: 1.0.3
Sentence Transformers: 3.0.1
Transformers: 4.39.0
PyTorch: 2.4.1+cu121
Datasets: 2.21.0
Tokenizers: 0.15.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}