Rolshoven commited on
Commit
6d30b51
·
verified ·
1 Parent(s): ed1ad8a

Align the card with the paper; update licensor

Browse files
Files changed (2) hide show
  1. NOTICE +1 -1
  2. README.md +26 -11
NOTICE CHANGED
@@ -1,5 +1,5 @@
1
  TENDRIL most-embed-de
2
- Copyright 2026 The TENDRIL authors (IntelliProcure project)
3
 
4
  This model is an adapted version of malteos/most-embed-de
5
  (https://huggingface.co/malteos/most-embed-de). The fine-tuning contribution of most-embed-de is
 
1
  TENDRIL most-embed-de
2
+ Copyright 2026 Institute for Public Sector Transformation, Bern University of Applied Sciences
3
 
4
  This model is an adapted version of malteos/most-embed-de
5
  (https://huggingface.co/malteos/most-embed-de). The fine-tuning contribution of most-embed-de is
README.md CHANGED
@@ -40,17 +40,21 @@ L2-normalized and have 2,048 dimensions, and inputs are truncated at 32,768 toke
40
 
41
  ## Results
42
 
43
- Test set: [IntelliProcure/TENDRIL](https://huggingface.co/datasets/IntelliProcure/TENDRIL), 35 projects. Scores are
44
- macro-averaged over projects, and 95% intervals come from 1,000 project-level bootstrap resamples.
 
45
 
46
  | Model | nDCG@10 | Recall@10 | Recall@100 | MRR@10 |
47
  |---|---|---|---|---|
48
- | BM25 | 0.437 [0.388, 0.485] | 0.533 | 0.920 | 0.476 |
49
- | most-embed-de (base) | 0.644 [0.593, 0.695] | 0.763 | 0.977 | 0.665 |
50
- | **TENDRIL most-embed-de** | **0.660 [0.614, 0.710]** | **0.788** | **0.987** | **0.671** |
51
 
52
- The gain over the base model on the test set is small, and the confidence intervals overlap. On the validation
53
- projects, the fine-tuned model reaches a balanced nDCG@10 (the mean over areas of action and criteria) of 0.840.
 
 
 
54
 
55
  ## Training
56
 
@@ -58,7 +62,10 @@ projects, the fine-tuned model reaches a balanced nDCG@10 (the mean over areas o
58
  [TENDRIL-Train](https://huggingface.co/datasets/IntelliProcure/TENDRIL-Train) data: a query, one relevant passage
59
  and three hard negatives from the same project. Area-of-action and criterion examples were balanced, giving 9,000
60
  of 13,390 available examples. Validation used held-out projects (63 for areas of action, 62 for criteria).
61
- - **Loss.** TODO(sophie): name the loss function and how positives and hard negatives were selected.
 
 
 
62
  - **Hyperparameters.** Learning rate 1e-5 with linear decay and 10% warmup, AdamW without weight decay, effective
63
  batch size 32 (2 × 16 gradient accumulation), at most 2 epochs, bf16, seed 42. The model was evaluated every
64
  56 steps, and the checkpoint with the best balanced validation nDCG@10 (step 168) was kept.
@@ -75,9 +82,17 @@ projects, the fine-tuned model reaches a balanced nDCG@10 (the mean over areas o
75
 
76
  ## License
77
 
78
- Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC-4.0), the same as the base model. See
79
- [LICENSE](LICENSE) and [NOTICE](NOTICE) for the upstream attributions.
 
80
 
81
  ## Citation
82
 
83
- TODO: add the paper citation.
 
 
 
 
 
 
 
 
40
 
41
  ## Results
42
 
43
+ Test set: [IntelliProcure/TENDRIL](https://huggingface.co/datasets/IntelliProcure/TENDRIL) (revision `15eef0bc`),
44
+ 33 projects. Scores are macro-averaged over projects, and 95% intervals come from 10,000 project-level bootstrap
45
+ resamples.
46
 
47
  | Model | nDCG@10 | Recall@10 | Recall@100 | MRR@10 |
48
  |---|---|---|---|---|
49
+ | BM25 | 0.421 [0.381, 0.462] | 0.511 | 0.929 | 0.463 |
50
+ | most-embed-de (base) | 0.640 [0.588, 0.692] | 0.760 | 0.980 | 0.664 |
51
+ | **TENDRIL most-embed-de** | **0.660 [0.613, 0.707]** | **0.787** | **0.987** | **0.673** |
52
 
53
+ The gain over the base model is small: +0.020 nDCG@10 [−0.003, +0.044] in a two-sided paired bootstrap over projects.
54
+ After a Holm correction over the four fine-tuned models in the paper, it is significant only for precision@10
55
+ (p < 0.05). With an nDCG@10 of 0.660, the model is on par with the best proprietary model evaluated in the paper
56
+ (Voyage 4 Large, 0.649). On the validation projects, the fine-tuned model reaches a balanced nDCG@10 (the mean
57
+ over areas of action and criteria) of 0.840.
58
 
59
  ## Training
60
 
 
62
  [TENDRIL-Train](https://huggingface.co/datasets/IntelliProcure/TENDRIL-Train) data: a query, one relevant passage
63
  and three hard negatives from the same project. Area-of-action and criterion examples were balanced, giving 9,000
64
  of 13,390 available examples. Validation used held-out projects (63 for areas of action, 62 for criteria).
65
+ - **Loss.** InfoNCE over the positive and the three negatives, with the cosine similarity of L2-normalized embeddings
66
+ and a similarity scale of 20. Positives are passages the judge rated relevant (2); negatives are passages rated
67
+ irrelevant (0) for the same query, two from the positive's document and one from another document of the same
68
+ project (the other pool fills in when one has too few).
69
  - **Hyperparameters.** Learning rate 1e-5 with linear decay and 10% warmup, AdamW without weight decay, effective
70
  batch size 32 (2 × 16 gradient accumulation), at most 2 epochs, bf16, seed 42. The model was evaluated every
71
  56 steps, and the checkpoint with the best balanced validation nDCG@10 (step 168) was kept.
 
82
 
83
  ## License
84
 
85
+ Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC-4.0), the same as the base model. The
86
+ underlying Nemotron-3-Embed-1B is licensed under the OpenMDW License v1.1. See [LICENSE](LICENSE) and
87
+ [NOTICE](NOTICE) for the upstream attributions.
88
 
89
  ## Citation
90
 
91
+ ```bibtex
92
+ @misc{rolshoven2026tendril,
93
+ title = {{TENDRIL}: Benchmarking Sustainability-Focused Retrieval in Swiss Public Procurement},
94
+ author = {Rolshoven, Luca and Probst, Sophie and Matoshi, Veton and Ellendorff, Tilia and Meili, Rahel and
95
+ Hostettler, Sarah and Jann, Jeremy Austin and Jans, Meret and Tschumi, Pascal and St{\"u}rmer, Matthias},
96
+ year = {2026}
97
+ }
98
+ ```