hgissbkh commited on
Commit
be487aa
·
verified ·
1 Parent(s): c33db75

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +18 -16
README.md CHANGED
@@ -16,23 +16,9 @@ BERT-as-a-Judge is a family of encoder-based models designed for efficient, refe
16
  - **Model Type:** Encoder-based Judge (EuroBERT-210m backbone)
17
  - **Language:** English
18
 
19
- ## Naming Convention Breakdown
20
-
21
- Models follow a standardized naming structure: `BERTJudge-<Candidate_Format>-<Input_Structure>-<Additional_Info>`.
22
-
23
- * **Candidate Format:**
24
- * `Free`: Trained on unconstrained model generations.
25
- * `Formatted`: Trained on outputs that adhere to specific structural constraints. For optimized evaluation under the formatted setup, candidate outputs should ideally conclude with `"Final answer: <final_answer>"` (see the paper for details).
26
- * **Input Structure:**
27
- * `QCR`: The input sequence consists of [Question, Candidate, Reference].
28
- * `CR`: The input sequence consists only of [Candidate, Reference].
29
- * **Additional Info:**
30
- * `OOD`: Indicates evaluation of Out-of-Distribution performance (where specific generative models were withheld during training).
31
- * `100k/200k/500k`: Denotes the total training steps (default regime being 1 million).
32
-
33
  ## Intended Use
34
 
35
- These models are designed as sequence classifiers that output a sigmoid score indicating answer correctness. For inference, we suggest using the [BERT-as-a-Judge](https://github.com/artefactory/BERT-as-a-Judge) package. In most scenarios, we specifically recommend **BERTJudge-Free-QCR** for its superior and more robust evaluation performance.
36
 
37
  ### Installation
38
 
@@ -51,7 +37,7 @@ from bert_judge.judges import BERTJudge
51
 
52
  # 1) Initialize the judge
53
  judge = BERTJudge(
54
- model_path="artefactory/BERTJudge-Free-QCR",
55
  trust_remote_code=True,
56
  dtype="bfloat16",
57
  )
@@ -79,6 +65,22 @@ scores = judge.predict(
79
  print(scores)
80
  ```
81
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
  ## Citation
83
 
84
  If you find this model useful for your research, please consider citing:
 
16
  - **Model Type:** Encoder-based Judge (EuroBERT-210m backbone)
17
  - **Language:** English
18
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  ## Intended Use
20
 
21
+ BERTJudge models are designed as sequence classifiers that output a sigmoid score reflecting answer correctness. For inference, we suggest using the [BERT-as-a-Judge](https://github.com/artefactory/BERT-as-a-Judge) package.
22
 
23
  ### Installation
24
 
 
37
 
38
  # 1) Initialize the judge
39
  judge = BERTJudge(
40
+ model_path="artefactory/BERTJudge",
41
  trust_remote_code=True,
42
  dtype="bfloat16",
43
  )
 
65
  print(scores)
66
  ```
67
 
68
+ ## Naming Convention Breakdown
69
+
70
+ Models follow a standardized naming structure: `BERTJudge-<Candidate_Format>-<Input_Structure>-<Additional_Info>`.
71
+
72
+ * **Candidate Format:**
73
+ * `Free`: Trained on unconstrained model generations.
74
+ * `Formatted`: Trained on outputs that adhere to specific structural constraints. For optimized evaluation under the formatted setup, candidate outputs should ideally conclude with `"Final answer: <final_answer>"` (see the paper for details).
75
+ * **Input Structure:**
76
+ * `QCR`: The input sequence consists of [Question, Candidate, Reference].
77
+ * `CR`: The input sequence consists only of [Candidate, Reference].
78
+ * **Additional Info:**
79
+ * `OOD`: Indicates evaluation of Out-of-Distribution performance (where specific generative models were withheld during training).
80
+ * `100k/200k/500k`: Denotes the total training steps (default regime being 1 million).
81
+
82
+ **Note: For optimal evaluation performance, we recommend using `BERTJudge-Free-QCR`, available as `artefactory/BERTJudge`.**
83
+
84
  ## Citation
85
 
86
  If you find this model useful for your research, please consider citing: