dotvignesh commited on
Commit
33bfd88
·
verified ·
1 Parent(s): 2abb939

fixed incorrect results previously included from another model

Browse files
Files changed (1) hide show
  1. README.md +8 -7
README.md CHANGED
@@ -4,18 +4,19 @@ A generalist reasoning LLM trained on synthetic chain-of-thought traces over STE
4
 
5
  ## Overview
6
 
7
- Perry is a fine-tuned LLaMA model designed to improve reasoning capabilities through synthetic CoT supervision. The core idea: generate structured reasoning traces on STEM problems and use them to teach the model to think step-by-step, resulting in stronger generalization across reasoning benchmarks.
8
 
9
  Models were trained at 7B and 13B scales using compute-efficient methods.
10
 
11
  ## Results
12
 
13
- Improvements over baselines (as of Sep 2023):
14
 
15
- | Benchmark | Improvement |
16
- |-----------|-------------|
17
- | Winogrande | +4% |
18
- | ARC-Challenge | +6% |
 
19
 
20
  ## Usage
21
 
@@ -28,6 +29,6 @@ tokenizer = AutoTokenizer.from_pretrained("dotvignesh/perry-7b")
28
 
29
  ## Model Details
30
 
31
- - **Base model:** LLaMA
32
  - **Training data:** Synthetic CoT traces on STEM datasets
33
  - **Framework:** PyTorch / Transformers
 
4
 
5
  ## Overview
6
 
7
+ Perry is a fine-tuned LLaMA 2 7B model designed to improve reasoning capabilities through synthetic CoT supervision. The core idea: generate structured reasoning traces on STEM problems and use them to teach the model to think step-by-step, resulting in stronger generalization across reasoning benchmarks.
8
 
9
  Models were trained at 7B and 13B scales using compute-efficient methods.
10
 
11
  ## Results
12
 
13
+ Improvements over LLaMA 2 7B (as of Sep 2023):
14
 
15
+ | Benchmark | Perry-7B | LLaMA 2 7B | Delta |
16
+ |-----------|----------|------------|-------|
17
+ | MMLU (5-shot) | 46.18 | 43.80 | +2.38 |
18
+ | TruthfulQA (0-shot) | 40.08 | 38.98 | +1.10 |
19
+ | GSM8K (5-shot) | 10.31 | 5.38 | +4.93 |
20
 
21
  ## Usage
22
 
 
29
 
30
  ## Model Details
31
 
32
+ - **Base model:** LLaMA 2 7B
33
  - **Training data:** Synthetic CoT traces on STEM datasets
34
  - **Framework:** PyTorch / Transformers