Hugging Face's logo

TIGER-Lab
/
StructLM-7B

Text Generation
Transformers
Safetensors
English
llama
conversational
text-generation-inference
Model card Files
xet
Community
3
azhx commited on
Commit
a92109d
·
verified ·
1 Parent(s): d6b7884

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +30 -31
README.md CHANGED
@@ -5,7 +5,9 @@ datasets:
5
  language:
6
  - en
7
  ---
8
- # StructLM: Towards Building Generalist Models for Structured Knowledge Grounding
 
 
9
 
10
  Project Page: [https://tiger-ai-lab.github.io/StructLM/](https://tiger-ai-lab.github.io/StructLM/)
11
 
@@ -14,15 +16,17 @@ Paper: Arxiv link not yet announced
14
  Code: [https://github.com/TIGER-AI-Lab/StructLM](https://github.com/TIGER-AI-Lab/StructLM)
15
 
16
 
17
- ## Introduction
18
- StructLM, is a series of open-source large language models (LLMs) finetuned for structured knowledge grounding (SKG) tasks.
19
 
20
- We release 3 models:
21
 
22
- |-----|---------------------------------------------------------------|
23
- | 7B | [StructLM-7B](https://huggingface.co/TIGER-Lab/StructLM-7B) |
24
- | 13B | [StructLM-13B](https://huggingface.co/TIGER-Lab/StructLM-13B) |
25
- | 34B | [StructLM-34B](https://huggingface.co/TIGER-Lab/StructLM-34B) |
 
 
 
 
26
 
27
 
28
  ## Training Data
@@ -33,29 +37,24 @@ These models are trained on 🤗 [SKGInstruct Dataset](https://huggingface.co/da
33
  The models are fine-tuned with CodeLlama-Instruct-hf models as base models. Each model is trained for 3 epochs, and the best checkpoint is selected.
34
 
35
  ## Evaluation
36
- The models are evaluated using open-ended and multiple-choice math problems from several datasets. Here are the results:
37
-
38
-
39
- | **Model** | **Decoding** | **GSM** | **MATH** | **AQuA** | **NumG** | **SVA** | **Mat** | **Sim** | **SAT** | **MMLU** | **AVG** |
40
- |-----------------------|--------------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
41
- | **MAmmoTH-7B** | CoT | 50.5 | 10.4 | 43.7 | 44.0 | 47.3 | 9.2 | 18.9 | 32.7 | 39.9 | 33.0 |
42
- | | PoT | 51.6 | 28.7 | 43.3 | 52.3 | 65.1 | 41.9 | 48.2 | 39.1 | 44.6 | 46.1 |
43
- | | **Hybrid** | **53.6** | **31.5** | **44.5** | **61.2** | **67.7** | **46.3** | **41.2** | **42.7** | **42.6** | **47.9** |
44
- | **MAmmoTH-Coder-7B** | CoT | 22.4 | 7.9 | 36.2 | 36.0 | 37.0 | 8.2 | 7.2 | 32.7 | 34.6 | 24.7 |
45
- | | PoT | 58.8 | 32.1 | 47.2 | 57.1 | 71.1 | 53.9 | 44.6 | 40.0 | 47.8 | 50.3 |
46
- | | **Hybrid** | **59.4** | **33.4** | **47.2** | **66.4** | **71.4** | **55.4** | **45.9** | **40.5** | **48.3** | **52.0** |
47
- | **MAmmoTH-13B** | CoT | 56.3 | 12.9 | 45.3 | 45.6 | 53.8 | 11.7 | 22.4 | 43.6 | 42.3 | 37.1 |
48
- | | PoT | 61.3 | 32.6 | 48.8 | 59.6 | 72.2 | 48.5 | 40.3 | 46.8 | 45.4 | 50.6 |
49
- | | **Hybrid** | **62.0** | **34.2** | **51.6** | **68.7** | **72.4** | **49.2** | **43.2** | **46.8** | **47.6** | **52.9** |
50
- | **MAmmoTH-Coder-13B** | CoT | 32.1 | 10.2 | 40.6 | 36.2 | 43.0 | 9.6 | 10.1 | 40.9 | 36.6 | 28.8 |
51
- | | PoT | 64.3 | 35.2 | 46.8 | 54.2 | 73.2 | 60.0 | 44.2 | 48.2 | 48.2 | 52.7 |
52
- | | **Hybrid** | **64.7** | **36.3** | **46.9** | **66.8** | **73.7** | **61.5** | **47.1** | **48.6** | **48.3** | **54.9** |
53
- | **MAmmoTH-Coder-33B** | CoT | 34.3 | 11.6 | 39.0 | 36.2 | 44.6 | 10.8 | 10.9 | 46.4 | 42.9 | 30.7 |
54
- | | PoT | 72.3 | 42.8 | 53.8 | 59.6 | 84.0 | 64.7 | 50.6 | 58.6 | 52.7 | 59.9 |
55
- | | **Hybrid** | **72.7** | **43.6** | **54.7** | **71.6** | **84.3** | **65.4** | **51.8** | **60.9** | **53.8** | **62.1** |
56
- | **MAmmoTH-70B** | CoT | 72.4 | 21.1 | 57.9 | 58.9 | 71.6 | 20.0 | 31.9 | 57.3 | 52.1 | 49.2 |
57
- | | PoT | 76.7 | 40.1 | 60.2 | 64.3 | 81.7 | 55.3 | 45.3 | 64.1 | 53.5 | 60.1 |
58
- | | **Hybrid** | **76.9** | **41.8** | **65.0** | **74.4** | **82.4** | **55.6** | **51.4** | **66.4** | **56.7** | **63.4** |
59
 
60
  ## Usage
61
  You can use the models through Huggingface's Transformers library.
 
5
  language:
6
  - en
7
  ---
8
+ # 🏗️ StructLM: Towards Building Generalist Models for Structured Knowledge Grounding
9
+
10
+
11
 
12
  Project Page: [https://tiger-ai-lab.github.io/StructLM/](https://tiger-ai-lab.github.io/StructLM/)
13
 
 
16
  Code: [https://github.com/TIGER-AI-Lab/StructLM](https://github.com/TIGER-AI-Lab/StructLM)
17
 
18
 
19
+ ![Alt text](https://raw.githubusercontent.com/TIGER-AI-Lab/StructLM/gh-pages/static/images/thumbnail.drawio%20(1).png)
 
20
 
 
21
 
22
+ ## Introduction
23
+ StructLM, is a series of open-source large language models (LLMs) finetuned for structured knowledge grounding (SKG) tasks. We release 3 models:
24
+
25
+ 7B | [StructLM-7B](https://huggingface.co/TIGER-Lab/StructLM-7B)
26
+
27
+ 13B | [StructLM-13B](https://huggingface.co/TIGER-Lab/StructLM-13B)
28
+
29
+ 34B | [StructLM-34B](https://huggingface.co/TIGER-Lab/StructLM-34B)
30
 
31
 
32
  ## Training Data
 
37
  The models are fine-tuned with CodeLlama-Instruct-hf models as base models. Each model is trained for 3 epochs, and the best checkpoint is selected.
38
 
39
  ## Evaluation
40
+ Here are a subset of model evaluation results:
41
+
42
+ ### Held in
43
+
44
+ | **Model** | **ToTTo** | **GrailQA** | **CompWebQ** | **MMQA** | **Feverous** | **Spider** | **TabFact** | **Dart** |
45
+ |-----------------------|--------------|----------|----------|----------|----------|----------|----------|----------|
46
+ | **StructLM-7B** | 49.4 | 80.4 | 78.3 | 85.2 | 84.4 | 72.4 | 80.8 | 62.2 |
47
+ | **StructLM-13B** | 49.3 | 79.2 | 80.4 | 86.0 | 85.0 | 74.1 | 84.7 | 61.4 |
48
+ | **StructLM-34B** | 50.2 | 82.2 | 81.9 | 88.1 | 85.7 | 74.6 | 86.6 | 61.8 |
49
+
50
+
51
+ ### Held out
52
+ | **Model** | **BIRD** | **InfoTabs** | **FinQA** | **SQA** |
53
+ |-----------------------|--------------|----------|----------|----------|
54
+ | **StructLM-7B** | 22.3 | 55.3 | 27.3 | 49.7 |
55
+ | **StructLM-13B** | 22.8 | 58.1 | 25.6 | 36.1 |
56
+ | **StructLM-34B** | 24.7 | 61.8 | 36.2 | 44.2 |
57
+
 
 
 
 
 
58
 
59
  ## Usage
60
  You can use the models through Huggingface's Transformers library.