ACDRepo commited on
Commit
9d72d96
·
verified ·
1 Parent(s): ccfa311

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -8
README.md CHANGED
@@ -50,7 +50,7 @@ Use deterministic decoding for most evaluation-style tasks. Make sure special to
50
  from transformers import AutoModelForCausalLM, AutoTokenizer
51
  import torch
52
 
53
- model_id = "YOUR_ORG/permuformer"
54
 
55
  tokenizer = AutoTokenizer.from_pretrained(model_id)
56
  model = AutoModelForCausalLM.from_pretrained(model_id)
@@ -100,16 +100,14 @@ Algebraic operation example:
100
 
101
  ## Evaluation Notes
102
 
103
- The training code evaluates by exact match on the generated right-hand side after `=`. The local training log for this repository reports, at step 2,522,000 on a 2,560-example stratified evaluation sample:
104
 
105
  - Overall exact match: **98.44%**
106
  - Translation: **97.78%**
107
  - Property/statistic tasks: **99.17%**
108
  - Algebraic tasks: **98.36%**
109
 
110
- These figures are from the local log and should be treated as checkpoint-adjacent repository metadata, not a full benchmark report for every downstream setting.
111
-
112
- The paper also reports that PermuFormer is substantially more accurate than frontier general-purpose LLMs on a small held-out sample from the model's symbolic test distribution, while noting that the comparison is imperfect because PermuFormer was trained directly in this syntax.
113
 
114
  ## Finetuning
115
 
@@ -120,8 +118,6 @@ PermuFormer is designed to be finetuned on specialized permutation tasks. Experi
120
  - Schubert polynomial structure constants
121
  - Kazhdan-Lusztig polynomial degree prediction
122
 
123
- The repository's finetuning scripts compare starting from this pretrained checkpoint with training the same architecture from scratch.
124
-
125
  ## Limitations
126
 
127
  - This is a specialist symbolic model. It expects the exact whitespace-tokenized syntax used during training and is brittle to natural-language paraphrases or malformed prompts.
@@ -132,4 +128,4 @@ The repository's finetuning scripts compare starting from this pretrained checkp
132
 
133
  ## Citation
134
 
135
- If you use this model, please cite the accompanying PermuFormer paper once citation details are available.
 
50
  from transformers import AutoModelForCausalLM, AutoTokenizer
51
  import torch
52
 
53
+ model_id = "ACDRepo/permuformer"
54
 
55
  tokenizer = AutoTokenizer.from_pretrained(model_id)
56
  model = AutoModelForCausalLM.from_pretrained(model_id)
 
100
 
101
  ## Evaluation Notes
102
 
103
+ The training code evaluates by exact match on the generated right-hand side after `=`.
104
 
105
  - Overall exact match: **98.44%**
106
  - Translation: **97.78%**
107
  - Property/statistic tasks: **99.17%**
108
  - Algebraic tasks: **98.36%**
109
 
110
+ PermuFormer is substantially more accurate than frontier general-purpose LLMs on a small held-out sample from the model's symbolic test distribution (though note that this comparison is imperfect because PermuFormer is trained directly in this syntax and on these tasks).
 
 
111
 
112
  ## Finetuning
113
 
 
118
  - Schubert polynomial structure constants
119
  - Kazhdan-Lusztig polynomial degree prediction
120
 
 
 
121
  ## Limitations
122
 
123
  - This is a specialist symbolic model. It expects the exact whitespace-tokenized syntax used during training and is brittle to natural-language paraphrases or malformed prompts.
 
128
 
129
  ## Citation
130
 
131
+ Coming soon.