Hugging Face's logo

zai-org
/
chatglm3-6b

Transformers
PyTorch
Safetensors
Chinese
English
chatglm
glm
thudm
custom_code
Model card Files
xet
 Community
67

fix rmsnorm init weight bug.

#59
by Shan1990 - opened
base: refs/heads/main
←
from: refs/pr/59
Discussion Files changed
+1
-1
Shan1990

Using torch.ones to init rmsnorm weight. And torch.empty gets random weight tensor, which maybe out of float value limits.
fix rmsnorm init weight bug.9d3d7be5
Shan1990

@chielo pls review this pr, thx.
chielo

@Shan1990 Well. I don't belong to the organization, nor do I have the permission to merge.
But this PR looks good and friendly to initialize chatglm3 for training, etc.
It will be better to have this fix.
DuanKW

@zRzRzRzRzRzRzR pls review this pr, thx.
ZHANGYUXUAN-zR
 Z.ai org

check now
ZHANGYUXUAN-zR changed pull request status to merged

Sign up or log in to comment