stefaniancu commited on
Commit
bfc5158
·
verified ·
1 Parent(s): 9198e8e

The weights are bfloat16 now; say so instead of explaining float32

Browse files
Files changed (1) hide show
  1. README.md +7 -4
README.md CHANGED
@@ -51,10 +51,13 @@ out = model.generate(inputs["input_ids"].to("cuda"), max_new_tokens=128)
51
  print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
52
  ```
53
 
54
- `dtype=torch.bfloat16` is worth passing explicitly. The weights are stored as
55
- float32 because the release verification compares logits exactly and that
56
- comparison is only meaningful in float32; the model was trained in bfloat16 and
57
- loses nothing by being loaded that way.
 
 
 
58
 
59
  Measured on a single RTX 5070 (12 GB, SDPA path): **~91 tokens/second**, ~3.3 GB
60
  resident.
 
51
  print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
52
  ```
53
 
54
+ The weights are stored in bfloat16, the precision the model was trained in and the
55
+ precision the forward pass casts to regardless. They were published as float32 at first,
56
+ because release verification compares logits exactly and that is only meaningful in float32
57
+ but that made the files twice the size for no information: 47% of the parameters were already
58
+ bfloat16-exact, and under bfloat16 the two files produce bitwise identical logits. Loading in
59
+ float32 is still possible and shifts the top-8 probabilities by up to 1.7e-03, a mode the
60
+ model was never trained in.
61
 
62
  Measured on a single RTX 5070 (12 GB, SDPA path): **~91 tokens/second**, ~3.3 GB
63
  resident.