Hugging Face's logo

TevunahAi
/
NextCoder-7B-FP8

Text Generation
Transformers
Safetensors
qwen2
code
fp8
quantized
nextcoder
microsoft
conversational
text-generation-inference
compressed-tensors
Model card Files
xet
Community
1
rockylynnstein commited on
Commit
d9f7ee5
·
verified ·
1 Parent(s): 336e9a5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -49,7 +49,7 @@ import torch
49
  # Load model with FP8 quantization
50
  model = AutoModelForCausalLM.from_pretrained(
51
  "TevunahAi/NextCoder-7B-FP8",
52
- torch_dtype=torch.float8_e4m3fn, # FP8 dtype
53
  device_map="auto",
54
  low_cpu_mem_usage=True,
55
  )
 
49
  # Load model with FP8 quantization
50
  model = AutoModelForCausalLM.from_pretrained(
51
  "TevunahAi/NextCoder-7B-FP8",
52
+ torch_dtype=torch.bfloat16,
53
  device_map="auto",
54
  low_cpu_mem_usage=True,
55
  )