annekethvij commited on
Commit
e7fa016
·
verified ·
1 Parent(s): cdb81e8

Document inference precision and preserve FP32 expert biases

Browse files
Files changed (2) hide show
  1. README.md +7 -22
  2. modeling_afmoe.py +1 -0
README.md CHANGED
@@ -82,17 +82,14 @@ Try it out now at [chat.arcee.ai](http://chat.arcee.ai/)
82
 
83
  ## Transformers
84
 
85
- Use the `main` transformers branch
86
 
87
- ```
88
- git clone https://github.com/huggingface/transformers.git
89
- cd transformers
90
 
91
- # pip
92
- pip install '.[torch]'
93
 
94
- # uv
95
- uv pip install '.[torch]'
96
  ```
97
 
98
  ```python
@@ -104,7 +101,8 @@ tokenizer = AutoTokenizer.from_pretrained(model_id)
104
  model = AutoModelForCausalLM.from_pretrained(
105
  model_id,
106
  torch_dtype=torch.bfloat16,
107
- device_map="auto"
 
108
  )
109
 
110
  messages = [
@@ -130,19 +128,6 @@ response = tokenizer.decode(outputs[0], skip_special_tokens=True)
130
  print(response)
131
  ```
132
 
133
- If using a released transformers, simply pass "trust_remote_code=True":
134
-
135
- ```python
136
- model_id = "arcee-ai/Trinity-Mini"
137
- tokenizer = AutoTokenizer.from_pretrained(model_id)
138
- model = AutoModelForCausalLM.from_pretrained(
139
- model_id,
140
- torch_dtype=torch.bfloat16,
141
- device_map="auto",
142
- trust_remote_code=True
143
- )
144
- ```
145
-
146
  ## VLLM
147
 
148
  Supported in VLLM release 0.11.1
 
82
 
83
  ## Transformers
84
 
85
+ **Inference precision:** For unquantized BF16 inference, the intended parameter dtypes are BF16 for normalization and router weights, and FP32 for expert correction biases (`expert_bias`). Preserve FP32 checkpoint bias values during loading.
86
 
87
+ This checkpoint stores `expert_bias` in BF16; upcast these values to FP32 for inference.
 
 
88
 
89
+ Use the bundled model implementation with `transformers==4.57.1` and `trust_remote_code=True`.
 
90
 
91
+ ```bash
92
+ pip install "transformers==4.57.1" "accelerate==1.11.0"
93
  ```
94
 
95
  ```python
 
101
  model = AutoModelForCausalLM.from_pretrained(
102
  model_id,
103
  torch_dtype=torch.bfloat16,
104
+ device_map="auto",
105
+ trust_remote_code=True
106
  )
107
 
108
  messages = [
 
128
  print(response)
129
  ```
130
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
  ## VLLM
132
 
133
  Supported in VLLM release 0.11.1
modeling_afmoe.py CHANGED
@@ -469,6 +469,7 @@ class AfmoeDecoderLayer(GradientCheckpointingLayer):
469
 
470
 
471
  class AfmoePreTrainedModel(PreTrainedModel):
 
472
  config_class = AfmoeConfig
473
  base_model_prefix = "model"
474
  _no_split_modules = ["AfmoeDecoderLayer"]
 
469
 
470
 
471
  class AfmoePreTrainedModel(PreTrainedModel):
472
+ _keep_in_fp32_modules_strict = ["expert_bias"]
473
  config_class = AfmoeConfig
474
  base_model_prefix = "model"
475
  _no_split_modules = ["AfmoeDecoderLayer"]