Cadododoom

/

agents-a1-modelopt-nvfp4

like 0

Hey, just curious if there was some part of this I did not quite understand. I tried mimicing these configurations in my vllm serving, and I always result in a default over to ModelOpt (even when pushing the compressed-tensors flag) and then furthermore some errors relating to rope, kwargs, pretty much an all-around requirement for forcing the max model len override env var, and then lastly - it spectacularly ends in a python traceback right at the first shard load.

Here's the most curious snippet of the crashing serve, albeit I believe I was somehow ignoring the kwarg errors anyways, just the only independent variable outside of the fact that it's the shard loads that's the problem (separate from below). Just never encountered anything similar, also, latest vllm 26 nightly

(APIServer pid=2412710) [transformers] Unrecognized keys in `rope_parameters` fo
r 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=2412710) [transformers] Unrecognized keys in `rope_parameters` fo
r 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=2412710) INFO 08-03 09:04:48 [model.py:645] Resolved architecture
: Qwen3_5MoeForCausalLM
(APIServer pid=2412710) WARNING 08-03 09:04:48 [model.py:2377] User-specified ma
x_model_len (112000) is greater than the derived max_model_len (max_position_emb
eddings=32768.0 or model_max_length=None in model's config.json). VLLM_ALLOW_LON
G_MAX_MODEL_LEN must be used with extreme caution. If the model uses relative po
sition encoding (RoPE), positions exceeding derived_max_model_len lead to nan. I
f the model uses absolute position encoding, positions exceeding derived_max_mod
el_len will cause a CUDA array out-of-bounds error.
(APIServer pid=2412710) INFO 08-03 09:04:48 [model.py:1883] Using max model len
112000
(APIServer pid=2412710) INFO 08-03 09:04:48 [cache.py:296] Using fp8 data type t
o store kv cache. It reduces the GPU memory footprint and boosts the performance
. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=2412710) INFO 08-03 09:04:48 [scheduler.py:242] Chunked prefill i
s enabled with max_num_batched_tokens=4096.
(APIServer pid=2412710) WARNING 08-03 09:04:48 [modelopt.py:1019] Detected Model
Opt NVFP4 checkpoint (quant_algo=W4A16_NVFP4). Please note that the format is ex
perimental and could change in future.
(APIServer pid=2412710) INFO 08-03 09:04:48 [kernel.py:306] Final IR op priority
 after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_
add_rms_norm=['native'])
(APIServer pid=2412710) [transformers] Unrecognized keys in `rope_parameters` fo
r 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
(APIServer pid=2412710) [transformers] Unrecognized keys in `rope_parameters` fo
r 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id
Ignored unknown kwarg option id

...............