PumpkinCat/Qwen3-8B-FPA

Qwen AR-FPA CPT Inference Package

This folder contains the converted Hugging Face safetensors checkpoint plus the tokenizer with the added communication token <|comm|>.

Important: do not use plain AutoModelForCausalLM.generate() if you want the AR-FPA behavior. Plain Hugging Face generation will load the weights with normal causal attention and will not insert communication tokens.

Use ar_fpa_generate.py, which:

• loads the checkpoint with plain transformers.AutoModelForCausalLM
• uses the local standalone AR-FPA attention implementation
• inserts <|comm|> tokens into each AR-FPA block
• hides communication tokens from the user-visible output

Requirements

Install the normal Hugging Face/PyTorch dependencies:

pip install -r requirements-ar-fpa.txt

No nemo_automodel / Automodel repository is required for inference. The standalone AR-FPA model definition and kernel are included in modeling_qwen3_ar_fpa.py.

Example

CUDA_VISIBLE_DEVICES=0 python ar_fpa_generate.py \
  --model-dir /path/to/qwen_ar_fpa_cpt_inference_package \
  --prompt "Explain why antibiotics do not work against viral infections." \
  --max-new-tokens 128

You can also load the model directly with Hugging Face remote code:

from transformers import AutoModelForCausalLM, AutoTokenizer

path = "/path/to/qwen_ar_fpa_cpt_inference_package"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True, torch_dtype="auto").cuda()

For correct AR-FPA generation you still need to insert communication tokens. ar_fpa_generate.py provides that wrapper.

AR-FPA Settings

The checkpoint was trained/evaluated with:

{
  "attention": "ar_fpa",
  "block_size": 128,
  "num_comm_tokens": 4,
  "comm_token": "<|comm|>",
  "max_seq_length": 4096
}

The tokenizer contains <|comm|> at id 151669. The checkpoint embedding matrix is padded to vocab size 151936; this is expected.