facebook

/

omniASR-LLM-300M

like 6

Follow

AI at Meta 13.9k

Hi Meta team,

I've been working on integrating OmniASR into vLLM
(see https://github.com/vllm-project/vllm/pull/36864), and ran into a
few items that prevent the model from loading via the standard
HuggingFace pipeline directly from this repo.

Issues found

1. Missing config.json: The repo currently only has the .pt
   checkpoint, no config.json describing the architecture, model_type,
   encoder/decoder hyperparameters, etc.

2. Tokenizer files: No tokenizer.model / tokenizer_config.json
   in the repo; users need to source these separately.

Here are some special configurations I found for this model:

1. RoPE convention: The reference implementation uses GPT-J style
   interleaved RoPE, but most HF / vLLM LLaMA implementations default to
   NeoX style. I worked around this with a one-time Q/K weight
   permutation at load time (mathematically equivalent, bit-exact verified).

2. rms_norm_eps: Reference code uses 1e-5

Working fork

I've published a working version at
ziywang50/omniASR-LLM-300M
that includes:

• model.safetensors (converted from .pt)
• config.json with all encoder + text_config (LLaMA decoder) fields
• tokenizer.model and tokenizer_config.json
  I have also forked 1B, 3B, and 7B models.

vLLM tests pass with this fork (English + multilingual French samples
verified bit-exact against the reference fairseq2 implementation).

Offer

Happy to open a PR to this repo with the missing config + tokenizer
files (the weights themselves don't need to change). Would that help?

Best,
Ziyue