README.md

metadata

language:
  - en
  - zh
license: mit
library_name: mlx
tags:
  - mlx
  - longcat
  - lsa
  - moe
  - ngram-embedding
base_model: meituan-longcat/LongCat-Flash-Lite-Sparse
pipeline_tag: text-generation

LongCat-Flash-Lite-Sparse-6bit (MLX)

6-bit MLX quantization of meituan-longcat/LongCat-Flash-Lite-Sparse (69B-A3B, LongcatCausalLM).

6-bit (52 GB of weights) is the middle-ground variant, for a 96 GB Mac. Also available: 8-bit (68 GB, 128 GB Macs, near-lossless) and 4-bit (~36 GB, 64 GB Macs, fastest).

What's in this checkpoint

LongCat-Flash-Lite-Sparse adds three things vanilla LongCat-Flash lacks:

• LongCat Sparse Attention (LSA) — a DeepSeek-style lightning indexer over MLA, with streaming-aware indexing (fixed sink + local window) and cross-layer index reuse. Native long context.
• Zero-computation (identity) experts in the ScMoE decoder (256 routed + 128 identity, top-12).
• N-gram ("oe") input embedding — ~46% of the parameters, fused into the token embedding.

The n-gram fix

The oe embedding hash and tables are identical to the published n-gram references (the Scaling Embeddings paper, mlx-lm, SGLang, llama.cpp, Meituan's dense modeling). The one difference in LongcatCausalLM is the fusion: it keeps the word embedding at full scale — word + Σ projections / (1 + num_embedders) — rather than the dense form (word + Σ projections) / (1 + num_embedders). Dividing the word by 1 + num_embedders garbles generation; this build applies the correct fusion.

Usage

Requires mlx-vlm with longcat_flash_sparse support (PR #2063):

pip install git+https://github.com/Lazarus-931/mlx-vlm@add-longcat-flash

from mlx_vlm import load, generate
model, processor = load("AlazarM/LongCat-Flash-Lite-Sparse-6bit", trust_remote_code=True)
tok = processor.tokenizer
text = tok.apply_chat_template(
    [{"role": "user", "content": "What is the capital of France?"}],
    tokenize=False, add_generation_prompt=True,
)
print(generate(model, processor, text, max_tokens=64, temperature=0.0))
# -> The capital of France is Paris.

Throughput (M5 Max, 128 GB, batch 1, greedy)

Decode tok/s across the published quantizations:

ctx	4-bit	6-bit	8-bit
512	112	87	80
2048	85	72	65
8192	83	71	65
32768	73	64	60

Batch-1 decode is partly weight-bandwidth-bound, so lower precision is faster (~30% spread 4→8-bit); LSA keeps all three nearly flat as context grows. Peak memory across 512→32k: 4-bit ~39–45 GB, 6-bit ~56–63 GB, 8-bit ~74–80 GB. 6-bit is the balance point — most of 8-bit's quality at ~⅔ the footprint.

License

MIT, inherited from the base model.