shadowrock-io/LocateAnything-3B-Community-NVFP4

LocateAnything-3B — Community NVFP4

Unofficial community quantization — not an NVIDIA release.

NVFP4 W4A4 build of nvidia/LocateAnything-3B (revision c32291ca), GPTQ-rounded and validated against a BF16 baseline of the published checkpoint. All credit for the model belongs to NVIDIA; this repo changes only the numeric precision of the decoder weights. Checkpoint size drops from 7.6 GB to 2.9 GB. On our COCO val2017 grounding benchmark the two NVFP4 seeds straddle the BF16 model's own seed-to-seed variance band: one seed lands inside it on every metric, the other dips 0.002–0.003 below the band floor on F1/recall — an order of magnitude less than the BF16 seed spread itself.

Pick this variant for Blackwell GPUs where memory is the constraint. The companion FP8 build is closer to lossless on the fixture-level gates and runs natively on Ada/Hopper too.

What is quantized

Scheme: NVFP4 via llm-compressor 0.11 — FP4 (E2M1) weights and activations in groups of 16 with FP8 per-group scales and a global scale. Weights are rounded with GPTQ (error-compensating, 128 image-grounded calibration samples, hessians accumulated through the model's own multimodal forward path); activation global scales are calibrated on the same set. Only the Qwen2.5-3B language-model decoder linears are quantized. Everything the grounding head depends on stays BF16:

• MoonViT vision tower and the mlp1 connector
• embed_tokens / lm_head (tied), which protects the <0>–<1000> coordinate token rows

Calibration prompts are the committed 128-sample COCO val2017 calibset (image URLs only, no images redistributed), forwarded with precomputed BF16 vision features — the same path the evaluation harness uses.

Validation

All evaluation drives the model through its own hybrid parallel-box-decoding runtime (batch_utils.generate_batch_hybrid) under the model's canonical sampling settings (temperature 0.7, top_p 0.9, repetition_penalty 1.1) with the torch RNG reseeded per case, which makes runs bit-reproducible on a fixed stack.

Grounding benchmark, 2×2+2 design. 100 COCO val2017 images (first 100 ids with annotations, ascending — a deterministic slice), one query per ground-truth category per image, 288 queries. Two seeds per model, so the BF16 seed-to-seed spread is measured rather than assumed. LocateAnything emits boxes without confidence scores, so we report score-free greedy-matched precision/recall/F1 instead of score-ranked mAP; the paper's own headline metric family is mean F1.

model / seed	P@0.5	R@0.5	F1@0.5	mean F1 (0.50:0.95)	mean matched IoU
BF16, seed A	0.7345	0.5501	0.6290	0.4761	0.8536
BF16, seed B	0.7329	0.5303	0.6154	0.4691	0.8559
NVFP4, seed A	0.7333	0.5275	0.6136	0.4668	0.8547
NVFP4, seed B	0.7545	0.5374	0.6277	0.4763	0.8539

The BF16 model moves 0.0136 F1@0.5 between seeds. NVFP4 seed B sits inside that band on every metric (precision and mean F1 at or above the band top); seed A lands 0.0018 F1 below the band floor. Matched IoU — box quality on the boxes it does find — is indistinguishable from BF16 on both seeds. The paper reports COCO mean F1 ≈ 52.1 for the full protocol; our BF16 baseline lands ≈ 47.6 under the deviations listed below, and the number that matters here is NVFP4-vs-BF16 under an identical protocol.

Contract test. The quantized export passes the same gate as the BF16 snapshot: parallel-box-decoding forwards active (mtp_forwards > 0), coordinate token ids intact and finite, tied lm_head verified, valid labeled boxes on the fixture set.

Fixture parity. On the 12-case fixture set, 7/12 responses are exact or near-exact (the FP8 companion scores 10/12 on the same gate). 4-bit weights perturb sampled trajectories more than FP8; the benchmark above measures what that costs in grounding quality — close to nothing.

Teacher-forced diagnostics. Full-sequence forced decoding over the BF16 golden responses shows every top-1 flip confined to coordinate positions: zero label or structure flips.

Raw results ship in this repo under evidence/: per-query benchmark predictions for both seeds, fixture parity report, and the teacher-forced capture. Reproduction scripts (quantizer, contract test, teacher-forced capture, COCO benchmark harness) live in the source repo.

Protocol deviations from the paper

• 100-image deterministic slice of val2017, not the full split
• greedy best-first IoU matching (score-free), not a ranked matcher
• queries are per-GT-category per image (assumes category presence is known)

Usage

vLLM (true W4A4)

vLLM reads the compressed-tensors config and runs real FP4 kernels (Blackwell GPUs run native NVFP4 GEMMs; earlier architectures use weight-only FP4):

from vllm import LLM
llm = LLM(model="shadowrock-io/LocateAnything-3B-Community-NVFP4", trust_remote_code=True)

transformers (decompress-on-load)

run_compressed: false is set in config.json: transformers decompresses the FP4 weights to BF16 at load. Numerics of the quantized weights are preserved, but activations run in BF16, so this path measures weight-quantization error only. Pin transformers==4.57.1 and compressed-tensors==0.12.2, and disable torch.compile for the load — compressed-tensors 0.12.2 decorates its FP4 unpacking with torch.compile(fullgraph=True), which crashes on meta tensors during from_pretrained:

import os
os.environ["TORCH_COMPILE_DISABLE"] = "1"
os.environ["TORCHDYNAMO_DISABLE"] = "1"

from transformers import AutoModel
model = AutoModel.from_pretrained(
    "shadowrock-io/LocateAnything-3B-Community-NVFP4",
    trust_remote_code=True, torch_dtype="bfloat16")

Numerics are unaffected; vLLM does not need the workaround. The model's inference contract (sampling settings, hybrid decode runtime) is documented in the upstream README, which ships in this repo together with the pinned model code. Greedy decoding is not supported by the base model (it never emits the stop token); use the canonical sampling settings above.

Prior art

An NVFP4 export of this model exists (switzerchees/LocateAnything-3B-NVFP4, ModelOpt default recipe, synthetic calibration) along with GGUF/MLX/ONNX variants. None we found publish accuracy measurements against the BF16 baseline; this release's purpose is 4-bit quantization with evidence: GPTQ rounding, image-grounded calibration through the real multimodal path, and the benchmark above. Reproduction scripts (quantizer, contract test, teacher-forced capture, COCO benchmark) live in the source repo.

Intended use & limitations

Intended uses are the base model's: open-vocabulary visual grounding and object detection from natural-language queries. The base card's intended-use, safety, and coverage statements — nvidia/LocateAnything-3B — carry over unchanged; quantization alters none of the model's behavior boundaries, only its numeric precision. Our evaluation establishes parity on the COCO benchmark above and nothing beyond it: other image domains, query styles, and hardware/runtime combinations inherit the base model's behavior with quantization noise that we have not measured there.

Attribution & citation

Quantization, validation harness, and card by Matt Busi (@mattbusi on Hugging Face) at ShadowRock. If you use this build, cite the NVIDIA base model — the grounding capability is theirs:

@misc{nvidia2026locateanything,
  title  = {LocateAnything-3B},
  author = {NVIDIA},
  year   = {2026},
  url    = {https://huggingface.co/nvidia/LocateAnything-3B}
}

License

NVIDIA non-commercial license, inherited unchanged from the base model — see LICENSE. Quantized weights are a derivative of NVIDIA's checkpoint and carry the same terms. No COCO images are redistributed here; the benchmark fetches them by id from cocodataset.org. Community build by ShadowRock; no NVIDIA affiliation or endorsement.

About ShadowRock

ShadowRock is an AI-specialized systems integrator and Zendesk Premier Partner. We help businesses get real value from their go-to-market technology, from CRM and support platforms to applied AI like the models in this collection. Find us at shadowrock.io or on LinkedIn.