--- license: mit language: - en - zh base_model: - nvidia/LocateAnything-3B pipeline_tag: zero-shot-object-detection library_name: transformers tags: - LocateAnything-3B - Int4 - VLM - GPTQ --- # LocateAnything-3B This version of LocateAnything-3B have been converted to run on the Axera NPU using **w4a16** quantization. Compatible with Pulsar2 version: 6.0 ## Convert tools links: For those who are interested in model conversion, you can try to export axmodel through the original repo : - https://huggingface.co/nvidia/LocateAnything-3B [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) [AXera NPU HOST LLM Runtime](TODO) ## Support Platform - AX650 - AX650N DEMO Board - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) - [M.2 Accelerator card](https://docs.m5stack.com/zh_CN/ai_hardware/LLM-8850_Card) **Image Process** |Chips| input size | image num | image encoder | ttft(493 tokens) | w4a16 | CMM | Flash | |--|--|--|--|--|--|--|--| |AX650| 560*560 | 1 | 1152.583 ms | 2072.06 ms | 10.61 tokens/sec| 2.9GiB | 3.2GiB | The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value. ## 模型下载（Hugging Face） 先创建模型目录并进入，然后下载到该目录： ```shell mkdir -p AXERA-TECH/LocateAnything-3B cd AXERA-TECH/LocateAnything-3B hf download AXERA-TECH/LocateAnything-3B --local-dir . # structure of the downloaded files tree -L 3 . └── AXERA-TECH └── LocateAnything-3B |-- assert |-- config.json |-- gradio_locateanything_axengine.py |-- image_encoder_mlp.axmodel |-- infer_locateanything_axengine.py |-- model.embed_tokens.weight.bfloat16.bin |-- post_config.json |-- qwen2.5_tokenizer |-- qwen2_5_tokenizer.txt |-- qwen2_p128_l0_together.axmodel |-- qwen2_p128_l10_together.axmodel |-- qwen2_p128_l11_together.axmodel |-- qwen2_p128_l12_together.axmodel |-- qwen2_p128_l13_together.axmodel |-- qwen2_p128_l14_together.axmodel |-- qwen2_p128_l15_together.axmodel |-- qwen2_p128_l16_together.axmodel |-- qwen2_p128_l17_together.axmodel |-- qwen2_p128_l18_together.axmodel |-- qwen2_p128_l19_together.axmodel |-- qwen2_p128_l1_together.axmodel |-- qwen2_p128_l20_together.axmodel |-- qwen2_p128_l21_together.axmodel |-- qwen2_p128_l22_together.axmodel |-- qwen2_p128_l23_together.axmodel |-- qwen2_p128_l24_together.axmodel |-- qwen2_p128_l25_together.axmodel |-- qwen2_p128_l26_together.axmodel |-- qwen2_p128_l27_together.axmodel |-- qwen2_p128_l28_together.axmodel |-- qwen2_p128_l29_together.axmodel |-- qwen2_p128_l2_together.axmodel |-- qwen2_p128_l30_together.axmodel |-- qwen2_p128_l31_together.axmodel |-- qwen2_p128_l32_together.axmodel |-- qwen2_p128_l33_together.axmodel |-- qwen2_p128_l34_together.axmodel |-- qwen2_p128_l35_together.axmodel |-- qwen2_p128_l3_together.axmodel |-- qwen2_p128_l4_together.axmodel |-- qwen2_p128_l5_together.axmodel |-- qwen2_p128_l6_together.axmodel |-- qwen2_p128_l7_together.axmodel |-- qwen2_p128_l8_together.axmodel |-- qwen2_p128_l9_together.axmodel |-- qwen2_post.axmodel |-- results `-- test_data 4 directories, 44 files ``` ## Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board ### Gradio Demo ```shell (base) root@ax650:~/LocateAnything# python gradio_locateanything_axengine.py [INFO] Available providers: ['AxEngineExecutionProvider', 'AXCLRTExecutionProvider'] [Gradio] starting LocateAnything UI [Gradio] local: http://127.0.0.1:7860 [Gradio] LAN: http://10.126.29.50:7860 [Gradio] LAN: http://10.126.29.68:7860 [Gradio] LAN: http://172.17.0.1:7860 [Gradio] Use another computer in the same LAN to open the LAN URL. * Running on local URL: http://0.0.0.0:7860 * To create a public link, set `share=True` in `launch()`. ``` Output: detection: ![detection](./assert/person.jpg) ocr: ![ocr](./assert/ocr.jpg) phrase grounding: ![phrase grounding](./assert/phrase_grounding.jpg) ### WebUI (via ax-llm serve) Besides the Gradio demo, [ax-llm](https://github.com/AXERA-TECH/ax-llm) provides an OpenAI-compatible `serve` plus a lightweight, dependency-free (Python stdlib only) web front-end — `scripts/locateanything_webui.py` — that draws detection boxes in real time as they stream. ![webui](./web.jpeg) #### 1. Start the model service with ax-llm Build / obtain the `axllm` binary from [ax-llm](https://github.com/AXERA-TECH/ax-llm) (AX650 host build), then serve this model. The folder already contains everything `serve` needs (`config.json`, `qwen2_5_tokenizer.txt`, `post_config.json` and the axmodels): ```shell # on the AX650 host (M4N-Dock / AX650N DEMO Board / M.2 card) LD_LIBRARY_PATH=/soc/lib ./axllm serve /path/to/AXERA-TECH/LocateAnything-3B --port 8010 # ... loading ... # OpenAI API Server starting on http://0.0.0.0:8010 # Models: AXERA-TECH/LocateAnything-3B ``` #### 2. Start the WebUI The `pexels-images/` sample images (with per-image `tags.json` presets) ship in this repo. Point the webui at the running serve and at that image folder: ```shell AXLLM_SERVE_URL=http://127.0.0.1:8010 \ AXLLM_IMAGE_DIR=/path/to/AXERA-TECH/LocateAnything-3B/pexels-images \ python3 scripts/locateanything_webui.py --host 0.0.0.0 --port 7861 ``` Then open `http://:7861` from any machine on the same LAN. - Pick a preset thumbnail from the top banner, or **Upload** your own image. - **Object detection** — edit the category chips (one query per category, - **Phrase grounding** — type a description (e.g. `the dog on the left`) to locate a specific instance. - Press **Detect**; boxes are drawn one-by-one as they stream in. **Stop** Flags: `--host`, `--port`, `--serve-url`, `--image-dir`, `--model`.