Instructions to use Akfmsk/LocateAnything-3B-custom with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Akfmsk/LocateAnything-3B-custom with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Akfmsk/LocateAnything-3B-custom", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Akfmsk/LocateAnything-3B-custom", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Akfmsk/LocateAnything-3B-custom with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Akfmsk/LocateAnything-3B-custom"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Akfmsk/LocateAnything-3B-custom",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Akfmsk/LocateAnything-3B-custom

SGLang

How to use Akfmsk/LocateAnything-3B-custom with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Akfmsk/LocateAnything-3B-custom" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Akfmsk/LocateAnything-3B-custom",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Akfmsk/LocateAnything-3B-custom" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Akfmsk/LocateAnything-3B-custom",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Akfmsk/LocateAnything-3B-custom with Docker Model Runner:
```
docker model run hf.co/Akfmsk/LocateAnything-3B-custom
```

LocateAnything-3B-custom

Commit History

Jul 3

21:55
Guarantee completely safe memory bounds by chunking to 1024 and compiling every single layer independently, preventing the XLA scheduler from overlapping chunk lifetimes across layers

Akfmsk

7976ec3
21:48
Revert padding and use a 4-chunk Memory-Efficient Attention to perfectly balance the 4.2GB peak memory and 16-node XLA graph size, compiling instantly without OOMing the TPU or Host RAM

Akfmsk

5aa7afb
21:41
Fix severe XLA compiler memory leak and 10-minute hang by padding both head_dim and seq_len to multiples of 128 to perfectly satisfy the TPU Flash Attention kernel, completely eliminating the fallback graph explosion

Akfmsk

fe892dd
21:17
Fix type error for Transformers 5.12+ where DynamicCache uses '.layers' instead of '.key_cache'

Akfmsk

3f5fa6d
21:08
Fix type error: 'DynamicCache' object is not subscriptable by explicitly accessing .key_cache and .value_cache on newer Transformers versions

Akfmsk

7e3f58a
06:47
Implement manual Memory-Efficient Attention by chunking the Query sequence dimension to forcefully bound the memory of PyTorch XLA's naive fallback kernel from 17.2 GB down to safe 1 GB blocks

Akfmsk

2e46ef6
06:36
Pad head_dim from 72 to 128 to trick PyTorch XLA into mapping the attention calculation to the highly-optimized Pallas TPU Flash Attention kernel, completely avoiding the catastrophic O(N^2) memory fallback

Akfmsk

8f6ba09
06:28
Trigger PyTorch XLA Flash Attention kernel natively by calling F.scaled_dot_product_attention without any explicit masks or eager math, resolving the 8.26GB Out-Of-Memory failure and restoring the chunked graph execution

Akfmsk

d72b51f
06:15
Inject xm.mark_step() every 4 layers inside the Vision Model encoder to prevent PyTorch XLA from accumulating 8.3GB of intermediate activations during graph compilation and execution

Akfmsk

703d98a
05:42
Defeat XLA compiler graph rewriter by injecting mathematical fusion breakers into the vision model batched matmuls, preventing XLA from aggressively flattening them back into a single massive 16416x16416 SDPA

Akfmsk

672baa4
05:31
Fix massive OOM in the vision model by replacing the 16416x16416 block-diagonal attention mask with a memory-efficient batched slice SDPA

Akfmsk

e640dfc
05:21
Sever XLA upper bound tracking on chunk_embeds slice to prevent the compiler from padding the attention matrix back to 16416x16416

Akfmsk

2d02f79
04:13
Switch to dynamic torch.where fusion breaker to guarantee XLA algebraic simplifier cannot bypass it

Akfmsk

3b4b10e
04:05
Disable PyTorch XLA dot_product_attention_fusion pattern matching by adding a mathematically invisible constant, preventing the TPU compiler from padding q_len to kv_len in the SDPA CustomCall fallback

Akfmsk

0075ff2
03:50
Force eager attention mode to bypass PyTorch XLA Math SDPA padding bug which allocates a square matrix for dynamic sequence lengths

Akfmsk

8ed3055
03:40
Pad final chunk to keep q_len perfectly static, preventing PyTorch XLA from bucketing q_len to 16416 and allocating a 17GB square matrix during Math SDPA

Akfmsk

4c381d5
03:30
Remove forced attention_mask=None to prevent PyTorch XLA Math SDPA from allocating a full max_len^2 causal mask during generation chunks

Akfmsk

deef94a
03:14
Move 16416-sized StaticCache and padded_input_ids allocations after the chunked prefill loop to prevent XLA dynamic shape padding from blowing up the Math SDPA matrix to 17GB

Akfmsk

c0aebd8
03:05
Implement chunked prefill (4096 tokens per chunk) to avoid Math backend OOM on TPU

Akfmsk

cbd3875
02:55
Explicitly set attention_mask=None for SDPA prefill to trigger Pallas FlashAttention and avoid OOM

Akfmsk

1c64569
02:45
Fix DynamicCache iteration in transformers 5.0

Akfmsk

40a8955
02:37
Fix DynamicCache indexing

Akfmsk

cceb21b
02:31
Fix next_decoder_cache.to_legacy_cache() error in Qwen2Model

Akfmsk

01e5143
02:23
Fix max_cache_len property conflict in LocateAnythingStaticCache

Akfmsk

a5ae7ae

Jul 2

23:23
Fix TPU OOM by bypassing StaticCache for prefill and implementing custom caching logic without explicit 4D masks

Akfmsk

98e1225
23:08
Safely retrieve max_cache_length in Qwen2ForCausalLM

Akfmsk

6ef731e
22:54
Use get_seq_length() instead of seen_tokens in Qwen2ForCausalLM

Akfmsk

00c4f1b
22:42
Call post_init in LocateAnythingForConditionalGeneration constructor

Akfmsk

aa5099a
22:37
Remove _tied_weights_keys from Qwen2ForCausalLM

Akfmsk

cf5963c
22:30
Use getattr for pad_token_id in Qwen2Model

Akfmsk

306fd68
22:25
Set default attention implementation to sdpa

Akfmsk

d208661
22:19
Use relative import for Qwen2Config in configuration_locateanything.py

Akfmsk

e5f776d
22:13
Set _MAGI_AVAILABLE = False in modeling_qwen2.py

Akfmsk

e125991

Jul 1

02:07
Restore _CONFIG_FOR_DOC and other doc strings in modeling_qwen2.py

Akfmsk

75157c2

Jun 30

23:59
Initial custom version

Akfmsk

6fd1cee

Jun 12

10:51
Document LA Flash implementation details

ShihaoW

Verified signature
c32291c
10:37
Refine LA Flash plan documentation

ShihaoW

Verified signature
fb9f8b1
10:28
Update README.md

ShihaoW

Verified signature
fd5777e
10:19
Add batch inference and LA Flash runtime

ShihaoW

Verified signature
8d21b77

Jun 8

19:55
Update README.md

Zhiding

Verified signature
272068e

May 27

08:30
Update README.md

ShihaoW

Verified signature
7a81d81
08:27
Update README.md

ShihaoW

Verified signature
567ab66

May 25

13:55
Initial snapshot

exiawsh

d979fab

Commit History

Jul 3

Guarantee completely safe memory bounds by chunking to 1024 and compiling every single layer independently, preventing the XLA scheduler from overlapping chunk lifetimes across layers

Revert padding and use a 4-chunk Memory-Efficient Attention to perfectly balance the 4.2GB peak memory and 16-node XLA graph size, compiling instantly without OOMing the TPU or Host RAM

Fix severe XLA compiler memory leak and 10-minute hang by padding both head_dim and seq_len to multiples of 128 to perfectly satisfy the TPU Flash Attention kernel, completely eliminating the fallback graph explosion

Fix type error for Transformers 5.12+ where DynamicCache uses '.layers' instead of '.key_cache'

Fix type error: 'DynamicCache' object is not subscriptable by explicitly accessing .key_cache and .value_cache on newer Transformers versions

Implement manual Memory-Efficient Attention by chunking the Query sequence dimension to forcefully bound the memory of PyTorch XLA's naive fallback kernel from 17.2 GB down to safe 1 GB blocks

Pad head_dim from 72 to 128 to trick PyTorch XLA into mapping the attention calculation to the highly-optimized Pallas TPU Flash Attention kernel, completely avoiding the catastrophic O(N^2) memory fallback

Trigger PyTorch XLA Flash Attention kernel natively by calling F.scaled_dot_product_attention without any explicit masks or eager math, resolving the 8.26GB Out-Of-Memory failure and restoring the chunked graph execution

Inject xm.mark_step() every 4 layers inside the Vision Model encoder to prevent PyTorch XLA from accumulating 8.3GB of intermediate activations during graph compilation and execution

Defeat XLA compiler graph rewriter by injecting mathematical fusion breakers into the vision model batched matmuls, preventing XLA from aggressively flattening them back into a single massive 16416x16416 SDPA

Fix massive OOM in the vision model by replacing the 16416x16416 block-diagonal attention mask with a memory-efficient batched slice SDPA

Sever XLA upper bound tracking on chunk_embeds slice to prevent the compiler from padding the attention matrix back to 16416x16416

Switch to dynamic torch.where fusion breaker to guarantee XLA algebraic simplifier cannot bypass it

Disable PyTorch XLA dot_product_attention_fusion pattern matching by adding a mathematically invisible constant, preventing the TPU compiler from padding q_len to kv_len in the SDPA CustomCall fallback

Force eager attention mode to bypass PyTorch XLA Math SDPA padding bug which allocates a square matrix for dynamic sequence lengths

Pad final chunk to keep q_len perfectly static, preventing PyTorch XLA from bucketing q_len to 16416 and allocating a 17GB square matrix during Math SDPA

Remove forced attention_mask=None to prevent PyTorch XLA Math SDPA from allocating a full max_len^2 causal mask during generation chunks

Move 16416-sized StaticCache and padded_input_ids allocations after the chunked prefill loop to prevent XLA dynamic shape padding from blowing up the Math SDPA matrix to 17GB

Implement chunked prefill (4096 tokens per chunk) to avoid Math backend OOM on TPU

Explicitly set attention_mask=None for SDPA prefill to trigger Pallas FlashAttention and avoid OOM

Fix DynamicCache iteration in transformers 5.0

Fix DynamicCache indexing

Fix next_decoder_cache.to_legacy_cache() error in Qwen2Model

Fix max_cache_len property conflict in LocateAnythingStaticCache

Jul 2

Fix TPU OOM by bypassing StaticCache for prefill and implementing custom caching logic without explicit 4D masks

Safely retrieve max_cache_length in Qwen2ForCausalLM

Use get_seq_length() instead of seen_tokens in Qwen2ForCausalLM

Call post_init in LocateAnythingForConditionalGeneration constructor

Remove _tied_weights_keys from Qwen2ForCausalLM

Use getattr for pad_token_id in Qwen2Model

Set default attention implementation to sdpa

Use relative import for Qwen2Config in configuration_locateanything.py

Set _MAGI_AVAILABLE = False in modeling_qwen2.py

Jul 1

Restore _CONFIG_FOR_DOC and other doc strings in modeling_qwen2.py

Jun 30

Initial custom version

Jun 12

Document LA Flash implementation details

Refine LA Flash plan documentation

Update README.md

Add batch inference and LA Flash runtime

Jun 8

Update README.md

May 27

Update README.md

Update README.md

May 25

Initial snapshot