Transformers
GGUF
sparse-attention
approximate-nearest-neighbors
faiss
qwen3
long-context
conversational
Instructions to use datasysdev/ann-sparseattention with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use datasysdev/ann-sparseattention with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("datasysdev/ann-sparseattention", dtype="auto") - llama-cpp-python
How to use datasysdev/ann-sparseattention with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="datasysdev/ann-sparseattention", filename="gguf/Qwen3-4B-Instruct-2507-F16-ann-6layer-k128-v2.gguf", )
llm.create_chat_completion( messages = "No input example has been defined for this model task." )
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use datasysdev/ann-sparseattention with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf datasysdev/ann-sparseattention:F16 # Run inference directly in the terminal: llama cli -hf datasysdev/ann-sparseattention:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf datasysdev/ann-sparseattention:F16 # Run inference directly in the terminal: llama cli -hf datasysdev/ann-sparseattention:F16
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf datasysdev/ann-sparseattention:F16 # Run inference directly in the terminal: ./llama-cli -hf datasysdev/ann-sparseattention:F16
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf datasysdev/ann-sparseattention:F16 # Run inference directly in the terminal: ./build/bin/llama-cli -hf datasysdev/ann-sparseattention:F16
Use Docker
docker model run hf.co/datasysdev/ann-sparseattention:F16
- LM Studio
- Jan
- Ollama
How to use datasysdev/ann-sparseattention with Ollama:
ollama run hf.co/datasysdev/ann-sparseattention:F16
- Unsloth Studio
How to use datasysdev/ann-sparseattention with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for datasysdev/ann-sparseattention to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for datasysdev/ann-sparseattention to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for datasysdev/ann-sparseattention to start chatting
- Pi
How to use datasysdev/ann-sparseattention with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf datasysdev/ann-sparseattention:F16
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "datasysdev/ann-sparseattention:F16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use datasysdev/ann-sparseattention with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf datasysdev/ann-sparseattention:F16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default datasysdev/ann-sparseattention:F16
Run Hermes
hermes
- Atomic Chat new
- OpenClaw new
How to use datasysdev/ann-sparseattention with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf datasysdev/ann-sparseattention:F16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "datasysdev/ann-sparseattention:F16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Docker Model Runner
How to use datasysdev/ann-sparseattention with Docker Model Runner:
docker model run hf.co/datasysdev/ann-sparseattention:F16
- Lemonade
How to use datasysdev/ann-sparseattention with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull datasysdev/ann-sparseattention:F16
Run and chat with the model
lemonade run user.ann-sparseattention-F16
List all available models
lemonade list
Update model card with all32 status
Browse files
README.md
CHANGED
|
@@ -131,6 +131,14 @@ Per-layer step-500 mass@K at K=128:
|
|
| 131 |
|
| 132 |
The next run reserves `[0, 1, 2, 35]` and trains layers `3..34`.
|
| 133 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
## Positioning against related methods
|
| 135 |
|
| 136 |
The paper frames this method as closest in asymptotic shape to Reformer and
|
|
@@ -152,6 +160,14 @@ superiority. The clean result proves the approach for the six-layer pilot; the
|
|
| 152 |
active all32 reserved-layer run tests whether broad near-whole-model
|
| 153 |
substitution can preserve that quality.
|
| 154 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 155 |
## Checkpoints
|
| 156 |
|
| 157 |
Important checkpoint paths in this HF repo:
|
|
|
|
| 131 |
|
| 132 |
The next run reserves `[0, 1, 2, 35]` and trains layers `3..34`.
|
| 133 |
|
| 134 |
+
First diagnostic from the active all32 run:
|
| 135 |
+
|
| 136 |
+
| Step | Recall@K eval | PPL gap | Read |
|
| 137 |
+
|---:|---:|---:|---|
|
| 138 |
+
| 250 | 0.812 | +2.28% | already better than all36 best training eval |
|
| 139 |
+
|
| 140 |
+
This is not a final result; the run is continuing toward step 1000.
|
| 141 |
+
|
| 142 |
## Positioning against related methods
|
| 143 |
|
| 144 |
The paper frames this method as closest in asymptotic shape to Reformer and
|
|
|
|
| 160 |
active all32 reserved-layer run tests whether broad near-whole-model
|
| 161 |
substitution can preserve that quality.
|
| 162 |
|
| 163 |
+
This method targets a different deployment scenario than native
|
| 164 |
+
sliding-window/state-space/hybrid architectures such as Mistral-style sliding
|
| 165 |
+
window, Mamba, or Qwen3.6 Gated DeltaNet hybrids. Those models are trained from
|
| 166 |
+
scratch with their sparse or hybrid mechanism in place. This work is post-hoc:
|
| 167 |
+
train a base model with full attention for maximum expressivity, then add
|
| 168 |
+
lightweight retrieval projections afterward to make inference sub-linear without
|
| 169 |
+
changing base weights.
|
| 170 |
+
|
| 171 |
## Checkpoints
|
| 172 |
|
| 173 |
Important checkpoint paths in this HF repo:
|