Instructions to use keepsloading/icml_repro_scratch with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use keepsloading/icml_repro_scratch with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="keepsloading/icml_repro_scratch")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("keepsloading/icml_repro_scratch")
model = AutoModelForCausalLM.from_pretrained("keepsloading/icml_repro_scratch", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use keepsloading/icml_repro_scratch with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "keepsloading/icml_repro_scratch"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/keepsloading/icml_repro_scratch

SGLang

How to use keepsloading/icml_repro_scratch with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "keepsloading/icml_repro_scratch" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "keepsloading/icml_repro_scratch" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use keepsloading/icml_repro_scratch with Docker Model Runner:
```
docker model run hf.co/keepsloading/icml_repro_scratch
```

icml_repro_scratch

File size: 2,666 Bytes

46b9eea

{
  "release": "jiamingshan/AHA-L2A-Qwen3-1.7B-repro",
  "source_git": {
    "repository": "https://github.com/shanjiaming/AHA",
    "branch": "codex/l2a-router-granularity",
    "commit": "b47a549"
  },
  "base_checkpoint": {
    "location": "repository root",
    "description": "Qwen3-1.7B tuned vanilla, global step 6911",
    "sha256_model_safetensors": "22c971a65f7f1835b0e2a38b45f2f92191f5d8b63b30969f2f61a69d30afbc05"
  },
  "execution": {
    "distributed_training": false,
    "world_size_per_arm": 1,
    "gpu_count_per_arm": 1,
    "one_gpu_machine": "run AHA then L2A-style sequentially on GPU 0",
    "two_or_more_gpu_machine": "run AHA on GPU 0 and L2A-style on GPU 1 concurrently; remaining GPUs are unused",
    "arm_definition": "one independent experimental variant, not a processor architecture or GPU group"
  },
  "arms": {
    "aha": {
      "router_granularity": "token_kv_head",
      "description": "one native gate per token, KV head, and layer"
    },
    "l2a_style": {
      "router_granularity": "token",
      "description": "one native gate per token and layer, shared across all heads",
      "claim_boundary": "L2A-style shared-gate; not an official L2A reproduction"
    }
  },
  "common": {
    "local_attention": {
      "kind": "sink_recent",
      "sink_tokens": 64,
      "recent_tokens": 256
    },
    "gate_initialization": {
      "weight": "zeros",
      "full_probability": 0.9
    },
    "effective_sparsity_denominator": "token x KV-head x layer"
  },
  "stage_1": {
    "dataset": "recipe/data/am_distilled_long_mix",
    "rows": 1024,
    "max_length": 8192,
    "batch_size": 1,
    "optimizer_steps": 300,
    "epochs": 0.29296875,
    "trainable": "native gate rows only",
    "learning_rate": 0.00003,
    "warmup_ratio": 0.1,
    "hidden_state_distillation": 1.0,
    "regularizer": 0.1,
    "ce": 0.0,
    "train_threshold": 0.5,
    "seed": 42
  },
  "stage_2": {
    "dataset": "same 1024 rows",
    "max_length": 8192,
    "batch_size": 1,
    "optimizer_steps_run": 75,
    "selected_checkpoint_step": 25,
    "selected_epochs": 0.0244140625,
    "gate_learning_rate": 0.000003,
    "backbone_learning_rate": 0.0000003,
    "ce": 1.0,
    "attention_distillation": 0.5,
    "regularizer": 0.01,
    "train_threshold": 0.58,
    "freeze_embeddings_and_lm_head": true,
    "seed": 47
  },
  "evaluation": {
    "thresholds": [0.45, 0.5, 0.525, 0.55, 0.575, 0.6, 0.625, 0.65],
    "strict_prefill_and_decode_sparse_routing": true,
    "force_full_heads": false,
    "full_decode": false,
    "headline": "highest measured effective sparsity whose score is at least 95% of tuned vanilla on every suite"
  }
}