Instructions to use ArchSpace-Collection/SiameseNorm-DepthAttention with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ArchSpace-Collection/SiameseNorm-DepthAttention with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ArchSpace-Collection/SiameseNorm-DepthAttention")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ArchSpace-Collection/SiameseNorm-DepthAttention", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ArchSpace-Collection/SiameseNorm-DepthAttention with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ArchSpace-Collection/SiameseNorm-DepthAttention"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArchSpace-Collection/SiameseNorm-DepthAttention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ArchSpace-Collection/SiameseNorm-DepthAttention

SGLang

How to use ArchSpace-Collection/SiameseNorm-DepthAttention with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ArchSpace-Collection/SiameseNorm-DepthAttention" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArchSpace-Collection/SiameseNorm-DepthAttention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ArchSpace-Collection/SiameseNorm-DepthAttention" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArchSpace-Collection/SiameseNorm-DepthAttention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ArchSpace-Collection/SiameseNorm-DepthAttention with Docker Model Runner:
```
docker model run hf.co/ArchSpace-Collection/SiameseNorm-DepthAttention
```

SiameseNorm-DepthAttention / README.md

arch-space

Add files using upload-large-folder tool

92a0653 verified 5 days ago

preview code

Raw

History Blame Contribute Delete

2.69 kB

	---
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- olmo3
	- custom-code
	- siamese-norm
	- depth-attention
	- sliding-window-attention
	---

	# OLMo 3 with SiameseNorm and DepthAttention

	This repository contains the 1B checkpoints from the four-stage OLMo 3
	training pipeline with SiameseNorm and DepthAttention.

	## Checkpoints

	\| Checkpoint \| Hub subfolder \| Context length \|
	\|---\|---\|---:\|
	\| Stage 1 pretraining \| `olmo3/1b/stage1` \| 8,192 \|
	\| Stage 2 mid-training \| `olmo3/1b/stage2` \| 8,192 \|
	\| Stage 3 long-context training \| `olmo3/1b/stage3` \| 65,536 \|
	\| Stage 4 Think SFT \| `olmo3/1b/stage4/think` \| 65,536 \|
	\| Stage 4 Instruct SFT \| `olmo3/1b/stage4/instruct` \| 65,536 \|

	Stage 3 and Stage 4 apply YaRN only to Full-attention layers. Sliding-window
	attention layers retain the original RoPE and a 4,096-token window.

	## Loading

	Select one checkpoint through `subfolder`. SDPA is the recommended and
	release-validated BF16 inference backend:

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	repo_id = "ArchSpace-Collection/SiameseNorm-DepthAttention"
	subfolder = "olmo3/1b/stage4/instruct"

	tokenizer = AutoTokenizer.from_pretrained(
	repo_id,
	subfolder=subfolder,
	trust_remote_code=True,
	fix_mistral_regex=False,
	)
	model = AutoModelForCausalLM.from_pretrained(
	repo_id,
	subfolder=subfolder,
	trust_remote_code=True,
	dtype=torch.bfloat16,
	attn_implementation="sdpa",
	)
	```

	`fix_mistral_regex=False` is intentional and preserves the tokenizer behavior
	used during training.

	The eager backend can also load these checkpoints. For BF16 generation, SDPA
	is recommended because eager cache partitioning can introduce small rounding
	differences when the leading logits are nearly tied; in that narrow case,
	greedy generation can select a different token. This is a numerical
	backend/cache-partition effect, not a checkpoint conversion or weight-integrity
	problem.

	The repository root contains the shared custom modeling code required by
	Transformers remote-code loading. Each checkpoint subfolder also contains a
	self-contained copy of its configuration, tokenizer, modeling code, and
	weights.

	## Architecture

	- 16 transformer layers
	- hidden size 2,048
	- intermediate size 8,192
	- 16 query heads and 16 key/value heads
	- 128-dimensional attention heads
	- 3:1 sliding-window/full-attention pattern
	- 4,096-token sliding window
	- reordered RMSNorm, SiameseNorm, and DepthAttention

	The Hugging Face implementation is intended for inference and generation.
	Exact continuation of the native distributed training objective should use
	the accompanying MindSpeed/Megatron training pipeline.