---
license: mit
base_model:
- Qwen/Qwen3-VL-30B-A3B-Instruct
library_name: transformers
---

# Qwen3-VL-1.0B-A0.4B-Instruct

This is a tiny version of [Qwen/Qwen3-VL-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct) created for testing and development.

## Model Details

- **Base Model**: Qwen/Qwen3-VL-30B-A3B-Instruct
- **Architecture**: qwen3_vl_moe
- **Total Parameters**: 1.434B
- **Activated Parameters**: ~0.4B (8 of 128 experts active per token)

## Configuration Changes

The following parameters were reduced from the original model:

| Parameter | Original | Tiny |
|-----------|----------|------|
| `text_config.num_hidden_layers` | 48 | 1 |
| `vision_config.depth` | 27 | 4 |
| `vision_config.deepstack_visual_indexes` | [8, 16, 24] | [1, 2, 3] |
| `text_config.hidden_size` | 2048 | 2048 (unchanged) |
| `text_config.num_local_experts` | 128 | 128 (unchanged) |
| `text_config.num_experts_per_tok` | 8 | 8 (unchanged) |
| `text_config.moe_intermediate_size` | 768 | 768 (unchanged) |
| `text_config.num_attention_heads` | 32 | 32 (unchanged) |
| `text_config.num_key_value_heads` | 4 | 4 (unchanged) |
| `vision_config.hidden_size` | 1152 | 1152 (unchanged) |

## Checkpoint Structure

The model is saved as a single `model.safetensors` file (89 tensors). The checkpoint key structure matches the original `Qwen/Qwen3-VL-30B-A3B-Instruct` exactly.

## Usage

```python
from transformers import Qwen3VLMoeForConditionalGeneration, AutoTokenizer

model = Qwen3VLMoeForConditionalGeneration.from_pretrained(
    "inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
```

Note: `AutoModelForCausalLM` does not support this model type. Use `Qwen3VLMoeForConditionalGeneration` or `AutoModelForImageTextToText` directly.

## Creation Process

This model was created using the llm-compressor `create-tiny-model` claude skill.

1. Config fetched from `Qwen/Qwen3-VL-30B-A3B-Instruct` without downloading weights (`skip_weights_download`)
2. Text depth reduced from 48 → 1 layers; vision depth reduced from 27 → 4 blocks
3. All parameters randomly initialized with `initializer_range=0.02` (matching the base model)
4. MoE expert parameters (`gate_up_proj`, `down_proj`, `gate.weight`) explicitly initialized since they are bare `nn.Parameter` tensors, not `nn.Linear` modules
5. Fine-tuned on a toy text dataset (internet copypastas) until perplexity ≤ 3.0

## Validation

```
Success: perplexity=1.0001 <= 10.0
Generated: According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little
```

## Notes

- This is a **text-only fine-tuned** model. The vision encoder weights are randomly initialized and not fine-tuned; the model is intended for text-based testing only.
- The MoE routing architecture (128 experts, 8 active per token) is preserved in full to match the original model's structural properties.
- Fine-tuning converged in ~160 steps with learning rate 1e-4.