Falcon-H1-0.5B Rust Distilled

A Rust/CUDA-compatible, sequence-level distilled version of Falcon-H1-0.5B-Instruct.

The model uses the original Falcon-H1 hybrid architecture, combining grouped-query attention with Mamba-2 state-space layers.

The implementation performs Falcon-H1 inference using Rust, Candle and CUDA, including fused Mamba execution.

Model provenance

Original pretrained student

tiiuae/Falcon-H1-0.5B-Instruct

Teacher

tiiuae/Falcon-H1-1.5B-Deep-Instruct

Dataset

C4 English (allenai/c4, subset en).

Rust implementation

https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API

Distillation method

The Python teacher generates text continuations from C4 prompts.

Because teacher and student tokenizers differ, teacher-generated continuations are retokenized using the student tokenizer.

The Rust/CUDA student learns from these continuations using masked autoregressive cross-entropy.

The pretrained Falcon-H1 backbone is frozen.

A rank-16 trainable correction is applied to the output head. The correction is trained using Rust, Candle and CUDA.

After training, the correction is merged into the original pretrained output-head weights.

Thus the published checkpoint retains the original Falcon-H1-0.5B architecture and parameter shapes.

This is output-head adaptation using sequence-level knowledge distillation, not full-backbone fine-tuning.

Training

Setting Value
Teacher Falcon-H1-1.5B-Deep-Instruct
Student Falcon-H1-0.5B-Instruct
Dataset C4 English
Student framework Rust / Candle / CUDA
Training GPU NVIDIA L4
Trainable component Rank-16 output-head correction
Training target tokens in final run 97,706
Held-out target tokens 4,161
Selected V3 epoch 5
Training batch size 128

The final run continued from the previously selected V2 adapter. The original pretrained backbone was retained.

Held-out C4 evaluation

Evaluation uses masked next-token prediction on 4,161 held-out teacher-continuation tokens.

Model Perplexity
Original Falcon-H1-0.5B 3.3695
Distilled V2 3.2312
Distilled V3 3.2182

The V3 adapter was selected using the lowest validation cross-entropy, at epoch 5.

These results measure agreement with teacher-generated continuations on the held-out C4 set.

They do not establish improvements in general reasoning, instruction following, or independent downstream benchmarks.

Download

The primary model file is:

falcon-h1-c4-distilled-rust-v3.f32pack

This is a custom FH1W float32 checkpoint, not a standard Hugging Face Transformers safetensors checkpoint.

Download it with:

hf download Angshul/Falcon-H1-0.5B-Rust-Distilled \
  falcon-h1-c4-distilled-rust-v3.f32pack \
  --local-dir ./model

Rust CUDA inference

Clone the Rust implementation:

git clone https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API
cd Falcon-H1-Rust-Transformer-API

Compile and run the API with an NVIDIA CUDA GPU:

cargo run --release --features cuda --bin falcon-h1-api -- \
  --checkpoint ./model/falcon-h1-c4-distilled-rust-v3.f32pack \
  --device cuda \
  --bind 127.0.0.1:8080

The API accepts token IDs and supports generation through its HTTP endpoints.

Use the tokenizer associated with the original Falcon-H1-0.5B-Instruct student.

Limitations

  • The model is specialized using C4 English text continuations.
  • Only the output-head correction was trained.
  • The backbone is unchanged from the pretrained student.
  • The released .f32pack requires the compatible Rust loader.
  • The reported perplexity is measured on teacher-generated continuation targets, not a standard language-model benchmark.
  • General downstream improvements have not been established.

License and attribution

The original Falcon-H1 model weights were developed by the Technology Innovation Institute (TII).

The underlying model weights and derived checkpoint are subject to the Falcon-LLM License.

The separate Rust implementation is available under its own repository license.

Original Falcon-H1 model:

https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct

References

Falcon-H1 technical report: https://arxiv.org/abs/2507.22448

Rust implementation: https://github.com/AngshulMajumdar/Falcon-H1-Rust-Transformer-API

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Angshul/Falcon-H1-0.5B-Rust-Distilled

Finetuned
(7)
this model

Dataset used to train Angshul/Falcon-H1-0.5B-Rust-Distilled

Paper for Angshul/Falcon-H1-0.5B-Rust-Distilled