arudradey/qwen-formula-engine

Qwen 0.5B Formula Engine

🧮 Neural Network Weights as Mathematical Formulas

This repository contains compressed formula representations of the Qwen2.5-0.5B-Instruct model weights.

What is this?

Instead of storing the full 942 MB model weights, we store compact mathematical formulas that can reconstruct the weights on-the-fly:

	Original	Formula-Compressed
Size	942.3 MB	474.1 MB
Savings	—	49.7%
Quality	Baseline	99.99% cosine similarity

Formula Types Used

• 4-bit Quantization: W ≈ scale × W_q + zero_point (169 layers)
• SVD Factorization: W ≈ U_r × diag(S_r) × V_r^T (available for rectangular matrices)
• Raw Storage: Tiny tensors stored as-is (121 layers - biases, norms)

How to use

import torch
from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer

# Load formula file
packed = torch.load("formula_weights_packed.pt", map_location="cpu", weights_only=True)
index = packed["index"]
weights_data = packed["weights"]

# Reconstruct weights
def reconstruct(data):
    if data["type"] == "quantize":
        return (data["W_q"].float() * data["scale"].float() + data["w_min"].float()).half()
    elif data["type"] == "raw":
        return data["data"]
    elif data["type"] == "svd":
        return (data["U"].float() @ torch.diag(data["S"].float()) @ data["Vh"].float()).half()

# Build model
config = AutoConfig.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForCausalLM.from_config(config)
state_dict = {name: reconstruct(weights_data[name]) for name in index}
model.load_state_dict(state_dict, strict=False)
model.eval()

# Chat!
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
# ... use as normal

Demo Space

Try the live chatbot: Formula Engine Chatbot

Credits

• Base model: Qwen/Qwen2.5-0.5B-Instruct (Apache 2.0)
• Compression: Formula Engine (SVD + Quantization)

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern