trentzap/qtensor-tinyllama-hybrid-v2

QTensor Asymmetric 2.0 (Hybrid Bridge)

This model is a highly compressed version of TinyLlama-1.1B-Chat-v1.0, utilizing the QTensor Asymmetric 2.0 framework. It achieves massive VRAM reductions by applying mathematically distinct compression topologies to different architectural components, maintaining zero-shot performance and fluency.

Architecture

• Attention Layers (The Eyes): Compressed via Block-SVD (r=16) + 3.0% SpLoRA.
• MLP Layers (The Muscle): Compressed via Channel-Wise INT4 + LoRA.
• Harmonization: Employs a 1D Rank-1 Subspace Bridge (subspace_bridge) to harmonize variance mismatches between the SVD residual stream and INT4 inputs.

Performance & Footprint

• VRAM Footprint: ~1.16 GB
• HellaSwag (Zero-Shot): 38.0%
• WikiText-2 (Zero-Shot Perplexity): 168.46

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "your-username/qtensor-tinyllama-hybrid-v2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    trust_remote_code=True, 
    torch_dtype=torch.bfloat16
)