README.md

metadata

license: mit
datasets:
  - tokyotech-llm/swallow-code-v2
language:
  - en
  - zh
metrics:
  - accuracy
base_model:
  - Qwen/Qwen2.5-Coder-1.5B
library_name: transformers
tags:
  - Qwen
  - HybridArch
  - sinkAttention
  - MLA
  - GQA

PyraCode-1.5B

🌟 Model Overview

This is a custom-architected model based on Qwen2.5-Coder-1.5B. We introduced a novel Asymmetric Hybrid Architecture (GQA + MLA) with Cross-Layer Shared Latent Gates and Attention Sinks, enabling efficient feature communication and reduced KV-Cache memory footprint.

🏗️ Architecture Innovations

(这里插入你用 picture.py 生成的架构图，可以把图片拖进 Hugging Face 网页版的编辑框里自动生成链接)

Unlike standard Qwen2 models, this Hybrid-v9 backbone features:

1. Asymmetric Layers:
   • L0-L6: Standard GQA (Grouped-Query Attention) for robust low-level feature extraction.
   • L7 (Shared Hub): Generates a global latent vector $c_{kv}$ (Rank 320).
   • L8-L27: Soft MLA (Multi-Head Latent Attention) with SVD-initialized low-rank projections.
2. Shared Latent Gate: Deep layers can dynamically access the global latent vector from L7 via a learnable gating mechanism (warmup_alpha).
3. HybridCache & Attention Sinks: Implements a sliding window (8192) alongside a 64-token attention sink to maintain generation stability at infinite sequence lengths.

🚀 Quick Start

⚠️ IMPORTANT: Because this model uses a custom architecture, you MUST pass trust_remote_code=True when loading it.

Prerequisites

pip install transformers torch