Update README.md

README.md

@@ -1,3 +1,100 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+---
+# HRWKV7-Reka-Flash3.1-Preview
+
+<div align="center">
+  <img src="./hxa079.png" style="border-radius: 15px; width: 60%; height: 60%; object-fit: cover;  box-shadow: 10px 10px 20px rgba(0, 0, 0, 0.5); border: 2px solid white;" alt="PRWKV" />
+</div>
+
+### Model Description
+
+HRWKV7-Reka-Flash3.1-Preview is an RNN hybrid architecture model that combines RWKV v7's linear attention mechanism with Group Query Attention (GQA) layers. Built upon the Reka-flash3.1 21B foundation, this model replaces most Transformer attention blocks with RWKV blocks while strategically maintaining some GQA layers to enhance performance on specific tasks.
+
+- **Developed by:** OpenMOSE
+- **Model type:** Hybrid Linear-Attention Language Model
+- **Language(s):** Multilingual (inherited from Reka-flash3.1 21B)
+- **License:** Apache-2.0
+- **Base Model:** Reka-flash3 21B(https://huggingface.co/RekaAI/reka-flash-3.1)
+- **Year:** 2025
+
+### Architecture Specifications
+
+- **Architecture:** RWKV v7 based "hxa079" Architecture + Group Query Attention Hybrid
+- **Total Layers:** 44 layers (L44D6114)
+  - 38 RWKV layers (with Rope)
+  - 6 GQA layers (No Rope, No Position Embeddings)
+- **Hidden Dimension:** 6144
+- **Training Context Window:** 4096 tokens
+- **Inference Context Window** 32768+
+- **Training Strategy** Following RADLADS method based knowledge distillation
+
+## Technical Innovation
+
+### RWKV "hxa079" Architecture
+
+The model implements several key improvements over original RWKV architectures:
+
+1. **Token Shift Removal**: In order to effectively inherit the teacher model weights, we removed the residual connection one token ago.
+2. **GroupNorm Removal**: Helps improve training stability issues
+3. **k_first Introduction**: Experimentally adopted the approach of residually connecting k layers in layer 0.
+
+### Hybrid Design Benefits
+
+- **Linear Attention Inference**: RWKV blocks enable O(1) memory complexity during inference, and the hybrid approach reduces the KVCache to 1/7 of full GQA.
+- **Enhanced Needle Tasks**: Strategic placement of GQA layers significantly improves performance on needle-in-haystack retrieval tasks, addressing a known limitation of pure linear attention models
+- **Implicit Position Encoding**: Interestingly, the model achieves better performance when RoPE (Rotary Position Embedding) is not applied to GQA layers, suggesting that RWKV blocks provide implicit positional encoding capabilities
+
+## Intended Use
+
+This is an **experimental research model** designed to explore hybrid architectures combining linear and quadratic attention mechanisms. It is intended for:
+
+- Research into efficient attention mechanisms
+- Benchmarking hybrid architecture performance
+- Exploring linear attention limitations and solutions
+- Academic and industrial R&D purposes
+
+## Limitations
+
+- **Experimental Status**: This model is in experimental stages and may exhibit unexpected behaviors
+- **Context Window**: Limited to 4096 tokens during training, though RWKV architecture theoretically supports longer sequences
+- **Performance Variability**: As a hybrid model, performance may vary significantly across different task types
+
+## Training Details
+
+- **Training Context Window:** 4096 tokens
+- **Training GPU** AMD MI300X x 1(takes 70hrs) AMD Developer Cloud.
+- **Training Strategy** 8bit MLP Quant, frozen emb,mlp,head, Deepspeed Stage1, Stage1 100M, Stage2 360M
+- **Base Model Initialization:** Weights initialized from Reka-flash3.1 21B
+- **Architecture Conversion:** Transformer attention blocks systematically replaced with RWKV blocks, except for 6 strategically placed GQA layers
+
+## Evaluation
+
+Performance evaluation is ongoing. The model shows promising results in:
+- Maintaining base model capabilities while achieving linear attention efficiency
+- Significantly improved needle-in-haystack task performance compared to pure RWKV architectures
+- Competitive performance on standard language modeling benchmarks
+
+
+## Run
+- RWKV-Infer now support hxa079
+```bash
+curl http://127.0.0.1:9000/loadmodel -X POST -H "Content-Type: application/json" -d '{"model_filename":"/home/client/Projects/llm/hxa079-reka-flash3-stage2-hybrid.pth","model_viewname":"RWKV HXA079 L38T6 Reka Flash3","model_strategy":"int8","adapter_filename":"","adapter_mode":"", "template":"rekaflash3", "endtoken":"\n <sep>","default_temperature":"0.2", "default_top_p":"0.3", "rope_theta":"8000000.0", "rms_norm_eps":"1e-5"}'
+
+```
+
+## Thank you for Big help :)
+- SmerkyG Inspired by RADLADS (https://arxiv.org/abs/2505.03005)
+- https://github.com/recursal/RADLADS-paper
+
+## Training Code
+- https://github.com/OpenMOSE/RWKVInside (still buggy)
+
+
+## Model Card Contact
+
+OpenMOSE - 2025
+
+---
+
+*Note: This is an experimental model. Performance characteristics and behaviors may differ from both pure RWKV and standard Transformer architectures. Users should thoroughly evaluate the model for their specific use cases.*