| # SAGE — Self-Adaptive General Engine |
|
|
| **SAGE** is a senior-grade, production-structured Large Language Model (LLM) system built entirely from scratch using Python and PyTorch. It implements modern transformer architectures including Mixture of Experts (MoE), Rotary Positional Embeddings (RoPE), and Low-Rank Adaptation (LoRA). |
|
|
| Designed to be both educational and functional, SAGE can be trained, fine-tuned, quantized, and deployed on a single consumer GPU (e.g., NVIDIA T4 with 16GB VRAM). |
|
|
| --- |
| --- |
|
|
| ## 🚀 Key Features |
|
|
| - **Decoder-Only Transformer**: A GPT-style architecture with pre-layer normalization. |
| - **Mixture of Experts (MoE)**: Efficient scaling with a learned router selecting top-k experts per token. |
| - **Rotary Positional Embeddings (RoPE)**: Enhanced long-sequence generalization. |
| - **KV-Cache Inference**: O(1) time-per-token generation for high-speed response. |
| - **Retrieval-Augmented Generation (RAG)**: Integration with FAISS for document-based context lookup. |
| - **Efficient Fine-Tuning**: Support for LoRA and instruction tuning with loss masking. |
| - **Post-Training Quantization**: INT8 support to reduce memory footprint. |
| - **Interactive CLI**: A full REPL (Read-Eval-Print Loop) for chatting and system management. |
|
|
| --- |
|
|
| ## 📂 Project Structure |
|
|
|
|
| --- |
|
|
| ## 🛠️ Installation & Setup |
|
|
| ### 1. Requirements |
| Ensure you have Python 3.9+ and a CUDA-compatible GPU (recommended). |
|
|
| ```bash |
| # Clone the repository (GitHub) |
| git clone https://github.com/er-del/sage.git |
| cd sage |
| |
| # OR Clone from Hugging Face |
| git clone https://huggingface.co/sage002/sage |
| cd sage |
| |
| # Install dependencies |
| pip install -r requirements.txt |
| ``` |
|
|
| ### 2. Dependencies |
| - **PyTorch**: Core deep learning framework. |
| - **tiktoken**: Fast BPE tokenization (OpenAI's cl100k_base). |
| - **datasets**: For streaming training data from HuggingFace. |
| - **faiss-cpu**: For vector-based retrieval (RAG). |
| - **tqdm**: Progress bars for training. |
| - **bitsandbytes**: (Optional) For advanced quantization. |
| |
| --- |
| |
| ## 🎮 Getting Started |
| |
| |
| ### Basic Chat |
| Once launched, simply type your message to chat with SAGE. The system uses a rolling conversation history to maintain context. |
| |
| --- |
| |
| ## 👨🏫 Training SAGE |
| |
| SAGE supports real-time training either directly from the interactive REPL or via simple one-liner CLI commands (useful for background scripts). |
| |
| ### Non-Interactive "One-Liner" Commands |
| |
| --- |
| |
| ## 🧠 Advanced Commands |
| |
| --- |
| |
| ## 🏗️ Architecture Details |
| |
| --- |
| |
| ## 🤗 Hugging Face Model Hub |
| |
| This project is actively maintained on Hugging Face. You can find pre-trained checkpoints, datasets, and community discussions here: |
| |
| 🔗 **[huggingface.co/sage002/sage](https://huggingface.co/sage002/sage)** |
| |
| |
| --- |
| |
| ## 📜 Disclaimer |
| SAGE is an experimental engine. While architecturally complete, the quality of generated responses depends heavily on the amount of training data and compute steps provided. |
| |
