Upload folder using huggingface_hub

README.md

@@ -0,0 +1,299 @@
+# Sage - Custom LLM Model
+
+Sage is a custom-built transformer language model designed for text generation tasks. This model demonstrates the full lifecycle of building and publishing a custom AI model to Hugging Face.
+
+## 📊 Model Overview
+
+- **Model Type**: Transformer-based language model
+- **Architecture**: Decoder-only transformer
+- **Vocabulary Size**: 40 characters
+- **Hidden Size**: 256
+- **Number of Layers**: 4
+- **Number of Attention Heads**: 8
+- **Feedforward Size**: 1024
+- **Max Sequence Length**: 64
+- **Parameters**: ~3.2M
+- **Training Framework**: PyTorch
+- **License**: MIT
+
+## 📚 Training Data
+
+Sage was trained on a curated dataset of example sentences covering:
+- Conversational phrases and greetings
+- Weather and environmental descriptions
+- Machine learning and AI concepts
+- Deep learning architectures (transformers, neural networks)
+- Natural language processing applications
+- Model development and deployment practices
+
+The dataset consists of 10 carefully crafted examples designed to teach the model patterns in technical and conversational English.
+
+## 🔧 Technical Specifications
+
+### Model Architecture
+```
+TransformerLM(
+  (embedding): Embedding(40, 256)
+  (pos_embedding): Embedding(64, 256)
+  (transformer_encoder): TransformerEncoder(
+    (layers): ModuleList(
+      (0-3): TransformerEncoderLayer(
+        (self_attn): MultiheadAttention(
+          (embed_dim): 256
+          (num_heads): 8
+        )
+        (linear1): Linear(in_features=256, out_features=1024, bias=True)
+        (linear2): Linear(in_features=1024, out_features=256, bias=True)
+        (norm1): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
+        (norm2): LayerNorm((256,), eps=1e-05, elementwise_affine=True)
+        (dropout): Dropout(p=0.1, inplace=False)
+        (dropout1): Dropout(p=0.1, inplace=False)
+        (dropout2): Dropout(p=0.1, inplace=False)
+      )
+    )
+  )
+  (output_layer): Linear(in_features=256, out_features=40, bias=True)
+)
+```
+
+### Tokenization
+Sage uses a character-level tokenizer with:
+- Vocabulary: 40 unique characters including special tokens
+- Special tokens: `<PAD>` (0), `<UNK>` (1)
+- Encoding: UTF-8 character mapping
+- Maximum sequence length: 64 tokens
+
+## 🚀 Usage
+
+### With Transformers Library
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+# Load model and tokenizer
+model_name = "itriedcoding/Sage"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+
+# Generate text
+def generate_text(prompt, max_length=50, temperature=0.8):
+    inputs = tokenizer.encode(prompt, return_tensors="pt")
+    
+    with torch.no_grad():
+        outputs = model.generate(
+            inputs,
+            max_length=max_length,
+            temperature=temperature,
+            do_sample=True,
+            pad_token_id=tokenizer.eos_token_id
+        )
+    
+    return tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+# Examples
+print(generate_text("Hello"))
+print(generate_text("The weather"))
+print(generate_text("Deep learning"))
+```
+
+### Direct PyTorch Usage
+```python
+import torch
+from modeling_transformer_lm import TransformerLM
+import json
+import pickle
+
+# Load model components
+with open('config.json', 'r') as f:
+    config_dict = json.load(f)
+
+# For actual usage, you would load the tokenizer similarly
+# This example shows the structure
+model = TransformerLM.from_pretrained("itriedcoding/Sage")
+```
+
+## 🏗️ Model Card Metadata
+
+```yaml
+---
+library_name: transformers
+license: MIT
+base_model: custom-built
+tags:
+- text-generation
+- transformer
+- character-level
+- custom-model
+- educational
+pipeline_tag: text-generation
+widget:
+- example: Hello
+  parameters: {max_length: 30, temperature: 0.7}
+- example: The weather
+  parameters: {max_length: 30, temperature: 0.7}
+- example: Deep learning
+  parameters: {max_length: 30, temperature: 0.7}
+---
+```
+
+## 🤗 Hugging Face Spaces Deployment
+
+You can run this model in various Hugging Face Spaces templates:
+
+### Streamlit Space
+Create a `streamlit_app.py`:
+```python
+import streamlit as st
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+@st.cache_resource
+def load_model():
+    model_name = "itriedcoding/Sage"
+    tokenizer = AutoTokenizer.from_pretrained(model_name)
+    model = AutoModelForCausalLM.from_pretrained(model_name)
+    return tokenizer, model
+
+def main():
+    st.title("🤖 Sage Text Generator")
+    st.write("A custom character-level language model")
+    
+    tokenizer, model = load_model()
+    
+    prompt = st.text_input("Enter your prompt:", "Hello")
+    max_length = st.slider("Max length:", 10, 100, 30)
+    temperature = st.slider("Temperature:", 0.1, 2.0, 0.8)
+    
+    if st.button("Generate"):
+        with st.spinner("Generating..."):
+            inputs = tokenizer.encode(prompt, return_tensors="pt")
+            with torch.no_grad():
+                outputs = model.generate(
+                    inputs,
+                    max_length=max_length,
+                    temperature=temperature,
+                    do_sample=True,
+                    pad_token_id=tokenizer.eos_token_id
+                )
+            result = tokenizer.decode(outputs[0], skip_special_tokens=True)
+            st.write("**Generated text:**")
+            st.write(result)
+
+if __name__ == "__main__":
+    main()
+```
+
+### Gradio Space
+Create an `app.py`:
+```python
+import gradio as gr
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+def load_model():
+    model_name = "itriedcoding/Sage"
+    tokenizer = AutoTokenizer.from_pretrained(model_name)
+    model = AutoModelForCausalLM.from_pretrained(model_name)
+    return tokenizer, model
+
+def generate_text(prompt, max_length, temperature):
+    tokenizer, model = load_model()
+    inputs = tokenizer.encode(prompt, return_tensors="pt")
+    
+    with torch.no_grad():
+        outputs = model.generate(
+            inputs,
+            max_length=int(max_length),
+            temperature=temperature,
+            do_sample=True,
+            pad_token_id=tokenizer.eos_token_id
+        )
+    
+    return tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+demo = gr.Interface(
+    fn=generate_text,
+    inputs=[
+        gr.Textbox(label="Prompt", value="Hello"),
+        gr.Slider(minimum=10, maximum=100, value=30, label="Max Length"),
+        gr.Slider(minimum=0.1, maximum=2.0, value=0.8, label="Temperature")
+    ],
+    outputs=gr.Textbox(label="Generated Text"),
+    title="🤖 Sage Text Generator",
+    description="Custom character-level language model for text generation"
+)
+
+if __name__ == "__main__":
+    demo.launch()
+```
+
+## 📦 GGUF Quantization
+
+For efficient deployment, Sage is available in GGUF format:
+
+### Available Quantizations
+- `sage-q4_0.gguf` - 4-bit quantization (balanced quality/size)
+- `sage-q5_0.gguf` - 5-bit quantization (higher quality)
+- `sage-q8_0.gguf` - 8-bit quantization (near-full precision)
+- `sage-f16.gguf` - Float16 (full precision)
+
+### Using GGUF with llama.cpp
+```bash
+# Install llama.cpp
+git clone https://github.com/ggerganov/llama.cpp
+cd llama.cpp
+make
+
+# Run the model
+./main -m sage-q4_0.gguf -p "Hello" -n 30
+```
+
+## 📈 Performance & Limitations
+
+### Intended Use
+- Educational demonstrations of transformer architectures
+- Character-level language modeling experiments
+- Prototyping and testing custom model pipelines
+- Learning about model deployment on Hugging Face
+
+### Limitations
+- Small vocabulary (character-level only limits coherence)
+- Limited training data (10 examples)
+- Small model size (3.2M parameters)
+- Not suitable for production NLP applications
+- Best for short text generation (<50 tokens)
+
+### Bias & Ethics
+As a small educational model trained on curated technical text:
+- Minimal harmful bias expected
+- Should not be used for decision-making applications
+- Outputs should be reviewed for appropriateness
+- Model reflects patterns in its limited training data
+
+## 📝 Citation
+
+```bibtex
+@misc{sage_model_2026,
+  author = {itriedcoding},
+  title = {Sage: Custom Character-Level Language Model},
+  year = {2026},
+  publisher = {Hugging Face},
+  journal = {Hugging Face Model Hub},
+  doi = {10.57967/hf/0000},
+  url = {https://huggingface.co/itriedcoding/Sage}
+}
+```
+
+## 🔄 Training Reproducibility
+
+To reproduce this model:
+1. Clone this repository
+2. Install requirements: `pip install torch torchvision torchaudio pandas`
+3. Run training: `python train_model.py`
+4. The model will be saved as `custom_llm_model.pth`
+
+## 📞 Contact
+
+For questions or collaboration opportunities:
+- Hugging Face: https://huggingface.co/itriedcoding
+- Model Issues: Use the "Issues" tab on this model page

__init__.py

@@ -0,0 +1,3 @@
+from .modeling_transformer_lm import TransformerLM, TransformerLMConfig
+
+__all__ = ["TransformerLM", "TransformerLMConfig"]

config.json

@@ -0,0 +1,14 @@
+{
+  "architectures": ["TransformerLM"],
+  "model_type": "transformer_lm",
+  "vocab_size": 40,
+  "hidden_size": 256,
+  "num_hidden_layers": 4,
+  "num_attention_heads": 8,
+  "intermediate_size": 1024,
+  "max_position_embeddings": 64,
+  "pad_token_id": 0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "torch_dtype": "float32"
+}

modeling_transformer_lm.py

@@ -0,0 +1,109 @@
+import torch
+import torch.nn as nn
+import math
+from transformers import PreTrainedModel
+from transformers.modeling_utils import PretrainedConfig
+
+class TransformerLMConfig(PretrainedConfig):
+    model_type = "transformer_lm"
+    
+    def __init__(
+        self,
+        vocab_size=40,
+        hidden_size=256,
+        num_hidden_layers=4,
+        num_attention_heads=8,
+        intermediate_size=1024,
+        max_position_embeddings=64,
+        pad_token_id=0,
+        bos_token_id=1,
+        eos_token_id=2,
+        **kwargs
+    ):
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            **kwargs
+        )
+        
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.intermediate_size = intermediate_size
+        self.max_position_embeddings = max_position_embeddings
+
+class TransformerLM(PreTrainedModel):
+    config_class = TransformerLMConfig
+    
+    def __init__(self, config):
+        super().__init__(config)
+        self.config = config
+        
+        self.embedding = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.pos_embedding = nn.Embedding(config.max_position_embeddings, config.hidden_size)
+        
+        encoder_layer = nn.TransformerEncoderLayer(
+            d_model=config.hidden_size,
+            nhead=config.num_attention_heads,
+            dim_feedforward=config.intermediate_size,
+            batch_first=True
+        )
+        self.transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=config.num_hidden_layers)
+        self.output_layer = nn.Linear(config.hidden_size, config.vocab_size)
+        
+        self.max_position_embeddings = config.max_position_embeddings
+        
+    def forward(self, input_ids, attention_mask=None, labels=None):
+        seq_len = input_ids.size(1)
+        pos = torch.arange(0, seq_len, device=input_ids.device).unsqueeze(0)
+        
+        # Embedding + positional encoding
+        src_emb = self.embedding(input_ids) * math.sqrt(self.config.hidden_size)
+        pos_emb = self.pos_embedding(pos)
+        src_emb = src_emb + pos_emb
+        
+        # Create key padding mask for transformer (True where we should mask)
+        if attention_mask is not None:
+            # Transformer expects True for positions to mask
+            src_key_padding_mask = ~attention_mask.bool()
+        else:
+            src_key_padding_mask = None
+        
+        # Transformer encoder
+        output = self.transformer_encoder(src_emb, src_key_padding_mask=src_key_padding_mask)
+        
+        # Output projection
+        logits = self.output_layer(output)
+        
+        loss = None
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
+        
+        return {
+            "loss": loss,
+            "logits": logits
+        }
+        
+    def prepare_inputs_for_generation(self, input_ids, **kwargs):
+        # Only last token for inputs_ids if past is defined in kwargs
+        if "past_key_values" in kwargs:
+            input_ids = input_ids[:, -1].unsqueeze(-1)
+        
+        attention_mask = kwargs.get("attention_mask", None)
+        position_ids = kwargs.get("position_ids", None)
+        
+        # if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly
+        if attention_mask is not None:
+            attention_mask = attention_mask
+        
+        return {
+            "input_ids": input_ids,
+            "attention_mask": attention_mask,
+            "position_ids": position_ids,
+        }

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:accd9d82bd55ee686643f9e889f53e3d9938197f30fea126df1b596090c70382
+size 12805265