Add OpenLLM Small Extended 6k model

OpenLLM Small Extended model trained for 6,000 steps.

- Model: GPT-style transformer (35.8M parameters)
- Training: 6,000 steps on SQUAD Wikipedia passages
- Tokenizer: SentencePiece BPE (32k vocabulary)
- License: GPL-3.0 / Commercial available

For more details, see: https://github.com/louischua/openllm

README.md

@@ -0,0 +1,111 @@
+# OpenLLM Small Extended 6k
+
+This is the OpenLLM Small Extended model trained for 6,000 steps on Wikipedia passages from the SQUAD dataset.
+
+## Model Details
+
+- **Model Type:** GPT-style Transformer
+- **Architecture:** Small (35.8M parameters)
+- **Training Steps:** 6,000
+- **Training Data:** ~41k Wikipedia passages from SQUAD dataset
+- **Tokenizer:** SentencePiece BPE (32k vocabulary)
+- **License:** GPL-3.0 (Open Source) / Commercial License available
+
+## Model Performance
+
+- **Final Training Loss:** 5.4302
+- **Model Parameters:** 35,823,616
+- **Context Length:** 512 tokens
+- **Training Hardware:** CPU/GPU compatible
+
+## Usage
+
+### Using Transformers
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+# Load model and tokenizer
+model_name = "lemms/openllm-small-extended-6k"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+
+# Generate text
+prompt = "The history of artificial intelligence"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        inputs.input_ids,
+        max_new_tokens=50,
+        temperature=0.7,
+        top_k=40,
+        do_sample=True
+    )
+
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(generated_text)
+```
+
+### Using the Custom Loader
+
+```python
+# Use the provided load_hf_model.py script
+from load_hf_model import load_model_and_tokenizer
+
+model, tokenizer = load_model_and_tokenizer()
+# ... rest of usage
+```
+
+## Training Details
+
+This model was trained using the OpenLLM training pipeline:
+
+1. **Data Preparation:** SQUAD dataset processing (~41k passages)
+2. **Tokenizer Training:** SentencePiece BPE with 32k vocabulary
+3. **Model Training:** GPT-style transformer for 6,000 steps
+4. **Evaluation:** Perplexity and text generation quality assessment
+
+## Model Architecture
+
+- **Layers:** 12 transformer layers
+- **Attention Heads:** 12
+- **Hidden Size:** 768
+- **Intermediate Size:** 3072
+- **Activation:** GELU
+- **Layer Norm:** Pre-norm
+
+## Limitations
+
+- **Training Data:** Limited to Wikipedia passages
+- **Context Length:** 512 tokens maximum
+- **Model Size:** Small model with 35.8M parameters
+- **Performance:** Basic text generation capabilities
+
+## License
+
+This model is dual-licensed:
+- **Open Source:** GPL-3.0 for research and community use
+- **Commercial:** Commercial license available for enterprise use
+
+For commercial licensing, contact: louischua@gmail.com
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@misc{openllm2024,
+  title={OpenLLM: Open Source Large Language Model},
+  author={Louis Chua Bean Chong},
+  year={2024},
+  url={https://github.com/louischua/openllm}
+}
+```
+
+## Links
+
+- **Repository:** https://github.com/louischua/openllm
+- **Documentation:** https://github.com/louischua/openllm/docs
+- **Training Pipeline:** https://github.com/louischua/openllm/docs/training_pipeline.md

config.json

@@ -0,0 +1,18 @@
+{
+  "architectures": [
+    "GPTModel"
+  ],
+  "model_type": "gpt",
+  "vocab_size": 32000,
+  "n_layer": 6,
+  "n_head": 8,
+  "n_embd": 512,
+  "block_size": 1024,
+  "dropout": 0.1,
+  "bias": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.0.0",
+  "openllm_version": "0.1.0",
+  "training_steps": 6000,
+  "model_size": "small"
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "max_length": 512,
+  "max_new_tokens": 256,
+  "temperature": 0.7,
+  "top_k": 40,
+  "top_p": 0.9,
+  "do_sample": true,
+  "pad_token_id": 0,
+  "eos_token_id": 1,
+  "bos_token_id": 2
+}

load_hf_model.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+"""
+Hugging Face Compatible Loader for OpenLLM
+
+Usage:
+    # Using transformers library (if you implement custom model class)
+    # from transformers import AutoModel, AutoTokenizer
+    # model = AutoModel.from_pretrained(".")
+    # tokenizer = AutoTokenizer.from_pretrained(".")
+    
+    # Manual loading
+    from load_hf_model import load_model_manual
+    model, tokenizer = load_model_manual(".")
+"""
+
+import torch
+import json
+import sentencepiece as smp
+from pathlib import Path
+
+def load_model_manual(model_dir="."):
+    """Manually load model in HF format."""
+    model_dir = Path(model_dir)
+    
+    # Load config
+    with open(model_dir / "config.json", 'r') as f:
+        config = json.load(f)
+    
+    # Load model weights
+    state_dict = torch.load(model_dir / "pytorch_model.bin", map_location='cpu')
+    
+    # Load tokenizer
+    tokenizer = smp.SentencePieceProcessor()
+    tokenizer.load(str(model_dir / "tokenizer.model"))
+    
+    print(f"Loaded model: {config['model_type']} with {config['n_layer']} layers")
+    print(f"Vocabulary size: {config['vocab_size']}")
+    
+    return state_dict, tokenizer
+
+if __name__ == "__main__":
+    state_dict, tokenizer = load_model_manual()
+    print(f"Model weights loaded: {len(state_dict)} parameters")
+    print(f"Tokenizer vocabulary: {tokenizer.vocab_size()}")

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccade877ad32abcabfee7ab6eb99cbfad84dad5c68cdcc71720d8d526de0fa87
+size 168490621

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6efb1da9b0e667cee37b23f4240e0bd34fbfb20e1faebcb8d299a7598c0635f3
+size 547695

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "tokenizer_class": "SentencePieceTokenizer",
+  "model_max_length": 1024,
+  "vocab_size": 32000,
+  "unk_token": "<unk>",
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "pad_token": "<pad>"
+}