Upload folder using huggingface_hub

README.md

@@ -0,0 +1,40 @@
+---
+license: mit
+tags:
+  - llama
+  - pre-trained
+  - from-scratch
+  - pytorch
+---
+
+# LLaMA-355M Base Model
+
+A 355M parameter LLaMA-style language model trained from scratch.
+
+## Architecture
+- **Type**: LLaMA-style Transformer
+- **Parameters**: 355M
+- **Layers**: 24
+- **Heads**: 16
+- **Hidden dim**: 1024
+- **Context**: 512 tokens
+- **Vocab**: 50257 (tiktoken GPT-2 BPE)
+
+## Features
+- RMSNorm (instead of LayerNorm)
+- Rotary Position Embeddings (RoPE)
+- SwiGLU activation
+- Flash Attention
+- No bias terms
+
+## Training
+- Pre-trained on a mix of OpenWebText, AutoMathText, WikiText, HackerNews
+- Fine-tuned on: claude-reasoning
+- Trained on RTX 3080 Ti
+
+## Usage
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model = AutoModelForCausalLM.from_pretrained("YOUR_USER/llama-355m")
+tokenizer = AutoTokenizer.from_pretrained("gpt2")
+```

config.json

@@ -0,0 +1,20 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "model_type": "llama",
+  "hidden_size": 1024,
+  "intermediate_size": 2752,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 16,
+  "max_position_embeddings": 512,
+  "vocab_size": 50257,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "float32",
+  "hidden_act": "silu",
+  "bos_token_id": 0,
+  "eos_token_id": 0
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "max_new_tokens": 256,
+  "temperature": 0.7,
+  "top_p": 0.9,
+  "top_k": 40,
+  "do_sample": true
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2900bbe7bd6134ba51b7eb59d66b8a8337657198bbc389160dbbd5293501246
+size 1420375090

tokenizer_config.json

@@ -0,0 +1,4 @@
+{
+  "model_type": "gpt2",
+  "tokenizer_class": "GPT2Tokenizer"
+}