Upload 5 files

README.md

@@ -1,3 +1,53 @@
 ---
 license: apache-2.0
+tags:
+  - llama
+  - text-generation
+  - pytorch
+language:
+  - en
+pipeline_tag: text-generation
 ---
+
+# Vertex 0.1 (10M)
+
+Small LLaMA-architecture causal language model, ~10.9M parameters.
+
+## Model Details
+
+- **Architecture:** LlamaForCausalLM
+- **Parameters:** ~10.9M
+- **Hidden size:** 256
+- **Layers:** 4
+- **Attention heads:** 4 (2 KV heads, GQA)
+- **Head dim:** 64
+- **Intermediate size:** 640
+- **Max position embeddings:** 1024
+- **Vocab size:** 32000
+- **Dtype:** float32
+
+## Training
+
+- **Epochs:** 3
+- **Global steps:** 3207
+- **Batch size:** 8
+- **Estimated tokens seen:** ~26M total (~8.8M tokens/epoch) — estimated from batch size × seq len × steps; not logged exactly by the trainer.
+- **Training time:** ~1 Hour
+- **Hardware:** M5 Pro 24gb Macbook
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("path/or/repo-id")
+tokenizer = AutoTokenizer.from_pretrained("path/or/repo-id")
+
+inputs = tokenizer("Once upon a time", return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=50)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Limitations
+
+This is a very small model (~10.9M params) trained on a small dataset. Expect limited coherence, knowledge, and generalization compared to larger models. Intended for experimentation, education, and low-resource testing, not production use.

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 640,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 2,
+  "pad_token_id": 1,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 1,
+  "transformers_version": "5.14.1",
+  "use_cache": false
+}

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 5120,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}