Add particle-1.0 weights (from-scratch 100M Llama-style, MIT)

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 particle-1.0 contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,77 @@
+---
+license: mit
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - llama
+  - from-scratch
+  - smol
+---
+
+# particle-1.0
+
+~100M-parameter Llama-style chat model trained **from scratch** (random init). Not a fine-tune of Llama, SmolLM, or any Hub base.
+
+Weights are MIT. Training data still needs attribution (below).
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "prathamkode/particle-1.0"
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo)
+
+messages = [{"role": "user", "content": "hello"}]
+prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+ids = tok(prompt, return_tensors="pt")
+out = model.generate(**ids, max_new_tokens=64, temperature=0.7)
+print(tok.decode(out[0], skip_special_tokens=False))
+```
+
+Chat format:
+
+```
+<|user|>
+hello
+<|assistant|>
+```
+
+## Model details
+
+| | |
+|---|---|
+| Architecture | Llama-style decoder (RoPE, SwiGLU, RMSNorm, tied embeddings) |
+| Parameters | ~100M (12 layers, 768 hidden, 12 heads) |
+| Context | 2048 tokens |
+| Tokenizer | Custom 32k byte-level BPE (not Llama / GPT-2 vocab) |
+| Init | Random `N(0, 0.02)` — trained from scratch |
+| Precision | BF16 training; Hub weights `bfloat16` |
+
+## Training
+
+1. **Tokenizer** trained from scratch on a FineWeb-Edu sample (~2GB text).
+2. **Pretrain** next-token prediction on [`HuggingFaceFW/fineweb_edu_100BT-shuffled`](https://huggingface.co/datasets/HuggingFaceFW/fineweb_edu_100BT-shuffled), first ~2B tokens.
+3. **SFT** on [`HuggingFaceTB/smol-smoltalk`](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) (first user/assistant turn + a few greeting seeds).
+
+SFT used that dataset as **text only**. No teacher model weights were copied.
+
+## Intended use
+
+Research / demo small chat model. Expect short replies, mistakes, and weak reasoning.
+
+## Limitations
+
+- Very small capacity
+- May hallucinate
+- English-centric FineWeb-Edu subset
+- No RLHF / preference tuning
+
+## License
+
+- **These weights:** [MIT](LICENSE)
+- **FineWeb-Edu:** ODC-By (attribute)
+- **smol-smoltalk:** follow the dataset card

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 1,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.47.0",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 0,
+  "pad_token_id": 1,
+  "transformers_version": "4.47.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9673d133bae4eee60f8a539b025338299d3dca6a886474d652d1991fac40420
+size 219071992

special_tokens_map.json

@@ -0,0 +1,9 @@
+{
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>"
+  ],
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|padding|>",
+  "unk_token": "<|unk|>"
+}

tokenizer_config.json

@@ -0,0 +1,57 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|unk|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'user' %}<|user|>\n{{ message['content'] }}\n{% elif message['role'] == 'assistant' %}<|assistant|>\n{{ message['content'] }}{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>\n{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 2048,
+  "pad_token": "<|padding|>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<|unk|>"
+}