Create README.md

README.md

@@ -0,0 +1,40 @@
+Model Info:
+- Size: 7B
+- Dataset: The Pile v2
+  - `contaminated(P3) + lower_code(5%) + wiki(fixed) + books3(fixed & broken)`
+- Batch size (in tokens): 8M
+- Checkpoint path (AWS East): `/fsx/ckpts/7b_tok=neox_data=pilev2-recontam_lower-code_bs=8m_tp=4_pp=1_init=wang-small-init/global_step69000_hf`
+
+Notes:
+- Trained for 36k steps with incorrectly tokenized Books3 dataset (GPT-2 tokenizer instead of NeoX tokenizer)
+- tp=2 (not 4)
+
+W&B Report: https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-7B-alpha---Vmlldzo2MjA
+
+Usage:
+
+```python
+import transformers
+
+model = transformers.AutoModelForCausalLM.from_pretrained("CarperAI/7b-alpha")
+tokenizer = transformers.AutoTokenizer.from_pretrained("CarperAI/7b-alpha")
+tokenizer.pad_token = tokenizer.eos_token
+tokenizer.paddding_side = "left"
+
+prompts = [
+    "User1: The dog sat on a man's lap and barked 3 times.\nUser2: How many times did the dog bark?"
+    "Curious Person Question: A group of genetically identical individuals is called what?\nSmart Person Answer: a clone\n\nCurious Person Question: Who proposed the theory of evolution by natural selection?\nSmart Person Answer:"
+]
+batch_encoding = tokenizer(prompts, return_tensors="pt", padding=True)
+
+print(f"Generating {len(prompts)} prompts...")
+samples = model.generate(
+    **batch_encoding,
+    max_new_tokens=64,
+    temperature=0.0,
+    do_sample=False,
+)
+samples = tokenizer.batch_decode(samples, skip_special_tokens=True)
+for i, sample in enumerate(samples):
+    print(f"Prompt: {prompts[i]}\nSample: {sample}\n")
+```