Initial upload of fine‑tuned Gemma + custom tokenizer

README.md

@@ -0,0 +1,176 @@
+### tsor13/Special12b
+The following is a a model trained by [...suspense...] that is meant to:
+- follow instructions better than pretrained models and be more diverse / less mode-collapsed than instruct models;
+- be a really good, approximately bayesian in-context learner;
+- fit an data generation process
+- be calibrated over distributions of possible outputs wrt a population or epistemic uncertainty
+It is initialized from the `gemma-3-1b-pt` checkpoint.
+
+Loading model example:
+```
+from transformers import AutoTokenizer, AutoModelForCausalLM
+tokenizer = AutoTokenizer.from_pretrained("tsor13/special12b", trust_remote_code=True) # custom tokenizer for handling messages / loss
+model = AutoModelForCausalLM.from_pretrained("tsor13/special12b", device_map="auto")
+```
+
+It has its own chat-style input messages, with the following roles:
+- `description`(optional): A description of the generating process, or some information meant to instantiate a prior
+- `input` (optional): Any variables that a model is not responsible for predicting, but could be used to condition generation somehow;
+- `output`: This is what the model will actually predict / generate.
+
+For example,
+```
+messages = [
+    {"role": "description", "content": "Capitals"},
+    {"role": "input", "content": "France"},
+    {"role": "output", "content": "Paris"},
+    {"role": "input", "content": "Japan"},
+]
+```
+
+To templatize the messages, you can use the tokenizer:
+```
+formatted_prompt = tokenizer.messages_to_text(messages, start_generation=True)
+print(formatted_prompt) # start_generation adds the <start_of_turn> token to condition the model for generation
+```
+Output:
+```
+This is a test task
+What is 2+2?
+<start_of_turn>4<end_of_turn>
+What is 3+3?
+<start_of_turn>
+```
+The data for the model to emulate / generate is wrapped in `<start_of_turn>` / `<end_of_turn>` tokens.
+Description and input is not wrapped in anything. Thus, do not expect the m
+Messages are separated by newlines.
+ 
+In training, loss is ONLY calculated on the output tokens and the `<end_of_turn>` token. Thus, the model is only designed to generate / predict probabilities after `<start_of_turn>` and until `<end_of_turn>` - everything else is out of distribution for the model and not recommended.
+
+Once you have the formatted text, you can tokenize as normal:
+```
+inputs = tokenizer(formatted_prompt, return_tensors="pt").to(model.device)
+```
+
+Let's look at what the model does. In this case, there is a single correct answer. Let's look at model probabilities after `<start_of_turn>`:
+```
+import torch
+with torch.no_grad():
+    output = model(**inputs)
+    logits = output.logits[0, -1, :]
+    probs = torch.nn.functional.softmax(logits, dim=-1)
+    top_probs, top_indices = torch.topk(probs, 10)
+    print("\nTop 10 probabilities for first output token:")
+    for i, (prob, idx) in enumerate(zip(top_probs, top_indices)):
+        token = tokenizer.decode(idx)
+        print(f"{i+1:2d}. '{token}' -> {prob.item():.4f}")
+```
+
+Output:
+```
+Top 10 probabilities for first output token:
+ 1. 'Tokyo' -> 0.9792
+ 2. 'Tok' -> 0.0059
+ 3. '東京' -> 0.0028
+ 4. 'Ky' -> 0.0019
+ 5. 'T' -> 0.0011
+ 6. ' Tokyo' -> 0.0011
+ 7. 'Osaka' -> 0.0009
+ 8. 'To' -> 0.0008
+ 9. 'Toy' -> 0.0006
+10. 'Nag' -> 0.0005
+```
+
+Great! Almost all of the probability mass is on the correct answer, Tokyo.
+
+Let's try an example with many possible reasonable choices / a harder to describe distribution. For example, say that I'm interested in modeling "board games that I like". I may be hard-pressed to actually describe what it is that I like about games - but I could provide a few examples pretty easily.
+
+```
+messages = [
+    {"role": "output", "content": "Dune: Imperium"},
+    {"role": "output", "content": "Acquire"},
+    {"role": "output", "content": "Catan"},
+    {"role": "output", "content": "Tigris and Euphrates"},
+    {"role": "output", "content": "Brass: Birmingham"},
+]
+```
+
+Given these example outputs, the model will try to generate more outputs like these outputs.
+```
+formatted_prompt = tokenizer.messages_to_text(messages, start_generation=True)
+n_gens = 4
+inputs = tokenizer([formatted_prompt] * n_gens, return_tensors="pt").to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=10, stop_strings=["<end_of_turn>"], tokenizer=tokenizer)
+for i in range(n_gens):
+    print(tokenizer.decode(outputs[i][inputs["input_ids"][i].shape[0]:], skip_special_tokens=True))
+```
+
+Outputs:
+```
+7 Wonders: Duel
+Dominion
+Terraforming Mars
+Agricola: All Creatures Big and Small
+```
+Not too bad!
+
+You can also specify just the description:
+Input:
+```
+messages = [
+    {"role": "description", "content": "Metaphors about life."},
+]
+
+formatted_prompt = tokenizer.messages_to_text(messages, start_generation=True)
+n_gens = 4
+inputs = tokenizer([formatted_prompt] * n_gens, return_tensors="pt").to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=10, stop_strings=["<end_of_turn>"], tokenizer=tokenizer)
+for i in range(n_gens):
+    print(tokenizer.decode(outputs[i][inputs["input_ids"][i].shape[0]:], skip_special_tokens=True))
+    print()
+```
+
+Finally, let's look at a synthetic data generation task. For example, maybe we want to generate situations to do social reasoning over, along with whether or not they are awkward. When there are multiple variables to condition on or generat, the model is used to json format.
+
+Input:
+```
+import json
+messages = [
+    {"role": "description", "content": "Situations to do social reasoning over, along with whether or not it is an awkward situation."},
+    {"role": "output", "content": json.dumps({
+        "situation": "You're at a party and you realize that your shirt is on backwards.",
+        "is_awkward": True,
+    })},
+    {"role": "output", "content": json.dumps({
+        "situation": "While at work, your boss commends you on a job well done.",
+        "is_awkward": False,
+    })},
+    {"role": "output", "content": json.dumps({
+        "situation": "Realizing you forgot to bring your passport to the airport.",
+        "is_awkward": True,
+    })},
+]
+
+formatted_prompt = tokenizer.messages_to_text(messages, start_generation=True)
+n_gens = 4
+inputs = tokenizer([formatted_prompt] * n_gens, return_tensors="pt").to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=40, stop_strings=["<end_of_turn>"], tokenizer=tokenizer)
+for i in range(n_gens):
+    print(tokenizer.decode(outputs[i][inputs["input_ids"][i].shape[0]:], skip_special_tokens=True))
+```
+Output:
+```
+{"situation": "Noticing that you've been holding the wrong end of your toothbrush.", "is_awkward": true}
+
+{"situation": "Your teacher gives you extra credit for participating in class.", "is_awkward": false}
+
+{"situation": "Your friend doesn't recognize you because of your new look.", "is_awkward": true}
+
+{"situation": "Your friend comes over and asks you to help them move some furniture, but you have other plans.", "is_awkward": true}
+```
+
+
+