Initial upload of fine‑tuned Gemma + custom tokenizer

README.md

@@ -37,15 +37,17 @@ print(formatted_prompt) # start_generation adds the <start_of_turn> token to con
 ```
 Output:
 ```
-Capitals
-France
-<start_of_turn>Paris<end_of_turn>
-Japan
-<start_of_turn>
-```
-The data for the model to emulate / generate is wrapped in `<start_of_turn>` / `<end_of_turn>` tokens.
-Description and input is not wrapped in anything. Thus, do not expect the model to generate these tokens - instead focus on the wrapped output tokens.
-Messages are separated by newlines.
+<start_of_turn>descriptions
+Capitals<end_of_turn>
+<start_of_turn>input
+France<end_of_turn>
+<start_of_turn>output
+Paris<end_of_turn>
+<start_of_turn>input
+Japan<end_of_turn>
+<start_of_turn>output
+
+```
  
 In training, loss is ONLY calculated on the output tokens and the `<end_of_turn>` token. Thus, the model is only designed to generate / predict probabilities after `<start_of_turn>` and until `<end_of_turn>` - everything else is out of distribution for the model and not recommended.
 
@@ -70,17 +72,17 @@ with torch.no_grad():
 
 Output:
 ```
-Top 10 probabilities for first output token:
- 1. 'Tokyo' -> 0.9764
- 2. 'Tok' -> 0.0070
- 3. '東京' -> 0.0026
- 4. 'Ky' -> 0.0019
- 5. 'T' -> 0.0014
- 6. ' Tokyo' -> 0.0014
- 7. 'To' -> 0.0011
- 8. 'Osaka' -> 0.0009
- 9. 'Toy' -> 0.0007
-10. 'tok' -> 0.0005
+Top 10 probabilities for first output token:                                                                                               
+ 1. 'Tokyo' -> 0.9846                                                                                                                      
+ 2. '東京' -> 0.0032                                                                                                                       
+ 3. 'Tok' -> 0.0023                                                                                                                        
+ 4. 'Ky' -> 0.0023                                                                                                                         
+ 5. 'tok' -> 0.0012                                                                                                                        
+ 6. 'TO' -> 0.0011                                                                                                                         
+ 7. 'T' -> 0.0009                                                                                                                          
+ 8. 'To' -> 0.0005                                                                                                                         
+ 9. 'Toy' -> 0.0004                                                                                                                        
+10. '东京' -> 0.0002       
 ```
 
 Great! Almost all of the probability mass is on the correct answer, Tokyo.
@@ -110,10 +112,10 @@ for i in range(n_gens):
 
 Outputs:
 ```
-Catan: Rivals for Catan
-Gloomhaven
-Great Western Trail
-Azul
+Twilight Struggle
+Ark Nova
+Bardsung
+Carcassonne
 ```
 Not too bad!
 
@@ -135,10 +137,10 @@ for i in range(n_gens):
 ```
 Output:
 ```
-Deep Sea Blue
-Gray#222222
-Gold, Red, Black
-I can’t believe we’re already talking about color theory. How is this possible? Can time go any faster? Also how does your body    
+Light Green
+#ffff00
+yellow
+Bordeaux
 ```
 
 By default, the model is only trained to do 1) either emulate outputs if examples are provided, or 2) generate data based on the description. Because of this, the model always expects EITHER a description OR examples. If you want it to act slightly more like an instruction following chat model, you can add a description such as the following:
@@ -164,14 +166,23 @@ for i in range(n_gens):
 Some example generations:
 ```
 Generation 0:
-A deep-sea creature, silent and fierce, Shivers through water, its body sleek. Its jaws, a vice, its eyes cold steel, The shark moves with grace, never to feel.
-of power and danger,
+Swimming gracefully in the deep blue sea,
+A shark glides by so elegantly.
+Its sleek body moves with precision,
+As it searches for its next meal with attention.
+With fins that
 Generation 1:
-The great white shark lurks in the deep, with teeth so sharp, it could cut a whale in half. Its dorsal fin slices through the water, like a knife through butter, and its tail
+There once was a shark named Sandy, With a smile that would make you drooly. He swam in the deep blue sea, With friends who loved him oh so free.
 Generation 2:
-The shark swam in the sea, With a toothy grin, as if it could be glee. It was the top of the food chain, The apex of the sea's terrain. With sleek  
+I was walking through the forest
+and came across a shark.
+I was terrified and ran away,
+leaving the shark alone.
 Generation 3:
-I am a gentle, tranquil wave, gliding smoothly across the ocean's expanse. Yet deep within me lies a secret, a hidden power, a creature of the sea, fierce and agile. It
+In the depths of the ocean, where shadows roam,
+Lurks a creature of ancient lore,
+With jaws that snap and teeth like razors,
+A shark swims through the deep,
 ```
 
 
@@ -208,10 +219,10 @@ for i in range(n_gens):
 ```
 Output:
 ```
-{"situation": "While walking on the street, someone waves and smiles at you, but you don't know them.", "is_awkward": false}                        
-{"situation": "Taking a cab and giving the driver wrong directions.", "is_awkward": true}                                                           
-{"situation": "Being told that an individual you've had a long-term crush on is also crushing on someone else.", "is_awkward": true}                
-{"situation": "Watching a loved one get proposed to.", "is_awkward": false}  
+{"situation": "Your family surprised you with tickets to your favorite bands concert.", "is_awkward": false}                              
+{"situation": "Your boss is talking about how she is going on a trip and she starts to talk about how she is going to a fancy restaurant.", "is_awkward": false}
+{"situation": "You show up at the hospital for an appointment and realize you are one day late.", "is_awkward": true}                     
+{"situation": "Your crush tells you that they had a terrible day.", "is_awkward": true}     
 ```
 
 A few tips and tricks: