Upload folder using huggingface_hub

README.md

@@ -1,146 +1,58 @@
 ---
 base_model: HuggingFaceTB/SmolLM2-360M
 library_name: transformers
-model_name: SmolLM2-360M-tldr-sft-2025-02-12_15-13
+model_name: SmolLM2-360M-tldr-sft-2025-05-28_17-48
 tags:
 - generated_from_trainer
 - trl
 - sft
-license: mit
-datasets:
-- davanstrien/hub-tldr-dataset-summaries-llama
-- davanstrien/hub-tldr-model-summaries-llama
+licence: license
 ---
 
-# Smol-Hub-tldr 
+# Model Card for SmolLM2-360M-tldr-sft-2025-05-28_17-48
 
-<div style="float: right; margin-left: 1em;">
-<img src="https://cdn-uploads.huggingface.co/production/uploads/60107b385ac3e86b3ea4fc34/dD9vx3VOPB0Tf6C_ZjJT2.png" alt="Model visualization" width="200"/>
-</div>
+This model is a fine-tuned version of [HuggingFaceTB/SmolLM2-360M](https://huggingface.co/HuggingFaceTB/SmolLM2-360M).
+It has been trained using [TRL](https://github.com/huggingface/trl).
 
-This model is a fine-tuned version of [HuggingFaceTB/SmolLM2-360M](https://huggingface.co/HuggingFaceTB/SmolLM2-360M). The model is focused on generating concise, one-sentence summaries of model and dataset cards from the Hugging Face Hub. These summaries are intended to be used for:
-
-- creating useful tl;dr descriptions that can give you a quick sense of what a dataset or model is for
-- as input text for creating embeddings for semantic search. You can see a demo of this in [librarian-bots/huggingface-datasets-semantic-search](https://huggingface.co/spaces/librarian-bots/huggingface-datasets-semantic-search).
-
-The model was trained using supervised fine-tuning (SFT) with [TRL](https://github.com/huggingface/trl). 
-
-A meta example of a summary generated for this card: 
-
-> This model is a fine-tuned version of SmolLM2-360M for generating concise, one-sentence summaries of model and dataset cards from the Hugging Face Hub.
-
-
-## Intended Use
-
-The model is designed to generate brief, informative summaries of:
-- Model cards: Focusing on key capabilities and characteristics
-- Dataset cards: Capturing essential dataset characteristics and purposes
-
-## Training Data
-
-The model was trained on:
-- Model card summaries generated by Llama 3.3 70B
-- Dataset card summaries generated by Llama 3.3 70B
-
-Model context length: the model was trained with cards up to a length of 2048 tokens
-
-## Usage
-
-Using the chat template when using the model in inference is recommended. Additionally, you should prepend either `<MODEL_CARD>` or `<DATASET_CARD>` to the start of the card you want to summarize. The training data used the body of the model or dataset card (i.e., the part after the YAML, so you will likely get better results only by passing this part of the card. 
-
-I have so far found that a low temperature of `0.4` generates better results. 
-
-Example:
+## Quick start
 
 ```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from huggingface_hub import ModelCard
-
-card = ModelCard.load("davanstrien/Smol-Hub-tldr")
-
-# Load tokenizer and model
-tokenizer = AutoTokenizer.from_pretrained("davanstrien/Smol-Hub-tldr")
-model = AutoModelForCausalLM.from_pretrained("davanstrien/Smol-Hub-tldr")
-
-# Format input according to the chat template
-messages = [{"role": "user", "content": f"<MODEL_CARD>{card.text}"}]
-# Encode with the chat template
-inputs = tokenizer.apply_chat_template(
-    messages, add_generation_prompt=True, return_tensors="pt"
-)
-
-# Generate with stop tokens
-outputs = model.generate(
-    inputs,
-    max_new_tokens=60,
-    pad_token_id=tokenizer.pad_token_id,
-    eos_token_id=tokenizer.eos_token_id,
-    temperature=0.4,
-    do_sample=True,
-)
+from transformers import pipeline
 
-input_length = inputs.shape[1]
-response = tokenizer.decode(outputs[0][input_length:], skip_special_tokens=False)
-
-# Extract just the summary part
-summary = response.split("<CARD_SUMMARY>")[-1].split("</CARD_SUMMARY>")[0]
-print(summary)
->>> "The Smol-Hub-tldr model is a fine-tuned version of SmolLM2-360M designed to generate concise, one-sentence summaries of model and dataset cards from the Hugging Face Hub."
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="davanstrien/SmolLM2-360M-tldr-sft-2025-05-28_17-48", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
 ```
 
-The model currently should close its summary with a `</CARD_SUMMARY>` (cooking some more with this...), so you can also use this as a stopping criterion when using `pipeline` inference. 
-
-```python
-from transformers import pipeline, StoppingCriteria, StoppingCriteriaList
-import torch
+## Training procedure
 
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/davanstrien/huggingface/runs/qf0k0s0e) 
 
-class StopOnTokens(StoppingCriteria):
-    def __init__(self, tokenizer, stop_token_ids):
-        self.stop_token_ids = stop_token_ids
-        self.tokenizer = tokenizer
 
-    def __call__(
-        self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs
-    ) -> bool:
-        for stop_id in self.stop_token_ids:
-            if input_ids[0][-1] == stop_id:
-                return True
-        return False
+This model was trained with SFT.
 
+### Framework versions
 
-# Initialize pipeline
-pipe = pipeline("text-generation", "davanstrien/Smol-Hub-tldr")
-tokenizer = pipe.tokenizer
+- TRL: 0.19.0
+- Transformers: 4.52.3
+- Pytorch: 2.7.0
+- Datasets: 3.6.0
+- Tokenizers: 0.21.1
 
-# Get the token IDs for stopping
-stop_token_ids = [
-    tokenizer.encode("</CARD_SUMMARY>", add_special_tokens=True)[-1],
-    tokenizer.eos_token_id,
-]
+## Citations
 
-# Create stopping criteria
-stopping_criteria = StoppingCriteriaList([StopOnTokens(tokenizer, stop_token_ids)])
 
-# Generate with stopping criteria
-response = pipe(
-    messages,
-    max_new_tokens=50,
-    do_sample=True,
-    temperature=0.7,
-    stopping_criteria=stopping_criteria,
-    return_full_text=False,
-)
-
-# Clean up the response
-summary = response[0]["generated_text"]
-print(summary)
->>> "This model is a fine-tuned version of SmolLM2-360M for generating concise, one-sentence summaries of model and dataset cards from the Hugging Face Hub."
-```
 
-## Framework Versions
-- TRL 0.14.0
-- Transformers 4.48.3
-- PyTorch 2.6.0
-- Datasets 3.2.0
-- Tokenizers 0.21.0
+Cite TRL as:
+    
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

added_tokens.json

@@ -1,8 +1,9 @@
 {
-  "</CARD>": 49153,
-  "</CARD_SUMMARY>": 49155,
-  "<CARD>": 49152,
-  "<CARD_SUMMARY>": 49154,
-  "<DATASET_CARD>": 49156,
-  "<MODEL_CARD>": 49157
+  "</CARD>": 49154,
+  "</CARD_SUMMARY>": 49156,
+  "<CARD>": 49153,
+  "<CARD_SUMMARY>": 49155,
+  "<DATASET_CARD>": 49157,
+  "<MODEL_CARD>": 49158,
+  "[PAD]": 49152
 }

chat_template.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{% if message['role'] == 'user' %}<CARD>{{ message['content'] }}</CARD>{% elif message['role'] == 'assistant' %}<CARD_SUMMARY>{{ message['content'] }}</CARD_SUMMARY>{% endif %}{% endfor %}{% if not add_generation_prompt %}<|endoftext|>{% endif %}{% if add_generation_prompt %}<CARD_SUMMARY>{% endif %}

config.json

@@ -1,5 +1,4 @@
 {
-  "_name_or_path": "HuggingFaceTB/SmolLM2-360M",
   "architectures": [
     "LlamaForCausalLM"
   ],
@@ -26,7 +25,7 @@
   "rope_theta": 100000,
   "tie_word_embeddings": true,
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.48.3",
+  "transformers_version": "4.52.3",
   "use_cache": true,
-  "vocab_size": 49158
+  "vocab_size": 49159
 }

generation_config.json

@@ -2,5 +2,5 @@
   "_from_model_config": true,
   "bos_token_id": 0,
   "eos_token_id": 0,
-  "transformers_version": "4.48.3"
+  "transformers_version": "4.52.3"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:458b1458e6d5eb6c17779eff4a8f82e6df895ad0ce8fc06a8f04f0dc7a3d4377
-size 723686432
+oid sha256:a5ad1d90198dfce88e6ee1f90539332d12d087f3a59da4f6abdf436813ff622c
+size 723688352

runs/May28_17-48-55_r-davanstrien-jupyterlab-svh3scip-91c05-sx22q/events.out.tfevents.1748447366.r-davanstrien-jupyterlab-svh3scip-91c05-sx22q.1156.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e473849f0b893c6dad386796492a653251310f5d85bb1a862d98be3b90c16e7
+size 18148

special_tokens_map.json

@@ -57,7 +57,13 @@
     "rstrip": false,
     "single_word": false
   },
-  "pad_token": "<|endoftext|>",
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
   "unk_token": {
     "content": "<|endoftext|>",
     "lstrip": false,

tokenizer.json

@@ -1,11 +1,6 @@
 {
   "version": "1.0",
-  "truncation": {
-    "direction": "Right",
-    "max_length": 8192,
-    "strategy": "LongestFirst",
-    "stride": 0
-  },
+  "truncation": null,
   "padding": null,
   "added_tokens": [
     {
@@ -163,7 +158,7 @@
     },
     {
       "id": 49152,
-      "content": "<CARD>",
+      "content": "[PAD]",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
@@ -172,7 +167,7 @@
     },
     {
       "id": 49153,
-      "content": "</CARD>",
+      "content": "<CARD>",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
@@ -181,7 +176,7 @@
     },
     {
       "id": 49154,
-      "content": "<CARD_SUMMARY>",
+      "content": "</CARD>",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
@@ -190,7 +185,7 @@
     },
     {
       "id": 49155,
-      "content": "</CARD_SUMMARY>",
+      "content": "<CARD_SUMMARY>",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
@@ -199,7 +194,7 @@
     },
     {
       "id": 49156,
-      "content": "<DATASET_CARD>",
+      "content": "</CARD_SUMMARY>",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
@@ -208,6 +203,15 @@
     },
     {
       "id": 49157,
+      "content": "<DATASET_CARD>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 49158,
       "content": "<MODEL_CARD>",
       "single_word": false,
       "lstrip": false,

tokenizer_config.json

@@ -138,7 +138,7 @@
       "special": true
     },
     "49152": {
-      "content": "<CARD>",
+      "content": "[PAD]",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -146,7 +146,7 @@
       "special": true
     },
     "49153": {
-      "content": "</CARD>",
+      "content": "<CARD>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -154,7 +154,7 @@
       "special": true
     },
     "49154": {
-      "content": "<CARD_SUMMARY>",
+      "content": "</CARD>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -162,7 +162,7 @@
       "special": true
     },
     "49155": {
-      "content": "</CARD_SUMMARY>",
+      "content": "<CARD_SUMMARY>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -170,7 +170,7 @@
       "special": true
     },
     "49156": {
-      "content": "<DATASET_CARD>",
+      "content": "</CARD_SUMMARY>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -178,6 +178,14 @@
       "special": true
     },
     "49157": {
+      "content": "<DATASET_CARD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49158": {
       "content": "<MODEL_CARD>",
       "lstrip": false,
       "normalized": false,
@@ -195,12 +203,11 @@
     "<MODEL_CARD>"
   ],
   "bos_token": "<|endoftext|>",
-  "chat_template": "{% for message in messages %}{% if message['role'] == 'user' %}<CARD>{{ message['content'] }}</CARD>{% elif message['role'] == 'assistant' %}<CARD_SUMMARY>{{ message['content'] }}</CARD_SUMMARY>{% endif %}{% endfor %}{% if not add_generation_prompt %}<|endoftext|>{% endif %}{% if add_generation_prompt %}<CARD_SUMMARY>{% endif %}",
   "clean_up_tokenization_spaces": false,
   "eos_token": "<|endoftext|>",
   "extra_special_tokens": {},
   "model_max_length": 8192,
-  "pad_token": "<|endoftext|>",
+  "pad_token": "[PAD]",
   "tokenizer_class": "GPT2Tokenizer",
   "unk_token": "<|endoftext|>",
   "vocab_size": 49152

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dc065d067b0b20c9ee90c61037b138a771ed147cc184d5523c1e1e87c5446736
-size 5688
+oid sha256:75bc5c4c8f5f8eed5a885964a0e2fea3ca3737607c6f9890a35ef46b516d96d9
+size 6097