README.md

---
base_model: HuggingFaceTB/SmolLM2-1.7B-Instruct
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
- code
- superthoughts
- cot
- reasoning
license: apache-2.0
language:
- en
pipeline_tag: text-generation
model-index:
- name: Superthoughts-lite-1.8B-experimental-o1
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: IFEval (0-Shot)
      type: wis-k/instruction-following-eval
      split: train
      args:
        num_few_shot: 0
    metrics:
    - type: inst_level_strict_acc and prompt_level_strict_acc
      value: 3.75
      name: averaged accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Pinkstack%2FSuperthoughts-lite-1.8B-experimental-o1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: BBH (3-Shot)
      type: SaylorTwift/bbh
      split: test
      args:
        num_few_shot: 3
    metrics:
    - type: acc_norm
      value: 9.13
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Pinkstack%2FSuperthoughts-lite-1.8B-experimental-o1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MATH Lvl 5 (4-Shot)
      type: lighteval/MATH-Hard
      split: test
      args:
        num_few_shot: 4
    metrics:
    - type: exact_match
      value: 1.06
      name: exact match
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Pinkstack%2FSuperthoughts-lite-1.8B-experimental-o1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GPQA (0-shot)
      type: Idavidrein/gpqa
      split: train
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 3.36
      name: acc_norm
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Pinkstack%2FSuperthoughts-lite-1.8B-experimental-o1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MuSR (0-shot)
      type: TAUR-Lab/MuSR
      args:
        num_few_shot: 0
    metrics:
    - type: acc_norm
      value: 1.76
      name: acc_norm
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Pinkstack%2FSuperthoughts-lite-1.8B-experimental-o1
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU-PRO (5-shot)
      type: TIGER-Lab/MMLU-Pro
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 9.45
      name: accuracy
    source:
      url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=Pinkstack%2FSuperthoughts-lite-1.8B-experimental-o1
      name: Open LLM Leaderboard
---
![superthoughtslight.png](https://cdn-uploads.huggingface.co/production/uploads/6710ba6af1279fe0dfe33afe/2LuPB_ZPCGni3-PyCkL0-.png)
# Information
Advanced, high-quality and lite reasoning for a tiny size that you can run locally in Q8 on your phone! 😲

⚠️This is an experimental version: it may not always answer your question properly or correctly. currently reasoning may not always work on long conversations, as we've trained it on single turn conversations only.
SmolLM2-1.7B-Instruct on an advanced reasoning pattern dataset (half synthetic, half written manually by us.) to create this model. Supposed to output like this:
```
<|im_start|>user
What are you<|im_end|>
<|im_start|>assistant
<think>
Alright, the user just asked 'What are you', meaning they want to know who I am. I think my name is Superthoughts (lite version), created by Pinkstack on January 2025. I'm ready to answer their question.
</think>
Welcome! I'm Superthoughts (lite) created by Pinkstack in January 2025. Ready to help you with whatever you need!<|im_end|>
```
# Evaluation (soon)

# Examples:
all responses below generated with no system prompt, 400 maximum tokens and a temperature of 0.7 (not recommended, 0.3 - 0.5 is better):
Generated inside the android application, Pocketpal via GGUF Q8, using the model's prompt format.
1)
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6710ba6af1279fe0dfe33afe/wh33o-vjxIePfPqoN3q1z.png)
2)
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6710ba6af1279fe0dfe33afe/Y8optw73kTgqMnZKj3wKj.png)
3)
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6710ba6af1279fe0dfe33afe/6lywy3IYEIgzPnUIJ5RvF.png)
4)
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6710ba6af1279fe0dfe33afe/0K2rR9osmT20JrDvZuptV.png)

# Uploaded  model

- **Developed by:** Pinkstack
- **License:** apache-2.0
- **Finetuned from model :** HuggingFaceTB/SmolLM2-1.7B-Instruct

This smollm2 model was trained with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/Pinkstack__Superthoughts-lite-1.8B-experimental-o1-details)!
Summarized results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/contents/viewer/default/train?q=Pinkstack%2FSuperthoughts-lite-1.8B-experimental-o1&sort[column]=Average%20%E2%AC%86%EF%B8%8F&sort[direction]=desc)!

|      Metric       |Value (%)|
|-------------------|--------:|
|**Average**        |     4.75|
|IFEval (0-Shot)    |     3.75|
|BBH (3-Shot)       |     9.13|
|MATH Lvl 5 (4-Shot)|     1.06|
|GPQA (0-shot)      |     3.36|
|MuSR (0-shot)      |     1.76|
|MMLU-PRO (5-shot)  |     9.45|