OxxoCodes/distil-SmolLM2-135M-Instruct

Distil-SmolLM2-135M

Distil-SmolLM2-135M is a distilled version of SmolLM2-1.7B-Instruct, trained on a filtered subset of the Smoltalk dataset. This release aims to provide a more capable and performant ultra-small 135M generative large language model for small tasks on-edge or at-scale.

Model Details

• Student Model: SmolLM2-135M-Instruct (this model)
• Teacher Model: SmolLM2-1.7B-Instruct
• Parameters: ~135 Million
• Language: English
• Description: This model was created by distilling the knowledge from SmolLM2-1.7B-Instruct into SmolLM2-135M-Instruct. The distillation process utilized the Smoltalk dataset, with specific exclusions.

Intended Uses & Limitations

Intended Uses: This model is intended for research, experimentation, and general use in instruction-following and chat applications where a smaller model footprint is desired. It can be used for:

• Answering questions based on provided context.
• Classifying text.
• Simple conversational tasks.
• More complex tasks upon further fine-tuning.

Limitations:

• Reduced Capacity: Being a smaller model (135M parameters), its performance will generally be significantly lower than its larger teacher model (1.7B parameters) and other state-of-the-art large language models, especially on complex reasoning or knowledge-intensive tasks.
• Hallucinations: Like all LLMs, this model can generate incorrect or nonsensical information (hallucinate).
• Bias: The model may reflect biases present in the training data.
• Safety: The model has not undergone extensive safety fine-tuning or alignment beyond the original instruction tuning of the teacher. It may generate harmful, unethical, or offensive content. Use with caution and appropriate safeguards.
• Dataset Exclusions: The model was not trained on the apigen-80k or longalign sources from Smoltalk, which might affect its performance on tasks related to function calling or very long context alignment.

How to Get Started

You can use this model with the transformers library for text generation tasks.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "OxxoCodes/distil-SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Make sure to use a prompt format the model was trained on
# This example uses a generic instruction format.
# Refer to SmolLM2-1.7B-Instruct or SmolLM2-135M-Instruct for specific prompt templates if applicable.
prompt = f"<|im_start|>system\nYou are a helpful AI assistant.<|im_end|>\n<|im_start|>user\nWhat is the world's largest sea mammal?\n<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=50, num_return_sequences=1, temperature=0.3, top_p=0.95)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))