Llama-8B-Distill-CoT

Llama-8B-Distill-CoT is based on the Llama [ KT ] model, distilled by DeepSeek-R1-Distill-Llama-8B. It has been fine-tuned on the long chain-of-thought reasoning model and specialized datasets, focusing on chain-of-thought (CoT) reasoning for problem-solving. This model is optimized for tasks requiring logical reasoning, detailed explanations, and multi-step problem-solving, making it ideal for applications such as instruction-following, text generation, and complex reasoning tasks.

Use with transformers

Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.

Make sure to update your transformers installation via pip install --upgrade transformers.

import transformers
import torch

model_id = "prithivMLmods/Llama-8B-Distill-CoT"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Intended Use:

1. Instruction-Following: The model is designed to handle detailed instructions, making it ideal for virtual assistants, automation tools, and educational platforms.
2. Problem-Solving: Its fine-tuning on chain-of-thought (CoT) reasoning allows it to tackle multi-step problem-solving in domains such as mathematics, logic, and programming.
3. Text Generation: Capable of generating coherent and contextually relevant content, it is suitable for creative writing, documentation, and report generation.
4. Education and Training: Provides step-by-step explanations and logical reasoning, making it a useful tool for teaching and learning.
5. Research and Analysis: Supports researchers and professionals by generating detailed analyses and structured arguments for complex topics.
6. Programming Assistance: Helps in generating, debugging, and explaining code, as well as creating structured outputs like JSON or XML.

Limitations:

1. Resource Intensive: Requires high computational resources to run efficiently, which may limit accessibility for small-scale deployments.
2. Hallucination Risk: May generate incorrect or misleading information, especially when handling ambiguous or poorly framed prompts.
3. Domain-Specific Gaps: While fine-tuned for reasoning, it may not perform well in specialized domains outside its training data.
4. Bias in Training Data: The model's responses can reflect biases present in the datasets it was trained on, potentially leading to biased or inappropriate outputs.
5. Dependence on Input Quality: Performance heavily depends on clear, structured inputs. Ambiguous or vague queries can result in suboptimal outputs.
6. Limited Real-Time Context: The model cannot access real-time information or updates beyond its training data, potentially affecting its relevance for time-sensitive queries.
7. Scalability for Long-Context: While capable of multi-step reasoning, its ability to handle extremely long or complex contexts may be limited compared to larger, more specialized models.