README.md

---
base_model:
- Qwen/Qwen2.5-3B-Instruct
library_name: transformers
license: mit
language:
- zho
- eng
- fra
- spa
- por
- deu
- ita
- rus
- jpn
- kor
- vie
- tha
- ara
tags:
- chemistry
- biology
- code
- text-generation-inference
- STEM
- unsloth
---
<div align="center">
<span style="font-family: default; font-size: 1.5em;">Athena-3</span>
<div>
🚀 Faster, Sharper, Smarter than Athena 1 and Athena 2🌟
</div>
</div>
<br>
<div align="center" style="line-height: 1;">
  <a href="https://github.com/Aayan-Mishra/Maverick-Search" style="margin: 2px;">
    <img alt="Github Page" src="https://img.shields.io/badge/Toolkit-000000?style=for-the-badge&logo=github&logoColor=000&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://aayanmishra.com/blog/athena-3" target="_blank" style="margin: 2px;">
    <img alt="Blogpost" src="https://img.shields.io/badge/Blogpost-%23000000.svg?style=for-the-badge&logo=notion&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://huggingface.co/Spestly/Athena-3-3B" style="margin: 2px;">
    <img alt="HF Page" src="https://img.shields.io/badge/Athena-fcd022?style=for-the-badge&logo=huggingface&logoColor=000&labelColor" style="display: inline-block; vertical-align: middle;"/>
  </a>
</div>

# **Athena-3-3B Model Card**

*Athena generated this model card!*

## **Model Overview**

**Athena-3-3B** is a 3.09-billion-parameter causal language model fine-tuned from Qwen2.5-3B-Instruct. This model is designed to excel in various natural language processing tasks, offering enhanced reasoning and instruction-following capabilities.

## **Model Details**

- **Model Developer:** Aayan Mishra
- **Model Type:** Causal Language Model
- **Architecture:** Transformer with Rotary Position Embeddings (RoPE), SwiGLU activation, RMSNorm, Attention QKV bias, and tied word embeddings
- **Parameters:** 3.09 billion total (2.77 billion non-embedding)
- **Layers:** 36
- **Attention Heads:** 16 for query and 2 for key-value (Grouped Query Attention)
- **Vocabulary Size:** Approximately 151,646 tokens
- **Context Length:** Supports up to 32,768 tokens
- **Languages Supported:** Primarily English, with basic support for other languages
- **License:** MIT

## **Training Details**

Athena-3-3B was fine-tuned using the Unsloth framework on a single NVIDIA A100 GPU. The fine-tuning process spanned approximately 90 minutes over 60 epochs, utilizing a curated dataset focused on instruction-following and general NLP tasks. This approach aimed to enhance the model's performance in complex reasoning and academic tasks.

## **Intended Use**

Athena-3-3B is designed for a range of applications, including but not limited to:

- **General NLP Tasks:** Engaging in text completion, summarization, and question-answering tasks.
- **Academic Assistance:** Providing support for tutoring, essay composition, and research inquiries.
- **Data Analysis:** Offering insights and interpretations of data-centric queries.

While Athena-3-3B is a powerful tool for various applications, it is not intended for real-time, safety-critical systems or for processing sensitive personal information.

## **How to Use**

To utilize Athena-3-3B, ensure that you have the latest version of the `transformers` library installed:

```bash
pip install transformers
```

Here's an example of how to load the Athena-3-3B model and generate a response:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Spestly/Athena-3-3B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Explain the concept of entropy in thermodynamics."
messages = [
    {"role": "system", "content": "You are Maverick, an AI assistant designed to be helpful."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
```
### **Maverick Search usage 🔍**

To use this model with Maverick Search, please refer to this [repository](https://github.com/Aayan-Mishra/Maverick-Search)

## **Limitations**

Users should be aware of the following limitations:

- **Biases:** Athena-3-3B may exhibit biases present in its training data. Users should critically assess outputs, especially in sensitive contexts.
- **Knowledge Cutoff:** The model's knowledge is current up to August 2024. It may not be aware of events or developments occurring after this date.
- **Language Support:** While primarily trained on English data, performance in other languages may be inconsistent.

## **Acknowledgements**

Athena-3-3B builds upon the work of the Qwen team. Gratitude is also extended to the open-source AI community for their contributions to tools and frameworks that facilitated the development of Athena-3-3B.

## **License**

Athena-3-3B is released under the MIT License, permitting wide usage with proper attribution.

## **Contact**

- Email: maverick@aayanmishra.com