|
|
--- |
|
|
license: cc-by-4.0 |
|
|
datasets: |
|
|
- sixf0ur/ScentSet |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- chemistry |
|
|
- biology |
|
|
- climate |
|
|
- medical |
|
|
- text-generation-inference |
|
|
- tiny |
|
|
- scent |
|
|
- smell |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# ScentLLaMA |
|
|
|
|
|
A tiny LLaMA-based language model with 600k parameters, pretrained specifically on the synthetic ScentSet dataset (572k entries, ~15M tokens). |
|
|
Designed exclusively to describe and classify smells and aromas. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Parameters:** ~600,000 |
|
|
- **Task:** Text generation of smell descriptions |
|
|
- **Training data:** ScentSet (synthetic dataset of smell descriptions) |
|
|
- **Training date:** July 2025 |
|
|
- **License:** CC BY 4.0 |
|
|
|
|
|
### 📉 Training & Evaluation Loss |
|
|
|
|
|
The following plot shows the training and evaluation loss over time. |
|
|
Training was performed for approximately **160,000 steps**. |
|
|
|
|
|
The evaluation loss remains consistently close to the training loss throughout training (within ~0.01), |
|
|
indicating that the model generalizes well and shows no signs of overfitting. |
|
|
Training arguments can be seen below: |
|
|
```python |
|
|
TRAINING_ARGS = TrainingArguments( |
|
|
output_dir=OUTPUT_DIR, |
|
|
overwrite_output_dir=True, |
|
|
num_train_epochs=20, |
|
|
per_device_train_batch_size=16, |
|
|
per_device_eval_batch_size=16, |
|
|
learning_rate=1e-4, |
|
|
warmup_steps=500, |
|
|
lr_scheduler_type="cosine", |
|
|
weight_decay=0.01, |
|
|
max_grad_norm=1.0, |
|
|
logging_dir=os.path.join(OUTPUT_DIR, "logs"), |
|
|
logging_steps=100, |
|
|
save_steps=500, |
|
|
eval_steps=500, |
|
|
eval_strategy="steps", |
|
|
load_best_model_at_end=True, |
|
|
metric_for_best_model="eval_loss", |
|
|
greater_is_better=False, |
|
|
save_total_limit=2, |
|
|
fp16=True, |
|
|
report_to="tensorboard", |
|
|
) |
|
|
```` |
|
|
 |
|
|
|
|
|
|
|
|
## Usage |
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
model_name = "sixf0ur/ScentLLaMA" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
model = AutoModelForCausalLM.from_pretrained(model_name) |
|
|
|
|
|
prompt = "A fresh and fruity aroma with hints of" |
|
|
inputs = tokenizer(prompt, return_token_type_ids=False, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=25) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
|
|
|
# > A fresh and fruity aroma with hints of green leaves and a hint of something earthy. It is a ripe plum. |
|
|
``` |
|
|
|
|
|
|
|
|
### Citation |
|
|
```json |
|
|
@misc{ScentLLaMA_2025, |
|
|
author = {David S.}, |
|
|
title = {ScentLLaMA: A tiny LLaMA Model for Smell Description Generation}, |
|
|
year = {2025}, |
|
|
publisher = {Hugging Face Models}, |
|
|
howpublished = {\url{https://huggingface.co/sixf0ur/ScentLLaMA}}, |
|
|
note = {Pretrained on the ScentSet dataset to generate natural language descriptions of smells} |
|
|
} |
|
|
``` |
