README.md

---
base_model: unsloth/llama-3.2-3b-instruct-bnb-4bit
library_name: peft
---

# Model Card for Lyric Generation Model

A fine-tuned language model for generating song lyrics based on musical features extracted from audio.

## Model Details

### Model Description

This model is fine-tuned to generate song lyrics conditioned on musical features extracted from audio files. The model takes musical characteristics as input and produces contextually appropriate lyrics that match the musical style and mood.

- **Developed by:** umerbappi
- **Model type:** Text Generation (Fine-tuned Language Model)
- **Language(s) (NLP):** English
- **License:** No specific license
- **Finetuned from model [optional]:** unsloth/llama-3.2-3b-instruct-bnb-4bit

### Model Sources [optional]

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

### Direct Use

This model can be used to generate song lyrics based on musical features. Input the musical characteristics of an audio track, and the model will generate appropriate lyrics that match the musical style, tempo, and mood.

### Downstream Use [optional]

The model can be integrated into music production workflows, songwriting applications, or creative tools for musicians and content creators.

### Out-of-Scope Use

This model should not be used for generating lyrics that promote harmful content, hate speech, or copyright infringement. The model is designed for creative and educational purposes.

## Bias, Risks, and Limitations

The model is trained on existing song lyrics and may reflect biases present in the training data. Generated lyrics should be reviewed for appropriateness and potential copyright concerns before commercial use.

### Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Review generated content for appropriateness and potential copyright issues before use.

## How to Get Started with the Model

Use the code below to get started with the model.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load the model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("umerbappi/YOUR_MODEL_NAME")
model = AutoModelForCausalLM.from_pretrained("umerbappi/YOUR_MODEL_NAME")

def generate_lyrics(musical_features: str, max_length: int = 200) -> str:
    """Generate lyrics using the trained model."""
    
    # Create Llama-formatted prompt
    instruction = f"Generate lyrics for a song with these characteristics:\n{musical_features}"
    prompt = f"<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n{instruction}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
    
    # Tokenize
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    # Generate
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_length,
            temperature=0.8,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
            eos_token_id=tokenizer.eos_token_id,
        )
    
    # Decode and clean
    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    lyrics = generated_text.replace(prompt, "").strip()
    
    return lyrics

# Example usage
test_features = """Tempo: 120.0 BPM
Duration: 3:30
Loudness: -8.5 dB
Key: G major
Time Signature: 4/4"""

sample_lyrics = generate_lyrics(test_features)
print(sample_lyrics)
```

## Training Details

### Training Data

The model was trained on the LyricGen dataset, which combines musical features from the Million Song Dataset with corresponding lyrics obtained through the Genius API.

**Dataset:** [umerbappi/LyricGen](https://huggingface.co/datasets/umerbappi/LyricGen)

The dataset contains:
- Musical features extracted from the Million Song Dataset
- Song names and metadata
- Corresponding lyrics retrieved via Genius API

### Training Procedure

The model was fine-tuned using the PEFT (Parameter Efficient Fine-Tuning) library on the base model unsloth/llama-3.2-3b-instruct-bnb-4bit.

#### Preprocessing [optional]

Musical features were extracted and formatted into structured text inputs including:
- Tempo (BPM)
- Duration 
- Loudness (dB)
- Key signature
- Time signature

The data was formatted using Llama's chat format with specific user/assistant tokens for instruction following.

#### Training Hyperparameters

- **Base Model:** unsloth/Llama-3.2-3B-Instruct-bnb-4bit
- **Max Sequence Length:** 2048
- **Load in 4-bit:** True
- **LoRA r:** 32
- **LoRA alpha:** 64
- **LoRA dropout:** 0.05
- **Target modules:** ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
- **Number of epochs:** 15
- **Batch size:** 6
- **Evaluation batch size:** 4
- **Gradient accumulation steps:** 3 (effective batch size: 18)
- **Learning rate:** 1e-4
- **Weight decay:** 0.02
- **Warmup steps:** 300
- **Optimizer:** adamw_torch_fused
- **LR scheduler:** cosine_with_restarts
- **Training regime:** bf16 mixed precision
- **Gradient checkpointing:** True
- **Max gradient norm:** 0.5

#### Speeds, Sizes, Times [optional]

[More Information Needed]

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

[More Information Needed]

#### Factors

[More Information Needed]

#### Metrics

[More Information Needed]

### Results

No formal evaluation metrics were computed for this model.

#### Summary

The model was trained to generate lyrics conditioned on musical features but no quantitative evaluation was performed.

## Model Examination [optional]

[More Information Needed]

## Environmental Impact

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

Based on Llama 3.2 3B architecture, fine-tuned for conditional text generation where musical features serve as conditioning input for lyric generation.

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[More Information Needed]

### Framework versions

- PEFT 0.15.2