README.md

---
library_name: transformers
tags:
  - text-generation
  - ad-generation
  - marketing
  - transformers
  - pytorch
  - beam-search
---

# # Model Card for Falcon-RW-1B Fine-Tuned Model

This model is a fine-tuned version of `tiiuae/falcon-rw-1b` trained on an advertising-related dataset to generate ad text based on prompts.



## Model Details

### Model Description

This model is a fine-tuned version of the Falcon-RW-1B model, specifically adapted for generating advertising content. The fine-tuning process utilized a dataset containing ad-related text, formatted as structured prompt-response pairs.

- **Developed by:** Adnane Touiyate
- **Funded by [optional]:** [Adnane10](https://huggingface.co/Adnane10)
- **Shared by [optional]:** [Adnane10](https://huggingface.co/Adnane10)
- **Model type:** Falcon-RW-1B (Causal Language Model)
- **Language(s) (NLP):** English
- **License:** MIT
- **Finetuned from model [optional]:** `tiiuae/falcon-rw-1b`


## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

This model can be used for generating advertising content based on structured prompts. It is useful for marketers and advertisers who need AI-generated ad copies.


### Downstream Use [optional]

The model can be further fine-tuned for specific ad categories or integrated into larger marketing automation workflows.


### Out-of-Scope Use

This model is not intended for generating non-advertising-related content, and its performance may be suboptimal in general text generation tasks beyond its training scope.


## Bias, Risks, and Limitations

Since the model has been fine-tuned on advertising content, it may inherit biases present in the dataset. Users should be cautious when generating ads to ensure they meet ethical and regulatory standards.

### Recommendations

Users should validate the generated content for appropriateness, compliance, and factual accuracy before using it in real-world applications.

## How to Get Started with the Model

Use the code below to load and use the model:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-rw-1b")
model = AutoModelForCausalLM.from_pretrained("path_to_finetuned_model")

def generate_ad(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to('cuda')
    outputs = model.generate(**inputs, max_length=100)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

print(generate_ad("Introducing our latest product: "))
```

## Training Details

### Training Data

The model was trained on `fixed_ads_list.json`, a dataset containing structured ad-related prompts and responses.

### Training Procedure

- **Preprocessing:** Tokenized text in the format `### Prompt: [User Input] ### Response: [Ad Text]`
- **Quantization:** Used 4-bit quantization (NF4) with `bitsandbytes` for efficiency.
- **Fine-tuning method:** LoRA (Low-Rank Adaptation) for efficient adaptation.
- **Hardware:** GPU-accelerated training.

#### Training Hyperparameters

- **Learning Rate:** 1e-4
- **Batch Size:** 2 (per device)
- **Gradient Accumulation:** 8 steps
- **Epochs:** 6
- **Precision:** BF16
- **Evaluation Strategy:** Epoch-based
- **Early Stopping:** Enabled after 2 epochs without improvement

## Evaluation

### Testing Data, Factors & Metrics

- **Metrics:** BLEU and ROUGE scores
- **Results:** Sample evaluation showed:


## Environmental Impact

- **Hardware Type:** NVIDIA P100 GPU
- **Hours used:** ~54 minutes
- **Cloud Provider:** Kaggle

### Model Architecture and Objective

The Falcon-RW-1B model is a causal language model optimized for text generation.

### Compute Infrastructure

#### Hardware

- GPUs (NVIDIA P100)
- Used `bitsandbytes` for memory-efficient training

#### Software

- `transformers`
- `datasets`
- `peft`
- `torch`
- `accelerate`
- `bitsandbytes`

## Model Card Authors

**Adnane Touiyate** ([@Adnane10](https://huggingface.co/Adnane10))

## Contact

For questions or collaborations, reach out via [LinkedIn](https://www.linkedin.com/in/adnanetouiyate/) or email: [adnanetouiayte11@gmail.com](mailto:adnanetouiayte11@gmail.com)