---
library_name: transformers
tags: [causal-lm, text-generation, fine-tuned, falcon, lora, imdb, sentiment-analysis]
---

# Falcon-RW-1B Fine-tuned with LoRA on IMDb Sentiment Dataset

This is a Falcon-RW-1B language model fine-tuned using LoRA (Low-Rank Adaptation) for causal language modeling, trained on a subset of the IMDb movie reviews dataset for sentiment-related text generation tasks.

## Model Details

### Model Description

This model is based on the Falcon-RW-1B pretrained causal language model, fine-tuned with parameter-efficient LoRA adapters targeting the "query_key_value" modules. Training was performed on a small subset of the IMDb dataset (1,000 samples) with sequences truncated/padded to 128 tokens.

- **Developed by:** Sujith Somanunnithan
- **Model type:** Causal Language Model (Transformer)
- **Language:** English
- **License:** Apache 2.0
- **Finetuned from:** `tiiuae/falcon-rw-1b`
- **Fine-tuning method:** LoRA (using PEFT library)

### Model Sources

- **Repository:** Sujithadr/Falc-Lora-imdb
- **Base model:** https://huggingface.co/tiiuae/falcon-rw-1b
- **Dataset:** IMDb movie reviews — https://huggingface.co/datasets/imdb

## Uses

### Direct Use

This model can be used for generating or completing English text sequences related to movie reviews, sentiment analysis prompts, or similar NLP causal language modeling tasks.

### Downstream Use

The LoRA adapters allow further parameter-efficient fine-tuning for other NLP tasks or domain adaptation, leveraging the Falcon-RW-1B base.

### Out-of-Scope Use

- This model is **not optimized for zero-shot classification or tasks outside of causal language modeling.**
- Not suitable for languages other than English.
- The small training subset limits generalization; performance on real-world text may vary.

## Bias, Risks, and Limitations

- The base Falcon-RW-1B model inherits biases present in the pretraining data.
- The fine-tuning on IMDb is limited in scope and size; results may be biased toward movie review sentiment.
- Use caution when deploying in production or sensitive applications.

## How to Get Started with the Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "your-hf-username/your-falcon-lora-model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

inputs = tokenizer("The movie was", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))
```

## Training Details

### Training Data

- Dataset: IMDb movie reviews (subset of 1000 training samples)
- Text sequences truncated/padded to max length 128

### Training Procedure

- Fine-tuned on Falcon-RW-1B using LoRA adapters targeting "query_key_value"
- Training arguments: batch size 2, gradient accumulation 4, 1 epoch, mixed precision (fp16)
- Trainer API from Hugging Face Transformers with PEFT integration

### Training Hyperparameters

- LoRA config: r=8, lora_alpha=16, dropout=0.1
- Optimized with AdamW (default Trainer)
- Single epoch training on a small dataset for demonstration

## Evaluation

### Testing Data, Factors & Metrics

- No formal evaluation metrics reported for this demo model
- Intended for proof-of-concept fine-tuning and further downstream adaptation

## Environmental Impact

- Training performed on a GPU-enabled machine with mixed precision to reduce energy consumption.
- Approximate compute and carbon footprint unknown; training on a small subset minimizes impact.

## Technical Specifications

### Model Architecture and Objective

- Falcon-RW-1B causal LM architecture based on transformer decoder blocks
- Objective: language modeling via cross-entropy loss on next-token prediction

### Compute Infrastructure

- Training performed on a single GPU with mixed precision (fp16)
- Software: Transformers, PEFT, PyTorch

## Citation

If you use this model, please cite:

```
@misc{somanunnithan2025falconlora,
  title={Falcon-RW-1B fine-tuned with LoRA on IMDb dataset},
  author={Sujith Somanunnithan},
  year={2025},
  howpublished={\url{https://huggingface.co/your-hf-username/your-falcon-lora-model}}
}
```

## Model Card Authors

- Sujith Somanunnithan

## Model Card Contact

- sujith.adr@gmail.com