Falcon-RW-1B Fine-tuned with LoRA on IMDb Sentiment Dataset
This is a Falcon-RW-1B language model fine-tuned using LoRA (Low-Rank Adaptation) for causal language modeling, trained on a subset of the IMDb movie reviews dataset for sentiment-related text generation tasks.
Model Details
Model Description
This model is based on the Falcon-RW-1B pretrained causal language model, fine-tuned with parameter-efficient LoRA adapters targeting the "query_key_value" modules. Training was performed on a small subset of the IMDb dataset (1,000 samples) with sequences truncated/padded to 128 tokens.
- Developed by: Sujith Somanunnithan
- Model type: Causal Language Model (Transformer)
- Language: English
- License: Apache 2.0
- Finetuned from:
tiiuae/falcon-rw-1b - Fine-tuning method: LoRA (using PEFT library)
Model Sources
- Repository: Sujithadr/Falc-Lora-imdb
- Base model: https://huggingface.co/tiiuae/falcon-rw-1b
- Dataset: IMDb movie reviews — https://huggingface.co/datasets/imdb
Uses
Direct Use
This model can be used for generating or completing English text sequences related to movie reviews, sentiment analysis prompts, or similar NLP causal language modeling tasks.
Downstream Use
The LoRA adapters allow further parameter-efficient fine-tuning for other NLP tasks or domain adaptation, leveraging the Falcon-RW-1B base.
Out-of-Scope Use
- This model is not optimized for zero-shot classification or tasks outside of causal language modeling.
- Not suitable for languages other than English.
- The small training subset limits generalization; performance on real-world text may vary.
Bias, Risks, and Limitations
- The base Falcon-RW-1B model inherits biases present in the pretraining data.
- The fine-tuning on IMDb is limited in scope and size; results may be biased toward movie review sentiment.
- Use caution when deploying in production or sensitive applications.
How to Get Started with the Model
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "your-hf-username/your-falcon-lora-model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
inputs = tokenizer("The movie was", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))
Training Details
Training Data
- Dataset: IMDb movie reviews (subset of 1000 training samples)
- Text sequences truncated/padded to max length 128
Training Procedure
- Fine-tuned on Falcon-RW-1B using LoRA adapters targeting "query_key_value"
- Training arguments: batch size 2, gradient accumulation 4, 1 epoch, mixed precision (fp16)
- Trainer API from Hugging Face Transformers with PEFT integration
Training Hyperparameters
- LoRA config: r=8, lora_alpha=16, dropout=0.1
- Optimized with AdamW (default Trainer)
- Single epoch training on a small dataset for demonstration
Evaluation
Testing Data, Factors & Metrics
- No formal evaluation metrics reported for this demo model
- Intended for proof-of-concept fine-tuning and further downstream adaptation
Environmental Impact
- Training performed on a GPU-enabled machine with mixed precision to reduce energy consumption.
- Approximate compute and carbon footprint unknown; training on a small subset minimizes impact.
Technical Specifications
Model Architecture and Objective
- Falcon-RW-1B causal LM architecture based on transformer decoder blocks
- Objective: language modeling via cross-entropy loss on next-token prediction
Compute Infrastructure
- Training performed on a single GPU with mixed precision (fp16)
- Software: Transformers, PEFT, PyTorch
Citation
If you use this model, please cite:
@misc{somanunnithan2025falconlora,
title={Falcon-RW-1B fine-tuned with LoRA on IMDb dataset},
author={Sujith Somanunnithan},
year={2025},
howpublished={\url{https://huggingface.co/your-hf-username/your-falcon-lora-model}}
}
Model Card Authors
- Sujith Somanunnithan