| |
|
| | --- |
| | library_name: transformers |
| | tags: [causal-lm, text-generation, fine-tuned, falcon, lora, imdb, sentiment-analysis] |
| | --- |
| | |
| | # Falcon-RW-1B Fine-tuned with LoRA on IMDb Sentiment Dataset |
| |
|
| | This is a Falcon-RW-1B language model fine-tuned using LoRA (Low-Rank Adaptation) for causal language modeling, trained on a subset of the IMDb movie reviews dataset for sentiment-related text generation tasks. |
| |
|
| | ## Model Details |
| |
|
| | ### Model Description |
| |
|
| | This model is based on the Falcon-RW-1B pretrained causal language model, fine-tuned with parameter-efficient LoRA adapters targeting the "query_key_value" modules. Training was performed on a small subset of the IMDb dataset (1,000 samples) with sequences truncated/padded to 128 tokens. |
| |
|
| | - **Developed by:** Sujith Somanunnithan |
| | - **Model type:** Causal Language Model (Transformer) |
| | - **Language:** English |
| | - **License:** Apache 2.0 |
| | - **Finetuned from:** `tiiuae/falcon-rw-1b` |
| | - **Fine-tuning method:** LoRA (using PEFT library) |
| |
|
| | ### Model Sources |
| |
|
| | - **Repository:** Sujithadr/Falc-Lora-imdb |
| | - **Base model:** https://huggingface.co/tiiuae/falcon-rw-1b |
| | - **Dataset:** IMDb movie reviews — https://huggingface.co/datasets/imdb |
| |
|
| | ## Uses |
| |
|
| | ### Direct Use |
| |
|
| | This model can be used for generating or completing English text sequences related to movie reviews, sentiment analysis prompts, or similar NLP causal language modeling tasks. |
| |
|
| | ### Downstream Use |
| |
|
| | The LoRA adapters allow further parameter-efficient fine-tuning for other NLP tasks or domain adaptation, leveraging the Falcon-RW-1B base. |
| |
|
| | ### Out-of-Scope Use |
| |
|
| | - This model is **not optimized for zero-shot classification or tasks outside of causal language modeling.** |
| | - Not suitable for languages other than English. |
| | - The small training subset limits generalization; performance on real-world text may vary. |
| |
|
| | ## Bias, Risks, and Limitations |
| |
|
| | - The base Falcon-RW-1B model inherits biases present in the pretraining data. |
| | - The fine-tuning on IMDb is limited in scope and size; results may be biased toward movie review sentiment. |
| | - Use caution when deploying in production or sensitive applications. |
| |
|
| | ## How to Get Started with the Model |
| |
|
| | ```python |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | model_id = "your-hf-username/your-falcon-lora-model" |
| | tokenizer = AutoTokenizer.from_pretrained(model_id) |
| | model = AutoModelForCausalLM.from_pretrained(model_id) |
| | |
| | inputs = tokenizer("The movie was", return_tensors="pt") |
| | outputs = model.generate(**inputs, max_length=50) |
| | print(tokenizer.decode(outputs[0])) |
| | ``` |
| |
|
| | ## Training Details |
| |
|
| | ### Training Data |
| |
|
| | - Dataset: IMDb movie reviews (subset of 1000 training samples) |
| | - Text sequences truncated/padded to max length 128 |
| |
|
| | ### Training Procedure |
| |
|
| | - Fine-tuned on Falcon-RW-1B using LoRA adapters targeting "query_key_value" |
| | - Training arguments: batch size 2, gradient accumulation 4, 1 epoch, mixed precision (fp16) |
| | - Trainer API from Hugging Face Transformers with PEFT integration |
| |
|
| | ### Training Hyperparameters |
| |
|
| | - LoRA config: r=8, lora_alpha=16, dropout=0.1 |
| | - Optimized with AdamW (default Trainer) |
| | - Single epoch training on a small dataset for demonstration |
| | |
| | ## Evaluation |
| | |
| | ### Testing Data, Factors & Metrics |
| | |
| | - No formal evaluation metrics reported for this demo model |
| | - Intended for proof-of-concept fine-tuning and further downstream adaptation |
| | |
| | ## Environmental Impact |
| | |
| | - Training performed on a GPU-enabled machine with mixed precision to reduce energy consumption. |
| | - Approximate compute and carbon footprint unknown; training on a small subset minimizes impact. |
| | |
| | ## Technical Specifications |
| | |
| | ### Model Architecture and Objective |
| | |
| | - Falcon-RW-1B causal LM architecture based on transformer decoder blocks |
| | - Objective: language modeling via cross-entropy loss on next-token prediction |
| | |
| | ### Compute Infrastructure |
| | |
| | - Training performed on a single GPU with mixed precision (fp16) |
| | - Software: Transformers, PEFT, PyTorch |
| | |
| | ## Citation |
| | |
| | If you use this model, please cite: |
| | |
| | ``` |
| | @misc{somanunnithan2025falconlora, |
| | title={Falcon-RW-1B fine-tuned with LoRA on IMDb dataset}, |
| | author={Sujith Somanunnithan}, |
| | year={2025}, |
| | howpublished={\url{https://huggingface.co/your-hf-username/your-falcon-lora-model}} |
| | } |
| | ``` |
| | |
| | ## Model Card Authors |
| | |
| | - Sujith Somanunnithan |
| | |
| | ## Model Card Contact |
| | |
| | - sujith.adr@gmail.com |
| | |
