M4 Submission
Collection
A collection of the assignments from the module M4 • 5 items • Updated • 1
Green Patent Classification using LoRA Fine-Tuning
Overview
This project implements a parameter-efficient fine-tuning approach to classify patents as environmentally sustainable ("green") or not green. A pretrained DistilGPT2 model was adapted using Low-Rank Adaptation (LoRA) to create an LLM-based patent judge.
The model is trained to output a binary decision (0 or 1) based on patent text input.
Objective
The goal of this assignment is to:
- Apply parameter-efficient fine-tuning (LoRA) to a pretrained language model.
- Build an LLM-based classifier for green technology patents.
- Generate machine-assisted supervision (MAS) labels.
- Improve dataset quality by overriding silver labels with LLM-generated gold labels.
Base Model
- Base model: distilgpt2
- Total parameters: ~82M
- Trainable parameters (LoRA):
147K (0.18%)
Only LoRA adapter weights were trained. The base model remains frozen.
Methodology
1. Dataset
- Source: 50,000 patent documents
- Label type: Binary (
is_green) - Training split: Silver-labeled data
- Evaluation subset: Top 100 high-priority patents
2. Fine-Tuning Strategy
- Applied LoRA using the PEFT library
- Supervised fine-tuning (SFT)
- Instruction-style prompt formatting
- Binary JSON output format
3. LLM-Based Labeling
The trained model was used as a judge with the following prompt structure:
"You are a patent judge. Return ONLY JSON with key is_green (0 or 1)."
The model generated labels for 100 selected patents, creating a gold-standard subset.
Results
Gold label distribution (100 patents):
- Non-green (0): 53
- Green (1): 47
The generated gold labels were used to override corresponding silver labels in the full dataset, improving label quality.
Key Contributions
- Demonstrates parameter-efficient fine-tuning
- Builds a domain-specific LLM classifier
- Implements machine-assisted supervision
- Shows practical use of LoRA for real-world NLP tasks
Limitations
- Binary classification may oversimplify sustainability
- LLM outputs occasionally require fallback parsing
- Silver labels may still contain noise outside the gold subset
Future Improvements
- Use larger base models
- Apply 8-bit or 4-bit quantization for efficiency
- Add human validation for gold labels
- Evaluate performance using precision/recall metrics
Usage
To load the model with PEFT:
from transformers import AutoModelForCausalLM
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("distilgpt2")
model = PeftModel.from_pretrained(base_model, "your-username/your-repo-name")
Author
Assignment submission for Green Patent Classification using LLMs and LoRA fine-tuning.
Link to video
Model tree for houlie3/LoRA_LLM_training
Base model
distilbert/distilgpt2