Upload folder using huggingface_hub

README.md

@@ -0,0 +1,137 @@
+---
+license: mit
+base_model: gpt2-large
+tags:
+- natural-language-inference
+- lora
+- peft
+- gpt2
+- multinli
+- text-classification
+datasets:
+- nyu-mll/multi_nli
+language:
+- en
+pipeline_tag: text-classification
+---
+
+# GPT2-Large LoRA Fine-tuned for Natural Language Inference
+
+This model is a LoRA (Low-Rank Adaptation) fine-tuned version of GPT2-large for Natural Language Inference (NLI) on the MultiNLI dataset.
+
+## Model Details
+
+- **Base Model**: GPT2-large (774M parameters)
+- **Fine-tuning Method**: LoRA (Low-Rank Adaptation)
+- **Trainable Parameters**: ~2.3M (0.3% of total parameters)
+- **Dataset**: MultiNLI (50K training samples)
+- **Task**: Natural Language Inference (3-class classification)
+
+## Performance
+
+- **Test Accuracy (Matched)**: ~79.22%
+- **Test Accuracy (Mismatched)**: ~78.xx%
+- **Training Method**: Parameter-efficient fine-tuning with LoRA
+- **Hardware**: Trained on 36G vGPU
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+
+# Load tokenizer and base model
+tokenizer = AutoTokenizer.from_pretrained("gpt2-large")
+base_model = AutoModelForCausalLM.from_pretrained("gpt2-large")
+
+# Load LoRA adapter
+model = PeftModel.from_pretrained(base_model, "hilaryc112/LoRA-GPT2-Project")
+
+# Format input
+premise = "A person is outdoors, on a horse."
+hypothesis = "A person is at a diner, ordering an omelette."
+input_text = f"Premise: {premise}\nHypothesis: {hypothesis}\nRelationship:"
+
+# Tokenize and generate
+inputs = tokenizer(input_text, return_tensors="pt")
+with torch.no_grad():
+    outputs = model.generate(**inputs, max_new_tokens=10, pad_token_id=tokenizer.eos_token_id)
+    
+prediction = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True).strip()
+print(f"Prediction: {prediction}")  # Should output: contradiction, neutral, or entailment
+```
+
+## Training Configuration
+
+{
+  "model_name": "gpt2-large",
+  "max_length": 512,
+  "lora_r": 16,
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "target_modules": [
+    "c_attn",
+    "c_proj",
+    "c_fc"
+  ],
+  "num_epochs": 3,
+  "train_batch_size": 4,
+  "eval_batch_size": 12,
+  "gradient_accumulation_steps": 6,
+  "learning_rate": 0.0002,
+  "weight_decay": 0.01,
+  "max_grad_norm": 1.0,
+  "use_fp16": true,
+  "gradient_checkpointing": true,
+  "logging_steps": 100,
+  "eval_steps": 500,
+  "save_steps": 500,
+  "save_total_limit": 3,
+  "early_stopping_patience": 5,
+  "data_dir": "./processed_data",
+  "output_dir": "./gpt2_lora_multinli",
+  "seed": 42,
+  "use_wandb": false,
+  "_comments": {
+    "effective_batch_size": "6 * 6 = 36 (optimized for 36G vGPU)",
+    "memory_optimization": "FP16 + gradient checkpointing enabled",
+    "lora_config": "Rank 16 with alpha 32 for good performance/efficiency balance",
+    "target_modules": "GPT2 attention and MLP layers for comprehensive adaptation",
+    "training_data": "Uses 50K samples from MultiNLI training set (configured in preprocessing)",
+    "evaluation_data": "Uses local dev files for matched/mismatched evaluation",
+    "training_adjustments": "Reduced epochs to 2 and LR to 1e-4 for better training with real data",
+    "eval_frequency": "Less frequent evaluation (every 500 steps) due to larger dataset"
+  }
+}
+
+## Dataset Format
+
+The model was trained on text-to-text format:
+```
+Premise: [premise text]
+Hypothesis: [hypothesis text]
+Relationship: [entailment/neutral/contradiction]
+```
+
+## Files
+
+- `adapter_config.json`: LoRA adapter configuration
+- `adapter_model.safetensors`: LoRA adapter weights
+- `training_config.json`: Training hyperparameters and settings
+
+## Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{gpt2-lora-multinli,
+  title={GPT2-Large LoRA Fine-tuned for Natural Language Inference},
+  author={CSIT6000R Individual Project},
+  year={2024},
+  howpublished={\url{https://huggingface.co/hilaryc112/LoRA-GPT2-Project}}
+}
+```
+
+## License
+
+This model is released under the MIT License.

adapter_config.json

@@ -0,0 +1,38 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "gpt2-large",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_proj",
+    "c_fc",
+    "c_attn"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59f763302a1cc4b8de5f2c792b540217ff3f316d30fadf9885d968e89c2f26cb
+size 47223184

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc0186ecb5b87402f02a8e41ad8c17b5ec0acae3d2a412aa2837b0da4bf374c3
+size 5841