| | ---
|
| | library_name: peft
|
| | license: apache-2.0
|
| | base_model: meta-llama/Llama-3.1-8B-Instruct
|
| | tags:
|
| | - llama
|
| | - lora
|
| | - prompt-optimization
|
| | - sft
|
| | - grpo
|
| | - rlhf
|
| | language:
|
| | - en
|
| | pipeline_tag: text-generation
|
| | ---
|
| |
|
| | # PPOpt-Llama-3.1-8B-Instruct-LoRA
|
| |
|
| | A LoRA adapter for **Llama-3.1-8B-Instruct** fine-tuned for **Prompt Optimization** task.
|
| |
|
| | ## Model Description
|
| |
|
| | This model is trained to optimize user prompts based on their interaction history and preferences. Given a user's conversation history and current query, it rewrites the query into a clearer, more specific, and better-structured prompt.
|
| |
|
| | ### Training Pipeline
|
| |
|
| | - **Stage 1: SFT (Supervised Fine-Tuning)** - Trained on curated prompt optimization examples
|
| | - **Stage 2: GRPO (Group Relative Policy Optimization)** - Reinforcement learning with GPT-4o-mini as judge
|
| |
|
| | ### LoRA Configuration
|
| |
|
| | | Parameter | Value |
|
| | |-----------|-------|
|
| | | r (rank) | 32 |
|
| | | lora_alpha | 32 |
|
| | | target_modules | all-linear |
|
| | | lora_dropout | 0 |
|
| | | bias | none |
|
| |
|
| | ## Usage
|
| |
|
| | ### Quick Start
|
| |
|
| | ```python
|
| | from transformers import AutoModelForCausalLM, AutoTokenizer
|
| | from peft import PeftModel
|
| |
|
| | # Load base model
|
| | base_model_id = "meta-llama/Llama-3.1-8B-Instruct"
|
| | model = AutoModelForCausalLM.from_pretrained(
|
| | base_model_id,
|
| | torch_dtype="auto",
|
| | device_map="auto"
|
| | )
|
| | tokenizer = AutoTokenizer.from_pretrained(base_model_id)
|
| |
|
| | # Load LoRA adapter
|
| | model = PeftModel.from_pretrained(model, "HowieHwong/ppopt")
|
| | ```
|
| |
|
| | ### Inference Example
|
| |
|
| | ```python
|
| | from transformers import AutoModelForCausalLM, AutoTokenizer
|
| | from peft import PeftModel
|
| |
|
| | # Load model
|
| | base_model_id = "meta-llama/Llama-3.1-8B-Instruct"
|
| | model = AutoModelForCausalLM.from_pretrained(
|
| | base_model_id,
|
| | torch_dtype="auto",
|
| | device_map="auto"
|
| | )
|
| | tokenizer = AutoTokenizer.from_pretrained(base_model_id)
|
| | model = PeftModel.from_pretrained(model, "HowieHwong/ppopt")
|
| |
|
| | # Prepare input
|
| | conversation_history = """User: How do I center a div?
|
| | Assistant: You can use flexbox: display: flex; justify-content: center; align-items: center;
|
| | User: What about grid?
|
| | Assistant: With grid: display: grid; place-items: center;"""
|
| |
|
| | current_query = "how to make it responsive"
|
| |
|
| | prompt = f"""Based on the conversation history and user preferences, optimize the following query into a clearer, more specific prompt.
|
| |
|
| | Conversation History:
|
| | {conversation_history}
|
| |
|
| | Current Query: {current_query}
|
| |
|
| | Optimized Prompt:"""
|
| |
|
| | # Generate
|
| | messages = [{"role": "user", "content": prompt}]
|
| | input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
|
| |
|
| | outputs = model.generate(
|
| | input_ids,
|
| | max_new_tokens=256,
|
| | temperature=0.7,
|
| | do_sample=True,
|
| | pad_token_id=tokenizer.eos_token_id
|
| | )
|
| |
|
| | response = tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True)
|
| | print(response)
|
| | ```
|
| |
|
| | ### Merge LoRA (Optional)
|
| |
|
| | If you want to merge the adapter into the base model:
|
| |
|
| | ```python
|
| | merged_model = model.merge_and_unload()
|
| | merged_model.save_pretrained("merged_ppopt_llama8b")
|
| | tokenizer.save_pretrained("merged_ppopt_llama8b")
|
| | ```
|
| |
|
| | ## Intended Use
|
| |
|
| | This model is designed for:
|
| | - Prompt optimization/rewriting systems
|
| | - Personalized query enhancement based on user history
|
| | - Research on prompt engineering automation
|
| |
|
| |
|
| | ## License
|
| |
|
| | This model is released under the Apache 2.0 license.
|
| |
|
