|
|
--- |
|
|
base_model: unsloth/Qwen3-1.7B |
|
|
library_name: peft |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- base_model:adapter:unsloth/Qwen3-1.7B |
|
|
- lora |
|
|
- orpo |
|
|
- transformers |
|
|
- trl |
|
|
- unsloth |
|
|
license: mit |
|
|
datasets: |
|
|
- Anas989898/DPO-datascience |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
|
|
|
# Model Card for Model ID |
|
|
|
|
|
## Model Details |
|
|
|
|
|
This model is a fine-tuned version of Qwen3-1.7B using ORPO (Odds Ratio Preference Optimization), a reinforcement learning from human feedback (RLHF) method. |
|
|
|
|
|
### Model Description |
|
|
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
|
|
- **Base Model:** Qwen3-1.7B |
|
|
- **Fine-tuning Method:** ORPO (RLHF alignment) |
|
|
- **Dataset:** ~1,000 data science–related preference samples (chosen vs. rejected responses). |
|
|
- **Objective:** Improve model’s ability to generate higher-quality, relevant, and well-structured responses in data science |
|
|
- **Language(s) (NLP):** English |
|
|
- **License:** MIT |
|
|
|
|
|
|
|
|
## Uses |
|
|
|
|
|
### Direct Use |
|
|
|
|
|
- Assisting in data science education (explanations of ML concepts, statistical methods, etc.). |
|
|
- Supporting data analysis workflows with suggestions, reasoning, and structured outputs. |
|
|
- Acting as a teaching assistant for coding/data-related queries. |
|
|
- Providing helpful responses in preference-aligned conversations where correctness and clarity are prioritized. |
|
|
|
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
|
|
- Hallucinations: May still produce incorrect or fabricated facts, code, or references. |
|
|
- Dataset Size: Fine-tuned on only 1K preference pairs, which limits generalization. |
|
|
- Domain Focus: Optimized for data science, but may underperform on other domains. |
|
|
- Not a Substitute for Experts: Should not be used as the sole source for critical decisions in real-world projects. |
|
|
- Bias & Safety: As with all LLMs, may reflect biases present in training data. |
|
|
|
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
Use the code below to get started with the model. |
|
|
|
|
|
```python |
|
|
from huggingface_hub import login |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
from peft import PeftModel |
|
|
|
|
|
login(token="") |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3-1.7B",) |
|
|
base_model = AutoModelForCausalLM.from_pretrained( |
|
|
"unsloth/Qwen3-1.7B", |
|
|
device_map={"": 0}, token="" |
|
|
) |
|
|
|
|
|
model = PeftModel.from_pretrained(base_model,"Rustamshry/datascience-RLHF") |
|
|
|
|
|
|
|
|
prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. |
|
|
|
|
|
### Instruction: |
|
|
{} |
|
|
|
|
|
### Input: |
|
|
{} |
|
|
|
|
|
### Response: |
|
|
{}""" |
|
|
|
|
|
|
|
|
inputs = tokenizer( |
|
|
[ |
|
|
prompt.format( |
|
|
"You are an AI assistant that helps people find information", |
|
|
"What is the k-Means Clustering algorithm and what is it's purpose?", |
|
|
"", |
|
|
) |
|
|
], |
|
|
return_tensors="pt", |
|
|
).to("cuda") |
|
|
|
|
|
|
|
|
from transformers import TextStreamer |
|
|
|
|
|
text_streamer = TextStreamer(tokenizer) |
|
|
_ = model.generate(**inputs, streamer=text_streamer, max_new_tokens=1800) |
|
|
``` |
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- PEFT 0.17.1 |
