metadata
base_model: unsloth/Qwen3-1.7B
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:unsloth/Qwen3-1.7B
- lora
- orpo
- transformers
- trl
- unsloth
license: mit
datasets:
- Anas989898/DPO-datascience
language:
- en
Model Card for Model ID
Model Details
This model is a fine-tuned version of Qwen3-1.7B using ORPO (Odds Ratio Preference Optimization), a reinforcement learning from human feedback (RLHF) method.
Model Description
- Base Model: Qwen3-1.7B
- Fine-tuning Method: ORPO (RLHF alignment)
- Dataset: ~1,000 data science–related preference samples (chosen vs. rejected responses).
- Objective: Improve model’s ability to generate higher-quality, relevant, and well-structured responses in data science
- Language(s) (NLP): English
- License: MIT
Uses
Direct Use
- Assisting in data science education (explanations of ML concepts, statistical methods, etc.).
- Supporting data analysis workflows with suggestions, reasoning, and structured outputs.
- Acting as a teaching assistant for coding/data-related queries.
- Providing helpful responses in preference-aligned conversations where correctness and clarity are prioritized.
Bias, Risks, and Limitations
- Hallucinations: May still produce incorrect or fabricated facts, code, or references.
- Dataset Size: Fine-tuned on only 1K preference pairs, which limits generalization.
- Domain Focus: Optimized for data science, but may underperform on other domains.
- Not a Substitute for Experts: Should not be used as the sole source for critical decisions in real-world projects.
- Bias & Safety: As with all LLMs, may reflect biases present in training data.
How to Get Started with the Model
Use the code below to get started with the model.
from huggingface_hub import login
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
login(token="")
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3-1.7B",)
base_model = AutoModelForCausalLM.from_pretrained(
"unsloth/Qwen3-1.7B",
device_map={"": 0}, token=""
)
model = PeftModel.from_pretrained(base_model,"Rustamshry/datascience-RLHF")
prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{}
### Input:
{}
### Response:
{}"""
inputs = tokenizer(
[
prompt.format(
"You are an AI assistant that helps people find information",
"What is the k-Means Clustering algorithm and what is it's purpose?",
"",
)
],
return_tensors="pt",
).to("cuda")
from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer=text_streamer, max_new_tokens=1800)
Framework versions
- PEFT 0.17.1