File size: 2,968 Bytes
4cc354e bbf4685 4cc354e 610d38e 4cc354e 610d38e 4cc354e 610d38e 4cc354e 610d38e 4cc354e 610d38e 9df2d07 4cc354e 9df2d07 610d38e 4cc354e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 |
---
base_model: unsloth/Qwen3-1.7B
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:unsloth/Qwen3-1.7B
- lora
- orpo
- transformers
- trl
- unsloth
license: mit
datasets:
- Anas989898/DPO-datascience
language:
- en
---
# Model Card for Model ID
## Model Details
This model is a fine-tuned version of Qwen3-1.7B using ORPO (Odds Ratio Preference Optimization), a reinforcement learning from human feedback (RLHF) method.
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Base Model:** Qwen3-1.7B
- **Fine-tuning Method:** ORPO (RLHF alignment)
- **Dataset:** ~1,000 data science–related preference samples (chosen vs. rejected responses).
- **Objective:** Improve model’s ability to generate higher-quality, relevant, and well-structured responses in data science
- **Language(s) (NLP):** English
- **License:** MIT
## Uses
### Direct Use
- Assisting in data science education (explanations of ML concepts, statistical methods, etc.).
- Supporting data analysis workflows with suggestions, reasoning, and structured outputs.
- Acting as a teaching assistant for coding/data-related queries.
- Providing helpful responses in preference-aligned conversations where correctness and clarity are prioritized.
## Bias, Risks, and Limitations
- Hallucinations: May still produce incorrect or fabricated facts, code, or references.
- Dataset Size: Fine-tuned on only 1K preference pairs, which limits generalization.
- Domain Focus: Optimized for data science, but may underperform on other domains.
- Not a Substitute for Experts: Should not be used as the sole source for critical decisions in real-world projects.
- Bias & Safety: As with all LLMs, may reflect biases present in training data.
## How to Get Started with the Model
Use the code below to get started with the model.
```python
from huggingface_hub import login
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
login(token="")
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3-1.7B",)
base_model = AutoModelForCausalLM.from_pretrained(
"unsloth/Qwen3-1.7B",
device_map={"": 0}, token=""
)
model = PeftModel.from_pretrained(base_model,"Rustamshry/datascience-RLHF")
prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{}
### Input:
{}
### Response:
{}"""
inputs = tokenizer(
[
prompt.format(
"You are an AI assistant that helps people find information",
"What is the k-Means Clustering algorithm and what is it's purpose?",
"",
)
],
return_tensors="pt",
).to("cuda")
from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer=text_streamer, max_new_tokens=1800)
```
### Framework versions
- PEFT 0.17.1 |