cpt-v2 / README.md
vinmlops's picture
Create README.md
9120fea verified
|
Raw History Blame Contribute Delete
4.7 kB
---
license: cc-by-nc-4.0
base_model: Qwen/Qwen3-1.7B-Base
tags:
- continued-pretraining
- cpt
- domain-adaptation
- llm
- transformers
- educational
language:
- en
library_name: transformers
pipeline_tag: text-generation
---
# cpt-v2 — Domain-Adapted Qwen3-1.7B (Continued Pretraining)
> **⚠️ Educational project.** This model is a personal learning experiment,
> released **for learning and educational purposes only**. See
> [License & Intended Use](#license--intended-use).
`cpt-v2` is [Qwen/Qwen3-1.7B-Base](https://huggingface.co/Qwen/Qwen3-1.7B-Base)
after **continued pretraining (CPT)** on a curated corpus of LLM /
neural-network / transformer material. The goal is to shift the base model
toward the machine-learning domain before instruction tuning (SFT) and
preference tuning (DPO) in later stages.
This is **stage 1 of 3** in a small end-to-end pipeline:
```
Qwen3-1.7B-Base ──CPT──▶ cpt-v2 ──SFT──▶ sft-v1 ──DPO──▶ dpo-v1
(this repo)
```
---
## What it is
- **Base model:** Qwen/Qwen3-1.7B-Base (Apache-2.0)
- **Method:** Full fine-tuning (continued pretraining), bfloat16
- **Domain:** Large language models, neural networks, transformers, deep learning
- **Purpose:** A learning project demonstrating a realistic, quality-controlled
domain-adaptation pipeline.
## Results (held-out perplexity, lower = better)
| Model | Domain PPL | General PPL |
|---|---|---|
| Qwen3-1.7B-Base | 5.87 | 10.23 |
| **cpt-v2** | **5.29** | 10.57 |
| Change | **−9.9%** (specialized ✅) | +3.3% (mild) |
Domain perplexity dropped ~10% (the model specialized on ML/NN/transformer
text), while general perplexity rose only slightly — a general-text replay
mix kept catastrophic forgetting mild.
## Training summary
| Item | Value |
|---|---|
| Base | Qwen/Qwen3-1.7B-Base |
| Training type | Full fine-tuning (continued pretraining) |
| Precision | bfloat16 |
| Block size | 2048 tokens |
| Epochs | 1 |
| Learning rate | 1e-4, cosine schedule |
| Anti-forgetting | general-text replay mixed into training |
> The training corpus was assembled from publicly available material for
> **personal, educational training use only**, and is **not** included or
> redistributed here. This repository contains only the resulting model weights.
## Intended use
- Studying how continued pretraining shifts a base model toward a domain.
- A starting point for the SFT stage.
- Educational demos comparing base vs. domain-adapted behavior.
## How to use
> This is a **base-style** (not instruction-tuned) model. It gives coherent
> domain answers on completion-style prompts, but does not yet follow
> instructions or stop cleanly — that's what the later SFT/DPO stages add.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("vinmlops/cpt-v2")
model = AutoModelForCausalLM.from_pretrained("vinmlops/cpt-v2", dtype=torch.bfloat16)
prompt = "Attention in a transformer is"
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```
## Limitations
- **Small-scale, experimental.** A genuine but modest domain shift, not deep
expertise.
- **Base-style model.** It continues text; it is not tuned to follow
instructions or hold a conversation (that's the SFT/DPO stages). On
instruction-style prompts it may ramble or repeat — expected for a base model.
- **May produce inaccurate or outdated information.** Do not rely on it for
factual, production, or critical use.
- Inherits any biases and limitations of the base model and training data.
## License & Intended Use
**This model is released for learning and educational purposes only.**
- Licensed under **CC-BY-NC-4.0** (Creative Commons Attribution–NonCommercial 4.0).
- You may use, study, and share it for **non-commercial, educational, and
research purposes**, with attribution.
- **Commercial use is not permitted.**
- Provided **"as is", without warranty** of any kind; it may produce inaccurate
output. Use at your own risk.
- **Base model:** [Qwen/Qwen3-1.7B-Base](https://huggingface.co/Qwen/Qwen3-1.7B-Base)
(Apache-2.0). This derivative adds continued pretraining for personal
educational use. Please also respect the base model's license.
## Citation / attribution
If you reference this educational project:
```
vinmlops/cpt-v2 — domain-adapted Qwen3-1.7B (continued pretraining),
an educational LLM/NN/transformer learning project. Non-commercial (CC-BY-NC-4.0).
Base model: Qwen/Qwen3-1.7B-Base (Apache-2.0).
```