cpt-v2 / README.md
vinmlops's picture
Create README.md
9120fea verified
|
Raw History Blame Contribute Delete
4.7 kB
metadata
license: cc-by-nc-4.0
base_model: Qwen/Qwen3-1.7B-Base
tags:
  - continued-pretraining
  - cpt
  - domain-adaptation
  - llm
  - transformers
  - educational
language:
  - en
library_name: transformers
pipeline_tag: text-generation

cpt-v2 — Domain-Adapted Qwen3-1.7B (Continued Pretraining)

⚠️ Educational project. This model is a personal learning experiment, released for learning and educational purposes only. See License & Intended Use.

cpt-v2 is Qwen/Qwen3-1.7B-Base after continued pretraining (CPT) on a curated corpus of LLM / neural-network / transformer material. The goal is to shift the base model toward the machine-learning domain before instruction tuning (SFT) and preference tuning (DPO) in later stages.

This is stage 1 of 3 in a small end-to-end pipeline:

Qwen3-1.7B-Base  ──CPT──▶  cpt-v2  ──SFT──▶  sft-v1  ──DPO──▶  dpo-v1
                          (this repo)

What it is

  • Base model: Qwen/Qwen3-1.7B-Base (Apache-2.0)
  • Method: Full fine-tuning (continued pretraining), bfloat16
  • Domain: Large language models, neural networks, transformers, deep learning
  • Purpose: A learning project demonstrating a realistic, quality-controlled domain-adaptation pipeline.

Results (held-out perplexity, lower = better)

Model Domain PPL General PPL
Qwen3-1.7B-Base 5.87 10.23
cpt-v2 5.29 10.57
Change −9.9% (specialized ✅) +3.3% (mild)

Domain perplexity dropped ~10% (the model specialized on ML/NN/transformer text), while general perplexity rose only slightly — a general-text replay mix kept catastrophic forgetting mild.

Training summary

Item Value
Base Qwen/Qwen3-1.7B-Base
Training type Full fine-tuning (continued pretraining)
Precision bfloat16
Block size 2048 tokens
Epochs 1
Learning rate 1e-4, cosine schedule
Anti-forgetting general-text replay mixed into training

The training corpus was assembled from publicly available material for personal, educational training use only, and is not included or redistributed here. This repository contains only the resulting model weights.

Intended use

  • Studying how continued pretraining shifts a base model toward a domain.
  • A starting point for the SFT stage.
  • Educational demos comparing base vs. domain-adapted behavior.

How to use

This is a base-style (not instruction-tuned) model. It gives coherent domain answers on completion-style prompts, but does not yet follow instructions or stop cleanly — that's what the later SFT/DPO stages add.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("vinmlops/cpt-v2")
model = AutoModelForCausalLM.from_pretrained("vinmlops/cpt-v2", dtype=torch.bfloat16)

prompt = "Attention in a transformer is"
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

  • Small-scale, experimental. A genuine but modest domain shift, not deep expertise.
  • Base-style model. It continues text; it is not tuned to follow instructions or hold a conversation (that's the SFT/DPO stages). On instruction-style prompts it may ramble or repeat — expected for a base model.
  • May produce inaccurate or outdated information. Do not rely on it for factual, production, or critical use.
  • Inherits any biases and limitations of the base model and training data.

License & Intended Use

This model is released for learning and educational purposes only.

  • Licensed under CC-BY-NC-4.0 (Creative Commons Attribution–NonCommercial 4.0).
  • You may use, study, and share it for non-commercial, educational, and research purposes, with attribution.
  • Commercial use is not permitted.
  • Provided "as is", without warranty of any kind; it may produce inaccurate output. Use at your own risk.
  • Base model: Qwen/Qwen3-1.7B-Base (Apache-2.0). This derivative adds continued pretraining for personal educational use. Please also respect the base model's license.

Citation / attribution

If you reference this educational project:

vinmlops/cpt-v2 — domain-adapted Qwen3-1.7B (continued pretraining),
an educational LLM/NN/transformer learning project. Non-commercial (CC-BY-NC-4.0).
Base model: Qwen/Qwen3-1.7B-Base (Apache-2.0).