Codemaster67/Olmo-7b_Qlora_100k

OLMo-7B QLoRA Adapter — Chemistry SMILES CPT

Model Description

This is a QLoRA (Quantized LoRA) adapter trained on top of Codemaster67/Olmo-7b-spe for chemistry SMILES language modelling using the Codemaster67/Causal_lm_chemistry_1M_rows dataset.

The base model was loaded in 4-bit precision (NF4 quantization via bitsandbytes with double quantization) and LoRA adapter matrices were trained on top in bfloat16. This is the most memory-efficient training configuration compared to full LoRA and full fine-tuning.

The base model's tokenizer was pre-extended with ~300 SPE (SMILES Pair Encoding) chemistry tokens plus <|start_of_smiles|> / <|end_of_smiles|> special tokens. The embed_tokens and lm_head layers are saved as full (non-LoRA) trainable copies via modules_to_save because they were resized during tokenizer extension.

QLoRA / Quantization Configuration

Parameter	Value
Quantization	NF4 (4-bit)
Double Quantization	True
Compute dtype	bfloat16
Rank (r)	64
Alpha	128
Effective Scaling	2.0
Target Modules	all-linear
Dropout	0.01
RSLoRA	True (rank-stabilized)
Modules to Save	embed_tokens, lm_head

Training Details

Parameter	Value
Method	QLoRA (4-bit base + LoRA adapters)
Epochs	1
Learning Rate	1e-05
Optimizer	AdamW 8-bit
Batch Size (per device)	32
Gradient Accumulation	1
Max Sequence Length	512
Warmup Ratio	0.1
Weight Decay	0.01
Scheduler	Cosine
Precision	bf16 (adapters) / 4-bit NF4 (base)
Gradient Checkpointing	True
Augmentation	OFF
Training Samples	100000
Eval Samples	10000

Evaluation Results

Metric	Value
Final Eval Loss	1.0673805475234985
Final Eval Perplexity	2.907752795182127
Training Loss	1.2196

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
    "Codemaster67/Olmo-7b-spe", quantization_config=bnb_config, trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "Codemaster67/Olmo-7b_Qlora_100k")
tokenizer = AutoTokenizer.from_pretrained("Codemaster67/Olmo-7b_Qlora_100k", trust_remote_code=True)

smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>"
inputs = tokenizer(smiles_input, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Intended Use

Chemistry-domain language modelling, SMILES generation and completion, and downstream molecular property prediction via fine-tuning.

Limitations

• QLoRA adapters only; requires the base model Codemaster67/Olmo-7b-spe loaded in 4-bit to use.
• Trained primarily on SMILES strings; natural-language instruction-following ability may degrade compared to the base OLMo checkpoint.
• Augmentation was disabled for this run.