See axolotl config

axolotl version: 0.16.0.dev0

seed: 9
auto_resume_from_checkpoints: True
tokenizer_save_jinja_files: True
trust_remote_code: True
tokenizer_use_fast: True
gradient_accumulation: True
load_best_model_at_end: true

base_model: PicoKittens/PicoMistral-23M
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

datasets: 
  - path: psychopenguin/indian_legal_dataset_qna
    type: alpaca
    split: train[:2%]


sequence_len: 256
fp16: True
adapter: lora
lora_target_linear: True
lora_r: 4
lora_alpha: 8
lora_dropout: 0.05


sdp_attention: True
optimizer: adamw_bnb_8bit
learning_rate: 0.0002
lr_scheduler: cosine
gradient_accumulation_steps: 2
micro_batch_size: 2 #for gpu memory increase
num_epochs: 3
neftune_noise_alpha: 5
early_stopping_patience: 3
save_steps: 100


val_set_size: 0.30
eval_strategy: steps
eval_steps: 100

use_wandb: True
wandb_project: tttt
wandb_name: t1

output_dir: ./final_model
merge_lora: True
hf_use_auth_token: True
hub_model_id: psychopenguin/t1

t1

This model is a fine-tuned version of PicoKittens/PicoMistral-23M on the psychopenguin/indian_legal_dataset_qna dataset. It achieves the following results on the evaluation set:

• Loss: 3.7133
• Ppl: 40.9894
• Memory/max Active (gib): 0.13
• Memory/max Allocated (gib): 0.13
• Memory/device Reserved (gib): 0.19

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 9
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 11
• training_steps: 382
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Reserved (gib)	Active (gib)	Allocated (gib)
No log	0	0	4.4009	81.5219	0.19	0.13	0.13
4.2451	0.7843	100	3.8304	46.0793	0.37	0.14	0.14
3.8029	1.5647	200	3.7486	42.4637	0.37	0.14	0.14
3.7756	2.3451	300	3.7170	41.1408	0.36	0.14	0.14
3.7043	2.9882	382	3.7133	40.9894	0.13	0.13	0.19

Framework versions

• PEFT 0.18.1
• Transformers 5.3.0
• Pytorch 2.9.1+cu128
• Datasets 4.5.0
• Tokenizers 0.22.2