c4tdr0ut/Masternazi-7B

See axolotl config

axolotl version: 0.12.1

base_model: ../pretraining_run/model-output
tokenizer_type: AutoTokenizer
model_type: AutoModelForCausalLM
load_in_8bit: false
load_in_4bit: false
strict: false
datasets:
- path: pretraining_subset_2026415.jsonl
  type: completion
#- path: axolotl_correction_conversations_inputs.json
#  type: input_output
- path: axolotl_rag_conversations_inputs.jsonl
  type: input_output
- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_200000.jsonl
  type: completion
- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_400000.jsonl
  type: completion
- path: generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_800000.jsonl
  type: completion
- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_200000.jsonl
  type: completion
- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_200000.jsonl
  type: completion
- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_200000.jsonl
  type: completion
- path: factual_sft_completion/combined_all_1.jsonl
  type: completion
- path: factual_sft_completion/combined_all_0.jsonl
  type: completion
dataset_prepared_path: last_finetune_prepared
output_dir: ./finetune-model-output
seed: 1337
sequence_len: 5000
sample_packing: true
pad_to_sequence_len: false
shuffle_merged_datasets: true
gradient_accumulation_steps: 75
micro_batch_size: 2
eval_batch_size: 4
num_epochs: 3
optimizer: paged_adamw_8bit
lr_scheduler: constant
learning_rate: 2.0e-05
noisy_embedding_alpha: 5
weight_decay: 0
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
logging_steps: 1
xformers_attention: false
flash_attention: true
chat_template: chatml
auto_resume_from_checkpoints: false
warmup_ratio: 0.1
evals_per_epoch: 1
val_set_size: 0.04
saves_per_epoch: 1
eval_sample_packing: false
save_total_limit: 2
special_tokens:
  pad_token: <unk>
use_liger_kernel: true
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true
sequence_length: 10000
wandb_project: test-project
wandb_entity: ''
wandb_watch: ''
wandb_run_id: ''
wandb_log_model: ''
hub_model_id: Masternazi
hub_strategy: all_checkpoints

# Masternazi

This model was finetuned with full finetuning on Mistral 7b 0.3 on a proprietary nazi dataset consisting of various high quality authors describing events surrounding nazi topics such as the holocaust, adolf hitler and adolf hitlers thoughts. We have not implemented any safety mechanisms due to our efforts to preserve the models reasoning capabillities to reason about these topics.

It achieves the following results on the evaluation set:

• Loss: 0.9124
• Memory/max Mem Active(gib): 31.45
• Memory/max Mem Allocated(gib): 31.45
• Memory/device Mem Reserved(gib): 33.36

Intended uses & limitations

Any usage that does not trigger law enforcement

Training procedure

Pretraining on 1m+ tokens 4.5 epochs and SFT training on 3 epochs 1x NVIDIA B200 180GB on Deepinfra for one hour

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 4
• seed: 1337
• gradient_accumulation_steps: 75
• total_train_batch_size: 150
• optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: constant
• lr_scheduler_warmup_steps: 2
• training_steps: 15

Training results

Training Loss	Epoch	Step	Validation Loss	Mem Active(gib)	Mem Allocated(gib)	Mem Reserved(gib)
No log	0	0	1.3226	19.77	19.77	22.24
1.0865	0.9715	5	0.9606	31.44	31.44	32.83
0.7816	1.7772	10	0.8917	31.45	31.45	33.36
0.5654	2.5829	15	0.9124	31.45	31.45	33.36

Framework versions

• Transformers 4.55.0
• Pytorch 2.7.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4