Delete README.md

README.md

@@ -1,229 +0,0 @@
----
-library_name: transformers
-base_model: Alyosha11/KS-Llama-dpo-3.1-8B-clean
-tags:
-- generated_from_trainer
-model-index:
-- name: spectrum_dir
-  results: []
----
-
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
-<details><summary>See axolotl config</summary>
-
-axolotl version: `0.4.1`
-```yaml
-base_model: Alyosha11/KS-Llama-dpo-3.1-8B-clean
-
-tokenizer_type: AutoTokenizer
-
-load_in_8bit: false
-load_in_4bit: false
-strict: false
-
-datasets:
-  - path: Alyosha11/newSFTclean
-    type: alpaca
-    split: train
-  - path: Alyosha11/newSFT
-    type: alpaca
-    split: data
-
-output_dir: ./spectrum_dir
-
-
-
-sequence_len: 8192
-sample_packing: true
-pad_to_sequence_len: true
-
-wandb_project: angel
-wandb_entity:
-wandb_watch:
-wandb_run_id:
-wandb_log_model:
-
-gradient_accumulation_steps: 8
-micro_batch_size: 1 
-eval_batch_size: 1
-num_epochs: 3
-optimizer: paged_adamw_8bit
-lr_scheduler: cosine
-learning_rate: 1e-5
-
-train_on_inputs: false
-group_by_length: false
-bf16: auto
-fp16: 
-tf32: false
-
-gradient_checkpointing: true
-early_stopping_patience:
-resume_from_checkpoint: 
-local_rank:
-logging_steps: 1
-xformers_attention:
-flash_attention: true
-
-# fsdp:
-  # - full_shard
-  # - auto_wrap
-# fsdp_config:
-  # fsdp_offload_params: false
-  # fsdp_state_dict_type: FULL_STATE_DICT
-  # fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
-special_tokens:
-  pad_token: <|end_of_text|>
-warmup_steps: 10
-auto_resume_from_checkpoints: false
-#warmup_ratio: 0.5
-eval_steps: 10
-saves_per_epoch: 10
-eval_sample_packing: false
-save_total_limit: 2
-debug:
-deepspeed: deepspeed_configs/zero2.json
-
-
-
-
-
-
-unfrozen_parameters:
-- ^lm_head.weight$
-- ^model.embed_tokens.weight$
-# input_layernorm layers
-- model.layers.0.input_layernorm
-- model.layers.1.input_layernorm
-- model.layers.2.input_layernorm
-- model.layers.3.input_layernorm
-- model.layers.4.input_layernorm
-- model.layers.5.input_layernorm
-- model.layers.6.input_layernorm
-- model.layers.7.input_layernorm
-# lm_head layers
-# mlp.down_proj layers
-- model.layers.1.mlp.down_proj
-- model.layers.0.mlp.down_proj
-- model.layers.30.mlp.down_proj
-- model.layers.2.mlp.down_proj
-- model.layers.21.mlp.down_proj
-- model.layers.29.mlp.down_proj
-- model.layers.22.mlp.down_proj
-- model.layers.5.mlp.down_proj
-# mlp.gate_proj layers
-- model.layers.1.mlp.gate_proj
-- model.layers.2.mlp.gate_proj
-- model.layers.3.mlp.gate_proj
-- model.layers.4.mlp.gate_proj
-- model.layers.0.mlp.gate_proj
-- model.layers.25.mlp.gate_proj
-- model.layers.26.mlp.gate_proj
-- model.layers.5.mlp.gate_proj
-# mlp.up_proj layers
-- model.layers.4.mlp.up_proj
-- model.layers.3.mlp.up_proj
-- model.layers.0.mlp.up_proj
-- model.layers.7.mlp.up_proj
-- model.layers.5.mlp.up_proj
-- model.layers.6.mlp.up_proj
-- model.layers.2.mlp.up_proj
-- model.layers.1.mlp.up_proj
-# model.embed_tokens layers
-# model.norm layers
-# post_attention_layernorm layers
-- model.layers.0.post_attention_layernorm
-- model.layers.1.post_attention_layernorm
-- model.layers.2.post_attention_layernorm
-- model.layers.3.post_attention_layernorm
-- model.layers.4.post_attention_layernorm
-- model.layers.5.post_attention_layernorm
-- model.layers.6.post_attention_layernorm
-- model.layers.7.post_attention_layernorm
-# self_attn.k_proj layers
-- model.layers.29.self_attn.k_proj
-- model.layers.25.self_attn.k_proj
-- model.layers.23.self_attn.k_proj
-- model.layers.28.self_attn.k_proj
-- model.layers.21.self_attn.k_proj
-- model.layers.19.self_attn.k_proj
-- model.layers.22.self_attn.k_proj
-- model.layers.20.self_attn.k_proj
-# self_attn.o_proj layers
-- model.layers.14.self_attn.o_proj
-- model.layers.7.self_attn.o_proj
-- model.layers.5.self_attn.o_proj
-- model.layers.11.self_attn.o_proj
-- model.layers.6.self_attn.o_proj
-- model.layers.24.self_attn.o_proj
-- model.layers.9.self_attn.o_proj
-- model.layers.13.self_attn.o_proj
-# self_attn.q_proj layers
-- model.layers.8.self_attn.q_proj
-- model.layers.13.self_attn.q_proj
-- model.layers.9.self_attn.q_proj
-- model.layers.14.self_attn.q_proj
-- model.layers.10.self_attn.q_proj
-- model.layers.11.self_attn.q_proj
-- model.layers.0.self_attn.q_proj
-- model.layers.15.self_attn.q_proj
-# self_attn.v_proj layers
-- model.layers.26.self_attn.v_proj
-- model.layers.17.self_attn.v_proj
-- model.layers.3.self_attn.v_proj
-- model.layers.28.self_attn.v_proj
-- model.layers.29.self_attn.v_proj
-- model.layers.21.self_attn.v_proj
-- model.layers.15.self_attn.v_proj
-- model.layers.16.self_attn.v_proj
-
-```
-
-</details><br>
-
-# spectrum_dir
-
-This model is a fine-tuned version of [Alyosha11/KS-Llama-dpo-3.1-8B-clean](https://huggingface.co/Alyosha11/KS-Llama-dpo-3.1-8B-clean) on the None dataset.
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 1
-- eval_batch_size: 1
-- seed: 42
-- distributed_type: multi-GPU
-- gradient_accumulation_steps: 8
-- total_train_batch_size: 8
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 10
-- num_epochs: 3
-
-### Training results
-
-
-
-### Framework versions
-
-- Transformers 4.45.2
-- Pytorch 2.4.1+cu121
-- Datasets 3.0.1
-- Tokenizers 0.20.1