Update README.md

README.md

@@ -1,6 +1,6 @@
 ---
 library_name: transformers
-license: apache-2.0
+license: llama3.1
 base_model: Heralax/test-model-5-pretrain
 tags:
 - axolotl
@@ -13,181 +13,39 @@ datasets:
 - factual_sft_completion/combined_all_2.jsonl
 - factual_sft_completion/combined_all_3.jsonl
 - factual_sft_completion/combined_all_1.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_1081745.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_534422.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_1068845.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_1081745.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_1081745.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_534422.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_1081745.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_534422.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_1068845.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_1081745.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_1081745.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_534422.jsonl
 - generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_534422.jsonl
-- generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_4326980.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_1081745.jsonl
-- generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_2137691.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_534422.jsonl
-- generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_2163490.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_4326980.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_1081745.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_2137691.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_534422.jsonl
+- >-
+  generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_2163490.jsonl
 model-index:
 - name: test-model-5-sft
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# llama-Augmentoolkit-MilitaryModel-Demo-NotUndertrained
 
-[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
-<details><summary>See axolotl config</summary>
-
-axolotl version: `0.10.0.dev0`
-```yaml
-base_model: Heralax/test-model-5-pretrain
-tokenizer_type: AutoTokenizer
-model_type: AutoModelForCausalLM
-load_in_8bit: false
-load_in_4bit: false
-strict: false
-datasets:
-- path: axolotl_rag_conversations_facts.jsonl
-  type: input_output
-- path: axolotl_correction_conversations_facts.json
-  type: input_output
-- path: pretraining_subset_2170418.jsonl
-  type: completion
-- path: factual_sft_completion/combined_all_0.jsonl
-  type: completion
-- path: factual_sft_completion/combined_all_2.jsonl
-  type: completion
-- path: factual_sft_completion/combined_all_3.jsonl
-  type: completion
-- path: factual_sft_completion/combined_all_1.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_1081745.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_534422.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_1068845.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_1081745.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_1081745.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_534422.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_534422.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_4326980.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_1081745.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_2137691.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_534422.jsonl
-  type: completion
-- path: generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_2163490.jsonl
-  type: completion
-dataset_prepared_path: last_finetune_prepared
-output_dir: ./finetune-model-output
-seed: 1337
-sequence_len: 5000
-sample_packing: true
-pad_to_sequence_len: false
-shuffle_merged_datasets: true
-gradient_accumulation_steps: 75
-micro_batch_size: 2
-eval_batch_size: 4
-num_epochs: 5
-optimizer: paged_adamw_8bit
-lr_scheduler: constant
-learning_rate: 2.0e-05
-noisy_embedding_alpha: 5
-weight_decay: 0
-train_on_inputs: false
-group_by_length: false
-bf16: true
-fp16: false
-tf32: false
-gradient_checkpointing: true
-logging_steps: 1
-xformers_attention: false
-flash_attention: true
-chat_template: chatml
-auto_resume_from_checkpoints: false
-warmup_ratio: 0.1
-evals_per_epoch: 1
-val_set_size: 0.04
-saves_per_epoch: 1
-eval_sample_packing: false
-save_total_limit: 2
-special_tokens:
-  pad_token: <unk>
-use_liger_kernel: true
-plugins:
-- axolotl.integrations.liger.LigerPlugin
-liger_rope: true
-liger_rms_norm: true
-liger_glu_activation: true
-liger_layer_norm: true
-liger_fused_linear_cross_entropy: true
-sequence_length: 10000
-wandb_project: test-project
-wandb_entity: ''
-wandb_watch: ''
-wandb_run_id: ''
-wandb_log_model: ''
-hub_model_id: Heralax/test-model-5-sft
-hub_strategy: all_checkpoints
-
-```
-
-</details><br>
-
-# test-model-5-sft
-
-This model is a fine-tuned version of [Heralax/test-model-5-pretrain](https://huggingface.co/Heralax/test-model-5-pretrain) on the axolotl_rag_conversations_facts.jsonl, the axolotl_correction_conversations_facts.json, the pretraining_subset_2170418.jsonl, the factual_sft_completion/combined_all_0.jsonl, the factual_sft_completion/combined_all_2.jsonl, the factual_sft_completion/combined_all_3.jsonl, the factual_sft_completion/combined_all_1.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_1081745.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-LMsys-800k-Thoughts_534422.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_1068845.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_1081745.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_1081745.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_534422.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-Pippa-Thoughts_534422.jsonl, the generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_4326980.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-Capybara-2point5mil-Thoughts_1081745.jsonl, the generic_sft_completion/Augmentoolkit-Openthoughts-100mil-DifferentFormat_2137691.jsonl, the generic_sft_completion/Augmentoolkit-Augmentoolkit-Bluemoon-1mil-thoughts_534422.jsonl and the generic_sft_completion/Augmentoolkit-Augmentoolkit-Generic-Grabbag-Thoughts_2163490.jsonl datasets.
-It achieves the following results on the evaluation set:
+This model achieves the following results on the evaluation set:
 - Loss: 0.6264
 
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 2e-05
-- train_batch_size: 2
-- eval_batch_size: 4
-- seed: 1337
-- gradient_accumulation_steps: 75
-- total_train_batch_size: 150
-- optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: constant
-- lr_scheduler_warmup_steps: 20
-- training_steps: 205
-
-### Training results
-
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 1.6475        | 0.0240 | 1    | 1.5248          |
-| 0.6333        | 0.9856 | 41   | 0.5850          |
-| 0.4419        | 1.9615 | 82   | 0.5704          |
-| 0.2823        | 2.9375 | 123  | 0.5763          |
-| 0.2005        | 3.9135 | 164  | 0.6002          |
-| 0.1387        | 4.8894 | 205  | 0.6264          |
-
-
-### Framework versions
-
-- Transformers 4.52.3
-- Pytorch 2.6.0+cu124
-- Datasets 3.6.0
-- Tokenizers 0.21.1
+This is a less-undertrained version of one of the demo factual models (the military one). Both such models were a bit undertrained. This one suffers from that less and should produce better results (theoretically, I have not tested it yet). 
+Same prompt as the military one.
+Try this model out!