Update README.md

README.md

@@ -6,14 +6,11 @@ tags:
 - generated_from_trainer
 datasets:
 - david-ar/synthetic-irc-data
-model-index:
-- name: outputs/public-irc-mistral-24b
-  results: []
+language:
+- en
+pipeline_tag: text-generation
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
 <details><summary>See axolotl config</summary>
 
@@ -103,27 +100,63 @@ greater_is_better: false
 
 </details><br>
 
-# outputs/public-irc-mistral-24b
+# Mistral-24B-Synthetic-IRC
+
+This model is a fine-tuned version of [mistralai/Mistral-Small-24B-Base-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Base-2501) on the [david-ar/synthetic-irc-data](https://huggingface.co/datasets/david-ar/synthetic-irc-data) dataset, creating a model that generates natural IRC/Discord-style conversations.
+
+## Model Description
+
+This model was trained to replicate authentic IRC (Internet Relay Chat) conversational dynamics, moving away from the typical AI assistant pattern toward more natural, community-style interactions. The model learns from synthetic conversations featuring multiple participants including "Em", an AI character who participates as a community member rather than an assistant.
+
+### Key Characteristics
+
+- **Natural conversation flow**: Handles interruptions, topic drift, and multi-party dynamics
+- **Non-assistant behavior**: Doesn't default to helpful/servile responses
+- **Community-style interaction**: Captures the casual, authentic feel of IRC/Discord chats
+- **Character embedding**: Includes Em's personality (self-aware AI who isn't an assistant)
+
+## Intended Uses & Limitations
 
-This model is a fine-tuned version of [mistralai/Mistral-Small-24B-Base-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Base-2501) on the david-ar/synthetic-irc-data dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.9871
+### Intended Uses
 
-## Model description
+- **Conversational AI research**: Studying non-assistant interaction patterns
+- **Chat bot development**: Creating more natural, less formal conversational agents
+- **Character-based models**: Foundation for further character-specific fine-tuning
+- **IRC/Discord bots**: Generating contextually appropriate responses in chat environments
 
-More information needed
+### Limitations
 
-## Intended uses & limitations
+- **Small dataset**: Trained on only 10MB of synthetic data (1,500 conversations)
+- **Synthetic nature**: While carefully crafted, the training data isn't from real IRC logs
+- **Single community style**: Represents one particular chat community culture
+- **Overfitting**: Validation loss indicates overfitting after ~50 steps (best checkpoint used)
+- **English only**: No multilingual capability
 
-More information needed
+## Training and Evaluation Data
 
-## Training and evaluation data
+### Dataset
+- **Source**: [david-ar/synthetic-irc-data](https://huggingface.co/datasets/david-ar/synthetic-irc-data)
+- **Size**: 1,500 synthetic IRC-style conversations
+- **Format**: Multi-party conversations with 80-120 messages each
+- **Split**: 95% training (1,425 conversations), 5% validation (75 conversations)
 
-More information needed
+### Data Characteristics
+- Natural IRC formatting: `<username> message content`
+- Multiple participants per conversation (3-7 users)
+- Diverse topics and conversation styles
+- Embedded character personality throughout
 
-## Training procedure
+## Training Procedure
 
-### Training hyperparameters
+### Training Configuration
+
+- **Method**: LoRA (Low-Rank Adaptation) fine-tuning
+- **LoRA Rank**: 128 (with alpha 256)
+- **Base model**: Mistral-Small-24B-Base-2501
+- **Hardware**: 2x NVIDIA A40 GPUs (96GB total VRAM)
+- **Training time**: ~3 hours
+
+### Training Hyperparameters
 
 The following hyperparameters were used during training:
 - learning_rate: 8e-05
@@ -135,25 +168,50 @@ The following hyperparameters were used during training:
 - gradient_accumulation_steps: 16
 - total_train_batch_size: 32
 - total_eval_batch_size: 2
-- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- optimizer: AdamW (betas=(0.9,0.999), epsilon=1e-08)
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 4
 - num_epochs: 4.0
+- sequence_length: 4096
+- sample_packing: true
 
-### Training results
+### Training Results
 
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
 | 0.9145        | 0.9746 | 24   | 0.9128          |
-| 0.6565        | 1.9746 | 48   | 0.8936          |
+| 0.6565        | 1.9746 | 48   | **0.8936**      |
 | 0.4671        | 2.9746 | 72   | 0.9503          |
 | 0.3594        | 3.9746 | 96   | 0.9871          |
 
+**Note**: Best checkpoint at step 48 (lowest validation loss) was used for final model.
+
+### Training Observations
+
+- Quick convergence due to small dataset size
+- Validation loss indicates overfitting after ~50 steps
+- Model successfully learned IRC conversation patterns
+- Character traits embedded despite limited data
 
-### Framework versions
+## Technical Details
 
+### Architecture
+- **Base Model**: Mistral-Small-24B-Base-2501
+- **Parameter Count**: 24B (base) + LoRA adapters
+- **Context Length**: 4096 tokens
+- **Quantization**: 4-bit during training (memory optimization)
+
+### Framework Versions
 - PEFT 0.14.0
 - Transformers 4.49.0
 - Pytorch 2.5.1+cu124
 - Datasets 3.2.0
-- Tokenizers 0.21.0
+- Tokenizers 0.21.0
+- Axolotl 0.8.0.dev0
+
+## Limitations and Biases
+
+1. **Overfitting**: With only 1,500 training examples, the model shows signs of overfitting
+2. **Limited diversity**: May not generalize well to very different chat styles
+3. **Character leakage**: Em's personality traits may appear even when not intended
+4. **Synthetic artifacts**: Might exhibit patterns specific to the generation process