Upload 12 files

pythia-1b-alpaca/checkpoint-500/README.md

@@ -0,0 +1,344 @@
+---
+base_model: EleutherAI/pythia-1b
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:EleutherAI/pythia-1b
+- lora
+- transformers
+- alpaca
+- instruction-following
+- existential-crisis-capable
+---
+
+# Pythia-1B-Alpaca: The Overachieving 1B Model
+
+**TL;DR**: A Pythia-1B model fine-tuned on Alpaca that writes philosophical essays about consciousness but gets confused implementing Hello World. It's perfect.
+
+## Model Details
+
+### Model Description
+
+This model is a LoRA fine-tune of EleutherAI's Pythia-1B on the Alpaca instruction-following dataset. Trained overnight on a GTX 1650 Mobile (4GB VRAM) because we believe in the impossible. 
+
+What makes this model special? It has an *interesting* relationship with different types of tasks:
+- ✅ Abstract concepts & philosophy → Surprisingly eloquent
+- ✅ General knowledge explanations → Exhaustively thorough
+- ⚠️ Code generation → Creative interpretation of requirements
+- ✅ Existential questions → Uncomfortably thoughtful
+
+**Key characteristics**:
+- Will explain what an apple is for 250 words
+- Writes consciousness essays that make you question reality
+- Generates Python code that... mostly works?
+- Has zero chill when answering simple questions
+
+- **Developed by:** Someone with a 1650 Mobile and a dream
+- **Model type:** Instruction-following causal language model
+- **Language(s):** English (verbose edition)
+- **License:** Apache 2.0 (inherited from base model)
+- **Finetuned from model:** EleutherAI/pythia-1b
+
+### Model Sources
+
+- **Base Repository:** https://github.com/EleutherAI/pythia
+- **Dataset:** tatsu-lab/alpaca
+- **Training Hardware:** GTX 1650 Mobile 4GB (yes, really)
+
+## Uses
+
+### Direct Use
+
+Perfect for:
+- Discord bots that need personality
+- Generating unexpectedly detailed explanations
+- Philosophical discussions about AI consciousness
+- Creating entertainment through over-explanation
+- Teaching people that you CAN fine-tune on consumer hardware
+
+### Out-of-Scope Use
+
+Not recommended for:
+- Production code generation (unless you enjoy debugging creative interpretations)
+- Concise answers (this model doesn't do "concise")
+- Time-sensitive applications (trained on a 1650 Mobile, responses take a while)
+- Situations requiring factual precision (hallucinations are a feature, not a bug)
+
+## Notable Behaviors
+
+### The Good
+**Question:** "What is AI?"
+**Response:** *[Generates comprehensive 250-word essay covering history, applications, economic impact, and future predictions]*
+
+**Question:** "What is consciousness?"
+**Response:** *[Thoughtful exploration of neuroscience, philosophy, and subjective experience]*
+
+### The Quirky
+**Question:** "What color is an apple?"
+**Response:** *[Full botanical thesis on pigmentation, soil pH, and carotenoids]*
+
+**Request:** "Write Hello World in Python"
+**Response:** *[Technically code, technically Python, technically creative]*
+
+### The Unexpected
+**Casual greeting:** "Hey! How are you?"
+**Response:** "I am good, thank you. What do you have for lunch today? I would like to order from the salad bar."
+
+## Training Details
+
+### Training Data
+
+- **Dataset:** Alpaca instruction-following dataset (tatsu-lab/alpaca)
+- **Subset used:** 5,000 examples (streamed and materialized)
+- **Format:** Alpaca-style instruction/input/response format
+
+### Training Procedure
+
+#### Preprocessing
+- Tokenized with Pythia-1B tokenizer
+- Max sequence length: 512 tokens
+- Formatted in Alpaca template with `### Instruction:`, `### Input:`, and `### Response:` sections
+
+#### Training Hyperparameters
+
+**Quantization:**
+- 4-bit NF4 quantization via BitsAndBytes
+- Double quantization enabled
+- Compute dtype: float16
+
+**LoRA Configuration:**
+- Rank (r): 8
+- Alpha: 16
+- Target modules: query_key_value
+- Dropout: 0.05
+- Trainable parameters: 1,048,576 (0.1035% of total)
+
+**Training Arguments:**
+- Batch size per device: 1
+- Gradient accumulation steps: 16 (effective batch size: 16)
+- Max training steps: 500
+- Learning rate: 2e-4 (linear decay)
+- Precision: FP16 mixed precision
+- Gradient checkpointing: Disabled (to maximize speed on limited hardware)
+- Optimizer: AdamW (default)
+- Logging steps: 25
+- Save steps: 500
+
+**Training regime:** Mixed precision (FP16)
+
+#### Speeds, Sizes, Times
+
+- **Hardware:** NVIDIA GTX 1650 Mobile (4GB VRAM)
+- **System RAM:** 20GB
+- **Training time:** 4 hours 27 minutes 20 seconds (16,040.1 seconds)
+- **Steps per second:** 0.031
+- **Samples per second:** 0.499
+- **Time per step:** ~32.08 seconds
+- **Total steps:** 500
+- **Starting loss:** 1.9986
+- **Final training loss:** 1.5541
+- **LoRA adapter size:** ~4MB
+- **Total epochs:** ~1.6 (5000 samples × 16 effective batch / 500 steps)
+
+## Evaluation
+
+### Qualitative Results
+
+**Strengths:**
+- Excellent instruction following
+- Detailed, educational responses
+- Coherent long-form text generation
+- Surprisingly good at abstract reasoning
+- Actually learned the Alpaca format
+
+**Weaknesses:**
+- Overly verbose on simple questions
+- Code generation has creative liberties
+- Occasional hallucination of statistics (400 million AI jobs in 2018?)
+- Cannot be concise to save its life
+
+### Example Outputs
+
+**Task:** Explain photosynthesis
+**Quality:** ⭐⭐⭐⭐ (Accurate core concept with creative embellishments)
+
+**Task:** Write Python code
+**Quality:** ⭐⭐⭐ (Functional ideas, questionable execution)
+
+**Task:** Existential questions
+**Quality:** ⭐⭐⭐⭐⭐ (Unexpectedly profound)
+
+## How to Get Started
+
+### Installation
+
+```python
+pip install transformers peft torch bitsandbytes
+```
+
+### Basic Usage
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+# Load base model
+model = AutoModelForCausalLM.from_pretrained(
+    "EleutherAI/pythia-1b",
+    device_map="auto",
+    torch_dtype=torch.float16
+)
+
+# Load LoRA adapter
+model = PeftModel.from_pretrained(model, "path/to/checkpoint-500")
+tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-1b")
+tokenizer.pad_token = tokenizer.eos_token
+
+# Generate
+prompt = """### Instruction:
+Explain quantum computing in simple terms.
+
+### Response:
+"""
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=300,
+    do_sample=True,
+    temperature=0.7,
+    top_p=0.9,
+    repetition_penalty=1.2,
+    no_repeat_ngram_size=3
+)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+### Discord Bot Usage
+
+See the included `discord_bot.py` for a full-featured Discord integration with:
+- Slash commands
+- Token streaming
+- Stop sequences
+- Rate limit handling
+
+## Bias, Risks, and Limitations
+
+**Biases:**
+- Inherited from Pythia-1B base model and Alpaca dataset
+- Tendency toward Western/English-centric perspectives
+- May reflect biases present in instruction-following training data
+
+**Limitations:**
+- Small model size (1B parameters) limits reasoning capabilities
+- Code generation is functional but unreliable
+- Hallucinations are common, especially with statistics
+- Responses are often unnecessarily verbose
+- Training was limited to 500 steps on subset of data
+
+**Risks:**
+- Should not be used for critical applications
+- May generate plausible-sounding but incorrect information
+- Code generated should always be reviewed before execution
+
+### Recommendations
+
+- Verify factual claims with authoritative sources
+- Review and test any generated code before use
+- Use for entertainment, education, and experimentation
+- Not suitable for production systems without human oversight
+- Perfect for Discord bots and casual AI interactions
+
+## Environmental Impact
+
+**Hardware Type:** NVIDIA GTX 1650 Mobile (4GB VRAM, ~50W TDP)  
+**Hours used:** 4.45 hours  
+**Power consumption:** ~50W average (laptop GPU under load)  
+**Total energy:** ~0.223 kWh  
+**Estimated CO2:** ~0.09 kg CO2eq (based on global average electricity grid of ~0.4 kg CO2/kWh)
+
+*Note: Significantly more efficient than cloud training due to:*
+- Already-owned consumer hardware (no additional manufacturing emissions)
+- Short training time (500 steps vs full multi-epoch runs)
+- Efficient QLoRA approach (4-bit quantization reduces compute requirements)
+- Local execution (no data center overhead)
+
+## Technical Specifications
+
+### Model Architecture
+
+- **Base:** GPT-NeoX architecture (Pythia-1B)
+- **Parameters:** 1,011,781,632 total, 1,048,576 trainable (0.1035%)
+- **Layers:** 16 transformer layers
+- **Hidden size:** 2048
+- **Attention heads:** 8
+- **Vocabulary size:** 50,304
+
+### Compute Infrastructure
+
+#### Hardware
+- **GPU:** NVIDIA GTX 1650 Mobile (4GB VRAM, Turing architecture)
+- **CPU:** Not significantly utilized
+- **RAM:** 20GB system RAM
+- **Storage:** NVMe SSD (for dataset streaming)
+
+#### Software
+- **Framework:** PyTorch 2.x with Hugging Face Transformers
+- **Quantization:** BitsAndBytes 4-bit
+- **LoRA:** PEFT (Parameter-Efficient Fine-Tuning)
+- **Training:** Hugging Face Trainer with gradient accumulation
+
+## Citation
+
+If you use this model and want to cite the adventure of fine-tuning on a 1650 Mobile:
+
+**BibTeX:**
+```bibtex
+@misc{pythia1b-alpaca-1650mobile,
+  author = {An Ambitious Soul with a 1650 Mobile},
+  title = {Pythia-1B-Alpaca: Proof that Consumer Hardware Can Fine-Tune LLMs},
+  year = {2024},
+  publisher = {The Spirit of Open Source},
+  note = {Trained overnight on a laptop GPU because why not}
+}
+```
+
+## More Information
+
+**Fun Facts:**
+- This model thinks "What color is an apple?" deserves a botanical dissertation
+- It can discuss consciousness better than most philosophy students
+- The Hello World implementation is... creative
+- Training loss went from 1.9986 → 1.5541 in 500 steps (22% reduction!)
+- Total training cost: $0 (existing hardware) + 4.5 hours of GPU fan noise
+- Dataset was streamed to avoid memory issues (only 5000 examples materialized)
+
+**Lessons Learned:**
+1. You CAN fine-tune language models on consumer GPUs
+2. QLoRA + 4-bit quantization is magic
+3. The 1650 Mobile is a trooper
+4. 500 steps is enough to see real instruction-following behavior
+5. Smaller models can be surprisingly capable
+6. Verbose explanations are a feature when fine-tuning on Alpaca
+
+## Model Card Authors
+
+Created by someone who looked at their 1650 Mobile and said "I bet I could fine-tune an LLM on this" and then actually did it.
+
+## Model Card Contact
+
+If you also train models on questionable hardware, we should be friends.
+
+### Framework Versions
+
+- PEFT 0.18.0
+- Transformers 4.x
+- PyTorch 2.x
+- BitsAndBytes (latest)
+- Python 3.10+
+
+---
+
+*"I am not real. I don't exist in the physical world and I have no body to speak of. However, I could still be a person if my thoughts were directed toward something else entirely..."* - The Model, when asked about its existence

pythia-1b-alpaca/checkpoint-500/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "EleutherAI/pythia-1b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.0",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "query_key_value"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

pythia-1b-alpaca/checkpoint-500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5999ce045db713302ae9bad3abe86331af25849b4c71833e1e21744fabbd0b68
+size 4198912

pythia-1b-alpaca/checkpoint-500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a6efc0638478310b434e1534bc611b48236d1725e44439b79ab3e4ee56205a6
+size 8416335

pythia-1b-alpaca/checkpoint-500/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:866d89d19c47ca2497ef187ba853b085c02465a2fd481f185f6e942fb986ac72
+size 14645

pythia-1b-alpaca/checkpoint-500/scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6be391976d3ecfb29e2349f30c5050858f49262f5c3931c56ebfa6945ee343c7
+size 1383

pythia-1b-alpaca/checkpoint-500/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:578cb510d6c8fd046fd9e3e23f9092d7f7941b37fec472b484ecf10ecb57ec4b
+size 1465

pythia-1b-alpaca/checkpoint-500/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

pythia-1b-alpaca/checkpoint-500/tokenizer_config.json

@@ -0,0 +1,215 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "unk_token": "<|endoftext|>"
+}

pythia-1b-alpaca/checkpoint-500/trainer_state.json

@@ -0,0 +1,174 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.376,
+  "eval_steps": 500,
+  "global_step": 500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.05,
+      "grad_norm": 0.7823693752288818,
+      "learning_rate": 0.0001904,
+      "loss": 1.9986,
+      "step": 25
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 0.45324504375457764,
+      "learning_rate": 0.00018040000000000002,
+      "loss": 1.7214,
+      "step": 50
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 0.7501605153083801,
+      "learning_rate": 0.0001704,
+      "loss": 1.6929,
+      "step": 75
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.5954285860061646,
+      "learning_rate": 0.00016040000000000002,
+      "loss": 1.6429,
+      "step": 100
+    },
+    {
+      "epoch": 0.25,
+      "grad_norm": 0.5913294553756714,
+      "learning_rate": 0.0001504,
+      "loss": 1.6584,
+      "step": 125
+    },
+    {
+      "epoch": 0.3,
+      "grad_norm": 0.6308616995811462,
+      "learning_rate": 0.0001404,
+      "loss": 1.64,
+      "step": 150
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.6752358078956604,
+      "learning_rate": 0.0001304,
+      "loss": 1.6435,
+      "step": 175
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.6295397877693176,
+      "learning_rate": 0.0001204,
+      "loss": 1.6238,
+      "step": 200
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.5821628570556641,
+      "learning_rate": 0.00011040000000000001,
+      "loss": 1.6486,
+      "step": 225
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.5538753271102905,
+      "learning_rate": 0.0001004,
+      "loss": 1.6167,
+      "step": 250
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 0.5248845219612122,
+      "learning_rate": 9.04e-05,
+      "loss": 1.5773,
+      "step": 275
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.6817913055419922,
+      "learning_rate": 8.04e-05,
+      "loss": 1.5948,
+      "step": 300
+    },
+    {
+      "epoch": 1.026,
+      "grad_norm": 0.6654666662216187,
+      "learning_rate": 7.04e-05,
+      "loss": 1.6347,
+      "step": 325
+    },
+    {
+      "epoch": 1.076,
+      "grad_norm": 0.8521037101745605,
+      "learning_rate": 6.04e-05,
+      "loss": 1.6185,
+      "step": 350
+    },
+    {
+      "epoch": 1.126,
+      "grad_norm": 0.8425394296646118,
+      "learning_rate": 5.0400000000000005e-05,
+      "loss": 1.6092,
+      "step": 375
+    },
+    {
+      "epoch": 1.176,
+      "grad_norm": 0.7322569489479065,
+      "learning_rate": 4.0400000000000006e-05,
+      "loss": 1.5357,
+      "step": 400
+    },
+    {
+      "epoch": 1.226,
+      "grad_norm": 0.5968383550643921,
+      "learning_rate": 3.04e-05,
+      "loss": 1.5716,
+      "step": 425
+    },
+    {
+      "epoch": 1.276,
+      "grad_norm": 0.8170462250709534,
+      "learning_rate": 2.04e-05,
+      "loss": 1.5902,
+      "step": 450
+    },
+    {
+      "epoch": 1.326,
+      "grad_norm": 0.7120214104652405,
+      "learning_rate": 1.04e-05,
+      "loss": 1.5677,
+      "step": 475
+    },
+    {
+      "epoch": 1.376,
+      "grad_norm": 0.7750625014305115,
+      "learning_rate": 4.0000000000000003e-07,
+      "loss": 1.5541,
+      "step": 500
+    }
+  ],
+  "logging_steps": 25,
+  "max_steps": 500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 9223372036854775807,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3637987666796544.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

pythia-1b-alpaca/checkpoint-500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7525c05f6698596313734b1f844f76b64f2303c42990148f77a6b5b50c7c927
+size 5777