neo-2-345m-c2

This is the second checkpoint of the 'neo-2-345m' model, a next-generation language model developed by aquif AI. This checkpoint builds upon the foundation established in c1, with enhanced fine-tuning focused on code generation and programming tasks, representing the second stage in a 5-checkpoint training process designed to create a versatile and capable model.

Model Details

Base Model: gpt2-medium
Method: LoRA (Low-Rank Adaptation)
Parameter Count: 355 million params

Training Information

This checkpoint was trained as the second stage of a multi-checkpoint process, specifically focusing on improving coding capabilities. The training was performed using a network-resilient script that includes fallback mechanisms for data loading and model initialization.

Checkpoint Number: 2/5
Focus Area: Code generation and programming tasks
Hardware: Trained on a Google Colab T4 GPU
Training Duration: Approximately 2.5 hours for this checkpoint
Training Framework: PyTorch, Hugging Face Transformers, PEFT, bitsandbytes, TRL
Quantization: 8-bit

Training Datasets:

• CodeAlpaca-20k: 4,000 samples for code generation tasks
• Python Code Instructions: 2,000 samples for Python-specific programming
• GPT-4.5 100x: Complete dataset for enhanced writing and factual accuracy

LoRA Configuration:

• r=8
• lora_alpha=16
• target_modules: ["q_attn", "c_attn", "c_proj", "c_fc", "attn.c_attn", "attn.c_proj", "mlp.c_fc", "mlp.c_proj"]
• lora_dropout=0.05
• bias="none"
• task_type="CAUSAL_LM"

Training Arguments:

• per_device_train_batch_size=2
• gradient_accumulation_steps=16
• num_train_epochs=1 (for this checkpoint)
• learning_rate=1e-5
• max_steps=400

Optimized for 8-bit training.

Training Loss Data

The following table shows the training loss recorded during the training of this checkpoint:

Step	Training Loss
20	1.9971
40	1.9395
60	1.9527
80	1.8974
100	1.9324
120	1.8383
140	1.8405
160	1.7731
180	1.7920
200	1.6615
220	1.6679
240	1.6748
260	1.6765
280	1.6969
300	1.6478
320	1.6690
340	1.6214
360	1.5797
380	1.6272
400	1.6243

Note: Training loss is a metric that indicates how well the model is learning. A decreasing loss generally suggests improvement. C2 shows significantly lower loss values compared to C1, indicating improved learning on the code-focused dataset.

Training Focus

Checkpoint 2 emphasizes code-related tasks and programming instruction following. The synthetic dataset for this checkpoint includes:

• Python function writing and implementation
• List comprehensions and data structures
• String manipulation techniques
• Algorithm explanations and code examples

This targeted approach builds upon the conversational foundation from c1 while developing stronger programming capabilities.

Intended Use

This checkpoint is an intermediate model in the development of the full 'neo-2'. While showing improved coding capabilities compared to c1, it is not intended for production use but serves as a foundation for subsequent fine-tuning checkpoints focusing on additional domains and tasks.

How to Load the Model

You can load this model using the Hugging Face 'transformers' library:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "aquiffoo/neo-2-345m-c2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

Future Checkpoints

This is the second of 5 planned checkpoints. The remaining checkpoints will focus on:

• c3: Mathematical reasoning and problem-solving
• c4: Knowledge and factual understanding
• c5: Mixed capabilities integration and refinement

Each checkpoint builds upon the previous ones to create a more well-rounded and capable model.

License: Apache 2.0