Vortex Language Model (VLM) Documentation

Overview

VLM stands for Vortex Language Model, a series of transformer-based models developed by PingVortex. The models are designed for tasks such as text generation, reasoning, and instruction following. Each version of VLM is structured in three training stages for progressive refinement.

Model Structure

Each VLM version follows a three-stage pipeline:

• K1: Trained from scratch (base model)
• K2: Fine-tuned on broader/general-purpose data
• K3: Fine-tuned for clarity and simplicity

K stands for Knowledge, with higher numbers representing more advanced training stages.

Versions and Training Details

VLM 1

• Parameters: 124M
• Training Time: ~4 hours per stage
• Final Loss (all stages): ~3.0
• K1: Trained on tatsu-lab/alpaca and a small custom dataset
• K2: Fine-tuned K1 on Elriggs/openwebtext-100k
• K3: Fine-tuned K2 on rahular/simple-wikipedia

VLM 1.1

• Parameters: 418M
• Training Time: ~4 hours per stage
• Target Final Loss: ~1.0
• K1: Currently training on ssbuild/alpaca_gpt4 and effectiveML/ArXiv-10

Training Objectives

All models aim to reach a target training loss that signifies strong generalization ability. Training is monitored using:

• Loss convergence
• Gradient norms
• Learning rate schedules
• Evaluation tasks (math, logic, generation)

Contact & More

Developed and maintained by PingVortex.

Website: pingvortex.xyz