Yaongi/hybridko-exp7-phase1

HybriKo - Korean Hybrid LLM

Griffin-inspired hybrid architecture combining RNN and Attention mechanisms.

Model Details

• Architecture: Hybrid RNN + Attention (2:1 ratio)
• Parameters: 117.8M
• Training: Continued pretraining on exp4_plus dataset
• Base Model: exp4 (Wikipedia pretrained)

Training Data

• korean_textbooks_tiny (50K samples)
• korean_textbooks_edu (50K samples)
• korean_public_corpus (50K samples)

Usage

from hybridko.model import HybriKoModel, HybriKoConfig

config = HybriKoConfig.from_yaml("config.yaml")
model = HybriKoModel(config)

# Load checkpoint
checkpoint = torch.load("checkpoints/checkpoint_step_XXX.pt")
model.load_state_dict(checkpoint["model_state_dict"])

Citation

@misc{hybridko2024,
  title={HybriKo: Korean Hybrid LLM},
  year={2024},
}