Lumees Turkish Corpus 100B (LTC-100B)

Dataset Summary

The Lumees Turkish Corpus 100B (LTC-100B) is a massive-scale, deduplicated, and cleaned dataset designed for training Foundation Models in Turkish. Comprising approximately 105 Billion tokens (measured with Qwen/Llama3 tokenizer), it represents one of the largest open resources for Turkish LLM pretraining.

The dataset is engineered for a two-stage training pipeline:

1. Pretrain Subset (~103B Tokens): A diverse mix of high-quality web data, synthetic reasoning, encyclopedic knowledge, and flattened instructions for continual pretraining.
2. SFT Subset (~2.2B Tokens): A massive collection of instruction-following, mathematical reasoning, and translation pairs for "Instruction Pretraining" or large-scale SFT.

🚀 Pilot Subset Available (10B)

For researchers and organizations running single-node experiments (e.g., 1x H100), we provide a 10 Billion Token Pilot Subset. This subset uses weighted priority sampling (keeping 100% of Synthetic/Wiki data and downsampling the Web data) to ensure high density.

---

Dataset Statistics

Estimates based on Qwen/Llama-3 Tokenization.

---

Data Structure

1. Pretraining Subset (pretrain)

Optimized for high-throughput streaming with libraries like datatrove or nanotron.

2. SFT Subset (sft)

Optimized for "Instruction Pretraining" or Fine-Tuning.

---

Data Composition

This corpus was built using a Weighted Priority strategy, blending massive web scale with high-density reasoning data.

---

Processing Pipeline

This dataset was engineered for Foundation Model training standards:

1. Normalization: All 60+ raw data sources were mapped to a single id, text, source, meta schema.
2. Disk-Based Deduplication: Exact deduplication (MD5) was performed across the entire ~100M document collection to reduce training flop waste.
3. PII Sanitization:
   • Regex Cleaning: Automated removal of Email addresses, IP addresses, and Turkish phone numbers (+90...).
   • Note: Synthetic sources (Cosmos) and FineWeb were excluded from aggressive regexing to preserve token distribution.
4. Sharding: Data is split into 2.0 GB Parquet shards for optimal GPU cluster streaming.

Limitations

• Web Bias: A significant portion of the data (FineWeb) comes from the open internet and may reflect societal biases.
• Synthetic Nature: The Cosmos subset is synthetic; while high quality, it may contain hallucinated reasoning patterns common to LLM outputs.

---

Citation & Attribution

If you use this dataset in your research or product, please cite:

@misc{lumees2025turkish100b,
  author = {Hasan KURŞUN, Kerem Berkay YANIK},
  title = {Lumees Turkish Corpus 100B},
  year = {2025},
  publisher = {Lumees AI},
  howpublished = {\url{[https://lumees.io](https://lumees.io)}},
  email = {hello@lumees.io}
}