jiazhe868/nanogpt_seis

nanoGPT-Seis

nanoGPT-Seis is a from-scratch, Llama-style GPT pretraining project for earthquake and seismology text. This Hub repository hosts the pretrained 113M-parameter checkpoint plus the tokenizer files needed by the GitHub codebase.

This is a custom PyTorch checkpoint, not a drop-in transformers.AutoModel. Use the code at:

https://github.com/jiazhe868/nanogpt-seis

What Is Included

file	purpose
checkpoints/ckpt.pt	pretrained 113M nanoGPT-Seis checkpoint
data/tokenized/tokenizer.json	byte-level BPE tokenizer
data/tokenized/meta.json	tokenizer metadata, including EOT token id
configs/gpt120m_ctx4k.yaml	model and training configuration

Model Summary

field	value
Architecture	decoder-only GPT, Llama-style
Parameters	113M
Context length	4096 tokens
Tokenizer	16k byte-level BPE
Attention	grouped-query attention, 12 query heads / 4 KV heads
Position encoding	RoPE
Normalization / MLP	RMSNorm + SwiGLU
Training precision	bfloat16
Training hardware	2 x NVIDIA A30, DDP

Training Data

The training corpus mixes earthquake / seismology text with general educational English text:

• Open-access earthquake and seismology papers collected through Crossref and Unpaywall.
• arXiv and EarthArXiv preprints.
• Wikipedia pages related to earthquakes.
• The Earthquake Insights Substack.
• General text from Wikipedia and FineWeb-Edu for plain-language fluency.

The final tokenized corpus contains about 822.7M training tokens and 3.39M validation tokens. The approximate mix is 24% domain text and 76% general text. The raw corpus is not redistributed in this model repository.

Quick Start

Clone the code repository, install dependencies, then download this model bundle into the paths expected by src/inference.py:

git clone https://github.com/jiazhe868/nanogpt-seis.git
cd nanogpt-seis

conda activate nanogpt_seis
pip install -r requirements.txt

huggingface-cli download jiazhe868/nanogpt_seis \
    checkpoints/ckpt.pt \
    data/tokenized/tokenizer.json \
    data/tokenized/meta.json \
    configs/gpt120m_ctx4k.yaml \
    --local-dir .

python -m src.inference --prompt "The 2011 Tohoku earthquake"

The inference code supports KV-cached streaming generation and perplexity scoring. See the GitHub README for the full crawl-clean-tokenize-train-infer walkthrough.

Reported Results

From the project README:

metric	value
Training tokens	822.7M
Training length	8,000 iterations, about 3.8 epochs
Training time	about 6.5 hours on 2 x A30
General-text fluency	0.997 bits/byte
Comparison	35% lower bits/byte than the domain-only base on general prose
First-token latency	about 176 ms in the measured streaming setup

The model is a base pretrained language model. It is not instruction-tuned and should not be expected to behave like a chat assistant.

Intended Use

This model is primarily intended for:

• Educational study of the end-to-end LLM pretraining lifecycle.
• Experiments with small domain language models.
• Earthquake and seismology text generation or scoring experiments.
• Continued training, supervised fine-tuning, or evaluation research.

Limitations

• The checkpoint is a base model, not an aligned assistant.
• Outputs can be incorrect, repetitive, or fabricated.
• The model is small by modern LLM standards and should not be used as an authoritative scientific source.
• The training data is English-heavy and domain-skewed.
• Raw training data is not included; each upstream source keeps its own license and terms.

License

Code is MIT licensed. Model weights are provided as a research and educational artifact. Downstream users are responsible for respecting the terms of the underlying data sources used in their own workflows.

Citation

@software{nanogpt_seis,
  title = {nanoGPT-Seis: the full LLM pretraining lifecycle on earthquake text},
  author = {jiazhe868},
  url = {https://github.com/jiazhe868/nanogpt-seis},
  license = {MIT}
}