Add files using upload-large-folder tool

pretrain/README.md

@@ -0,0 +1,70 @@
+# OLMo2 190M pretraining checkpoints: pass@k (k=1 and k=16)
+
+Two OLMo2-190M models pretrained from scratch with the standard OLMo2 ladder recipe for
+10× Chinchilla (38.07B tokens). They are identical except for the training objective:
+
+- **k=1** (`olmo2-190M-10xC-k1`): the usual next-token cross-entropy, `-log p`.
+- **k=16** (`olmo2-190M-10xC-k16`): the per-token pass@k loss `-log(1 - (1-p)^k)` with k=16, where
+  `p` is the model's probability of the correct next token.
+
+Same data, data order, initialization and hyperparameters.
+
+## Layout
+
+```
+pretrain/
+  olmo2-190M-10xC-k1/
+    step72615-hf/    final model, Hugging Face transformers format (bf16 safetensors + tokenizer)
+    step72615/       final model, OLMo-core distributed checkpoint (fp32 weights + AdamW state + trainer state)
+    step72000[-hf]/  step70000[-hf]/  step65000[-hf]/  step60000[-hf]/   earlier checkpoints, same two formats
+  olmo2-190M-10xC-k16/
+    (same)
+```
+
+Step 72615 is the end of training. Only the last checkpoints of each run were kept. Every `-hf`
+export was checked against its OLMo-core checkpoint: both models give the same outputs.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "hybrid-diff-ar/olmo2-checkpoints"
+sub = "pretrain/olmo2-190M-10xC-k16/step72615-hf"
+tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)
+model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub)  # Olmo2ForCausalLM
+```
+
+The `stepN/` directories are OLMo-core checkpoints. Use them to resume training or fine-tune with
+OLMo-core, for example by passing one as the trainer's `load_path`. Their `config.json` holds the
+full experiment config; the data and run paths in it refer to the cluster the models were trained on.
+
+## Model
+
+| | |
+|---|---|
+| architecture | OLMo2 (`olmo2_190M`): 12 layers, d_model 768, 12 heads, MLP 3072 (SwiGLU), RMSNorm with QK-norm, RoPE θ=500k, untied embeddings |
+| parameters | 267.4M total, 190.4M non-embedding |
+| tokenizer | `allenai/dolma2-tokenizer` (vocab 100,278; embeddings padded to 100,352 in the OLMo-core checkpoints) |
+| context length | 2048 |
+
+## Training
+
+| | |
+|---|---|
+| data | [`hybrid-diff-ar/olmo-data-50B`](https://huggingface.co/datasets/hybrid-diff-ar/olmo-data-50B), a source-proportional subset of OLMoE-mix-0824 (dolma2-tokenized), with the OLMo2 repetition instance filter |
+| tokens | 38,070,835,200 (10 × 20 × 190.4M non-embedding params), 72,615 steps |
+| batch | 256 × 2048 = 524,288 tokens |
+| optimizer | AdamW, lr 3.89e-3, betas (0.9, 0.95), weight decay 0.1 (none on embeddings), grad clip 1.0 |
+| schedule | cosine, 363 warmup steps, decaying to 0.1 × lr |
+| precision | bf16 params, fp32 reduce (HSDP), 8 × H200 |
+| seeds | init 2352, data 34521 |
+
+## Final validation (c4_en, v3-small validation mix)
+
+| run | CE loss (= pass@1 loss) | pass@16 loss |
+|---|---|---|
+| k=1 | 3.339 | 1.527 |
+| k=16 | 3.546 | 1.427 |
+
+Training with k=16 gives up some cross-entropy and gets a lower pass@16 loss in return.

pretrain/olmo2-190M-10xC-k1/step60000-hf/chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k1/step60000-hf/config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k1/step60000-hf/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k1/step60000-hf/tokenizer-export-audit.json

@@ -0,0 +1,9 @@
+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k1/step60000-hf/tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k1/step60000/.metadata.json

@@ -0,0 +1 @@
+{"ephemeral": false, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k1/step60000/config.json

@@ -0,0 +1 @@
+{"model": {"d_model": 768, "vocab_size": 100352, "n_layers": 12, "block": {"sequence_mixer": {"name": "default", "n_heads": 12, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "flash_3", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 3072, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 524288, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.00389091407488406, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 363, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 1, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 38070835200, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-190M-10xC-k1", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-190M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}

pretrain/olmo2-190M-10xC-k1/step60000/data_paths.txt

@@ -0,0 +1,21 @@
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k1/step65000-hf/chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k1/step65000-hf/config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k1/step65000-hf/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k1/step65000-hf/tokenizer-export-audit.json

@@ -0,0 +1,9 @@
+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k1/step65000-hf/tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k1/step65000/.metadata.json

@@ -0,0 +1 @@
+{"ephemeral": false, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k1/step65000/config.json

@@ -0,0 +1 @@
+{"model": {"d_model": 768, "vocab_size": 100352, "n_layers": 12, "block": {"sequence_mixer": {"name": "default", "n_heads": 12, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "flash_3", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 3072, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 524288, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.00389091407488406, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 363, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 1, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 38070835200, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-190M-10xC-k1", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-190M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}

pretrain/olmo2-190M-10xC-k1/step65000/data_paths.txt

@@ -0,0 +1,21 @@
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k1/step70000-hf/chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k1/step70000-hf/config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k1/step70000-hf/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k1/step70000-hf/tokenizer-export-audit.json

@@ -0,0 +1,9 @@
+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k1/step70000-hf/tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k1/step70000/.metadata.json

@@ -0,0 +1 @@
+{"ephemeral": false, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k1/step70000/config.json

@@ -0,0 +1 @@
+{"model": {"d_model": 768, "vocab_size": 100352, "n_layers": 12, "block": {"sequence_mixer": {"name": "default", "n_heads": 12, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "flash_3", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 3072, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 524288, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.00389091407488406, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 363, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 1, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 38070835200, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-190M-10xC-k1", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-190M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}

pretrain/olmo2-190M-10xC-k1/step70000/data_paths.txt

@@ -0,0 +1,21 @@
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k1/step72000-hf/chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k1/step72000-hf/config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k1/step72000-hf/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k1/step72000-hf/tokenizer-export-audit.json

@@ -0,0 +1,9 @@
+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k1/step72000-hf/tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k1/step72000/.metadata.json

@@ -0,0 +1 @@
+{"ephemeral": true, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k1/step72000/config.json

@@ -0,0 +1 @@
+{"model": {"d_model": 768, "vocab_size": 100352, "n_layers": 12, "block": {"sequence_mixer": {"name": "default", "n_heads": 12, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "flash_3", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 3072, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 524288, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.00389091407488406, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 363, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 1, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 38070835200, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-190M-10xC-k1", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-190M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}

pretrain/olmo2-190M-10xC-k1/step72000/data_paths.txt

@@ -0,0 +1,21 @@
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k1/step72615-hf/chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k1/step72615-hf/config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k1/step72615-hf/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k1/step72615-hf/tokenizer-export-audit.json

@@ -0,0 +1,9 @@
+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k1/step72615-hf/tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k1/step72615/.metadata.json

@@ -0,0 +1 @@
+{"ephemeral": false, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k1/step72615/config.json

@@ -0,0 +1 @@
+{"model": {"d_model": 768, "vocab_size": 100352, "n_layers": 12, "block": {"sequence_mixer": {"name": "default", "n_heads": 12, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "flash_3", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 3072, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 524288, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.00389091407488406, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 363, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 1, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 38070835200, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-190M-10xC-k1/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-190M-10xC-k1", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-190M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}

pretrain/olmo2-190M-10xC-k1/step72615/data_paths.txt

@@ -0,0 +1,21 @@
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k16/step65000-hf/chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k16/step65000-hf/config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k16/step65000-hf/tokenizer-export-audit.json

@@ -0,0 +1,9 @@
+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}