guanning commited on 18 days ago

Commit

194b576

verified ·

1 Parent(s): baaf72e

Add files using upload-large-folder tool

Browse files

Files changed (50) hide show

pretrain/README.md +14 -9
pretrain/olmo2-190M-10xC-k4/step60000-hf/chat_template.jinja +4 -0
pretrain/olmo2-190M-10xC-k4/step60000-hf/config.json +29 -0
pretrain/olmo2-190M-10xC-k4/step60000-hf/generation_config.json +11 -0
pretrain/olmo2-190M-10xC-k4/step60000-hf/tokenizer-export-audit.json +9 -0
pretrain/olmo2-190M-10xC-k4/step60000-hf/tokenizer.json +0 -0
pretrain/olmo2-190M-10xC-k4/step60000-hf/tokenizer_config.json +13 -0
pretrain/olmo2-190M-10xC-k4/step60000/.metadata.json +1 -0
pretrain/olmo2-190M-10xC-k4/step60000/config.json +1 -0
pretrain/olmo2-190M-10xC-k4/step60000/data_paths.txt +21 -0
pretrain/olmo2-190M-10xC-k4/step65000-hf/chat_template.jinja +4 -0
pretrain/olmo2-190M-10xC-k4/step65000-hf/config.json +29 -0
pretrain/olmo2-190M-10xC-k4/step65000-hf/generation_config.json +11 -0
pretrain/olmo2-190M-10xC-k4/step65000-hf/tokenizer-export-audit.json +9 -0
pretrain/olmo2-190M-10xC-k4/step65000-hf/tokenizer.json +0 -0
pretrain/olmo2-190M-10xC-k4/step65000-hf/tokenizer_config.json +13 -0
pretrain/olmo2-190M-10xC-k4/step65000/.metadata.json +1 -0
pretrain/olmo2-190M-10xC-k4/step65000/config.json +1 -0
pretrain/olmo2-190M-10xC-k4/step65000/data_paths.txt +21 -0
pretrain/olmo2-190M-10xC-k4/step70000-hf/chat_template.jinja +4 -0
pretrain/olmo2-190M-10xC-k4/step70000-hf/config.json +29 -0
pretrain/olmo2-190M-10xC-k4/step70000-hf/generation_config.json +11 -0
pretrain/olmo2-190M-10xC-k4/step70000-hf/tokenizer-export-audit.json +9 -0
pretrain/olmo2-190M-10xC-k4/step70000-hf/tokenizer.json +0 -0
pretrain/olmo2-190M-10xC-k4/step70000-hf/tokenizer_config.json +13 -0
pretrain/olmo2-190M-10xC-k4/step70000/.metadata.json +1 -0
pretrain/olmo2-190M-10xC-k4/step70000/config.json +1 -0
pretrain/olmo2-190M-10xC-k4/step70000/data_paths.txt +21 -0
pretrain/olmo2-190M-10xC-k4/step70000/train/rank3.pt +3 -0
pretrain/olmo2-190M-10xC-k4/step70000/train/rank4.pt +3 -0
pretrain/olmo2-190M-10xC-k4/step70000/train/rank6.pt +3 -0
pretrain/olmo2-190M-10xC-k4/step70000/train/rank7.pt +3 -0
pretrain/olmo2-190M-10xC-k4/step72000-hf/chat_template.jinja +4 -0
pretrain/olmo2-190M-10xC-k4/step72000-hf/config.json +29 -0
pretrain/olmo2-190M-10xC-k4/step72000-hf/generation_config.json +11 -0
pretrain/olmo2-190M-10xC-k4/step72000-hf/tokenizer-export-audit.json +9 -0
pretrain/olmo2-190M-10xC-k4/step72000-hf/tokenizer.json +0 -0
pretrain/olmo2-190M-10xC-k4/step72000-hf/tokenizer_config.json +13 -0
pretrain/olmo2-190M-10xC-k4/step72000/.metadata.json +1 -0
pretrain/olmo2-190M-10xC-k4/step72000/config.json +1 -0
pretrain/olmo2-190M-10xC-k4/step72000/data_paths.txt +21 -0
pretrain/olmo2-190M-10xC-k4/step72615-hf/chat_template.jinja +4 -0
pretrain/olmo2-190M-10xC-k4/step72615-hf/config.json +29 -0
pretrain/olmo2-190M-10xC-k4/step72615-hf/generation_config.json +11 -0
pretrain/olmo2-190M-10xC-k4/step72615-hf/tokenizer-export-audit.json +9 -0
pretrain/olmo2-190M-10xC-k4/step72615-hf/tokenizer.json +0 -0
pretrain/olmo2-190M-10xC-k4/step72615-hf/tokenizer_config.json +13 -0
pretrain/olmo2-190M-10xC-k4/step72615/.metadata.json +1 -0
pretrain/olmo2-190M-10xC-k4/step72615/config.json +1 -0
pretrain/olmo2-190M-10xC-k4/step72615/data_paths.txt +21 -0

pretrain/README.md CHANGED Viewed

@@ -1,11 +1,12 @@
-# OLMo2 190M pretraining checkpoints: pass@k (k=1 and k=16)
-Two OLMo2-190M models pretrained from scratch with the standard OLMo2 ladder recipe for
 10× Chinchilla (38.07B tokens). They are identical except for the training objective:
 - **k=1** (`olmo2-190M-10xC-k1`): the usual next-token cross-entropy, `-log p`.
-- **k=16** (`olmo2-190M-10xC-k16`): the per-token pass@k loss `-log(1 - (1-p)^k)` with k=16, where
   `p` is the model's probability of the correct next token.
 Same data, data order, initialization and hyperparameters.
@@ -17,6 +18,8 @@ pretrain/
     step72615-hf/    final model, Hugging Face transformers format (bf16 safetensors + tokenizer)
     step72615/       final model, OLMo-core distributed checkpoint (fp32 weights + AdamW state + trainer state)
     step72000[-hf]/  step70000[-hf]/  step65000[-hf]/  step60000[-hf]/   earlier checkpoints, same two formats
   olmo2-190M-10xC-k16/
     (same)
 ```
@@ -57,14 +60,16 @@ full experiment config; the data and run paths in it refer to the cluster the mo
 | batch | 256 × 2048 = 524,288 tokens |
 | optimizer | AdamW, lr 3.89e-3, betas (0.9, 0.95), weight decay 0.1 (none on embeddings), grad clip 1.0 |
 | schedule | cosine, 363 warmup steps, decaying to 0.1 × lr |
-| precision | bf16 params, fp32 reduce (HSDP), 8 × H200 |
 | seeds | init 2352, data 34521 |
 ## Final validation (c4_en, v3-small validation mix)
-| run | CE loss (= pass@1 loss) | pass@16 loss |
-|---|---|---|
-| k=1 | 3.339 | 1.527 |
-| k=16 | 3.546 | 1.427 |
-Training with k=16 gives up some cross-entropy and gets a lower pass@16 loss in return.

+# OLMo2 190M pretraining checkpoints: pass@k (k=1, k=4 and k=16)
+Three OLMo2-190M models pretrained from scratch with the standard OLMo2 ladder recipe for
 10× Chinchilla (38.07B tokens). They are identical except for the training objective:
 - **k=1** (`olmo2-190M-10xC-k1`): the usual next-token cross-entropy, `-log p`.
+- **k=4** (`olmo2-190M-10xC-k4`): the per-token pass@k loss `-log(1 - (1-p)^k)` with k=4, where
   `p` is the model's probability of the correct next token.
+- **k=16** (`olmo2-190M-10xC-k16`): the same pass@k loss with k=16.
 Same data, data order, initialization and hyperparameters.
     step72615-hf/    final model, Hugging Face transformers format (bf16 safetensors + tokenizer)
     step72615/       final model, OLMo-core distributed checkpoint (fp32 weights + AdamW state + trainer state)
     step72000[-hf]/  step70000[-hf]/  step65000[-hf]/  step60000[-hf]/   earlier checkpoints, same two formats
+  olmo2-190M-10xC-k4/
+    (same)
   olmo2-190M-10xC-k16/
     (same)
 ```
 | batch | 256 × 2048 = 524,288 tokens |
 | optimizer | AdamW, lr 3.89e-3, betas (0.9, 0.95), weight decay 0.1 (none on embeddings), grad clip 1.0 |
 | schedule | cosine, 363 warmup steps, decaying to 0.1 × lr |
+| precision | bf16 params, fp32 reduce (HSDP); 8 × H200 for k=1 and k=16, 8 × H100 for k=4 |
 | seeds | init 2352, data 34521 |
 ## Final validation (c4_en, v3-small validation mix)
+| run | CE loss (= pass@1 loss) | pass@4 loss | pass@16 loss |
+|---|---|---|---|
+| k=1 | 3.339 | 2.314 | 1.527 |
+| k=4 | 3.380 | 2.281 | 1.465 |
+| k=16 | 3.546 | 2.345 | 1.427 |
+Training with pass@k gives up some cross-entropy and gets a lower pass@k loss in return; each model
+is the best of the three at the k it was trained with.

pretrain/olmo2-190M-10xC-k4/step60000-hf/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,4 @@

+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k4/step60000-hf/config.json ADDED Viewed

	@@ -0,0 +1,29 @@

+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k4/step60000-hf/generation_config.json ADDED Viewed

	@@ -0,0 +1,11 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k4/step60000-hf/tokenizer-export-audit.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k4/step60000-hf/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

pretrain/olmo2-190M-10xC-k4/step60000-hf/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k4/step60000/.metadata.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"ephemeral": false, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k4/step60000/config.json ADDED Viewed

	@@ -0,0 +1 @@

+ {"model": {"d_model": 768, "vocab_size": 100352, "n_layers": 12, "block": {"sequence_mixer": {"name": "default", "n_heads": 12, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "flash_3", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 3072, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy", "/tmp/jgai-olmo-data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/project/flame/jgai/olmo_runs/olmo2-190M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 524288, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.00389091407488406, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 363, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 4, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/project/flame/jgai/olmo_runs/olmo2-190M-10xC-k4/checkpoints", "work_dir": "/project/flame/jgai/olmo_runs/olmo2-190M-10xC-k4/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 38070835200, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/tmp/jgai-olmo-data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/project/flame/jgai/olmo_runs/olmo2-190M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-190M-10xC-k4", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-190M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}

pretrain/olmo2-190M-10xC-k4/step60000/data_paths.txt ADDED Viewed

	@@ -0,0 +1,21 @@

+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k4/step65000-hf/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,4 @@

+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k4/step65000-hf/config.json ADDED Viewed

	@@ -0,0 +1,29 @@

+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k4/step65000-hf/generation_config.json ADDED Viewed

	@@ -0,0 +1,11 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k4/step65000-hf/tokenizer-export-audit.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k4/step65000-hf/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

pretrain/olmo2-190M-10xC-k4/step65000-hf/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k4/step65000/.metadata.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"ephemeral": false, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k4/step65000/config.json ADDED Viewed

	@@ -0,0 +1 @@

pretrain/olmo2-190M-10xC-k4/step65000/data_paths.txt ADDED Viewed

	@@ -0,0 +1,21 @@

+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k4/step70000-hf/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,4 @@

+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k4/step70000-hf/config.json ADDED Viewed

	@@ -0,0 +1,29 @@

+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k4/step70000-hf/generation_config.json ADDED Viewed

	@@ -0,0 +1,11 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k4/step70000-hf/tokenizer-export-audit.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k4/step70000-hf/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

pretrain/olmo2-190M-10xC-k4/step70000-hf/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k4/step70000/.metadata.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"ephemeral": false, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k4/step70000/config.json ADDED Viewed

	@@ -0,0 +1 @@

pretrain/olmo2-190M-10xC-k4/step70000/data_paths.txt ADDED Viewed

	@@ -0,0 +1,21 @@

+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k4/step70000/train/rank3.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:883700482dc47fb6c3b67c66b51c9afe696d3a2c748b4253e59d5637f4dbb2ce
+size 15317

pretrain/olmo2-190M-10xC-k4/step70000/train/rank4.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bb348cd283d9d8a565056d44cd5f7faf4de6833796c904749abadba27d9d9187
+size 15317

pretrain/olmo2-190M-10xC-k4/step70000/train/rank6.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:20e51fd307ec30e4c993efca721466109976e97bd2efc662312b9bf66213fbeb
+size 15317

pretrain/olmo2-190M-10xC-k4/step70000/train/rank7.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2c983176b7d9251781bace9bd15026bf0be05f566de06d2b5aef1c0be7ce65b7
+size 15317

pretrain/olmo2-190M-10xC-k4/step72000-hf/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,4 @@

+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k4/step72000-hf/config.json ADDED Viewed

	@@ -0,0 +1,29 @@

+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k4/step72000-hf/generation_config.json ADDED Viewed

	@@ -0,0 +1,11 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k4/step72000-hf/tokenizer-export-audit.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k4/step72000-hf/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

pretrain/olmo2-190M-10xC-k4/step72000-hf/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k4/step72000/.metadata.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"ephemeral": true, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k4/step72000/config.json ADDED Viewed

	@@ -0,0 +1 @@

pretrain/olmo2-190M-10xC-k4/step72000/data_paths.txt ADDED Viewed

	@@ -0,0 +1,21 @@

+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy

pretrain/olmo2-190M-10xC-k4/step72615-hf/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,4 @@

+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

pretrain/olmo2-190M-10xC-k4/step72615-hf/config.json ADDED Viewed

	@@ -0,0 +1,29 @@

+{
+  "architectures": [
+    "Olmo2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "eos_token_id": 100257,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 8192,
+  "model_type": "olmo2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pad_token_id": 100277,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.1",
+  "use_cache": true,
+  "vocab_size": 100278
+}

pretrain/olmo2-190M-10xC-k4/step72615-hf/generation_config.json ADDED Viewed

	@@ -0,0 +1,11 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100257,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 100277,
+  "transformers_version": "5.15.1",
+  "use_cache": true
+}

pretrain/olmo2-190M-10xC-k4/step72615-hf/tokenizer-export-audit.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+  "passed": true,
+  "source": "allenai/dolma2-tokenizer",
+  "revision": null,
+  "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
+  "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
+  "tokenizer_class": "TokenizersBackend",
+  "probe_count": 8
+}

pretrain/olmo2-190M-10xC-k4/step72615-hf/tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

pretrain/olmo2-190M-10xC-k4/step72615-hf/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>"
+}

pretrain/olmo2-190M-10xC-k4/step72615/.metadata.json ADDED Viewed

	@@ -0,0 +1 @@


1	+ {"ephemeral": false, "version": "3.0.0"}

pretrain/olmo2-190M-10xC-k4/step72615/config.json ADDED Viewed

	@@ -0,0 +1 @@

pretrain/olmo2-190M-10xC-k4/step72615/data_paths.txt ADDED Viewed

	@@ -0,0 +1,21 @@

+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.prefix9601222788.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix3022291736.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix1072279872.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix628969248.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix610056864.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.prefix406306580.npy
+/tmp/jgai-olmo-data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix189022960.npy