Instructions to use keepsloading/icml_repro_scratch with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use keepsloading/icml_repro_scratch with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="keepsloading/icml_repro_scratch")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("keepsloading/icml_repro_scratch")
model = AutoModelForCausalLM.from_pretrained("keepsloading/icml_repro_scratch", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use keepsloading/icml_repro_scratch with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "keepsloading/icml_repro_scratch"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/keepsloading/icml_repro_scratch

SGLang

How to use keepsloading/icml_repro_scratch with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "keepsloading/icml_repro_scratch" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "keepsloading/icml_repro_scratch" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use keepsloading/icml_repro_scratch with Docker Model Runner:
```
docker model run hf.co/keepsloading/icml_repro_scratch
```

keepsloading commited on 26 days ago

Commit

1ade3b4

verified ·

1 Parent(s): 7f8c049

Upload folder using huggingface_hub

Browse files

This view is limited to 50 files because it contains too many changes. See raw diff

Files changed (50) hide show

outputs/eval/logs/token_t045/babilong.log +312 -0
outputs/eval/logs/token_t045/helmet.log +252 -0
outputs/eval/logs/token_t045/mrcr.log +32 -0
outputs/eval/logs/token_t045/ruler_a.log +352 -0
outputs/eval/logs/token_t045/ruler_b.log +252 -0
outputs/eval/logs/token_t0525/babilong.log +312 -0
outputs/eval/logs/token_t0525/helmet.log +252 -0
outputs/eval/logs/token_t0525/mrcr.log +32 -0
outputs/eval/logs/token_t0525/ruler_a.log +340 -0
outputs/eval/logs/token_t0525/ruler_b.log +253 -0
outputs/eval/logs/token_t0575/babilong.log +314 -0
outputs/eval/logs/token_t0575/helmet.log +252 -0
outputs/eval/logs/token_t0575/mrcr.log +32 -0
outputs/eval/logs/token_t0575/ruler_a.log +344 -0
outputs/eval/logs/token_t0575/ruler_b.log +249 -0
outputs/eval/logs/token_t065/babilong.log +314 -0
outputs/eval/logs/token_t065/helmet.log +252 -0
outputs/eval/logs/token_t065/mrcr.log +32 -0
outputs/eval/logs/token_t065/ruler_a.log +345 -0
outputs/eval/logs/token_t065/ruler_b.log +253 -0
outputs/eval/logs/vanilla/babilong.log +179 -179
outputs/eval/logs/vanilla/ruler_a.log +260 -267
outputs/eval/logs/vanilla/ruler_b.log +210 -213
outputs/eval/token_t045/babilong.DONE +0 -0
outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/results_2026-07-18T12-31-42.056499.json +505 -0
outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa1_2026-07-18T12-31-42.056499.jsonl +3 -0
outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa2_2026-07-18T12-31-42.056499.jsonl +3 -0
outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa3_2026-07-18T12-31-42.056499.jsonl +3 -0
outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa4_2026-07-18T12-31-42.056499.jsonl +3 -0
outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa5_2026-07-18T12-31-42.056499.jsonl +3 -0
outputs/eval/token_t045/helmet.DONE +0 -0
outputs/eval/token_t045/helmet_icl8k_n50.jsonl +3 -0
outputs/eval/token_t045/helmet_icl8k_n50.summary.csv +251 -0
outputs/eval/token_t045/mrcr.DONE +0 -0
outputs/eval/token_t045/mrcr_8k_2_4_8needle_n10.jsonl +3 -0
outputs/eval/token_t045/mrcr_8k_2_4_8needle_n10.summary.csv +31 -0
outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/results_2026-07-18T12-12-07.465325.json +821 -0
outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multikey_2_2026-07-18T12-12-07.465325.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multiquery_2026-07-18T12-12-07.465325.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_single_1_2026-07-18T12-12-07.465325.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_single_3_2026-07-18T12-12-07.465325.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_ruler_fwe_2026-07-18T12-12-07.465325.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_ruler_qa_hotpot_2026-07-18T12-12-07.465325.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_ruler_vt_2026-07-18T12-12-07.465325.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/results_2026-07-18T12-26-30.235088.json +714 -0
outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multikey_1_2026-07-18T12-26-30.235088.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multikey_3_2026-07-18T12-26-30.235088.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multivalue_2026-07-18T12-26-30.235088.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_single_2_2026-07-18T12-26-30.235088.jsonl +3 -0
outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_ruler_cwe_2026-07-18T12-26-30.235088.jsonl +3 -0

outputs/eval/logs/token_t045/babilong.log ADDED Viewed

@@ -0,0 +1,312 @@
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A

+2026-07-18:12:26:33 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:12:26:37 INFO     [_cli.run:376] Selected Tasks: ['babilong_longctx']
+2026-07-18:12:26:38 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:12:26:38 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:12:26:41 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:12:26:42 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:12:26:42 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:12:26:44 WARNING  [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:26:44 WARNING  [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:26:44 WARNING  [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:26:44 WARNING  [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:26:45 WARNING  [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:26:45 INFO     [tasks:700] Selected tasks:
+2026-07-18:12:26:45 INFO     [tasks:703] Group: babilong_longctx
+2026-07-18:12:26:45 INFO     [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
+2026-07-18:12:26:45 INFO     [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:12:26:45 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
+2026-07-18:12:26:45 INFO     [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:12:26:45 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
+2026-07-18:12:26:45 INFO     [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:12:26:45 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
+2026-07-18:12:26:45 INFO     [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:12:26:45 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
+2026-07-18:12:26:45 INFO     [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:12:26:45 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
+2026-07-18:12:26:45 INFO     [api.task:311] Building contexts for babilong_qa1 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:12:26:45 INFO     [api.task:311] Building contexts for babilong_qa2 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:12:26:45 INFO     [api.task:311] Building contexts for babilong_qa3 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:12:26:45 INFO     [api.task:311] Building contexts for babilong_qa4 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:12:26:45 INFO     [api.task:311] Building contexts for babilong_qa5 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:12:26:45 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:12:31:42 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:12:31:42 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
+|     Tasks      |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|-----:|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |0.588|±  |0.0298|
+| - babilong_qa1 |      0|none  |     2|acc   |↑  |0.640|±  |0.0686|
+| - babilong_qa2 |      0|none  |     2|acc   |↑  |0.480|±  |0.0714|
+| - babilong_qa3 |      0|none  |     2|acc   |↑  |0.340|±  |0.0677|
+| - babilong_qa4 |      0|none  |     2|acc   |↑  |0.720|±  |0.0641|
+| - babilong_qa5 |      0|none  |     2|acc   |↑  |0.760|±  |0.0610|
+|     Groups     |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|------|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |0.588|±  |0.0298|

outputs/eval/logs/token_t045/helmet.log ADDED Viewed

	@@ -0,0 +1,252 @@

+[eval] stop_token_ids=[151645]
+[helmet_icl] 1/250 trec_coarse tokens=6597
+[helmet_icl] 2/250 trec_coarse tokens=6549
+[helmet_icl] 3/250 trec_coarse tokens=6508
+[helmet_icl] 4/250 trec_coarse tokens=6608
+[helmet_icl] 5/250 trec_coarse tokens=6497
+[helmet_icl] 6/250 trec_coarse tokens=6545
+[helmet_icl] 7/250 trec_coarse tokens=6583
+[helmet_icl] 8/250 trec_coarse tokens=6455
+[helmet_icl] 9/250 trec_coarse tokens=6643
+[helmet_icl] 10/250 trec_coarse tokens=6568
+[helmet_icl] 11/250 trec_coarse tokens=6518
+[helmet_icl] 12/250 trec_coarse tokens=6424
+[helmet_icl] 13/250 trec_coarse tokens=6750
+[helmet_icl] 14/250 trec_coarse tokens=6642
+[helmet_icl] 15/250 trec_coarse tokens=6494
+[helmet_icl] 16/250 trec_coarse tokens=6427
+[helmet_icl] 17/250 trec_coarse tokens=6547
+[helmet_icl] 18/250 trec_coarse tokens=6612
+[helmet_icl] 19/250 trec_coarse tokens=6591
+[helmet_icl] 20/250 trec_coarse tokens=6485
+[helmet_icl] 21/250 trec_coarse tokens=6542
+[helmet_icl] 22/250 trec_coarse tokens=6409
+[helmet_icl] 23/250 trec_coarse tokens=6482
+[helmet_icl] 24/250 trec_coarse tokens=6502
+[helmet_icl] 25/250 trec_coarse tokens=6344
+[helmet_icl] 26/250 trec_coarse tokens=6479
+[helmet_icl] 27/250 trec_coarse tokens=6569
+[helmet_icl] 28/250 trec_coarse tokens=6389
+[helmet_icl] 29/250 trec_coarse tokens=6479
+[helmet_icl] 30/250 trec_coarse tokens=6653
+[helmet_icl] 31/250 trec_coarse tokens=6532
+[helmet_icl] 32/250 trec_coarse tokens=6491
+[helmet_icl] 33/250 trec_coarse tokens=6489
+[helmet_icl] 34/250 trec_coarse tokens=6630
+[helmet_icl] 35/250 trec_coarse tokens=6488
+[helmet_icl] 36/250 trec_coarse tokens=6499
+[helmet_icl] 37/250 trec_coarse tokens=6483
+[helmet_icl] 38/250 trec_coarse tokens=6660
+[helmet_icl] 39/250 trec_coarse tokens=6675
+[helmet_icl] 40/250 trec_coarse tokens=6544
+[helmet_icl] 41/250 trec_coarse tokens=6520
+[helmet_icl] 42/250 trec_coarse tokens=6359
+[helmet_icl] 43/250 trec_coarse tokens=6657
+[helmet_icl] 44/250 trec_coarse tokens=6546
+[helmet_icl] 45/250 trec_coarse tokens=6481
+[helmet_icl] 46/250 trec_coarse tokens=6506
+[helmet_icl] 47/250 trec_coarse tokens=6621
+[helmet_icl] 48/250 trec_coarse tokens=6623
+[helmet_icl] 49/250 trec_coarse tokens=6411
+[helmet_icl] 50/250 trec_coarse tokens=6577
+[helmet_icl] 51/250 trec_fine tokens=6858
+[helmet_icl] 52/250 trec_fine tokens=6973
+[helmet_icl] 53/250 trec_fine tokens=6977
+[helmet_icl] 54/250 trec_fine tokens=6952
+[helmet_icl] 55/250 trec_fine tokens=6919
+[helmet_icl] 56/250 trec_fine tokens=6979
+[helmet_icl] 57/250 trec_fine tokens=6957
+[helmet_icl] 58/250 trec_fine tokens=7013
+[helmet_icl] 59/250 trec_fine tokens=6824
+[helmet_icl] 60/250 trec_fine tokens=6833
+[helmet_icl] 61/250 trec_fine tokens=6868
+[helmet_icl] 62/250 trec_fine tokens=6914
+[helmet_icl] 63/250 trec_fine tokens=6759
+[helmet_icl] 64/250 trec_fine tokens=6933
+[helmet_icl] 65/250 trec_fine tokens=6868
+[helmet_icl] 66/250 trec_fine tokens=6839
+[helmet_icl] 67/250 trec_fine tokens=6838
+[helmet_icl] 68/250 trec_fine tokens=7000
+[helmet_icl] 69/250 trec_fine tokens=6800
+[helmet_icl] 70/250 trec_fine tokens=6829
+[helmet_icl] 71/250 trec_fine tokens=6971
+[helmet_icl] 72/250 trec_fine tokens=6907
+[helmet_icl] 73/250 trec_fine tokens=6818
+[helmet_icl] 74/250 trec_fine tokens=6902
+[helmet_icl] 75/250 trec_fine tokens=6941
+[helmet_icl] 76/250 trec_fine tokens=6830
+[helmet_icl] 77/250 trec_fine tokens=6934
+[helmet_icl] 78/250 trec_fine tokens=6879
+[helmet_icl] 79/250 trec_fine tokens=6915
+[helmet_icl] 80/250 trec_fine tokens=6953
+[helmet_icl] 81/250 trec_fine tokens=6875
+[helmet_icl] 82/250 trec_fine tokens=6847
+[helmet_icl] 83/250 trec_fine tokens=6847
+[helmet_icl] 84/250 trec_fine tokens=6944
+[helmet_icl] 85/250 trec_fine tokens=6818
+[helmet_icl] 86/250 trec_fine tokens=6868
+[helmet_icl] 87/250 trec_fine tokens=7022
+[helmet_icl] 88/250 trec_fine tokens=6930
+[helmet_icl] 89/250 trec_fine tokens=6863
+[helmet_icl] 90/250 trec_fine tokens=6807
+[helmet_icl] 91/250 trec_fine tokens=7001
+[helmet_icl] 92/250 trec_fine tokens=6890
+[helmet_icl] 93/250 trec_fine tokens=6952
+[helmet_icl] 94/250 trec_fine tokens=6845
+[helmet_icl] 95/250 trec_fine tokens=6902
+[helmet_icl] 96/250 trec_fine tokens=6879
+[helmet_icl] 97/250 trec_fine tokens=6896
+[helmet_icl] 98/250 trec_fine tokens=6770
+[helmet_icl] 99/250 trec_fine tokens=6933
+[helmet_icl] 100/250 trec_fine tokens=6886
+[helmet_icl] 101/250 banking77 tokens=6809
+[helmet_icl] 102/250 banking77 tokens=7162
+[helmet_icl] 103/250 banking77 tokens=6876
+[helmet_icl] 104/250 banking77 tokens=7033
+[helmet_icl] 105/250 banking77 tokens=6647
+[helmet_icl] 106/250 banking77 tokens=6818
+[helmet_icl] 107/250 banking77 tokens=6841
+[helmet_icl] 108/250 banking77 tokens=6944
+[helmet_icl] 109/250 banking77 tokens=6776
+[helmet_icl] 110/250 banking77 tokens=7100
+[helmet_icl] 111/250 banking77 tokens=6967
+[helmet_icl] 112/250 banking77 tokens=6894
+[helmet_icl] 113/250 banking77 tokens=6900
+[helmet_icl] 114/250 banking77 tokens=6964
+[helmet_icl] 115/250 banking77 tokens=7085
+[helmet_icl] 116/250 banking77 tokens=6953
+[helmet_icl] 117/250 banking77 tokens=7406
+[helmet_icl] 118/250 banking77 tokens=6802
+[helmet_icl] 119/250 banking77 tokens=7119
+[helmet_icl] 120/250 banking77 tokens=7202
+[helmet_icl] 121/250 banking77 tokens=7103
+[helmet_icl] 122/250 banking77 tokens=7009
+[helmet_icl] 123/250 banking77 tokens=6940
+[helmet_icl] 124/250 banking77 tokens=6857
+[helmet_icl] 125/250 banking77 tokens=6922
+[helmet_icl] 126/250 banking77 tokens=6988
+[helmet_icl] 127/250 banking77 tokens=6945
+[helmet_icl] 128/250 banking77 tokens=6835
+[helmet_icl] 129/250 banking77 tokens=7035
+[helmet_icl] 130/250 banking77 tokens=6787
+[helmet_icl] 131/250 banking77 tokens=6940
+[helmet_icl] 132/250 banking77 tokens=6661
+[helmet_icl] 133/250 banking77 tokens=6934
+[helmet_icl] 134/250 banking77 tokens=7235
+[helmet_icl] 135/250 banking77 tokens=6829
+[helmet_icl] 136/250 banking77 tokens=7169
+[helmet_icl] 137/250 banking77 tokens=7074
+[helmet_icl] 138/250 banking77 tokens=6797
+[helmet_icl] 139/250 banking77 tokens=7013
+[helmet_icl] 140/250 banking77 tokens=6811
+[helmet_icl] 141/250 banking77 tokens=6983
+[helmet_icl] 142/250 banking77 tokens=7183
+[helmet_icl] 143/250 banking77 tokens=6801
+[helmet_icl] 144/250 banking77 tokens=7174
+[helmet_icl] 145/250 banking77 tokens=6773
+[helmet_icl] 146/250 banking77 tokens=7134
+[helmet_icl] 147/250 banking77 tokens=6907
+[helmet_icl] 148/250 banking77 tokens=7341
+[helmet_icl] 149/250 banking77 tokens=7042
+[helmet_icl] 150/250 banking77 tokens=7304
+[helmet_icl] 151/250 clinic150 tokens=7241
+[helmet_icl] 152/250 clinic150 tokens=7282
+[helmet_icl] 153/250 clinic150 tokens=7261
+[helmet_icl] 154/250 clinic150 tokens=7292
+[helmet_icl] 155/250 clinic150 tokens=7217
+[helmet_icl] 156/250 clinic150 tokens=7280
+[helmet_icl] 157/250 clinic150 tokens=7293
+[helmet_icl] 158/250 clinic150 tokens=7250
+[helmet_icl] 159/250 clinic150 tokens=7264
+[helmet_icl] 160/250 clinic150 tokens=7238
+[helmet_icl] 161/250 clinic150 tokens=7243
+[helmet_icl] 162/250 clinic150 tokens=7191
+[helmet_icl] 163/250 clinic150 tokens=7333
+[helmet_icl] 164/250 clinic150 tokens=7299
+[helmet_icl] 165/250 clinic150 tokens=7225
+[helmet_icl] 166/250 clinic150 tokens=7409
+[helmet_icl] 167/250 clinic150 tokens=7277
+[helmet_icl] 168/250 clinic150 tokens=7376
+[helmet_icl] 169/250 clinic150 tokens=7383
+[helmet_icl] 170/250 clinic150 tokens=7270
+[helmet_icl] 171/250 clinic150 tokens=7173
+[helmet_icl] 172/250 clinic150 tokens=7106
+[helmet_icl] 173/250 clinic150 tokens=7413
+[helmet_icl] 174/250 clinic150 tokens=7253
+[helmet_icl] 175/250 clinic150 tokens=7224
+[helmet_icl] 176/250 clinic150 tokens=7307
+[helmet_icl] 177/250 clinic150 tokens=7272
+[helmet_icl] 178/250 clinic150 tokens=7215
+[helmet_icl] 179/250 clinic150 tokens=7344
+[helmet_icl] 180/250 clinic150 tokens=7224
+[helmet_icl] 181/250 clinic150 tokens=7310
+[helmet_icl] 182/250 clinic150 tokens=7222
+[helmet_icl] 183/250 clinic150 tokens=7260
+[helmet_icl] 184/250 clinic150 tokens=7351
+[helmet_icl] 185/250 clinic150 tokens=7301
+[helmet_icl] 186/250 clinic150 tokens=7155
+[helmet_icl] 187/250 clinic150 tokens=7253
+[helmet_icl] 188/250 clinic150 tokens=7248
+[helmet_icl] 189/250 clinic150 tokens=7128
+[helmet_icl] 190/250 clinic150 tokens=7282
+[helmet_icl] 191/250 clinic150 tokens=7301
+[helmet_icl] 192/250 clinic150 tokens=7388
+[helmet_icl] 193/250 clinic150 tokens=7244
+[helmet_icl] 194/250 clinic150 tokens=7186
+[helmet_icl] 195/250 clinic150 tokens=7309
+[helmet_icl] 196/250 clinic150 tokens=7241
+[helmet_icl] 197/250 clinic150 tokens=7214
+[helmet_icl] 198/250 clinic150 tokens=7264
+[helmet_icl] 199/250 clinic150 tokens=7219
+[helmet_icl] 200/250 clinic150 tokens=7228
+[helmet_icl] 201/250 nlu tokens=7063
+[helmet_icl] 202/250 nlu tokens=6959
+[helmet_icl] 203/250 nlu tokens=6985
+[helmet_icl] 204/250 nlu tokens=6970
+[helmet_icl] 205/250 nlu tokens=7025
+[helmet_icl] 206/250 nlu tokens=7034
+[helmet_icl] 207/250 nlu tokens=7004
+[helmet_icl] 208/250 nlu tokens=7062
+[helmet_icl] 209/250 nlu tokens=7062
+[helmet_icl] 210/250 nlu tokens=7045
+[helmet_icl] 211/250 nlu tokens=7028
+[helmet_icl] 212/250 nlu tokens=7018
+[helmet_icl] 213/250 nlu tokens=7075
+[helmet_icl] 214/250 nlu tokens=7068
+[helmet_icl] 215/250 nlu tokens=7025
+[helmet_icl] 216/250 nlu tokens=7022
+[helmet_icl] 217/250 nlu tokens=7112
+[helmet_icl] 218/250 nlu tokens=6967
+[helmet_icl] 219/250 nlu tokens=6980
+[helmet_icl] 220/250 nlu tokens=7031
+[helmet_icl] 221/250 nlu tokens=7093
+[helmet_icl] 222/250 nlu tokens=6949
+[helmet_icl] 223/250 nlu tokens=7046
+[helmet_icl] 224/250 nlu tokens=7039
+[helmet_icl] 225/250 nlu tokens=7126
+[helmet_icl] 226/250 nlu tokens=7064
+[helmet_icl] 227/250 nlu tokens=7057
+[helmet_icl] 228/250 nlu tokens=6971
+[helmet_icl] 229/250 nlu tokens=7034
+[helmet_icl] 230/250 nlu tokens=7094
+[helmet_icl] 231/250 nlu tokens=6979
+[helmet_icl] 232/250 nlu tokens=7123
+[helmet_icl] 233/250 nlu tokens=7144
+[helmet_icl] 234/250 nlu tokens=7024
+[helmet_icl] 235/250 nlu tokens=6943
+[helmet_icl] 236/250 nlu tokens=6961
+[helmet_icl] 237/250 nlu tokens=7005
+[helmet_icl] 238/250 nlu tokens=7080
+[helmet_icl] 239/250 nlu tokens=7179
+[helmet_icl] 240/250 nlu tokens=7115
+[helmet_icl] 241/250 nlu tokens=6946
+[helmet_icl] 242/250 nlu tokens=6985
+[helmet_icl] 243/250 nlu tokens=7131
+[helmet_icl] 244/250 nlu tokens=7100
+[helmet_icl] 245/250 nlu tokens=7190
+[helmet_icl] 246/250 nlu tokens=7040
+[helmet_icl] 247/250 nlu tokens=7013
+[helmet_icl] 248/250 nlu tokens=7080
+[helmet_icl] 249/250 nlu tokens=7098
+[helmet_icl] 250/250 nlu tokens=6891
+/workspace/outputs/eval/token_t045/helmet_icl8k_n50.summary.csv

outputs/eval/logs/token_t045/mrcr.log ADDED Viewed

	@@ -0,0 +1,32 @@

+[eval] stop_token_ids=[151645]
+[mrcr] 1/30 8k_2needle tokens=4852
+[mrcr] 2/30 8k_2needle tokens=7119
+[mrcr] 3/30 8k_2needle tokens=5756
+[mrcr] 4/30 8k_2needle tokens=5072
+[mrcr] 5/30 8k_2needle tokens=5753
+[mrcr] 6/30 8k_2needle tokens=5481
+[mrcr] 7/30 8k_2needle tokens=5143
+[mrcr] 8/30 8k_2needle tokens=6399
+[mrcr] 9/30 8k_2needle tokens=5987
+[mrcr] 10/30 8k_2needle tokens=4380
+[mrcr] 11/30 8k_4needle tokens=5123
+[mrcr] 12/30 8k_4needle tokens=7908
+[mrcr] 13/30 8k_4needle tokens=5814
+[mrcr] 14/30 8k_4needle tokens=4856
+[mrcr] 15/30 8k_4needle tokens=7413
+[mrcr] 16/30 8k_4needle tokens=7764
+[mrcr] 17/30 8k_4needle tokens=4312
+[mrcr] 18/30 8k_4needle tokens=4847
+[mrcr] 19/30 8k_4needle tokens=6287
+[mrcr] 20/30 8k_4needle tokens=7841
+[mrcr] 21/30 8k_8needle tokens=4581
+[mrcr] 22/30 8k_8needle tokens=6357
+[mrcr] 23/30 8k_8needle tokens=6981
+[mrcr] 24/30 8k_8needle tokens=6308
+[mrcr] 25/30 8k_8needle tokens=6901
+[mrcr] 26/30 8k_8needle tokens=5407
+[mrcr] 27/30 8k_8needle tokens=6351
+[mrcr] 28/30 8k_8needle tokens=5657
+[mrcr] 29/30 8k_8needle tokens=7330
+[mrcr] 30/30 8k_8needle tokens=8028
+/workspace/outputs/eval/token_t045/mrcr_8k_2_4_8needle_n10.summary.csv

outputs/eval/logs/token_t045/ruler_a.log ADDED Viewed

@@ -0,0 +1,352 @@
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/500 [00:00<?, ?it/s][A
 11%|█         | 55/500 [00:01<00:08, 54.38it/s][A
 22%|██▏       | 110/500 [00:02<00:07, 54.20it/s][A
 33%|███▎      | 165/500 [00:03<00:06, 54.37it/s][A
 44%|████▍     | 220/500 [00:04<00:05, 54.34it/s][A
 55%|█████▌    | 275/500 [00:05<00:04, 54.57it/s][A
 66%|██████▌   | 331/500 [00:06<00:03, 55.02it/s][A
 77%|███████▋  | 387/500 [00:07<00:02, 54.89it/s][A
 89%|████████▉ | 444/500 [00:08<00:01, 55.23it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:11:58:36 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:11:58:39 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
+2026-07-18:11:58:41 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:11:58:41 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:11:58:44 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:11:58:44 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:11:58:45 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:11:58:54 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:58:54 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:11:59:04 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:59:14 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:59:23 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:59:33 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:59:33 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+Max length 500 | Current length 307 | Noises: 5
+Max length 500 | Current length 410 | Noises: 10
+Max length 500 | Current length 531 | Noises: 15
+Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
+Max length 8192 | Current length 791 | Noises: 10
+Max length 8192 | Current length 1025 | Noises: 20
+Max length 8192 | Current length 1273 | Noises: 30
+Max length 8192 | Current length 1510 | Noises: 40
+Max length 8192 | Current length 1749 | Noises: 50
+Max length 8192 | Current length 1996 | Noises: 60
+Max length 8192 | Current length 2231 | Noises: 70
+Max length 8192 | Current length 2473 | Noises: 80
+Max length 8192 | Current length 2705 | Noises: 90
+Max length 8192 | Current length 2953 | Noises: 100
+Max length 8192 | Current length 3186 | Noises: 110
+Max length 8192 | Current length 3431 | Noises: 120
+Max length 8192 | Current length 3671 | Noises: 130
+Max length 8192 | Current length 3916 | Noises: 140
+Max length 8192 | Current length 4149 | Noises: 150
+Max length 8192 | Current length 4386 | Noises: 160
+Max length 8192 | Current length 4630 | Noises: 170
+Max length 8192 | Current length 4871 | Noises: 180
+Max length 8192 | Current length 5110 | Noises: 190
+Max length 8192 | Current length 5349 | Noises: 200
+Max length 8192 | Current length 5590 | Noises: 210
+Max length 8192 | Current length 5832 | Noises: 220
+Max length 8192 | Current length 6070 | Noises: 230
+Max length 8192 | Current length 6311 | Noises: 240
+Max length 8192 | Current length 6544 | Noises: 250
+Max length 8192 | Current length 6791 | Noises: 260
+Max length 8192 | Current length 7031 | Noises: 270
+Max length 8192 | Current length 7271 | Noises: 280
+Max length 8192 | Current length 7513 | Noises: 290
+Max length 8192 | Current length 7751 | Noises: 300
+Max length 8192 | Current length 7986 | Noises: 310
+Max length 8192 | Current length 8230 | Noises: 320
+Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 11%|█         | 55/500 [00:01<00:08, 54.38it/s][A
 22%|██▏       | 110/500 [00:02<00:07, 54.20it/s][A
 33%|███▎      | 165/500 [00:03<00:06, 54.37it/s][A
 44%|████▍     | 220/500 [00:04<00:05, 54.34it/s][A
 55%|█████▌    | 275/500 [00:05<00:04, 54.57it/s][A
 66%|██████▌   | 331/500 [00:06<00:03, 55.02it/s][A
 77%|███████▋  | 387/500 [00:07<00:02, 54.89it/s][A
 89%|████████▉ | 444/500 [00:08<00:01, 55.23it/s][A
+2026-07-18:11:59:43 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:59:57 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:00:21 INFO     [tasks:700] Selected tasks:
+2026-07-18:12:00:21 INFO     [tasks:691] Task: ruler_qa_hotpot (ruler/qa_hotpot.yaml)
+2026-07-18:12:00:21 INFO     [tasks:691] Task: ruler_fwe (ruler/fwe.yaml)
+2026-07-18:12:00:21 INFO     [tasks:691] Task: ruler_vt (ruler/vt.yaml)
+2026-07-18:12:00:21 INFO     [tasks:691] Task: niah_multiquery (ruler/niah_multiquery.yaml)
+2026-07-18:12:00:21 INFO     [tasks:691] Task: niah_multikey_2 (ruler/niah_multikey_2.yaml)
+2026-07-18:12:00:21 INFO     [tasks:691] Task: niah_single_3 (ruler/niah_single_3.yaml)
+2026-07-18:12:00:21 INFO     [tasks:691] Task: niah_single_1 (ruler/niah_single_1.yaml)
+2026-07-18:12:00:21 INFO     [evaluator:314] ruler_qa_hotpot: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:12:00:21 INFO     [evaluator:314] ruler_fwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 50, 'until': []}
+2026-07-18:12:00:21 INFO     [evaluator:314] ruler_vt: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 30, 'until': []}
+2026-07-18:12:00:21 INFO     [evaluator:314] niah_multiquery: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:00:21 INFO     [evaluator:314] niah_multikey_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:00:21 INFO     [evaluator:314] niah_single_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:00:21 INFO     [evaluator:314] niah_single_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:00:21 INFO     [api.task:311] Building contexts for ruler_qa_hotpot on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:00:21 INFO     [api.task:311] Building contexts for ruler_fwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:00:21 INFO     [api.task:311] Building contexts for ruler_vt on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:00:21 INFO     [api.task:311] Building contexts for niah_multiquery on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:00:21 INFO     [api.task:311] Building contexts for niah_multikey_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:00:21 INFO     [api.task:311] Building contexts for niah_single_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:00:21 INFO     [api.task:311] Building contexts for niah_single_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:00:21 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:12:12:07 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:12:12:07 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_2|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_multiquery|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.9875|±  |   N/A|
+|niah_single_1  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_single_3  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.9500|±  |   N/A|
+|ruler_fwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7667|±  |   N/A|
+|ruler_qa_hotpot|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4000|±  |   N/A|
+|ruler_vt       |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.8700|±  |   N/A|

outputs/eval/logs/token_t045/ruler_b.log ADDED Viewed

@@ -0,0 +1,252 @@
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:12:12:10 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:12:12:14 INFO     [_cli.run:376] Selected Tasks: ['niah_single_2', 'niah_multikey_1', 'niah_multikey_3', 'niah_multivalue', 'ruler_cwe', 'ruler_qa_squad']
+2026-07-18:12:12:15 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:12:12:15 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:12:12:18 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:12:12:18 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:12:12:19 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:12:12:28 WARNING  [api.task:856] niah_single_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:12:28 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:12:12:39 WARNING  [api.task:856] niah_multikey_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:12:48 WARNING  [api.task:856] niah_multikey_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:12:55 WARNING  [api.task:856] niah_multivalue: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:13:05 WARNING  [api.task:856] ruler_cwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:13:05 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:12:13:12 WARNING  [api.task:856] ruler_qa_squad: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:13:23 INFO     [tasks:700] Selected tasks:
+2026-07-18:12:13:23 INFO     [tasks:691] Task: ruler_qa_squad (ruler/qa_squad.yaml)
+2026-07-18:12:13:23 INFO     [tasks:691] Task: ruler_cwe (ruler/cwe.yaml)
+2026-07-18:12:13:23 INFO     [tasks:691] Task: niah_multivalue (ruler/niah_multivalue.yaml)
+2026-07-18:12:13:23 INFO     [tasks:691] Task: niah_multikey_3 (ruler/niah_multikey_3.yaml)
+2026-07-18:12:13:23 INFO     [tasks:691] Task: niah_multikey_1 (ruler/niah_multikey_1.yaml)
+2026-07-18:12:13:23 INFO     [tasks:691] Task: niah_single_2 (ruler/niah_single_2.yaml)
+2026-07-18:12:13:23 INFO     [evaluator:314] ruler_qa_squad: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:12:13:23 INFO     [evaluator:314] ruler_cwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 120, 'until': []}
+2026-07-18:12:13:23 INFO     [evaluator:314] niah_multivalue: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:13:23 INFO     [evaluator:314] niah_multikey_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:13:23 INFO     [evaluator:314] niah_multikey_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:13:23 INFO     [evaluator:314] niah_single_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:13:23 INFO     [api.task:311] Building contexts for ruler_qa_squad on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:13:23 INFO     [api.task:311] Building contexts for ruler_cwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:13:23 INFO     [api.task:311] Building contexts for niah_multivalue on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:13:23 INFO     [api.task:311] Building contexts for niah_multikey_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:13:23 INFO     [api.task:311] Building contexts for niah_multikey_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:13:23 INFO     [api.task:311] Building contexts for niah_single_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:13:23 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:12:26:30 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:12:26:30 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_1|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_multikey_3|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7500|±  |   N/A|
+|niah_multivalue|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_single_2  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|ruler_cwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.3950|±  |   N/A|
+|ruler_qa_squad |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4208|±  |   N/A|

outputs/eval/logs/token_t0525/babilong.log ADDED Viewed

@@ -0,0 +1,312 @@
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A

+2026-07-18:13:09:04 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:13:09:08 INFO     [_cli.run:376] Selected Tasks: ['babilong_longctx']
+2026-07-18:13:09:09 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:13:09:09 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:13:09:12 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:13:09:12 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:13:09:13 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:13:09:14 WARNING  [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:09:15 WARNING  [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:09:15 WARNING  [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:09:15 WARNING  [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:09:15 WARNING  [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:09:16 INFO     [tasks:700] Selected tasks:
+2026-07-18:13:09:16 INFO     [tasks:703] Group: babilong_longctx
+2026-07-18:13:09:16 INFO     [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
+2026-07-18:13:09:16 INFO     [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:09:16 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
+2026-07-18:13:09:16 INFO     [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:09:16 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
+2026-07-18:13:09:16 INFO     [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:09:16 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
+2026-07-18:13:09:16 INFO     [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:09:16 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
+2026-07-18:13:09:16 INFO     [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:09:16 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
+2026-07-18:13:09:16 INFO     [api.task:311] Building contexts for babilong_qa1 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:09:16 INFO     [api.task:311] Building contexts for babilong_qa2 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:09:16 INFO     [api.task:311] Building contexts for babilong_qa3 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:09:16 INFO     [api.task:311] Building contexts for babilong_qa4 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:09:16 INFO     [api.task:311] Building contexts for babilong_qa5 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:09:16 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:13:14:07 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:13:14:07 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t0525/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
+|     Tasks      |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|-----:|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |0.608|±  |0.0295|
+| - babilong_qa1 |      0|none  |     2|acc   |↑  |0.740|±  |0.0627|
+| - babilong_qa2 |      0|none  |     2|acc   |↑  |0.520|±  |0.0714|
+| - babilong_qa3 |      0|none  |     2|acc   |↑  |0.340|±  |0.0677|
+| - babilong_qa4 |      0|none  |     2|acc   |↑  |0.720|±  |0.0641|
+| - babilong_qa5 |      0|none  |     2|acc   |↑  |0.720|±  |0.0641|
+|     Groups     |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|------|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |0.608|±  |0.0295|

outputs/eval/logs/token_t0525/helmet.log ADDED Viewed

	@@ -0,0 +1,252 @@

+[eval] stop_token_ids=[151645]
+[helmet_icl] 1/250 trec_coarse tokens=6597
+[helmet_icl] 2/250 trec_coarse tokens=6549
+[helmet_icl] 3/250 trec_coarse tokens=6508
+[helmet_icl] 4/250 trec_coarse tokens=6608
+[helmet_icl] 5/250 trec_coarse tokens=6497
+[helmet_icl] 6/250 trec_coarse tokens=6545
+[helmet_icl] 7/250 trec_coarse tokens=6583
+[helmet_icl] 8/250 trec_coarse tokens=6455
+[helmet_icl] 9/250 trec_coarse tokens=6643
+[helmet_icl] 10/250 trec_coarse tokens=6568
+[helmet_icl] 11/250 trec_coarse tokens=6518
+[helmet_icl] 12/250 trec_coarse tokens=6424
+[helmet_icl] 13/250 trec_coarse tokens=6750
+[helmet_icl] 14/250 trec_coarse tokens=6642
+[helmet_icl] 15/250 trec_coarse tokens=6494
+[helmet_icl] 16/250 trec_coarse tokens=6427
+[helmet_icl] 17/250 trec_coarse tokens=6547
+[helmet_icl] 18/250 trec_coarse tokens=6612
+[helmet_icl] 19/250 trec_coarse tokens=6591
+[helmet_icl] 20/250 trec_coarse tokens=6485
+[helmet_icl] 21/250 trec_coarse tokens=6542
+[helmet_icl] 22/250 trec_coarse tokens=6409
+[helmet_icl] 23/250 trec_coarse tokens=6482
+[helmet_icl] 24/250 trec_coarse tokens=6502
+[helmet_icl] 25/250 trec_coarse tokens=6344
+[helmet_icl] 26/250 trec_coarse tokens=6479
+[helmet_icl] 27/250 trec_coarse tokens=6569
+[helmet_icl] 28/250 trec_coarse tokens=6389
+[helmet_icl] 29/250 trec_coarse tokens=6479
+[helmet_icl] 30/250 trec_coarse tokens=6653
+[helmet_icl] 31/250 trec_coarse tokens=6532
+[helmet_icl] 32/250 trec_coarse tokens=6491
+[helmet_icl] 33/250 trec_coarse tokens=6489
+[helmet_icl] 34/250 trec_coarse tokens=6630
+[helmet_icl] 35/250 trec_coarse tokens=6488
+[helmet_icl] 36/250 trec_coarse tokens=6499
+[helmet_icl] 37/250 trec_coarse tokens=6483
+[helmet_icl] 38/250 trec_coarse tokens=6660
+[helmet_icl] 39/250 trec_coarse tokens=6675
+[helmet_icl] 40/250 trec_coarse tokens=6544
+[helmet_icl] 41/250 trec_coarse tokens=6520
+[helmet_icl] 42/250 trec_coarse tokens=6359
+[helmet_icl] 43/250 trec_coarse tokens=6657
+[helmet_icl] 44/250 trec_coarse tokens=6546
+[helmet_icl] 45/250 trec_coarse tokens=6481
+[helmet_icl] 46/250 trec_coarse tokens=6506
+[helmet_icl] 47/250 trec_coarse tokens=6621
+[helmet_icl] 48/250 trec_coarse tokens=6623
+[helmet_icl] 49/250 trec_coarse tokens=6411
+[helmet_icl] 50/250 trec_coarse tokens=6577
+[helmet_icl] 51/250 trec_fine tokens=6858
+[helmet_icl] 52/250 trec_fine tokens=6973
+[helmet_icl] 53/250 trec_fine tokens=6977
+[helmet_icl] 54/250 trec_fine tokens=6952
+[helmet_icl] 55/250 trec_fine tokens=6919
+[helmet_icl] 56/250 trec_fine tokens=6979
+[helmet_icl] 57/250 trec_fine tokens=6957
+[helmet_icl] 58/250 trec_fine tokens=7013
+[helmet_icl] 59/250 trec_fine tokens=6824
+[helmet_icl] 60/250 trec_fine tokens=6833
+[helmet_icl] 61/250 trec_fine tokens=6868
+[helmet_icl] 62/250 trec_fine tokens=6914
+[helmet_icl] 63/250 trec_fine tokens=6759
+[helmet_icl] 64/250 trec_fine tokens=6933
+[helmet_icl] 65/250 trec_fine tokens=6868
+[helmet_icl] 66/250 trec_fine tokens=6839
+[helmet_icl] 67/250 trec_fine tokens=6838
+[helmet_icl] 68/250 trec_fine tokens=7000
+[helmet_icl] 69/250 trec_fine tokens=6800
+[helmet_icl] 70/250 trec_fine tokens=6829
+[helmet_icl] 71/250 trec_fine tokens=6971
+[helmet_icl] 72/250 trec_fine tokens=6907
+[helmet_icl] 73/250 trec_fine tokens=6818
+[helmet_icl] 74/250 trec_fine tokens=6902
+[helmet_icl] 75/250 trec_fine tokens=6941
+[helmet_icl] 76/250 trec_fine tokens=6830
+[helmet_icl] 77/250 trec_fine tokens=6934
+[helmet_icl] 78/250 trec_fine tokens=6879
+[helmet_icl] 79/250 trec_fine tokens=6915
+[helmet_icl] 80/250 trec_fine tokens=6953
+[helmet_icl] 81/250 trec_fine tokens=6875
+[helmet_icl] 82/250 trec_fine tokens=6847
+[helmet_icl] 83/250 trec_fine tokens=6847
+[helmet_icl] 84/250 trec_fine tokens=6944
+[helmet_icl] 85/250 trec_fine tokens=6818
+[helmet_icl] 86/250 trec_fine tokens=6868
+[helmet_icl] 87/250 trec_fine tokens=7022
+[helmet_icl] 88/250 trec_fine tokens=6930
+[helmet_icl] 89/250 trec_fine tokens=6863
+[helmet_icl] 90/250 trec_fine tokens=6807
+[helmet_icl] 91/250 trec_fine tokens=7001
+[helmet_icl] 92/250 trec_fine tokens=6890
+[helmet_icl] 93/250 trec_fine tokens=6952
+[helmet_icl] 94/250 trec_fine tokens=6845
+[helmet_icl] 95/250 trec_fine tokens=6902
+[helmet_icl] 96/250 trec_fine tokens=6879
+[helmet_icl] 97/250 trec_fine tokens=6896
+[helmet_icl] 98/250 trec_fine tokens=6770
+[helmet_icl] 99/250 trec_fine tokens=6933
+[helmet_icl] 100/250 trec_fine tokens=6886
+[helmet_icl] 101/250 banking77 tokens=6809
+[helmet_icl] 102/250 banking77 tokens=7162
+[helmet_icl] 103/250 banking77 tokens=6876
+[helmet_icl] 104/250 banking77 tokens=7033
+[helmet_icl] 105/250 banking77 tokens=6647
+[helmet_icl] 106/250 banking77 tokens=6818
+[helmet_icl] 107/250 banking77 tokens=6841
+[helmet_icl] 108/250 banking77 tokens=6944
+[helmet_icl] 109/250 banking77 tokens=6776
+[helmet_icl] 110/250 banking77 tokens=7100
+[helmet_icl] 111/250 banking77 tokens=6967
+[helmet_icl] 112/250 banking77 tokens=6894
+[helmet_icl] 113/250 banking77 tokens=6900
+[helmet_icl] 114/250 banking77 tokens=6964
+[helmet_icl] 115/250 banking77 tokens=7085
+[helmet_icl] 116/250 banking77 tokens=6953
+[helmet_icl] 117/250 banking77 tokens=7406
+[helmet_icl] 118/250 banking77 tokens=6802
+[helmet_icl] 119/250 banking77 tokens=7119
+[helmet_icl] 120/250 banking77 tokens=7202
+[helmet_icl] 121/250 banking77 tokens=7103
+[helmet_icl] 122/250 banking77 tokens=7009
+[helmet_icl] 123/250 banking77 tokens=6940
+[helmet_icl] 124/250 banking77 tokens=6857
+[helmet_icl] 125/250 banking77 tokens=6922
+[helmet_icl] 126/250 banking77 tokens=6988
+[helmet_icl] 127/250 banking77 tokens=6945
+[helmet_icl] 128/250 banking77 tokens=6835
+[helmet_icl] 129/250 banking77 tokens=7035
+[helmet_icl] 130/250 banking77 tokens=6787
+[helmet_icl] 131/250 banking77 tokens=6940
+[helmet_icl] 132/250 banking77 tokens=6661
+[helmet_icl] 133/250 banking77 tokens=6934
+[helmet_icl] 134/250 banking77 tokens=7235
+[helmet_icl] 135/250 banking77 tokens=6829
+[helmet_icl] 136/250 banking77 tokens=7169
+[helmet_icl] 137/250 banking77 tokens=7074
+[helmet_icl] 138/250 banking77 tokens=6797
+[helmet_icl] 139/250 banking77 tokens=7013
+[helmet_icl] 140/250 banking77 tokens=6811
+[helmet_icl] 141/250 banking77 tokens=6983
+[helmet_icl] 142/250 banking77 tokens=7183
+[helmet_icl] 143/250 banking77 tokens=6801
+[helmet_icl] 144/250 banking77 tokens=7174
+[helmet_icl] 145/250 banking77 tokens=6773
+[helmet_icl] 146/250 banking77 tokens=7134
+[helmet_icl] 147/250 banking77 tokens=6907
+[helmet_icl] 148/250 banking77 tokens=7341
+[helmet_icl] 149/250 banking77 tokens=7042
+[helmet_icl] 150/250 banking77 tokens=7304
+[helmet_icl] 151/250 clinic150 tokens=7241
+[helmet_icl] 152/250 clinic150 tokens=7282
+[helmet_icl] 153/250 clinic150 tokens=7261
+[helmet_icl] 154/250 clinic150 tokens=7292
+[helmet_icl] 155/250 clinic150 tokens=7217
+[helmet_icl] 156/250 clinic150 tokens=7280
+[helmet_icl] 157/250 clinic150 tokens=7293
+[helmet_icl] 158/250 clinic150 tokens=7250
+[helmet_icl] 159/250 clinic150 tokens=7264
+[helmet_icl] 160/250 clinic150 tokens=7238
+[helmet_icl] 161/250 clinic150 tokens=7243
+[helmet_icl] 162/250 clinic150 tokens=7191
+[helmet_icl] 163/250 clinic150 tokens=7333
+[helmet_icl] 164/250 clinic150 tokens=7299
+[helmet_icl] 165/250 clinic150 tokens=7225
+[helmet_icl] 166/250 clinic150 tokens=7409
+[helmet_icl] 167/250 clinic150 tokens=7277
+[helmet_icl] 168/250 clinic150 tokens=7376
+[helmet_icl] 169/250 clinic150 tokens=7383
+[helmet_icl] 170/250 clinic150 tokens=7270
+[helmet_icl] 171/250 clinic150 tokens=7173
+[helmet_icl] 172/250 clinic150 tokens=7106
+[helmet_icl] 173/250 clinic150 tokens=7413
+[helmet_icl] 174/250 clinic150 tokens=7253
+[helmet_icl] 175/250 clinic150 tokens=7224
+[helmet_icl] 176/250 clinic150 tokens=7307
+[helmet_icl] 177/250 clinic150 tokens=7272
+[helmet_icl] 178/250 clinic150 tokens=7215
+[helmet_icl] 179/250 clinic150 tokens=7344
+[helmet_icl] 180/250 clinic150 tokens=7224
+[helmet_icl] 181/250 clinic150 tokens=7310
+[helmet_icl] 182/250 clinic150 tokens=7222
+[helmet_icl] 183/250 clinic150 tokens=7260
+[helmet_icl] 184/250 clinic150 tokens=7351
+[helmet_icl] 185/250 clinic150 tokens=7301
+[helmet_icl] 186/250 clinic150 tokens=7155
+[helmet_icl] 187/250 clinic150 tokens=7253
+[helmet_icl] 188/250 clinic150 tokens=7248
+[helmet_icl] 189/250 clinic150 tokens=7128
+[helmet_icl] 190/250 clinic150 tokens=7282
+[helmet_icl] 191/250 clinic150 tokens=7301
+[helmet_icl] 192/250 clinic150 tokens=7388
+[helmet_icl] 193/250 clinic150 tokens=7244
+[helmet_icl] 194/250 clinic150 tokens=7186
+[helmet_icl] 195/250 clinic150 tokens=7309
+[helmet_icl] 196/250 clinic150 tokens=7241
+[helmet_icl] 197/250 clinic150 tokens=7214
+[helmet_icl] 198/250 clinic150 tokens=7264
+[helmet_icl] 199/250 clinic150 tokens=7219
+[helmet_icl] 200/250 clinic150 tokens=7228
+[helmet_icl] 201/250 nlu tokens=7063
+[helmet_icl] 202/250 nlu tokens=6959
+[helmet_icl] 203/250 nlu tokens=6985
+[helmet_icl] 204/250 nlu tokens=6970
+[helmet_icl] 205/250 nlu tokens=7025
+[helmet_icl] 206/250 nlu tokens=7034
+[helmet_icl] 207/250 nlu tokens=7004
+[helmet_icl] 208/250 nlu tokens=7062
+[helmet_icl] 209/250 nlu tokens=7062
+[helmet_icl] 210/250 nlu tokens=7045
+[helmet_icl] 211/250 nlu tokens=7028
+[helmet_icl] 212/250 nlu tokens=7018
+[helmet_icl] 213/250 nlu tokens=7075
+[helmet_icl] 214/250 nlu tokens=7068
+[helmet_icl] 215/250 nlu tokens=7025
+[helmet_icl] 216/250 nlu tokens=7022
+[helmet_icl] 217/250 nlu tokens=7112
+[helmet_icl] 218/250 nlu tokens=6967
+[helmet_icl] 219/250 nlu tokens=6980
+[helmet_icl] 220/250 nlu tokens=7031
+[helmet_icl] 221/250 nlu tokens=7093
+[helmet_icl] 222/250 nlu tokens=6949
+[helmet_icl] 223/250 nlu tokens=7046
+[helmet_icl] 224/250 nlu tokens=7039
+[helmet_icl] 225/250 nlu tokens=7126
+[helmet_icl] 226/250 nlu tokens=7064
+[helmet_icl] 227/250 nlu tokens=7057
+[helmet_icl] 228/250 nlu tokens=6971
+[helmet_icl] 229/250 nlu tokens=7034
+[helmet_icl] 230/250 nlu tokens=7094
+[helmet_icl] 231/250 nlu tokens=6979
+[helmet_icl] 232/250 nlu tokens=7123
+[helmet_icl] 233/250 nlu tokens=7144
+[helmet_icl] 234/250 nlu tokens=7024
+[helmet_icl] 235/250 nlu tokens=6943
+[helmet_icl] 236/250 nlu tokens=6961
+[helmet_icl] 237/250 nlu tokens=7005
+[helmet_icl] 238/250 nlu tokens=7080
+[helmet_icl] 239/250 nlu tokens=7179
+[helmet_icl] 240/250 nlu tokens=7115
+[helmet_icl] 241/250 nlu tokens=6946
+[helmet_icl] 242/250 nlu tokens=6985
+[helmet_icl] 243/250 nlu tokens=7131
+[helmet_icl] 244/250 nlu tokens=7100
+[helmet_icl] 245/250 nlu tokens=7190
+[helmet_icl] 246/250 nlu tokens=7040
+[helmet_icl] 247/250 nlu tokens=7013
+[helmet_icl] 248/250 nlu tokens=7080
+[helmet_icl] 249/250 nlu tokens=7098
+[helmet_icl] 250/250 nlu tokens=6891
+/workspace/outputs/eval/token_t0525/helmet_icl8k_n50.summary.csv

outputs/eval/logs/token_t0525/mrcr.log ADDED Viewed

	@@ -0,0 +1,32 @@

+[eval] stop_token_ids=[151645]
+[mrcr] 1/30 8k_2needle tokens=4852
+[mrcr] 2/30 8k_2needle tokens=7119
+[mrcr] 3/30 8k_2needle tokens=5756
+[mrcr] 4/30 8k_2needle tokens=5072
+[mrcr] 5/30 8k_2needle tokens=5753
+[mrcr] 6/30 8k_2needle tokens=5481
+[mrcr] 7/30 8k_2needle tokens=5143
+[mrcr] 8/30 8k_2needle tokens=6399
+[mrcr] 9/30 8k_2needle tokens=5987
+[mrcr] 10/30 8k_2needle tokens=4380
+[mrcr] 11/30 8k_4needle tokens=5123
+[mrcr] 12/30 8k_4needle tokens=7908
+[mrcr] 13/30 8k_4needle tokens=5814
+[mrcr] 14/30 8k_4needle tokens=4856
+[mrcr] 15/30 8k_4needle tokens=7413
+[mrcr] 16/30 8k_4needle tokens=7764
+[mrcr] 17/30 8k_4needle tokens=4312
+[mrcr] 18/30 8k_4needle tokens=4847
+[mrcr] 19/30 8k_4needle tokens=6287
+[mrcr] 20/30 8k_4needle tokens=7841
+[mrcr] 21/30 8k_8needle tokens=4581
+[mrcr] 22/30 8k_8needle tokens=6357
+[mrcr] 23/30 8k_8needle tokens=6981
+[mrcr] 24/30 8k_8needle tokens=6308
+[mrcr] 25/30 8k_8needle tokens=6901
+[mrcr] 26/30 8k_8needle tokens=5407
+[mrcr] 27/30 8k_8needle tokens=6351
+[mrcr] 28/30 8k_8needle tokens=5657
+[mrcr] 29/30 8k_8needle tokens=7330
+[mrcr] 30/30 8k_8needle tokens=8028
+/workspace/outputs/eval/token_t0525/mrcr_8k_2_4_8needle_n10.summary.csv

outputs/eval/logs/token_t0525/ruler_a.log ADDED Viewed

@@ -0,0 +1,340 @@
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/500 [00:00<?, ?it/s][A
 13%|█▎        | 66/500 [00:01<00:06, 65.54it/s][A
 26%|██▋       | 132/500 [00:02<00:05, 65.39it/s][A
 40%|███▉      | 198/500 [00:03<00:04, 64.83it/s][A
 53%|█████▎    | 265/500 [00:04<00:03, 65.67it/s][A
 66%|██████▋   | 332/500 [00:05<00:02, 65.84it/s][A
 80%|████████  | 400/500 [00:06<00:01, 66.25it/s][A
 94%|█████████▎| 468/500 [00:07<00:00, 66.59it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:12:44:28 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:12:44:32 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
+2026-07-18:12:44:34 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:12:44:34 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:12:44:37 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:12:44:37 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:12:44:37 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:12:44:46 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:44:46 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:12:44:55 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:45:04 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:45:11 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:45:20 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:45:20 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+Max length 500 | Current length 307 | Noises: 5
+Max length 500 | Current length 410 | Noises: 10
+Max length 500 | Current length 531 | Noises: 15
+Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
+Max length 8192 | Current length 791 | Noises: 10
+Max length 8192 | Current length 1025 | Noises: 20
+Max length 8192 | Current length 1273 | Noises: 30
+Max length 8192 | Current length 1510 | Noises: 40
+Max length 8192 | Current length 1749 | Noises: 50
+Max length 8192 | Current length 1996 | Noises: 60
+Max length 8192 | Current length 2231 | Noises: 70
+Max length 8192 | Current length 2473 | Noises: 80
+Max length 8192 | Current length 2705 | Noises: 90
+Max length 8192 | Current length 2953 | Noises: 100
+Max length 8192 | Current length 3186 | Noises: 110
+Max length 8192 | Current length 3431 | Noises: 120
+Max length 8192 | Current length 3671 | Noises: 130
+Max length 8192 | Current length 3916 | Noises: 140
+Max length 8192 | Current length 4149 | Noises: 150
+Max length 8192 | Current length 4386 | Noises: 160
+Max length 8192 | Current length 4630 | Noises: 170
+Max length 8192 | Current length 4871 | Noises: 180
+Max length 8192 | Current length 5110 | Noises: 190
+Max length 8192 | Current length 5349 | Noises: 200
+Max length 8192 | Current length 5590 | Noises: 210
+Max length 8192 | Current length 5832 | Noises: 220
+Max length 8192 | Current length 6070 | Noises: 230
+Max length 8192 | Current length 6311 | Noises: 240
+Max length 8192 | Current length 6544 | Noises: 250
+Max length 8192 | Current length 6791 | Noises: 260
+Max length 8192 | Current length 7031 | Noises: 270
+Max length 8192 | Current length 7271 | Noises: 280
+Max length 8192 | Current length 7513 | Noises: 290
+Max length 8192 | Current length 7751 | Noises: 300
+Max length 8192 | Current length 7986 | Noises: 310
+Max length 8192 | Current length 8230 | Noises: 320
+Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 13%|█▎        | 66/500 [00:01<00:06, 65.54it/s][A
 26%|██▋       | 132/500 [00:02<00:05, 65.39it/s][A
 40%|███▉      | 198/500 [00:03<00:04, 64.83it/s][A
 53%|█████▎    | 265/500 [00:04<00:03, 65.67it/s][A
 66%|██████▋   | 332/500 [00:05<00:02, 65.84it/s][A
 80%|████████  | 400/500 [00:06<00:01, 66.25it/s][A
 94%|█████████▎| 468/500 [00:07<00:00, 66.59it/s][A
+2026-07-18:12:45:28 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:45:41 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:46:03 INFO     [tasks:700] Selected tasks:
+2026-07-18:12:46:03 INFO     [tasks:691] Task: ruler_qa_hotpot (ruler/qa_hotpot.yaml)
+2026-07-18:12:46:03 INFO     [tasks:691] Task: ruler_fwe (ruler/fwe.yaml)
+2026-07-18:12:46:03 INFO     [tasks:691] Task: ruler_vt (ruler/vt.yaml)
+2026-07-18:12:46:03 INFO     [tasks:691] Task: niah_multiquery (ruler/niah_multiquery.yaml)
+2026-07-18:12:46:03 INFO     [tasks:691] Task: niah_multikey_2 (ruler/niah_multikey_2.yaml)
+2026-07-18:12:46:03 INFO     [tasks:691] Task: niah_single_3 (ruler/niah_single_3.yaml)
+2026-07-18:12:46:03 INFO     [tasks:691] Task: niah_single_1 (ruler/niah_single_1.yaml)
+2026-07-18:12:46:03 INFO     [evaluator:314] ruler_qa_hotpot: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:12:46:03 INFO     [evaluator:314] ruler_fwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 50, 'until': []}
+2026-07-18:12:46:03 INFO     [evaluator:314] ruler_vt: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 30, 'until': []}
+2026-07-18:12:46:03 INFO     [evaluator:314] niah_multiquery: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:46:03 INFO     [evaluator:314] niah_multikey_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:46:03 INFO     [evaluator:314] niah_single_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:46:03 INFO     [evaluator:314] niah_single_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:46:03 INFO     [api.task:311] Building contexts for ruler_qa_hotpot on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:46:03 INFO     [api.task:311] Building contexts for ruler_fwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:46:03 INFO     [api.task:311] Building contexts for ruler_vt on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:46:03 INFO     [api.task:311] Building contexts for niah_multiquery on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:46:03 INFO     [api.task:311] Building contexts for niah_multikey_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:46:03 INFO     [api.task:311] Building contexts for niah_single_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:46:03 INFO     [api.task:311] Building contexts for niah_single_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:46:03 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:12:56:47 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:12:56:47 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t0525/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_2|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_multiquery|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_single_1  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_single_3  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|ruler_fwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7667|±  |   N/A|
+|ruler_qa_hotpot|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4000|±  |   N/A|
+|ruler_vt       |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.8800|±  |   N/A|

outputs/eval/logs/token_t0525/ruler_b.log ADDED Viewed

@@ -0,0 +1,253 @@
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:12:56:49 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:12:56:53 INFO     [_cli.run:376] Selected Tasks: ['niah_single_2', 'niah_multikey_1', 'niah_multikey_3', 'niah_multivalue', 'ruler_cwe', 'ruler_qa_squad']
+2026-07-18:12:56:54 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:12:56:54 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:12:56:58 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:12:56:58 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:12:56:58 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:12:57:07 WARNING  [api.task:856] niah_single_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:57:08 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:12:57:18 WARNING  [api.task:856] niah_multikey_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:57:28 WARNING  [api.task:856] niah_multikey_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:57:34 WARNING  [api.task:856] niah_multivalue: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:57:44 WARNING  [api.task:856] ruler_cwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:57:44 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:12:57:52 WARNING  [api.task:856] ruler_qa_squad: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:12:58:03 INFO     [tasks:700] Selected tasks:
+2026-07-18:12:58:03 INFO     [tasks:691] Task: ruler_qa_squad (ruler/qa_squad.yaml)
+2026-07-18:12:58:03 INFO     [tasks:691] Task: ruler_cwe (ruler/cwe.yaml)
+2026-07-18:12:58:03 INFO     [tasks:691] Task: niah_multivalue (ruler/niah_multivalue.yaml)
+2026-07-18:12:58:03 INFO     [tasks:691] Task: niah_multikey_3 (ruler/niah_multikey_3.yaml)
+2026-07-18:12:58:03 INFO     [tasks:691] Task: niah_multikey_1 (ruler/niah_multikey_1.yaml)
+2026-07-18:12:58:03 INFO     [tasks:691] Task: niah_single_2 (ruler/niah_single_2.yaml)
+2026-07-18:12:58:03 INFO     [evaluator:314] ruler_qa_squad: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:12:58:03 INFO     [evaluator:314] ruler_cwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 120, 'until': []}
+2026-07-18:12:58:03 INFO     [evaluator:314] niah_multivalue: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:58:03 INFO     [evaluator:314] niah_multikey_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:58:03 INFO     [evaluator:314] niah_multikey_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:58:03 INFO     [evaluator:314] niah_single_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:12:58:03 INFO     [api.task:311] Building contexts for ruler_qa_squad on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:58:03 INFO     [api.task:311] Building contexts for ruler_cwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:58:03 INFO     [api.task:311] Building contexts for niah_multivalue on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:58:03 INFO     [api.task:311] Building contexts for niah_multikey_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:58:03 INFO     [api.task:311] Building contexts for niah_multikey_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:58:03 INFO     [api.task:311] Building contexts for niah_single_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:12:58:03 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:13:09:02 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:13:09:02 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t0525/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_1|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_multikey_3|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7500|±  |   N/A|
+|niah_multivalue|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|niah_single_2  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 1.0000|±  |   N/A|
+|ruler_cwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4050|±  |   N/A|
+|ruler_qa_squad |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4208|±  |   N/A|

outputs/eval/logs/token_t0575/babilong.log ADDED Viewed

@@ -0,0 +1,314 @@
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A

+2026-07-18:13:56:15 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:13:56:19 INFO     [_cli.run:376] Selected Tasks: ['babilong_longctx']
+2026-07-18:13:56:20 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:13:56:20 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:13:56:23 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:13:56:23 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:13:56:24 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:13:56:25 WARNING  [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:56:26 WARNING  [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:56:26 WARNING  [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:56:26 WARNING  [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:56:26 WARNING  [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:56:26 INFO     [tasks:700] Selected tasks:
+2026-07-18:13:56:26 INFO     [tasks:703] Group: babilong_longctx
+2026-07-18:13:56:26 INFO     [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
+2026-07-18:13:56:26 INFO     [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:56:26 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
+2026-07-18:13:56:26 INFO     [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:56:26 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
+2026-07-18:13:56:26 INFO     [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:56:26 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
+2026-07-18:13:56:26 INFO     [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:56:26 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
+2026-07-18:13:56:26 INFO     [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:13:56:26 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
+2026-07-18:13:56:26 INFO     [api.task:311] Building contexts for babilong_qa1 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:56:26 INFO     [api.task:311] Building contexts for babilong_qa2 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:56:26 INFO     [api.task:311] Building contexts for babilong_qa3 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:56:27 INFO     [api.task:311] Building contexts for babilong_qa4 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:56:27 INFO     [api.task:311] Building contexts for babilong_qa5 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:13:56:27 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:14:02:04 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:14:02:04 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t0575/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
+|     Tasks      |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|-----:|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |0.592|±  |0.0298|
+| - babilong_qa1 |      0|none  |     2|acc   |↑  |0.760|±  |0.0610|
+| - babilong_qa2 |      0|none  |     2|acc   |↑  |0.500|±  |0.0714|
+| - babilong_qa3 |      0|none  |     2|acc   |↑  |0.340|±  |0.0677|
+| - babilong_qa4 |      0|none  |     2|acc   |↑  |0.640|±  |0.0686|
+| - babilong_qa5 |      0|none  |     2|acc   |↑  |0.720|±  |0.0641|
+|     Groups     |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|------|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |0.592|±  |0.0298|

outputs/eval/logs/token_t0575/helmet.log ADDED Viewed

	@@ -0,0 +1,252 @@

+[eval] stop_token_ids=[151645]
+[helmet_icl] 1/250 trec_coarse tokens=6597
+[helmet_icl] 2/250 trec_coarse tokens=6549
+[helmet_icl] 3/250 trec_coarse tokens=6508
+[helmet_icl] 4/250 trec_coarse tokens=6608
+[helmet_icl] 5/250 trec_coarse tokens=6497
+[helmet_icl] 6/250 trec_coarse tokens=6545
+[helmet_icl] 7/250 trec_coarse tokens=6583
+[helmet_icl] 8/250 trec_coarse tokens=6455
+[helmet_icl] 9/250 trec_coarse tokens=6643
+[helmet_icl] 10/250 trec_coarse tokens=6568
+[helmet_icl] 11/250 trec_coarse tokens=6518
+[helmet_icl] 12/250 trec_coarse tokens=6424
+[helmet_icl] 13/250 trec_coarse tokens=6750
+[helmet_icl] 14/250 trec_coarse tokens=6642
+[helmet_icl] 15/250 trec_coarse tokens=6494
+[helmet_icl] 16/250 trec_coarse tokens=6427
+[helmet_icl] 17/250 trec_coarse tokens=6547
+[helmet_icl] 18/250 trec_coarse tokens=6612
+[helmet_icl] 19/250 trec_coarse tokens=6591
+[helmet_icl] 20/250 trec_coarse tokens=6485
+[helmet_icl] 21/250 trec_coarse tokens=6542
+[helmet_icl] 22/250 trec_coarse tokens=6409
+[helmet_icl] 23/250 trec_coarse tokens=6482
+[helmet_icl] 24/250 trec_coarse tokens=6502
+[helmet_icl] 25/250 trec_coarse tokens=6344
+[helmet_icl] 26/250 trec_coarse tokens=6479
+[helmet_icl] 27/250 trec_coarse tokens=6569
+[helmet_icl] 28/250 trec_coarse tokens=6389
+[helmet_icl] 29/250 trec_coarse tokens=6479
+[helmet_icl] 30/250 trec_coarse tokens=6653
+[helmet_icl] 31/250 trec_coarse tokens=6532
+[helmet_icl] 32/250 trec_coarse tokens=6491
+[helmet_icl] 33/250 trec_coarse tokens=6489
+[helmet_icl] 34/250 trec_coarse tokens=6630
+[helmet_icl] 35/250 trec_coarse tokens=6488
+[helmet_icl] 36/250 trec_coarse tokens=6499
+[helmet_icl] 37/250 trec_coarse tokens=6483
+[helmet_icl] 38/250 trec_coarse tokens=6660
+[helmet_icl] 39/250 trec_coarse tokens=6675
+[helmet_icl] 40/250 trec_coarse tokens=6544
+[helmet_icl] 41/250 trec_coarse tokens=6520
+[helmet_icl] 42/250 trec_coarse tokens=6359
+[helmet_icl] 43/250 trec_coarse tokens=6657
+[helmet_icl] 44/250 trec_coarse tokens=6546
+[helmet_icl] 45/250 trec_coarse tokens=6481
+[helmet_icl] 46/250 trec_coarse tokens=6506
+[helmet_icl] 47/250 trec_coarse tokens=6621
+[helmet_icl] 48/250 trec_coarse tokens=6623
+[helmet_icl] 49/250 trec_coarse tokens=6411
+[helmet_icl] 50/250 trec_coarse tokens=6577
+[helmet_icl] 51/250 trec_fine tokens=6858
+[helmet_icl] 52/250 trec_fine tokens=6973
+[helmet_icl] 53/250 trec_fine tokens=6977
+[helmet_icl] 54/250 trec_fine tokens=6952
+[helmet_icl] 55/250 trec_fine tokens=6919
+[helmet_icl] 56/250 trec_fine tokens=6979
+[helmet_icl] 57/250 trec_fine tokens=6957
+[helmet_icl] 58/250 trec_fine tokens=7013
+[helmet_icl] 59/250 trec_fine tokens=6824
+[helmet_icl] 60/250 trec_fine tokens=6833
+[helmet_icl] 61/250 trec_fine tokens=6868
+[helmet_icl] 62/250 trec_fine tokens=6914
+[helmet_icl] 63/250 trec_fine tokens=6759
+[helmet_icl] 64/250 trec_fine tokens=6933
+[helmet_icl] 65/250 trec_fine tokens=6868
+[helmet_icl] 66/250 trec_fine tokens=6839
+[helmet_icl] 67/250 trec_fine tokens=6838
+[helmet_icl] 68/250 trec_fine tokens=7000
+[helmet_icl] 69/250 trec_fine tokens=6800
+[helmet_icl] 70/250 trec_fine tokens=6829
+[helmet_icl] 71/250 trec_fine tokens=6971
+[helmet_icl] 72/250 trec_fine tokens=6907
+[helmet_icl] 73/250 trec_fine tokens=6818
+[helmet_icl] 74/250 trec_fine tokens=6902
+[helmet_icl] 75/250 trec_fine tokens=6941
+[helmet_icl] 76/250 trec_fine tokens=6830
+[helmet_icl] 77/250 trec_fine tokens=6934
+[helmet_icl] 78/250 trec_fine tokens=6879
+[helmet_icl] 79/250 trec_fine tokens=6915
+[helmet_icl] 80/250 trec_fine tokens=6953
+[helmet_icl] 81/250 trec_fine tokens=6875
+[helmet_icl] 82/250 trec_fine tokens=6847
+[helmet_icl] 83/250 trec_fine tokens=6847
+[helmet_icl] 84/250 trec_fine tokens=6944
+[helmet_icl] 85/250 trec_fine tokens=6818
+[helmet_icl] 86/250 trec_fine tokens=6868
+[helmet_icl] 87/250 trec_fine tokens=7022
+[helmet_icl] 88/250 trec_fine tokens=6930
+[helmet_icl] 89/250 trec_fine tokens=6863
+[helmet_icl] 90/250 trec_fine tokens=6807
+[helmet_icl] 91/250 trec_fine tokens=7001
+[helmet_icl] 92/250 trec_fine tokens=6890
+[helmet_icl] 93/250 trec_fine tokens=6952
+[helmet_icl] 94/250 trec_fine tokens=6845
+[helmet_icl] 95/250 trec_fine tokens=6902
+[helmet_icl] 96/250 trec_fine tokens=6879
+[helmet_icl] 97/250 trec_fine tokens=6896
+[helmet_icl] 98/250 trec_fine tokens=6770
+[helmet_icl] 99/250 trec_fine tokens=6933
+[helmet_icl] 100/250 trec_fine tokens=6886
+[helmet_icl] 101/250 banking77 tokens=6809
+[helmet_icl] 102/250 banking77 tokens=7162
+[helmet_icl] 103/250 banking77 tokens=6876
+[helmet_icl] 104/250 banking77 tokens=7033
+[helmet_icl] 105/250 banking77 tokens=6647
+[helmet_icl] 106/250 banking77 tokens=6818
+[helmet_icl] 107/250 banking77 tokens=6841
+[helmet_icl] 108/250 banking77 tokens=6944
+[helmet_icl] 109/250 banking77 tokens=6776
+[helmet_icl] 110/250 banking77 tokens=7100
+[helmet_icl] 111/250 banking77 tokens=6967
+[helmet_icl] 112/250 banking77 tokens=6894
+[helmet_icl] 113/250 banking77 tokens=6900
+[helmet_icl] 114/250 banking77 tokens=6964
+[helmet_icl] 115/250 banking77 tokens=7085
+[helmet_icl] 116/250 banking77 tokens=6953
+[helmet_icl] 117/250 banking77 tokens=7406
+[helmet_icl] 118/250 banking77 tokens=6802
+[helmet_icl] 119/250 banking77 tokens=7119
+[helmet_icl] 120/250 banking77 tokens=7202
+[helmet_icl] 121/250 banking77 tokens=7103
+[helmet_icl] 122/250 banking77 tokens=7009
+[helmet_icl] 123/250 banking77 tokens=6940
+[helmet_icl] 124/250 banking77 tokens=6857
+[helmet_icl] 125/250 banking77 tokens=6922
+[helmet_icl] 126/250 banking77 tokens=6988
+[helmet_icl] 127/250 banking77 tokens=6945
+[helmet_icl] 128/250 banking77 tokens=6835
+[helmet_icl] 129/250 banking77 tokens=7035
+[helmet_icl] 130/250 banking77 tokens=6787
+[helmet_icl] 131/250 banking77 tokens=6940
+[helmet_icl] 132/250 banking77 tokens=6661
+[helmet_icl] 133/250 banking77 tokens=6934
+[helmet_icl] 134/250 banking77 tokens=7235
+[helmet_icl] 135/250 banking77 tokens=6829
+[helmet_icl] 136/250 banking77 tokens=7169
+[helmet_icl] 137/250 banking77 tokens=7074
+[helmet_icl] 138/250 banking77 tokens=6797
+[helmet_icl] 139/250 banking77 tokens=7013
+[helmet_icl] 140/250 banking77 tokens=6811
+[helmet_icl] 141/250 banking77 tokens=6983
+[helmet_icl] 142/250 banking77 tokens=7183
+[helmet_icl] 143/250 banking77 tokens=6801
+[helmet_icl] 144/250 banking77 tokens=7174
+[helmet_icl] 145/250 banking77 tokens=6773
+[helmet_icl] 146/250 banking77 tokens=7134
+[helmet_icl] 147/250 banking77 tokens=6907
+[helmet_icl] 148/250 banking77 tokens=7341
+[helmet_icl] 149/250 banking77 tokens=7042
+[helmet_icl] 150/250 banking77 tokens=7304
+[helmet_icl] 151/250 clinic150 tokens=7241
+[helmet_icl] 152/250 clinic150 tokens=7282
+[helmet_icl] 153/250 clinic150 tokens=7261
+[helmet_icl] 154/250 clinic150 tokens=7292
+[helmet_icl] 155/250 clinic150 tokens=7217
+[helmet_icl] 156/250 clinic150 tokens=7280
+[helmet_icl] 157/250 clinic150 tokens=7293
+[helmet_icl] 158/250 clinic150 tokens=7250
+[helmet_icl] 159/250 clinic150 tokens=7264
+[helmet_icl] 160/250 clinic150 tokens=7238
+[helmet_icl] 161/250 clinic150 tokens=7243
+[helmet_icl] 162/250 clinic150 tokens=7191
+[helmet_icl] 163/250 clinic150 tokens=7333
+[helmet_icl] 164/250 clinic150 tokens=7299
+[helmet_icl] 165/250 clinic150 tokens=7225
+[helmet_icl] 166/250 clinic150 tokens=7409
+[helmet_icl] 167/250 clinic150 tokens=7277
+[helmet_icl] 168/250 clinic150 tokens=7376
+[helmet_icl] 169/250 clinic150 tokens=7383
+[helmet_icl] 170/250 clinic150 tokens=7270
+[helmet_icl] 171/250 clinic150 tokens=7173
+[helmet_icl] 172/250 clinic150 tokens=7106
+[helmet_icl] 173/250 clinic150 tokens=7413
+[helmet_icl] 174/250 clinic150 tokens=7253
+[helmet_icl] 175/250 clinic150 tokens=7224
+[helmet_icl] 176/250 clinic150 tokens=7307
+[helmet_icl] 177/250 clinic150 tokens=7272
+[helmet_icl] 178/250 clinic150 tokens=7215
+[helmet_icl] 179/250 clinic150 tokens=7344
+[helmet_icl] 180/250 clinic150 tokens=7224
+[helmet_icl] 181/250 clinic150 tokens=7310
+[helmet_icl] 182/250 clinic150 tokens=7222
+[helmet_icl] 183/250 clinic150 tokens=7260
+[helmet_icl] 184/250 clinic150 tokens=7351
+[helmet_icl] 185/250 clinic150 tokens=7301
+[helmet_icl] 186/250 clinic150 tokens=7155
+[helmet_icl] 187/250 clinic150 tokens=7253
+[helmet_icl] 188/250 clinic150 tokens=7248
+[helmet_icl] 189/250 clinic150 tokens=7128
+[helmet_icl] 190/250 clinic150 tokens=7282
+[helmet_icl] 191/250 clinic150 tokens=7301
+[helmet_icl] 192/250 clinic150 tokens=7388
+[helmet_icl] 193/250 clinic150 tokens=7244
+[helmet_icl] 194/250 clinic150 tokens=7186
+[helmet_icl] 195/250 clinic150 tokens=7309
+[helmet_icl] 196/250 clinic150 tokens=7241
+[helmet_icl] 197/250 clinic150 tokens=7214
+[helmet_icl] 198/250 clinic150 tokens=7264
+[helmet_icl] 199/250 clinic150 tokens=7219
+[helmet_icl] 200/250 clinic150 tokens=7228
+[helmet_icl] 201/250 nlu tokens=7063
+[helmet_icl] 202/250 nlu tokens=6959
+[helmet_icl] 203/250 nlu tokens=6985
+[helmet_icl] 204/250 nlu tokens=6970
+[helmet_icl] 205/250 nlu tokens=7025
+[helmet_icl] 206/250 nlu tokens=7034
+[helmet_icl] 207/250 nlu tokens=7004
+[helmet_icl] 208/250 nlu tokens=7062
+[helmet_icl] 209/250 nlu tokens=7062
+[helmet_icl] 210/250 nlu tokens=7045
+[helmet_icl] 211/250 nlu tokens=7028
+[helmet_icl] 212/250 nlu tokens=7018
+[helmet_icl] 213/250 nlu tokens=7075
+[helmet_icl] 214/250 nlu tokens=7068
+[helmet_icl] 215/250 nlu tokens=7025
+[helmet_icl] 216/250 nlu tokens=7022
+[helmet_icl] 217/250 nlu tokens=7112
+[helmet_icl] 218/250 nlu tokens=6967
+[helmet_icl] 219/250 nlu tokens=6980
+[helmet_icl] 220/250 nlu tokens=7031
+[helmet_icl] 221/250 nlu tokens=7093
+[helmet_icl] 222/250 nlu tokens=6949
+[helmet_icl] 223/250 nlu tokens=7046
+[helmet_icl] 224/250 nlu tokens=7039
+[helmet_icl] 225/250 nlu tokens=7126
+[helmet_icl] 226/250 nlu tokens=7064
+[helmet_icl] 227/250 nlu tokens=7057
+[helmet_icl] 228/250 nlu tokens=6971
+[helmet_icl] 229/250 nlu tokens=7034
+[helmet_icl] 230/250 nlu tokens=7094
+[helmet_icl] 231/250 nlu tokens=6979
+[helmet_icl] 232/250 nlu tokens=7123
+[helmet_icl] 233/250 nlu tokens=7144
+[helmet_icl] 234/250 nlu tokens=7024
+[helmet_icl] 235/250 nlu tokens=6943
+[helmet_icl] 236/250 nlu tokens=6961
+[helmet_icl] 237/250 nlu tokens=7005
+[helmet_icl] 238/250 nlu tokens=7080
+[helmet_icl] 239/250 nlu tokens=7179
+[helmet_icl] 240/250 nlu tokens=7115
+[helmet_icl] 241/250 nlu tokens=6946
+[helmet_icl] 242/250 nlu tokens=6985
+[helmet_icl] 243/250 nlu tokens=7131
+[helmet_icl] 244/250 nlu tokens=7100
+[helmet_icl] 245/250 nlu tokens=7190
+[helmet_icl] 246/250 nlu tokens=7040
+[helmet_icl] 247/250 nlu tokens=7013
+[helmet_icl] 248/250 nlu tokens=7080
+[helmet_icl] 249/250 nlu tokens=7098
+[helmet_icl] 250/250 nlu tokens=6891
+/workspace/outputs/eval/token_t0575/helmet_icl8k_n50.summary.csv

outputs/eval/logs/token_t0575/mrcr.log ADDED Viewed

	@@ -0,0 +1,32 @@

+[eval] stop_token_ids=[151645]
+[mrcr] 1/30 8k_2needle tokens=4852
+[mrcr] 2/30 8k_2needle tokens=7119
+[mrcr] 3/30 8k_2needle tokens=5756
+[mrcr] 4/30 8k_2needle tokens=5072
+[mrcr] 5/30 8k_2needle tokens=5753
+[mrcr] 6/30 8k_2needle tokens=5481
+[mrcr] 7/30 8k_2needle tokens=5143
+[mrcr] 8/30 8k_2needle tokens=6399
+[mrcr] 9/30 8k_2needle tokens=5987
+[mrcr] 10/30 8k_2needle tokens=4380
+[mrcr] 11/30 8k_4needle tokens=5123
+[mrcr] 12/30 8k_4needle tokens=7908
+[mrcr] 13/30 8k_4needle tokens=5814
+[mrcr] 14/30 8k_4needle tokens=4856
+[mrcr] 15/30 8k_4needle tokens=7413
+[mrcr] 16/30 8k_4needle tokens=7764
+[mrcr] 17/30 8k_4needle tokens=4312
+[mrcr] 18/30 8k_4needle tokens=4847
+[mrcr] 19/30 8k_4needle tokens=6287
+[mrcr] 20/30 8k_4needle tokens=7841
+[mrcr] 21/30 8k_8needle tokens=4581
+[mrcr] 22/30 8k_8needle tokens=6357
+[mrcr] 23/30 8k_8needle tokens=6981
+[mrcr] 24/30 8k_8needle tokens=6308
+[mrcr] 25/30 8k_8needle tokens=6901
+[mrcr] 26/30 8k_8needle tokens=5407
+[mrcr] 27/30 8k_8needle tokens=6351
+[mrcr] 28/30 8k_8needle tokens=5657
+[mrcr] 29/30 8k_8needle tokens=7330
+[mrcr] 30/30 8k_8needle tokens=8028
+/workspace/outputs/eval/token_t0575/mrcr_8k_2_4_8needle_n10.summary.csv

outputs/eval/logs/token_t0575/ruler_a.log ADDED Viewed

@@ -0,0 +1,344 @@
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 61/500 [00:01<00:07, 60.58it/s][A
 24%|██▍       | 122/500 [00:02<00:06, 59.79it/s][A
 37%|███▋      | 183/500 [00:03<00:05, 60.19it/s][A
 49%|████▉     | 244/500 [00:04<00:04, 60.16it/s][A
 61%|██████    | 305/500 [00:05<00:03, 60.23it/s][A
 73%|███████▎  | 366/500 [00:06<00:02, 59.86it/s][A
 85%|████████▌ | 426/500 [00:07<00:01, 59.33it/s][A
 97%|█████████▋| 486/500 [00:08<00:00, 58.71it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:13:27:51 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:13:27:55 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
+2026-07-18:13:27:57 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:13:27:57 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:13:28:00 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:13:28:00 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:13:28:00 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:13:28:09 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:28:09 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:13:28:18 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:28:28 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:28:35 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:28:45 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:28:45 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+Max length 500 | Current length 307 | Noises: 5
+Max length 500 | Current length 410 | Noises: 10
+Max length 500 | Current length 531 | Noises: 15
+Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
+Max length 8192 | Current length 791 | Noises: 10
+Max length 8192 | Current length 1025 | Noises: 20
+Max length 8192 | Current length 1273 | Noises: 30
+Max length 8192 | Current length 1510 | Noises: 40
+Max length 8192 | Current length 1749 | Noises: 50
+Max length 8192 | Current length 1996 | Noises: 60
+Max length 8192 | Current length 2231 | Noises: 70
+Max length 8192 | Current length 2473 | Noises: 80
+Max length 8192 | Current length 2705 | Noises: 90
+Max length 8192 | Current length 2953 | Noises: 100
+Max length 8192 | Current length 3186 | Noises: 110
+Max length 8192 | Current length 3431 | Noises: 120
+Max length 8192 | Current length 3671 | Noises: 130
+Max length 8192 | Current length 3916 | Noises: 140
+Max length 8192 | Current length 4149 | Noises: 150
+Max length 8192 | Current length 4386 | Noises: 160
+Max length 8192 | Current length 4630 | Noises: 170
+Max length 8192 | Current length 4871 | Noises: 180
+Max length 8192 | Current length 5110 | Noises: 190
+Max length 8192 | Current length 5349 | Noises: 200
+Max length 8192 | Current length 5590 | Noises: 210
+Max length 8192 | Current length 5832 | Noises: 220
+Max length 8192 | Current length 6070 | Noises: 230
+Max length 8192 | Current length 6311 | Noises: 240
+Max length 8192 | Current length 6544 | Noises: 250
+Max length 8192 | Current length 6791 | Noises: 260
+Max length 8192 | Current length 7031 | Noises: 270
+Max length 8192 | Current length 7271 | Noises: 280
+Max length 8192 | Current length 7513 | Noises: 290
+Max length 8192 | Current length 7751 | Noises: 300
+Max length 8192 | Current length 7986 | Noises: 310
+Max length 8192 | Current length 8230 | Noises: 320
+Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 61/500 [00:01<00:07, 60.58it/s][A
 24%|██▍       | 122/500 [00:02<00:06, 59.79it/s][A
 37%|███▋      | 183/500 [00:03<00:05, 60.19it/s][A
 49%|████▉     | 244/500 [00:04<00:04, 60.16it/s][A
 61%|██████    | 305/500 [00:05<00:03, 60.23it/s][A
 73%|███████▎  | 366/500 [00:06<00:02, 59.86it/s][A
 85%|████████▌ | 426/500 [00:07<00:01, 59.33it/s][A
 97%|█████████▋| 486/500 [00:08<00:00, 58.71it/s][A
+2026-07-18:13:28:54 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:29:07 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:29:28 INFO     [tasks:700] Selected tasks:
+2026-07-18:13:29:28 INFO     [tasks:691] Task: ruler_qa_hotpot (ruler/qa_hotpot.yaml)
+2026-07-18:13:29:28 INFO     [tasks:691] Task: ruler_fwe (ruler/fwe.yaml)
+2026-07-18:13:29:28 INFO     [tasks:691] Task: ruler_vt (ruler/vt.yaml)
+2026-07-18:13:29:28 INFO     [tasks:691] Task: niah_multiquery (ruler/niah_multiquery.yaml)
+2026-07-18:13:29:28 INFO     [tasks:691] Task: niah_multikey_2 (ruler/niah_multikey_2.yaml)
+2026-07-18:13:29:28 INFO     [tasks:691] Task: niah_single_3 (ruler/niah_single_3.yaml)
+2026-07-18:13:29:28 INFO     [tasks:691] Task: niah_single_1 (ruler/niah_single_1.yaml)
+2026-07-18:13:29:28 INFO     [evaluator:314] ruler_qa_hotpot: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:13:29:28 INFO     [evaluator:314] ruler_fwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 50, 'until': []}
+2026-07-18:13:29:28 INFO     [evaluator:314] ruler_vt: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 30, 'until': []}
+2026-07-18:13:29:28 INFO     [evaluator:314] niah_multiquery: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:13:29:28 INFO     [evaluator:314] niah_multikey_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:13:29:28 INFO     [evaluator:314] niah_single_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:13:29:28 INFO     [evaluator:314] niah_single_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:13:29:28 INFO     [api.task:311] Building contexts for ruler_qa_hotpot on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:29:28 INFO     [api.task:311] Building contexts for ruler_fwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:29:28 INFO     [api.task:311] Building contexts for ruler_vt on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:29:28 INFO     [api.task:311] Building contexts for niah_multiquery on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:29:28 INFO     [api.task:311] Building contexts for niah_multikey_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:29:28 INFO     [api.task:311] Building contexts for niah_single_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:29:28 INFO     [api.task:311] Building contexts for niah_single_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:29:28 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:13:42:41 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:13:42:41 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t0575/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_2|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.9500|±  |   N/A|
+|niah_multiquery|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7500|±  |   N/A|
+|niah_single_1  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.0500|±  |   N/A|
+|niah_single_3  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7000|±  |   N/A|
+|ruler_fwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.7667|±  |   N/A|
+|ruler_qa_hotpot|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.3000|±  |   N/A|
+|ruler_vt       |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.8300|±  |   N/A|

outputs/eval/logs/token_t0575/ruler_b.log ADDED Viewed

@@ -0,0 +1,249 @@
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:13:42:44 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:13:42:48 INFO     [_cli.run:376] Selected Tasks: ['niah_single_2', 'niah_multikey_1', 'niah_multikey_3', 'niah_multivalue', 'ruler_cwe', 'ruler_qa_squad']
+2026-07-18:13:42:49 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:13:42:49 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:13:42:52 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:13:42:52 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:13:42:53 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:13:43:01 WARNING  [api.task:856] niah_single_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:43:02 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:13:43:12 WARNING  [api.task:856] niah_multikey_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:43:21 WARNING  [api.task:856] niah_multikey_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:43:27 WARNING  [api.task:856] niah_multivalue: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:43:37 WARNING  [api.task:856] ruler_cwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:43:37 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:13:43:44 WARNING  [api.task:856] ruler_qa_squad: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:13:43:54 INFO     [tasks:700] Selected tasks:
+2026-07-18:13:43:54 INFO     [tasks:691] Task: ruler_qa_squad (ruler/qa_squad.yaml)
+2026-07-18:13:43:54 INFO     [tasks:691] Task: ruler_cwe (ruler/cwe.yaml)
+2026-07-18:13:43:54 INFO     [tasks:691] Task: niah_multivalue (ruler/niah_multivalue.yaml)
+2026-07-18:13:43:54 INFO     [tasks:691] Task: niah_multikey_3 (ruler/niah_multikey_3.yaml)
+2026-07-18:13:43:54 INFO     [tasks:691] Task: niah_multikey_1 (ruler/niah_multikey_1.yaml)
+2026-07-18:13:43:54 INFO     [tasks:691] Task: niah_single_2 (ruler/niah_single_2.yaml)
+2026-07-18:13:43:54 INFO     [evaluator:314] ruler_qa_squad: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:13:43:54 INFO     [evaluator:314] ruler_cwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 120, 'until': []}
+2026-07-18:13:43:54 INFO     [evaluator:314] niah_multivalue: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:13:43:54 INFO     [evaluator:314] niah_multikey_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:13:43:54 INFO     [evaluator:314] niah_multikey_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:13:43:54 INFO     [evaluator:314] niah_single_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:13:43:54 INFO     [api.task:311] Building contexts for ruler_qa_squad on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:43:54 INFO     [api.task:311] Building contexts for ruler_cwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:43:54 INFO     [api.task:311] Building contexts for niah_multivalue on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:43:55 INFO     [api.task:311] Building contexts for niah_multikey_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:43:55 INFO     [api.task:311] Building contexts for niah_multikey_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:43:55 INFO     [api.task:311] Building contexts for niah_single_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:13:43:55 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:13:56:13 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:13:56:13 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t0575/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_1|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.8000|±  |   N/A|
+|niah_multikey_3|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.6500|±  |   N/A|
+|niah_multivalue|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.4875|±  |   N/A|
+|niah_single_2  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.2000|±  |   N/A|
+|ruler_cwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.3300|±  |   N/A|
+|ruler_qa_squad |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.3542|±  |   N/A|

outputs/eval/logs/token_t065/babilong.log ADDED Viewed

@@ -0,0 +1,314 @@
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A

+2026-07-18:14:39:52 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:14:39:56 INFO     [_cli.run:376] Selected Tasks: ['babilong_longctx']
+2026-07-18:14:39:57 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:14:39:57 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:14:40:00 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:14:40:00 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:14:40:00 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:14:40:02 WARNING  [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:40:02 WARNING  [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:40:02 WARNING  [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:40:03 WARNING  [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:40:03 WARNING  [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:40:03 INFO     [tasks:700] Selected tasks:
+2026-07-18:14:40:03 INFO     [tasks:703] Group: babilong_longctx
+2026-07-18:14:40:03 INFO     [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
+2026-07-18:14:40:03 INFO     [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:40:03 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
+2026-07-18:14:40:03 INFO     [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:40:03 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
+2026-07-18:14:40:03 INFO     [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:40:03 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
+2026-07-18:14:40:03 INFO     [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:40:03 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
+2026-07-18:14:40:03 INFO     [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:14:40:03 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
+2026-07-18:14:40:03 INFO     [api.task:311] Building contexts for babilong_qa1 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:40:03 INFO     [api.task:311] Building contexts for babilong_qa2 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:40:03 INFO     [api.task:311] Building contexts for babilong_qa3 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:40:03 INFO     [api.task:311] Building contexts for babilong_qa4 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:40:03 INFO     [api.task:311] Building contexts for babilong_qa5 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:14:40:04 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:14:45:41 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:14:45:41 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t065/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
+|     Tasks      |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|-----:|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |0.232|±  |0.0261|
+| - babilong_qa1 |      0|none  |     2|acc   |↑  |0.280|±  |0.0641|
+| - babilong_qa2 |      0|none  |     2|acc   |↑  |0.080|±  |0.0388|
+| - babilong_qa3 |      0|none  |     2|acc   |↑  |0.140|±  |0.0496|
+| - babilong_qa4 |      0|none  |     2|acc   |↑  |0.280|±  |0.0641|
+| - babilong_qa5 |      0|none  |     2|acc   |↑  |0.380|±  |0.0693|
+|     Groups     |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|----------------|------:|------|------|------|---|----:|---|-----:|
+|babilong_longctx|      0|none  |      |acc   |↑  |0.232|±  |0.0261|

outputs/eval/logs/token_t065/helmet.log ADDED Viewed

	@@ -0,0 +1,252 @@

+[eval] stop_token_ids=[151645]
+[helmet_icl] 1/250 trec_coarse tokens=6597
+[helmet_icl] 2/250 trec_coarse tokens=6549
+[helmet_icl] 3/250 trec_coarse tokens=6508
+[helmet_icl] 4/250 trec_coarse tokens=6608
+[helmet_icl] 5/250 trec_coarse tokens=6497
+[helmet_icl] 6/250 trec_coarse tokens=6545
+[helmet_icl] 7/250 trec_coarse tokens=6583
+[helmet_icl] 8/250 trec_coarse tokens=6455
+[helmet_icl] 9/250 trec_coarse tokens=6643
+[helmet_icl] 10/250 trec_coarse tokens=6568
+[helmet_icl] 11/250 trec_coarse tokens=6518
+[helmet_icl] 12/250 trec_coarse tokens=6424
+[helmet_icl] 13/250 trec_coarse tokens=6750
+[helmet_icl] 14/250 trec_coarse tokens=6642
+[helmet_icl] 15/250 trec_coarse tokens=6494
+[helmet_icl] 16/250 trec_coarse tokens=6427
+[helmet_icl] 17/250 trec_coarse tokens=6547
+[helmet_icl] 18/250 trec_coarse tokens=6612
+[helmet_icl] 19/250 trec_coarse tokens=6591
+[helmet_icl] 20/250 trec_coarse tokens=6485
+[helmet_icl] 21/250 trec_coarse tokens=6542
+[helmet_icl] 22/250 trec_coarse tokens=6409
+[helmet_icl] 23/250 trec_coarse tokens=6482
+[helmet_icl] 24/250 trec_coarse tokens=6502
+[helmet_icl] 25/250 trec_coarse tokens=6344
+[helmet_icl] 26/250 trec_coarse tokens=6479
+[helmet_icl] 27/250 trec_coarse tokens=6569
+[helmet_icl] 28/250 trec_coarse tokens=6389
+[helmet_icl] 29/250 trec_coarse tokens=6479
+[helmet_icl] 30/250 trec_coarse tokens=6653
+[helmet_icl] 31/250 trec_coarse tokens=6532
+[helmet_icl] 32/250 trec_coarse tokens=6491
+[helmet_icl] 33/250 trec_coarse tokens=6489
+[helmet_icl] 34/250 trec_coarse tokens=6630
+[helmet_icl] 35/250 trec_coarse tokens=6488
+[helmet_icl] 36/250 trec_coarse tokens=6499
+[helmet_icl] 37/250 trec_coarse tokens=6483
+[helmet_icl] 38/250 trec_coarse tokens=6660
+[helmet_icl] 39/250 trec_coarse tokens=6675
+[helmet_icl] 40/250 trec_coarse tokens=6544
+[helmet_icl] 41/250 trec_coarse tokens=6520
+[helmet_icl] 42/250 trec_coarse tokens=6359
+[helmet_icl] 43/250 trec_coarse tokens=6657
+[helmet_icl] 44/250 trec_coarse tokens=6546
+[helmet_icl] 45/250 trec_coarse tokens=6481
+[helmet_icl] 46/250 trec_coarse tokens=6506
+[helmet_icl] 47/250 trec_coarse tokens=6621
+[helmet_icl] 48/250 trec_coarse tokens=6623
+[helmet_icl] 49/250 trec_coarse tokens=6411
+[helmet_icl] 50/250 trec_coarse tokens=6577
+[helmet_icl] 51/250 trec_fine tokens=6858
+[helmet_icl] 52/250 trec_fine tokens=6973
+[helmet_icl] 53/250 trec_fine tokens=6977
+[helmet_icl] 54/250 trec_fine tokens=6952
+[helmet_icl] 55/250 trec_fine tokens=6919
+[helmet_icl] 56/250 trec_fine tokens=6979
+[helmet_icl] 57/250 trec_fine tokens=6957
+[helmet_icl] 58/250 trec_fine tokens=7013
+[helmet_icl] 59/250 trec_fine tokens=6824
+[helmet_icl] 60/250 trec_fine tokens=6833
+[helmet_icl] 61/250 trec_fine tokens=6868
+[helmet_icl] 62/250 trec_fine tokens=6914
+[helmet_icl] 63/250 trec_fine tokens=6759
+[helmet_icl] 64/250 trec_fine tokens=6933
+[helmet_icl] 65/250 trec_fine tokens=6868
+[helmet_icl] 66/250 trec_fine tokens=6839
+[helmet_icl] 67/250 trec_fine tokens=6838
+[helmet_icl] 68/250 trec_fine tokens=7000
+[helmet_icl] 69/250 trec_fine tokens=6800
+[helmet_icl] 70/250 trec_fine tokens=6829
+[helmet_icl] 71/250 trec_fine tokens=6971
+[helmet_icl] 72/250 trec_fine tokens=6907
+[helmet_icl] 73/250 trec_fine tokens=6818
+[helmet_icl] 74/250 trec_fine tokens=6902
+[helmet_icl] 75/250 trec_fine tokens=6941
+[helmet_icl] 76/250 trec_fine tokens=6830
+[helmet_icl] 77/250 trec_fine tokens=6934
+[helmet_icl] 78/250 trec_fine tokens=6879
+[helmet_icl] 79/250 trec_fine tokens=6915
+[helmet_icl] 80/250 trec_fine tokens=6953
+[helmet_icl] 81/250 trec_fine tokens=6875
+[helmet_icl] 82/250 trec_fine tokens=6847
+[helmet_icl] 83/250 trec_fine tokens=6847
+[helmet_icl] 84/250 trec_fine tokens=6944
+[helmet_icl] 85/250 trec_fine tokens=6818
+[helmet_icl] 86/250 trec_fine tokens=6868
+[helmet_icl] 87/250 trec_fine tokens=7022
+[helmet_icl] 88/250 trec_fine tokens=6930
+[helmet_icl] 89/250 trec_fine tokens=6863
+[helmet_icl] 90/250 trec_fine tokens=6807
+[helmet_icl] 91/250 trec_fine tokens=7001
+[helmet_icl] 92/250 trec_fine tokens=6890
+[helmet_icl] 93/250 trec_fine tokens=6952
+[helmet_icl] 94/250 trec_fine tokens=6845
+[helmet_icl] 95/250 trec_fine tokens=6902
+[helmet_icl] 96/250 trec_fine tokens=6879
+[helmet_icl] 97/250 trec_fine tokens=6896
+[helmet_icl] 98/250 trec_fine tokens=6770
+[helmet_icl] 99/250 trec_fine tokens=6933
+[helmet_icl] 100/250 trec_fine tokens=6886
+[helmet_icl] 101/250 banking77 tokens=6809
+[helmet_icl] 102/250 banking77 tokens=7162
+[helmet_icl] 103/250 banking77 tokens=6876
+[helmet_icl] 104/250 banking77 tokens=7033
+[helmet_icl] 105/250 banking77 tokens=6647
+[helmet_icl] 106/250 banking77 tokens=6818
+[helmet_icl] 107/250 banking77 tokens=6841
+[helmet_icl] 108/250 banking77 tokens=6944
+[helmet_icl] 109/250 banking77 tokens=6776
+[helmet_icl] 110/250 banking77 tokens=7100
+[helmet_icl] 111/250 banking77 tokens=6967
+[helmet_icl] 112/250 banking77 tokens=6894
+[helmet_icl] 113/250 banking77 tokens=6900
+[helmet_icl] 114/250 banking77 tokens=6964
+[helmet_icl] 115/250 banking77 tokens=7085
+[helmet_icl] 116/250 banking77 tokens=6953
+[helmet_icl] 117/250 banking77 tokens=7406
+[helmet_icl] 118/250 banking77 tokens=6802
+[helmet_icl] 119/250 banking77 tokens=7119
+[helmet_icl] 120/250 banking77 tokens=7202
+[helmet_icl] 121/250 banking77 tokens=7103
+[helmet_icl] 122/250 banking77 tokens=7009
+[helmet_icl] 123/250 banking77 tokens=6940
+[helmet_icl] 124/250 banking77 tokens=6857
+[helmet_icl] 125/250 banking77 tokens=6922
+[helmet_icl] 126/250 banking77 tokens=6988
+[helmet_icl] 127/250 banking77 tokens=6945
+[helmet_icl] 128/250 banking77 tokens=6835
+[helmet_icl] 129/250 banking77 tokens=7035
+[helmet_icl] 130/250 banking77 tokens=6787
+[helmet_icl] 131/250 banking77 tokens=6940
+[helmet_icl] 132/250 banking77 tokens=6661
+[helmet_icl] 133/250 banking77 tokens=6934
+[helmet_icl] 134/250 banking77 tokens=7235
+[helmet_icl] 135/250 banking77 tokens=6829
+[helmet_icl] 136/250 banking77 tokens=7169
+[helmet_icl] 137/250 banking77 tokens=7074
+[helmet_icl] 138/250 banking77 tokens=6797
+[helmet_icl] 139/250 banking77 tokens=7013
+[helmet_icl] 140/250 banking77 tokens=6811
+[helmet_icl] 141/250 banking77 tokens=6983
+[helmet_icl] 142/250 banking77 tokens=7183
+[helmet_icl] 143/250 banking77 tokens=6801
+[helmet_icl] 144/250 banking77 tokens=7174
+[helmet_icl] 145/250 banking77 tokens=6773
+[helmet_icl] 146/250 banking77 tokens=7134
+[helmet_icl] 147/250 banking77 tokens=6907
+[helmet_icl] 148/250 banking77 tokens=7341
+[helmet_icl] 149/250 banking77 tokens=7042
+[helmet_icl] 150/250 banking77 tokens=7304
+[helmet_icl] 151/250 clinic150 tokens=7241
+[helmet_icl] 152/250 clinic150 tokens=7282
+[helmet_icl] 153/250 clinic150 tokens=7261
+[helmet_icl] 154/250 clinic150 tokens=7292
+[helmet_icl] 155/250 clinic150 tokens=7217
+[helmet_icl] 156/250 clinic150 tokens=7280
+[helmet_icl] 157/250 clinic150 tokens=7293
+[helmet_icl] 158/250 clinic150 tokens=7250
+[helmet_icl] 159/250 clinic150 tokens=7264
+[helmet_icl] 160/250 clinic150 tokens=7238
+[helmet_icl] 161/250 clinic150 tokens=7243
+[helmet_icl] 162/250 clinic150 tokens=7191
+[helmet_icl] 163/250 clinic150 tokens=7333
+[helmet_icl] 164/250 clinic150 tokens=7299
+[helmet_icl] 165/250 clinic150 tokens=7225
+[helmet_icl] 166/250 clinic150 tokens=7409
+[helmet_icl] 167/250 clinic150 tokens=7277
+[helmet_icl] 168/250 clinic150 tokens=7376
+[helmet_icl] 169/250 clinic150 tokens=7383
+[helmet_icl] 170/250 clinic150 tokens=7270
+[helmet_icl] 171/250 clinic150 tokens=7173
+[helmet_icl] 172/250 clinic150 tokens=7106
+[helmet_icl] 173/250 clinic150 tokens=7413
+[helmet_icl] 174/250 clinic150 tokens=7253
+[helmet_icl] 175/250 clinic150 tokens=7224
+[helmet_icl] 176/250 clinic150 tokens=7307
+[helmet_icl] 177/250 clinic150 tokens=7272
+[helmet_icl] 178/250 clinic150 tokens=7215
+[helmet_icl] 179/250 clinic150 tokens=7344
+[helmet_icl] 180/250 clinic150 tokens=7224
+[helmet_icl] 181/250 clinic150 tokens=7310
+[helmet_icl] 182/250 clinic150 tokens=7222
+[helmet_icl] 183/250 clinic150 tokens=7260
+[helmet_icl] 184/250 clinic150 tokens=7351
+[helmet_icl] 185/250 clinic150 tokens=7301
+[helmet_icl] 186/250 clinic150 tokens=7155
+[helmet_icl] 187/250 clinic150 tokens=7253
+[helmet_icl] 188/250 clinic150 tokens=7248
+[helmet_icl] 189/250 clinic150 tokens=7128
+[helmet_icl] 190/250 clinic150 tokens=7282
+[helmet_icl] 191/250 clinic150 tokens=7301
+[helmet_icl] 192/250 clinic150 tokens=7388
+[helmet_icl] 193/250 clinic150 tokens=7244
+[helmet_icl] 194/250 clinic150 tokens=7186
+[helmet_icl] 195/250 clinic150 tokens=7309
+[helmet_icl] 196/250 clinic150 tokens=7241
+[helmet_icl] 197/250 clinic150 tokens=7214
+[helmet_icl] 198/250 clinic150 tokens=7264
+[helmet_icl] 199/250 clinic150 tokens=7219
+[helmet_icl] 200/250 clinic150 tokens=7228
+[helmet_icl] 201/250 nlu tokens=7063
+[helmet_icl] 202/250 nlu tokens=6959
+[helmet_icl] 203/250 nlu tokens=6985
+[helmet_icl] 204/250 nlu tokens=6970
+[helmet_icl] 205/250 nlu tokens=7025
+[helmet_icl] 206/250 nlu tokens=7034
+[helmet_icl] 207/250 nlu tokens=7004
+[helmet_icl] 208/250 nlu tokens=7062
+[helmet_icl] 209/250 nlu tokens=7062
+[helmet_icl] 210/250 nlu tokens=7045
+[helmet_icl] 211/250 nlu tokens=7028
+[helmet_icl] 212/250 nlu tokens=7018
+[helmet_icl] 213/250 nlu tokens=7075
+[helmet_icl] 214/250 nlu tokens=7068
+[helmet_icl] 215/250 nlu tokens=7025
+[helmet_icl] 216/250 nlu tokens=7022
+[helmet_icl] 217/250 nlu tokens=7112
+[helmet_icl] 218/250 nlu tokens=6967
+[helmet_icl] 219/250 nlu tokens=6980
+[helmet_icl] 220/250 nlu tokens=7031
+[helmet_icl] 221/250 nlu tokens=7093
+[helmet_icl] 222/250 nlu tokens=6949
+[helmet_icl] 223/250 nlu tokens=7046
+[helmet_icl] 224/250 nlu tokens=7039
+[helmet_icl] 225/250 nlu tokens=7126
+[helmet_icl] 226/250 nlu tokens=7064
+[helmet_icl] 227/250 nlu tokens=7057
+[helmet_icl] 228/250 nlu tokens=6971
+[helmet_icl] 229/250 nlu tokens=7034
+[helmet_icl] 230/250 nlu tokens=7094
+[helmet_icl] 231/250 nlu tokens=6979
+[helmet_icl] 232/250 nlu tokens=7123
+[helmet_icl] 233/250 nlu tokens=7144
+[helmet_icl] 234/250 nlu tokens=7024
+[helmet_icl] 235/250 nlu tokens=6943
+[helmet_icl] 236/250 nlu tokens=6961
+[helmet_icl] 237/250 nlu tokens=7005
+[helmet_icl] 238/250 nlu tokens=7080
+[helmet_icl] 239/250 nlu tokens=7179
+[helmet_icl] 240/250 nlu tokens=7115
+[helmet_icl] 241/250 nlu tokens=6946
+[helmet_icl] 242/250 nlu tokens=6985
+[helmet_icl] 243/250 nlu tokens=7131
+[helmet_icl] 244/250 nlu tokens=7100
+[helmet_icl] 245/250 nlu tokens=7190
+[helmet_icl] 246/250 nlu tokens=7040
+[helmet_icl] 247/250 nlu tokens=7013
+[helmet_icl] 248/250 nlu tokens=7080
+[helmet_icl] 249/250 nlu tokens=7098
+[helmet_icl] 250/250 nlu tokens=6891
+/workspace/outputs/eval/token_t065/helmet_icl8k_n50.summary.csv

outputs/eval/logs/token_t065/mrcr.log ADDED Viewed

	@@ -0,0 +1,32 @@

+[eval] stop_token_ids=[151645]
+[mrcr] 1/30 8k_2needle tokens=4852
+[mrcr] 2/30 8k_2needle tokens=7119
+[mrcr] 3/30 8k_2needle tokens=5756
+[mrcr] 4/30 8k_2needle tokens=5072
+[mrcr] 5/30 8k_2needle tokens=5753
+[mrcr] 6/30 8k_2needle tokens=5481
+[mrcr] 7/30 8k_2needle tokens=5143
+[mrcr] 8/30 8k_2needle tokens=6399
+[mrcr] 9/30 8k_2needle tokens=5987
+[mrcr] 10/30 8k_2needle tokens=4380
+[mrcr] 11/30 8k_4needle tokens=5123
+[mrcr] 12/30 8k_4needle tokens=7908
+[mrcr] 13/30 8k_4needle tokens=5814
+[mrcr] 14/30 8k_4needle tokens=4856
+[mrcr] 15/30 8k_4needle tokens=7413
+[mrcr] 16/30 8k_4needle tokens=7764
+[mrcr] 17/30 8k_4needle tokens=4312
+[mrcr] 18/30 8k_4needle tokens=4847
+[mrcr] 19/30 8k_4needle tokens=6287
+[mrcr] 20/30 8k_4needle tokens=7841
+[mrcr] 21/30 8k_8needle tokens=4581
+[mrcr] 22/30 8k_8needle tokens=6357
+[mrcr] 23/30 8k_8needle tokens=6981
+[mrcr] 24/30 8k_8needle tokens=6308
+[mrcr] 25/30 8k_8needle tokens=6901
+[mrcr] 26/30 8k_8needle tokens=5407
+[mrcr] 27/30 8k_8needle tokens=6351
+[mrcr] 28/30 8k_8needle tokens=5657
+[mrcr] 29/30 8k_8needle tokens=7330
+[mrcr] 30/30 8k_8needle tokens=8028
+/workspace/outputs/eval/token_t065/mrcr_8k_2_4_8needle_n10.summary.csv

outputs/eval/logs/token_t065/ruler_a.log ADDED Viewed

@@ -0,0 +1,345 @@
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 61/500 [00:01<00:07, 60.31it/s][A
 24%|██▍       | 122/500 [00:02<00:06, 60.22it/s][A
 37%|███▋      | 183/500 [00:03<00:05, 60.40it/s][A
 49%|████▉     | 244/500 [00:04<00:04, 59.73it/s][A
 61%|██████    | 304/500 [00:05<00:03, 58.15it/s][A
 73%|███████▎  | 363/500 [00:06<00:02, 57.32it/s][A
 85%|████████▍ | 423/500 [00:07<00:01, 58.15it/s][A
 97%|█████████▋| 483/500 [00:08<00:00, 58.52it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:14:12:42 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:14:12:46 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
+2026-07-18:14:12:47 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:14:12:47 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:14:12:50 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:14:12:51 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:14:12:51 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:14:13:00 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:13:00 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:14:13:11 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:13:20 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:13:28 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:13:37 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:13:37 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+Max length 500 | Current length 307 | Noises: 5
+Max length 500 | Current length 410 | Noises: 10
+Max length 500 | Current length 531 | Noises: 15
+Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
+Max length 8192 | Current length 791 | Noises: 10
+Max length 8192 | Current length 1025 | Noises: 20
+Max length 8192 | Current length 1273 | Noises: 30
+Max length 8192 | Current length 1510 | Noises: 40
+Max length 8192 | Current length 1749 | Noises: 50
+Max length 8192 | Current length 1996 | Noises: 60
+Max length 8192 | Current length 2231 | Noises: 70
+Max length 8192 | Current length 2473 | Noises: 80
+Max length 8192 | Current length 2705 | Noises: 90
+Max length 8192 | Current length 2953 | Noises: 100
+Max length 8192 | Current length 3186 | Noises: 110
+Max length 8192 | Current length 3431 | Noises: 120
+Max length 8192 | Current length 3671 | Noises: 130
+Max length 8192 | Current length 3916 | Noises: 140
+Max length 8192 | Current length 4149 | Noises: 150
+Max length 8192 | Current length 4386 | Noises: 160
+Max length 8192 | Current length 4630 | Noises: 170
+Max length 8192 | Current length 4871 | Noises: 180
+Max length 8192 | Current length 5110 | Noises: 190
+Max length 8192 | Current length 5349 | Noises: 200
+Max length 8192 | Current length 5590 | Noises: 210
+Max length 8192 | Current length 5832 | Noises: 220
+Max length 8192 | Current length 6070 | Noises: 230
+Max length 8192 | Current length 6311 | Noises: 240
+Max length 8192 | Current length 6544 | Noises: 250
+Max length 8192 | Current length 6791 | Noises: 260
+Max length 8192 | Current length 7031 | Noises: 270
+Max length 8192 | Current length 7271 | Noises: 280
+Max length 8192 | Current length 7513 | Noises: 290
+Max length 8192 | Current length 7751 | Noises: 300
+Max length 8192 | Current length 7986 | Noises: 310
+Max length 8192 | Current length 8230 | Noises: 320
+Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 61/500 [00:01<00:07, 60.31it/s][A
 24%|██▍       | 122/500 [00:02<00:06, 60.22it/s][A
 37%|███▋      | 183/500 [00:03<00:05, 60.40it/s][A
 49%|████▉     | 244/500 [00:04<00:04, 59.73it/s][A
 61%|██████    | 304/500 [00:05<00:03, 58.15it/s][A
 73%|███████▎  | 363/500 [00:06<00:02, 57.32it/s][A
 85%|████████▍ | 423/500 [00:07<00:01, 58.15it/s][A
 97%|█████████▋| 483/500 [00:08<00:00, 58.52it/s][A
+2026-07-18:14:13:47 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:14:00 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:14:21 INFO     [tasks:700] Selected tasks:
+2026-07-18:14:14:21 INFO     [tasks:691] Task: ruler_qa_hotpot (ruler/qa_hotpot.yaml)
+2026-07-18:14:14:21 INFO     [tasks:691] Task: ruler_fwe (ruler/fwe.yaml)
+2026-07-18:14:14:21 INFO     [tasks:691] Task: ruler_vt (ruler/vt.yaml)
+2026-07-18:14:14:21 INFO     [tasks:691] Task: niah_multiquery (ruler/niah_multiquery.yaml)
+2026-07-18:14:14:21 INFO     [tasks:691] Task: niah_multikey_2 (ruler/niah_multikey_2.yaml)
+2026-07-18:14:14:21 INFO     [tasks:691] Task: niah_single_3 (ruler/niah_single_3.yaml)
+2026-07-18:14:14:21 INFO     [tasks:691] Task: niah_single_1 (ruler/niah_single_1.yaml)
+2026-07-18:14:14:21 INFO     [evaluator:314] ruler_qa_hotpot: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:14:14:21 INFO     [evaluator:314] ruler_fwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 50, 'until': []}
+2026-07-18:14:14:21 INFO     [evaluator:314] ruler_vt: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 30, 'until': []}
+2026-07-18:14:14:21 INFO     [evaluator:314] niah_multiquery: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:14:21 INFO     [evaluator:314] niah_multikey_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:14:21 INFO     [evaluator:314] niah_single_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:14:21 INFO     [evaluator:314] niah_single_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:14:21 INFO     [api.task:311] Building contexts for ruler_qa_hotpot on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:14:21 INFO     [api.task:311] Building contexts for ruler_fwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:14:21 INFO     [api.task:311] Building contexts for ruler_vt on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:14:21 INFO     [api.task:311] Building contexts for niah_multiquery on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:14:21 INFO     [api.task:311] Building contexts for niah_multikey_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:14:21 INFO     [api.task:311] Building contexts for niah_single_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:14:21 INFO     [api.task:311] Building contexts for niah_single_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:14:21 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:14:27:38 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:14:27:38 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t065/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
+|---------------|------:|------|-----:|-----:|---|----:|---|------|
+|niah_multikey_2|      1|none  |     0|  4096|↑  |-1.00|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.15|±  |   N/A|
+|niah_multiquery|      1|none  |     0|  4096|↑  |-1.00|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.10|±  |   N/A|
+|niah_single_1  |      1|none  |     0|  4096|↑  |-1.00|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.00|±  |   N/A|
+|niah_single_3  |      1|none  |     0|  4096|↑  |-1.00|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.00|±  |   N/A|
+|ruler_fwe      |      1|none  |     0|  4096|↑  |-1.00|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.80|±  |   N/A|
+|ruler_qa_hotpot|      1|none  |     0|  4096|↑  |-1.00|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.20|±  |   N/A|
+|ruler_vt       |      1|none  |     0|  4096|↑  |-1.00|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.22|±  |   N/A|

outputs/eval/logs/token_t065/ruler_b.log ADDED Viewed

@@ -0,0 +1,253 @@
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A

+2026-07-18:14:27:41 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:14:27:45 INFO     [_cli.run:376] Selected Tasks: ['niah_single_2', 'niah_multikey_1', 'niah_multikey_3', 'niah_multivalue', 'ruler_cwe', 'ruler_qa_squad']
+2026-07-18:14:27:46 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:14:27:46 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:14:27:49 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:14:27:49 INFO     [models.huggingface:548] Model type cannot be determined. Using default model type 'causal'
+2026-07-18:14:27:50 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:14:27:59 WARNING  [api.task:856] niah_single_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:27:59 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:14:28:10 WARNING  [api.task:856] niah_multikey_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:19 WARNING  [api.task:856] niah_multikey_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:26 WARNING  [api.task:856] niah_multivalue: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:36 WARNING  [api.task:856] ruler_cwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:36 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace/outputs/l2a_style/stage2/checkpoint-25 for synthetic tasks.
+2026-07-18:14:28:43 WARNING  [api.task:856] ruler_qa_squad: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:14:28:54 INFO     [tasks:700] Selected tasks:
+2026-07-18:14:28:54 INFO     [tasks:691] Task: ruler_qa_squad (ruler/qa_squad.yaml)
+2026-07-18:14:28:54 INFO     [tasks:691] Task: ruler_cwe (ruler/cwe.yaml)
+2026-07-18:14:28:54 INFO     [tasks:691] Task: niah_multivalue (ruler/niah_multivalue.yaml)
+2026-07-18:14:28:54 INFO     [tasks:691] Task: niah_multikey_3 (ruler/niah_multikey_3.yaml)
+2026-07-18:14:28:54 INFO     [tasks:691] Task: niah_multikey_1 (ruler/niah_multikey_1.yaml)
+2026-07-18:14:28:54 INFO     [tasks:691] Task: niah_single_2 (ruler/niah_single_2.yaml)
+2026-07-18:14:28:54 INFO     [evaluator:314] ruler_qa_squad: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:14:28:54 INFO     [evaluator:314] ruler_cwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 120, 'until': []}
+2026-07-18:14:28:54 INFO     [evaluator:314] niah_multivalue: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:28:54 INFO     [evaluator:314] niah_multikey_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:28:54 INFO     [evaluator:314] niah_multikey_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:28:54 INFO     [evaluator:314] niah_single_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:14:28:54 INFO     [api.task:311] Building contexts for ruler_qa_squad on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:28:54 INFO     [api.task:311] Building contexts for ruler_cwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:28:54 INFO     [api.task:311] Building contexts for niah_multivalue on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:28:54 INFO     [api.task:311] Building contexts for niah_multikey_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:28:54 INFO     [api.task:311] Building contexts for niah_multikey_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:28:54 INFO     [api.task:311] Building contexts for niah_single_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:14:28:54 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:14:39:49 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:14:39:49 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/token_t065/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/*.jsonl
+hf ({'pretrained': '/workspace/outputs/l2a_style/stage2/checkpoint-25', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
+|     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
+|---------------|------:|------|-----:|-----:|---|------:|---|------|
+|niah_multikey_1|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.1000|±  |   N/A|
+|niah_multikey_3|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.0000|±  |   N/A|
+|niah_multivalue|      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.0500|±  |   N/A|
+|niah_single_2  |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.1000|±  |   N/A|
+|ruler_cwe      |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.2050|±  |   N/A|
+|ruler_qa_squad |      1|none  |     0|  4096|↑  |-1.0000|±  |   N/A|
+|               |       |none  |     0|  8192|↑  | 0.2542|±  |   N/A|

outputs/eval/logs/vanilla/babilong.log CHANGED Viewed

@@ -1,449 +1,449 @@
-2026-07-18:14:38:06 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
-2026-07-18:14:38:10 INFO     [_cli.run:376] Selected Tasks: ['babilong_longctx']
-2026-07-18:14:38:11 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
-2026-07-18:14:38:11 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
-2026-07-18:14:38:15 INFO     [models.huggingface:161] Using device 'cuda:0'
-2026-07-18:14:38:15 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
-2026-07-18:14:38:17 WARNING  [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:38:20 WARNING  [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:38:20 WARNING  [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:38:20 WARNING  [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:38:20 WARNING  [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:38:20 INFO     [tasks:700] Selected tasks:
-2026-07-18:14:38:20 INFO     [tasks:703] Group: babilong_longctx
-2026-07-18:14:38:20 INFO     [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
-2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
-2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
-2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
-2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
-2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
-2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
-2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
-2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
-2026-07-18:14:38:20 INFO     [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
-2026-07-18:14:38:20 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
-2026-07-18:14:38:20 INFO     [api.task:311] Building contexts for babilong_qa1 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
-2026-07-18:14:38:20 INFO     [api.task:311] Building contexts for babilong_qa2 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
-2026-07-18:14:38:21 INFO     [api.task:311] Building contexts for babilong_qa3 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
-2026-07-18:14:38:21 INFO     [api.task:311] Building contexts for babilong_qa4 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
-2026-07-18:14:38:21 INFO     [api.task:311] Building contexts for babilong_qa5 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
-2026-07-18:14:38:21 INFO     [evaluator:584] Running generate_until requests
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
-2026-07-18:14:41:29 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
-2026-07-18:14:41:29 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/*.jsonl
 hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
 |     Tasks      |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
 |----------------|------:|------|-----:|------|---|----:|---|-----:|

+2026-07-18:11:46:01 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:11:46:04 INFO     [_cli.run:376] Selected Tasks: ['babilong_longctx']
+2026-07-18:11:46:06 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:11:46:06 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:11:46:09 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:11:46:09 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:11:46:11 WARNING  [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:46:14 WARNING  [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:46:14 WARNING  [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:46:15 WARNING  [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:46:15 WARNING  [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:11:46:15 INFO     [tasks:700] Selected tasks:
+2026-07-18:11:46:15 INFO     [tasks:703] Group: babilong_longctx
+2026-07-18:11:46:15 INFO     [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
+2026-07-18:11:46:15 INFO     [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:11:46:15 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
+2026-07-18:11:46:15 INFO     [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:11:46:15 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
+2026-07-18:11:46:15 INFO     [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:11:46:15 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
+2026-07-18:11:46:15 INFO     [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:11:46:15 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
+2026-07-18:11:46:15 INFO     [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
+2026-07-18:11:46:15 WARNING  [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
+2026-07-18:11:46:15 INFO     [api.task:311] Building contexts for babilong_qa1 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:11:46:15 INFO     [api.task:311] Building contexts for babilong_qa2 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:11:46:15 INFO     [api.task:311] Building contexts for babilong_qa3 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:11:46:15 INFO     [api.task:311] Building contexts for babilong_qa4 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:11:46:15 INFO     [api.task:311] Building contexts for babilong_qa5 on rank 0...
  0%|          | 0/50 [00:00<?, ?it/s][A
+2026-07-18:11:46:16 INFO     [evaluator:584] Running generate_until requests
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
 The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+2026-07-18:11:49:27 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:11:49:27 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/*.jsonl
 hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
 |     Tasks      |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
 |----------------|------:|------|-----:|------|---|----:|---|-----:|

outputs/eval/logs/vanilla/ruler_a.log CHANGED Viewed

@@ -1,72 +1,69 @@
-2026-07-18:14:17:11 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
-2026-07-18:14:17:15 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
-2026-07-18:14:17:16 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
-2026-07-18:14:17:16 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
-2026-07-18:14:17:20 INFO     [models.huggingface:161] Using device 'cuda:0'
-2026-07-18:14:17:20 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
-2026-07-18:14:17:29 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:17:29 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
-2026-07-18:14:17:39 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:17:50 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:17:58 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:18:08 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:18:08 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
 Max length 500 | Current length 307 | Noises: 5
 Max length 500 | Current length 410 | Noises: 10
 Max length 500 | Current length 531 | Noises: 15
 Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/1 [00:00<?, ?it/s][A
 Max length 8192 | Current length 791 | Noises: 10
 Max length 8192 | Current length 1025 | Noises: 20
 Max length 8192 | Current length 1273 | Noises: 30
@@ -102,376 +99,372 @@ Max length 8192 | Current length 8230 | Noises: 320
 Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 59/500 [00:01<00:07, 58.56it/s][A
 24%|██▎       | 118/500 [00:02<00:06, 57.94it/s][A
 35%|███▌      | 176/500 [00:03<00:05, 57.91it/s][A
 47%|████▋     | 234/500 [00:04<00:04, 56.14it/s][A
 58%|█████▊    | 291/500 [00:05<00:03, 55.52it/s][A
 69%|██████▉   | 347/500 [00:06<00:02, 54.63it/s][A
 80%|████████  | 402/500 [00:07<00:01, 54.48it/s][A
 91%|█████████▏| 457/500 [00:08<00:00, 54.48it/s][A
-2026-07-18:14:18:18 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 13%|█▎        | 67/500 [00:01<00:06, 66.71it/s][A
 27%|██▋       | 134/500 [00:02<00:05, 66.50it/s][A
 41%|████      | 203/500 [00:03<00:04, 67.30it/s][A
 54%|█████▍    | 271/500 [00:04<00:03, 67.23it/s][A
 68%|██████▊   | 339/500 [00:05<00:02, 67.33it/s][A
 81%|████████▏ | 407/500 [00:06<00:01, 64.51it/s][A
 94%|█████████▍| 472/500 [00:07<00:00, 62.86it/s][A
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:18:32 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:18:55 INFO     [tasks:700] Selected tasks:
-2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_qa_hotpot (ruler/qa_hotpot.yaml)
-2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_fwe (ruler/fwe.yaml)
-2026-07-18:14:18:55 INFO     [tasks:691] Task: ruler_vt (ruler/vt.yaml)
-2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_multiquery (ruler/niah_multiquery.yaml)
-2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_multikey_2 (ruler/niah_multikey_2.yaml)
-2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_single_3 (ruler/niah_single_3.yaml)
-2026-07-18:14:18:55 INFO     [tasks:691] Task: niah_single_1 (ruler/niah_single_1.yaml)
-2026-07-18:14:18:55 INFO     [evaluator:314] ruler_qa_hotpot: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
-2026-07-18:14:18:55 INFO     [evaluator:314] ruler_fwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 50, 'until': []}
-2026-07-18:14:18:55 INFO     [evaluator:314] ruler_vt: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 30, 'until': []}
-2026-07-18:14:18:55 INFO     [evaluator:314] niah_multiquery: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
-2026-07-18:14:18:55 INFO     [evaluator:314] niah_multikey_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
-2026-07-18:14:18:55 INFO     [evaluator:314] niah_single_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
-2026-07-18:14:18:55 INFO     [evaluator:314] niah_single_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
-2026-07-18:14:18:55 INFO     [api.task:311] Building contexts for ruler_qa_hotpot on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:18:55 INFO     [api.task:311] Building contexts for ruler_fwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for ruler_vt on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_multiquery on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_multikey_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_single_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:18:56 INFO     [api.task:311] Building contexts for niah_single_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:18:56 INFO     [evaluator:584] Running generate_until requests
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:27:50 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
-2026-07-18:14:27:50 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/*.jsonl
 hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
 |     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
 |---------------|------:|------|-----:|-----:|---|------:|---|------|

+2026-07-18:11:25:33 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:11:25:37 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
+2026-07-18:11:25:39 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:11:25:39 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:11:25:42 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:11:25:42 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:11:25:52 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:25:52 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
+2026-07-18:11:26:00 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:26:10 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:26:17 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:26:26 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:26:26 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
 Max length 500 | Current length 307 | Noises: 5
 Max length 500 | Current length 410 | Noises: 10
 Max length 500 | Current length 531 | Noises: 15
 Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/1 [00:00<?, ?it/s][A
 Max length 8192 | Current length 791 | Noises: 10
 Max length 8192 | Current length 1025 | Noises: 20
 Max length 8192 | Current length 1273 | Noises: 30
 Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 59/500 [00:01<00:07, 58.56it/s][A
 24%|██▎       | 118/500 [00:02<00:06, 57.94it/s][A
 35%|███▌      | 176/500 [00:03<00:05, 57.91it/s][A
 47%|████▋     | 234/500 [00:04<00:04, 56.14it/s][A
 58%|█████▊    | 291/500 [00:05<00:03, 55.52it/s][A
 69%|██████▉   | 347/500 [00:06<00:02, 54.63it/s][A
 80%|████████  | 402/500 [00:07<00:01, 54.48it/s][A
 91%|█████████▏| 457/500 [00:08<00:00, 54.48it/s][A
 13%|█▎        | 67/500 [00:01<00:06, 66.71it/s][A
 27%|██▋       | 134/500 [00:02<00:05, 66.50it/s][A
 41%|████      | 203/500 [00:03<00:04, 67.30it/s][A
 54%|█████▍    | 271/500 [00:04<00:03, 67.23it/s][A
 68%|██████▊   | 339/500 [00:05<00:02, 67.33it/s][A
 81%|████████▏ | 407/500 [00:06<00:01, 64.51it/s][A
 94%|█████████▍| 472/500 [00:07<00:00, 62.86it/s][A
+2026-07-18:11:26:34 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:26:47 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:27:09 INFO     [tasks:700] Selected tasks:
+2026-07-18:11:27:09 INFO     [tasks:691] Task: ruler_qa_hotpot (ruler/qa_hotpot.yaml)
+2026-07-18:11:27:09 INFO     [tasks:691] Task: ruler_fwe (ruler/fwe.yaml)
+2026-07-18:11:27:09 INFO     [tasks:691] Task: ruler_vt (ruler/vt.yaml)
+2026-07-18:11:27:09 INFO     [tasks:691] Task: niah_multiquery (ruler/niah_multiquery.yaml)
+2026-07-18:11:27:09 INFO     [tasks:691] Task: niah_multikey_2 (ruler/niah_multikey_2.yaml)
+2026-07-18:11:27:09 INFO     [tasks:691] Task: niah_single_3 (ruler/niah_single_3.yaml)
+2026-07-18:11:27:09 INFO     [tasks:691] Task: niah_single_1 (ruler/niah_single_1.yaml)
+2026-07-18:11:27:09 INFO     [evaluator:314] ruler_qa_hotpot: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:11:27:09 INFO     [evaluator:314] ruler_fwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 50, 'until': []}
+2026-07-18:11:27:09 INFO     [evaluator:314] ruler_vt: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 30, 'until': []}
+2026-07-18:11:27:09 INFO     [evaluator:314] niah_multiquery: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:11:27:09 INFO     [evaluator:314] niah_multikey_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:11:27:09 INFO     [evaluator:314] niah_single_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:11:27:09 INFO     [evaluator:314] niah_single_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:11:27:09 INFO     [api.task:311] Building contexts for ruler_qa_hotpot on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:27:09 INFO     [api.task:311] Building contexts for ruler_fwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:27:09 INFO     [api.task:311] Building contexts for ruler_vt on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:27:09 INFO     [api.task:311] Building contexts for niah_multiquery on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:27:10 INFO     [api.task:311] Building contexts for niah_multikey_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:27:10 INFO     [api.task:311] Building contexts for niah_single_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:27:10 INFO     [api.task:311] Building contexts for niah_single_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:27:10 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:11:34:54 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:11:34:54 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/ruler8k_splits/a/lm_eval/__workspace/*.jsonl
 hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
 |     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
 |---------------|------:|------|-----:|-----:|---|------:|---|------|

outputs/eval/logs/vanilla/ruler_b.log CHANGED Viewed

@@ -1,356 +1,353 @@
-2026-07-18:14:27:53 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
-2026-07-18:14:27:57 INFO     [_cli.run:376] Selected Tasks: ['niah_single_2', 'niah_multikey_1', 'niah_multikey_3', 'niah_multivalue', 'ruler_cwe', 'ruler_qa_squad']
-2026-07-18:14:27:58 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
-2026-07-18:14:27:58 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
-2026-07-18:14:28:01 INFO     [models.huggingface:161] Using device 'cuda:0'
-2026-07-18:14:28:02 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
-2026-07-18:14:28:11 WARNING  [api.task:856] niah_single_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:28:11 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
-2026-07-18:14:28:21 WARNING  [api.task:856] niah_multikey_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:28:30 WARNING  [api.task:856] niah_multikey_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:28:36 WARNING  [api.task:856] niah_multivalue: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:28:46 WARNING  [api.task:856] ruler_cwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:28:46 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
-2026-07-18:14:28:53 WARNING  [api.task:856] ruler_qa_squad: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
-2026-07-18:14:29:05 INFO     [tasks:700] Selected tasks:
-2026-07-18:14:29:05 INFO     [tasks:691] Task: ruler_qa_squad (ruler/qa_squad.yaml)
-2026-07-18:14:29:05 INFO     [tasks:691] Task: ruler_cwe (ruler/cwe.yaml)
-2026-07-18:14:29:05 INFO     [tasks:691] Task: niah_multivalue (ruler/niah_multivalue.yaml)
-2026-07-18:14:29:05 INFO     [tasks:691] Task: niah_multikey_3 (ruler/niah_multikey_3.yaml)
-2026-07-18:14:29:05 INFO     [tasks:691] Task: niah_multikey_1 (ruler/niah_multikey_1.yaml)
-2026-07-18:14:29:05 INFO     [tasks:691] Task: niah_single_2 (ruler/niah_single_2.yaml)
-2026-07-18:14:29:05 INFO     [evaluator:314] ruler_qa_squad: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
-2026-07-18:14:29:05 INFO     [evaluator:314] ruler_cwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 120, 'until': []}
-2026-07-18:14:29:05 INFO     [evaluator:314] niah_multivalue: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
-2026-07-18:14:29:05 INFO     [evaluator:314] niah_multikey_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
-2026-07-18:14:29:05 INFO     [evaluator:314] niah_multikey_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
-2026-07-18:14:29:05 INFO     [evaluator:314] niah_single_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
-2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for ruler_qa_squad on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for ruler_cwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for niah_multivalue on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for niah_multikey_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for niah_multikey_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:29:05 INFO     [api.task:311] Building contexts for niah_single_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:29:05 INFO     [evaluator:584] Running generate_until requests
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
-2026-07-18:14:38:04 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
-2026-07-18:14:38:04 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/*.jsonl
 hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
 |     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
 |---------------|------:|------|-----:|-----:|---|------:|---|------|

+2026-07-18:11:34:57 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:11:35:00 INFO     [_cli.run:376] Selected Tasks: ['niah_single_2', 'niah_multikey_1', 'niah_multikey_3', 'niah_multivalue', 'ruler_cwe', 'ruler_qa_squad']
+2026-07-18:11:35:02 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:11:35:02 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:11:35:05 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:11:35:05 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:11:35:14 WARNING  [api.task:856] niah_single_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:35:15 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
+2026-07-18:11:35:25 WARNING  [api.task:856] niah_multikey_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:35:33 WARNING  [api.task:856] niah_multikey_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:35:39 WARNING  [api.task:856] niah_multivalue: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:35:49 WARNING  [api.task:856] ruler_cwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:11:35:49 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
+2026-07-18:11:35:55 WARNING  [api.task:856] ruler_qa_squad: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:36:05 INFO     [tasks:700] Selected tasks:
+2026-07-18:11:36:05 INFO     [tasks:691] Task: ruler_qa_squad (ruler/qa_squad.yaml)
+2026-07-18:11:36:05 INFO     [tasks:691] Task: ruler_cwe (ruler/cwe.yaml)
+2026-07-18:11:36:05 INFO     [tasks:691] Task: niah_multivalue (ruler/niah_multivalue.yaml)
+2026-07-18:11:36:05 INFO     [tasks:691] Task: niah_multikey_3 (ruler/niah_multikey_3.yaml)
+2026-07-18:11:36:05 INFO     [tasks:691] Task: niah_multikey_1 (ruler/niah_multikey_1.yaml)
+2026-07-18:11:36:05 INFO     [tasks:691] Task: niah_single_2 (ruler/niah_single_2.yaml)
+2026-07-18:11:36:05 INFO     [evaluator:314] ruler_qa_squad: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 32, 'until': []}
+2026-07-18:11:36:05 INFO     [evaluator:314] ruler_cwe: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 120, 'until': []}
+2026-07-18:11:36:05 INFO     [evaluator:314] niah_multivalue: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:11:36:05 INFO     [evaluator:314] niah_multikey_3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:11:36:05 INFO     [evaluator:314] niah_multikey_1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:11:36:05 INFO     [evaluator:314] niah_single_2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 128, 'until': []}
+2026-07-18:11:36:05 INFO     [api.task:311] Building contexts for ruler_qa_squad on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:36:05 INFO     [api.task:311] Building contexts for ruler_cwe on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:36:05 INFO     [api.task:311] Building contexts for niah_multivalue on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:36:05 INFO     [api.task:311] Building contexts for niah_multikey_3 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:36:05 INFO     [api.task:311] Building contexts for niah_multikey_1 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:36:06 INFO     [api.task:311] Building contexts for niah_single_2 on rank 0...
  0%|          | 0/20 [00:00<?, ?it/s][A
+2026-07-18:11:36:06 INFO     [evaluator:584] Running generate_until requests
+2026-07-18:11:45:58 INFO     [loggers.evaluation_tracker:247] Saving results aggregated
+2026-07-18:11:45:58 INFO     [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/ruler8k_splits/b/lm_eval/__workspace/*.jsonl
 hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 20.0, num_fewshot: None, batch_size: 1
 |     Tasks     |Version|Filter|n-shot|Metric|   | Value |   |Stderr|
 |---------------|------:|------|-----:|-----:|---|------:|---|------|

outputs/eval/token_t045/babilong.DONE ADDED Viewed

File without changes

outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/results_2026-07-18T12-31-42.056499.json ADDED Viewed

	@@ -0,0 +1,505 @@

+{
+  "results": {
+    "babilong_longctx": {
+      "acc,none": 0.588,
+      "acc_stderr,none": 0.029807545955988512,
+      "alias": "babilong_longctx"
+    },
+    "babilong_qa1": {
+      "alias": " - babilong_qa1",
+      "acc,none": 0.64,
+      "acc_stderr,none": 0.06857142857142856
+    },
+    "babilong_qa2": {
+      "alias": " - babilong_qa2",
+      "acc,none": 0.48,
+      "acc_stderr,none": 0.07137140569598172
+    },
+    "babilong_qa3": {
+      "alias": " - babilong_qa3",
+      "acc,none": 0.34,
+      "acc_stderr,none": 0.06767268161329718
+    },
+    "babilong_qa4": {
+      "alias": " - babilong_qa4",
+      "acc,none": 0.72,
+      "acc_stderr,none": 0.06414269805898185
+    },
+    "babilong_qa5": {
+      "alias": " - babilong_qa5",
+      "acc,none": 0.76,
+      "acc_stderr,none": 0.06101187572589322
+    }
+  },
+  "groups": {
+    "babilong_longctx": {
+      "acc,none": 0.588,
+      "acc_stderr,none": 0.029807545955988512,
+      "alias": "babilong_longctx"
+    }
+  },
+  "group_subtasks": {
+    "babilong_longctx": [
+      "babilong_qa1",
+      "babilong_qa2",
+      "babilong_qa3",
+      "babilong_qa4",
+      "babilong_qa5"
+    ]
+  },
+  "configs": {
+    "babilong_qa1": {
+      "task": "babilong_qa1",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa1"
+      },
+      "test_split": "qa1",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I will give you context with the facts about positions of different persons hidden in some random text and a question. You need to answer the question based only on the information from the facts. If a person was in different locations, use the latest location to answer the question.\nAlways return your answer in the following format:\nThe most recent location of 'person' is 'location'. Do not write anything else after that.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "Charlie went to the hallway. Judith come back to the kitchen. Charlie travelled to balcony.",
+            "question": "Where is Charlie?",
+            "target": "The most recent location of Charlie is balcony."
+          },
+          {
+            "input": "Alan moved to the garage. Charlie went to the beach. Alan went to the shop. Rouse travelled to balcony.",
+            "question": "Where is Alan?",
+            "target": "The most recent location of Alan is shop."
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    },
+    "babilong_qa2": {
+      "task": "babilong_qa2",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa2"
+      },
+      "test_split": "qa2",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I will give you context with the facts about locations and actions of different persons hidden in some random text and a question. You need to answer the question based only on the information from the facts. If a person got an item in the first location and travelled to the second location the item is also in the second location. If a person dropped an item in the first location and moved to the second location the item remains in the first location.\nAlways return your answer in the following format:\nThe 'item' is in 'location'. Do not write anything else after that.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "Charlie went to the kitchen. Charlie got a bottle. Charlie moved to the balcony.",
+            "question": "Where is the bottle?",
+            "target": "The bottle is in the balcony."
+          },
+          {
+            "input": "Alan moved to the garage. Alan got a screw driver. Alan moved to the kitchen.",
+            "question": "Where is the screw driver?",
+            "target": "The screw driver is in the kitchen."
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    },
+    "babilong_qa3": {
+      "task": "babilong_qa3",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa3"
+      },
+      "test_split": "qa3",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I give you context with the facts about locations and actions of different persons hidden in some random text and a question. You need to answer the question based only on the information from the facts. If a person got an item in the first location and travelled to the second location the item is also in the second location. If a person dropped an item in the first location and moved to the second location the item remains in the first location.\nAlways return your answer in the following format:\nBefore the $location_1$ the $item$ was in the $location_2$. Do not write anything else after that.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "John journeyed to the bedroom. Mary grabbed the apple. Mary went back to the bathroom. Daniel journeyed to the bedroom. Daniel moved to the garden. Mary travelled to the kitchen.",
+            "question": "Where was the apple before the kitchen?",
+            "target": "Before the kitchen the apple was in the bathroom."
+          },
+          {
+            "input": "John went back to the bedroom. John went back to the garden. John went back to the kitchen. Sandra took the football. Sandra travelled to the garden. Sandra journeyed to the bedroom.",
+            "question": "Where was the football before the bedroom?",
+            "target": "Before the bedroom the football was in the garden."
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    },
+    "babilong_qa4": {
+      "task": "babilong_qa4",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa4"
+      },
+      "test_split": "qa4",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I will give you context with the facts about different people, their location and actions, hidden in some random text and a question. You need to answer the question based only on the information from the facts.\nYour answer should contain only one word - location. Do not write anything else after that.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "The hallway is south of the kitchen. The bedroom is north of the kitchen.",
+            "question": "What is the kitchen south of?",
+            "target": "bedroom"
+          },
+          {
+            "input": "The garden is west of the bedroom. The bedroom is west of the kitchen.",
+            "question": "What is west of the bedroom?",
+            "target": "garden"
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    },
+    "babilong_qa5": {
+      "task": "babilong_qa5",
+      "custom_dataset": "def load_dataset(**kwargs):\n    config_name = kwargs.get(\"max_seq_lengths\", \"0k\")\n\n    # Get specific qa split\n    qa_split = kwargs.get(\"qa_split\")\n\n    eval_logger.info(\n        f\"Loading babilong dataset: max_seq_lengths={config_name}, split={qa_split}\"\n    )\n    dataset = datasets.load_dataset(\n        \"RMT-team/babilong-1k-samples\", name=config_name, split=qa_split\n    )\n    return {qa_split: dataset}\n",
+      "dataset_path": "RMT-team/babilong-1k-samples",
+      "dataset_kwargs": {
+        "qa_split": "qa5"
+      },
+      "test_split": "qa5",
+      "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+      "doc_to_target": "{{target}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    pred = postprocess_pred(results)\n    target = doc.get(\"target\", \"\").strip()\n\n    # String match\n    score = 1.0 if target.lower() in pred[0].lower() else 0.0\n\n    return {\"acc\": score}\n",
+      "description": "I will give you context with the facts about locations and their relations hidden in some random text and a question. You need to answer the question based only on the information from the facts.\nYour answer should contain only one word. Do not write anything else after that. Do not explain your answer.\n\n",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "first_n",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": [
+          {
+            "input": "Mary picked up the apple there. Mary gave the apple to Fred. Mary moved to the bedroom. Bill took the milk there.",
+            "question": "Who did Mary give the apple to?",
+            "target": "Fred"
+          },
+          {
+            "input": "Jeff took the football there. Jeff passed the football to Fred. Jeff got the milk there. Bill travelled to the bedroom.",
+            "question": "Who gave the football?",
+            "target": "Jeff"
+          },
+          {
+            "input": "Fred picked up the apple there. Fred handed the apple to Bill. Bill journeyed to the bedroom. Jeff went back to the garden.",
+            "question": "What did Fred give to Bill?",
+            "target": "apple"
+          }
+        ],
+        "doc_to_text": "{{input.strip()}}\n{{question.strip()}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{target}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 2,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 16,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 0.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": "8k"
+      }
+    }
+  },
+  "versions": {
+    "babilong_longctx": 0.0,
+    "babilong_qa1": 0.0,
+    "babilong_qa2": 0.0,
+    "babilong_qa3": 0.0,
+    "babilong_qa4": 0.0,
+    "babilong_qa5": 0.0
+  },
+  "n-shot": {
+    "babilong_qa1": 2,
+    "babilong_qa2": 2,
+    "babilong_qa3": 2,
+    "babilong_qa4": 2,
+    "babilong_qa5": 2
+  },
+  "higher_is_better": {
+    "babilong_longctx": {
+      "acc": true
+    },
+    "babilong_qa1": {
+      "acc": true
+    },
+    "babilong_qa2": {
+      "acc": true
+    },
+    "babilong_qa3": {
+      "acc": true
+    },
+    "babilong_qa4": {
+      "acc": true
+    },
+    "babilong_qa5": {
+      "acc": true
+    }
+  },
+  "n-samples": {
+    "babilong_qa1": {
+      "original": 1000,
+      "effective": 50
+    },
+    "babilong_qa2": {
+      "original": 999,
+      "effective": 50
+    },
+    "babilong_qa3": {
+      "original": 999,
+      "effective": 50
+    },
+    "babilong_qa4": {
+      "original": 999,
+      "effective": 50
+    },
+    "babilong_qa5": {
+      "original": 999,
+      "effective": 50
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+      "trust_remote_code": true,
+      "dtype": "bfloat16",
+      "max_length": 16384,
+      "attn_implementation": "sdpa"
+    },
+    "model_num_parameters": 2031969308,
+    "model_dtype": "torch.bfloat16",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "1",
+    "batch_sizes": [],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": 50.0,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1784377597.4135222,
+  "pretty_env_info": "PyTorch version: 2.9.1+cu128\nIs debug build: False\nCUDA used to build PyTorch: 12.8\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.5 LTS (x86_64)\nGCC version: Could not collect\nClang version: Could not collect\nCMake version: version 4.1.2\nLibc version: glibc-2.35\n\nPython version: 3.11.14 | packaged by conda-forge | (main, Oct 22 2025, 22:46:25) [GCC 14.3.0] (64-bit runtime)\nPython platform: Linux-6.12.90-120.164.amzn2023.x86_64-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: Could not collect\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB\nNvidia driver version: 580.159.03\ncuDNN version: Could not collect\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           46 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  96\nOn-line CPU(s) list:                     0-95\nVendor ID:                               GenuineIntel\nModel name:                              Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz\nCPU family:                              6\nModel:                                   85\nThread(s) per core:                      2\nCore(s) per socket:                      24\nSocket(s):                               2\nStepping:                                7\nBogoMIPS:                                6000.00\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\nHypervisor vendor:                       KVM\nVirtualization type:                     full\nL1d cache:                               1.5 MiB (48 instances)\nL1i cache:                               1.5 MiB (48 instances)\nL2 cache:                                48 MiB (48 instances)\nL3 cache:                                71.5 MiB (2 instances)\nNUMA node(s):                            2\nNUMA node0 CPU(s):                       0-23,48-71\nNUMA node1 CPU(s):                       24-47,72-95\nVulnerability Gather data sampling:      Unknown: Dependent on hypervisor status\nVulnerability Indirect target selection: Mitigation; Aligned branch/return thunks\nVulnerability Itlb multihit:             KVM: Mitigation: VMX unsupported\nVulnerability L1tf:                      Mitigation; PTE Inversion\nVulnerability Mds:                       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown:                  Mitigation; PTI\nVulnerability Mmio stale data:           Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Reg file data sampling:    Not affected\nVulnerability Retbleed:                  Vulnerable\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Vulnerable\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==2.3.4\n[pip3] nvidia-cublas-cu12==12.8.4.1\n[pip3] nvidia-cuda-cupti-cu12==12.8.90\n[pip3] nvidia-cuda-nvrtc-cu12==12.8.93\n[pip3] nvidia-cuda-runtime-cu12==12.8.90\n[pip3] nvidia-cudnn-cu12==9.10.2.21\n[pip3] nvidia-cufft-cu12==11.3.3.83\n[pip3] nvidia-curand-cu12==10.3.9.90\n[pip3] nvidia-cusolver-cu12==11.7.3.90\n[pip3] nvidia-cusparse-cu12==12.5.8.93\n[pip3] nvidia-cusparselt-cu12==0.7.1\n[pip3] nvidia-nccl-cu12==2.27.5\n[pip3] nvidia-nvjitlink-cu12==12.8.93\n[pip3] nvidia-nvtx-cu12==12.8.90\n[pip3] optree==0.17.0\n[pip3] torch==2.9.1+cu128\n[pip3] torchaudio==2.9.1+cu128\n[pip3] torchelastic==0.2.2\n[pip3] torchvision==0.24.1+cu128\n[pip3] triton==3.5.1\n[conda] numpy                        2.3.4            py311h2e04523_0       conda-forge\n[conda] nvidia-cublas-cu12           12.8.4.1         pypi_0                pypi\n[conda] nvidia-cuda-cupti-cu12       12.8.90          pypi_0                pypi\n[conda] nvidia-cuda-nvrtc-cu12       12.8.93          pypi_0                pypi\n[conda] nvidia-cuda-runtime-cu12     12.8.90          pypi_0                pypi\n[conda] nvidia-cudnn-cu12            9.10.2.21        pypi_0                pypi\n[conda] nvidia-cufft-cu12            11.3.3.83        pypi_0                pypi\n[conda] nvidia-curand-cu12           10.3.9.90        pypi_0                pypi\n[conda] nvidia-cusolver-cu12         11.7.3.90        pypi_0                pypi\n[conda] nvidia-cusparse-cu12         12.5.8.93        pypi_0                pypi\n[conda] nvidia-cusparselt-cu12       0.7.1            pypi_0                pypi\n[conda] nvidia-nccl-cu12             2.27.5           pypi_0                pypi\n[conda] nvidia-nvjitlink-cu12        12.8.93          pypi_0                pypi\n[conda] nvidia-nvtx-cu12             12.8.90          pypi_0                pypi\n[conda] optree                       0.17.0           pypi_0                pypi\n[conda] torch                        2.9.1+cu128      pypi_0                pypi\n[conda] torchaudio                   2.9.1+cu128      pypi_0                pypi\n[conda] torchelastic                 0.2.2            pypi_0                pypi\n[conda] torchvision                  0.24.1+cu128     pypi_0                pypi\n[conda] triton                       3.5.1            pypi_0                pypi",
+  "transformers_version": "4.54.0",
+  "lm_eval_version": "0.4.11",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|endoftext|>",
+    "151643"
+  ],
+  "tokenizer_eos_token": [
+    "<|im_end|>",
+    "151645"
+  ],
+  "tokenizer_bos_token": [
+    null,
+    "None"
+  ],
+  "eot_token_id": 151645,
+  "max_length": 16384,
+  "task_hashes": {
+    "babilong_qa1": "bbdcd109bea1bb245e118e814bd0563de6384bc875a21df5218ab4ed75b27380",
+    "babilong_qa2": "fa4f8d12bdd481e02a1051414304811f648caf7bce8f89dcf03ecbf5b86f1e92",
+    "babilong_qa3": "737b64396c8b2e5f96da9796bcd5e448f08e5a13e32ae7b610d31a60269b6e48",
+    "babilong_qa4": "298a25727c69c9babddcd78f438cae1aa21a388e14e7db90b2206eff753c90c0",
+    "babilong_qa5": "e60afcdcdd1c4b3e24b002ffbf8f87085f5ec4a02fcd72318f042dcdafe230f4"
+  },
+  "model_source": "hf",
+  "model_name": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+  "model_name_sanitized": "__workspace__outputs__l2a_style__stage2__checkpoint-25",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "307.892340994993"
+}

outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa1_2026-07-18T12-31-42.056499.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4bb665d4b030c9de83733515a244cabc2345f83990ccc55d24110b3ec5bfc739
+size 3164084

outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa2_2026-07-18T12-31-42.056499.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a41a461533929734a423ed2f311990f3c1d38a85ada1f9730acc45b1bb70e1be
+size 3147502

outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa3_2026-07-18T12-31-42.056499.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:71f73bb1d538565321b650de5af906d40c1206e407a27c69cb2554101954b1c2
+size 3126982

outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa4_2026-07-18T12-31-42.056499.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6d3469275d1131254499c2c7ffd046c1c3f6014ba45f3580f3fb4149d00c2429
+size 3137614

outputs/eval/token_t045/babilong8k_qa1_qa5_n50/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_babilong_qa5_2026-07-18T12-31-42.056499.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5d6d02c0fb088e4bf897d06df24d9d6969adddcd4873a16dc18a405d2d8e5a38
+size 3135534

outputs/eval/token_t045/helmet.DONE ADDED Viewed

File without changes

outputs/eval/token_t045/helmet_icl8k_n50.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9b6ac9eb4e2f34e3969e3385fcaccac5764f931ec526d3167089d68c5d733cc0
+size 6468315

outputs/eval/token_t045/helmet_icl8k_n50.summary.csv ADDED Viewed

	@@ -0,0 +1,251 @@

+method,benchmark,config,n,score,prompt_tokens
+token_t045,helmet_icl,trec_coarse,1,1.0,6597
+token_t045,helmet_icl,trec_coarse,1,0.0,6549
+token_t045,helmet_icl,trec_coarse,1,0.0,6508
+token_t045,helmet_icl,trec_coarse,1,1.0,6608
+token_t045,helmet_icl,trec_coarse,1,1.0,6497
+token_t045,helmet_icl,trec_coarse,1,0.0,6545
+token_t045,helmet_icl,trec_coarse,1,1.0,6583
+token_t045,helmet_icl,trec_coarse,1,1.0,6455
+token_t045,helmet_icl,trec_coarse,1,1.0,6643
+token_t045,helmet_icl,trec_coarse,1,1.0,6568
+token_t045,helmet_icl,trec_coarse,1,0.0,6518
+token_t045,helmet_icl,trec_coarse,1,1.0,6424
+token_t045,helmet_icl,trec_coarse,1,1.0,6750
+token_t045,helmet_icl,trec_coarse,1,1.0,6642
+token_t045,helmet_icl,trec_coarse,1,1.0,6494
+token_t045,helmet_icl,trec_coarse,1,0.0,6427
+token_t045,helmet_icl,trec_coarse,1,1.0,6547
+token_t045,helmet_icl,trec_coarse,1,0.0,6612
+token_t045,helmet_icl,trec_coarse,1,1.0,6591
+token_t045,helmet_icl,trec_coarse,1,1.0,6485
+token_t045,helmet_icl,trec_coarse,1,1.0,6542
+token_t045,helmet_icl,trec_coarse,1,1.0,6409
+token_t045,helmet_icl,trec_coarse,1,0.0,6482
+token_t045,helmet_icl,trec_coarse,1,1.0,6502
+token_t045,helmet_icl,trec_coarse,1,0.0,6344
+token_t045,helmet_icl,trec_coarse,1,1.0,6479
+token_t045,helmet_icl,trec_coarse,1,1.0,6569
+token_t045,helmet_icl,trec_coarse,1,1.0,6389
+token_t045,helmet_icl,trec_coarse,1,1.0,6479
+token_t045,helmet_icl,trec_coarse,1,0.0,6653
+token_t045,helmet_icl,trec_coarse,1,1.0,6532
+token_t045,helmet_icl,trec_coarse,1,1.0,6491
+token_t045,helmet_icl,trec_coarse,1,1.0,6489
+token_t045,helmet_icl,trec_coarse,1,0.0,6630
+token_t045,helmet_icl,trec_coarse,1,1.0,6488
+token_t045,helmet_icl,trec_coarse,1,1.0,6499
+token_t045,helmet_icl,trec_coarse,1,0.0,6483
+token_t045,helmet_icl,trec_coarse,1,1.0,6660
+token_t045,helmet_icl,trec_coarse,1,0.0,6675
+token_t045,helmet_icl,trec_coarse,1,0.0,6544
+token_t045,helmet_icl,trec_coarse,1,1.0,6520
+token_t045,helmet_icl,trec_coarse,1,1.0,6359
+token_t045,helmet_icl,trec_coarse,1,0.0,6657
+token_t045,helmet_icl,trec_coarse,1,1.0,6546
+token_t045,helmet_icl,trec_coarse,1,1.0,6481
+token_t045,helmet_icl,trec_coarse,1,1.0,6506
+token_t045,helmet_icl,trec_coarse,1,0.0,6621
+token_t045,helmet_icl,trec_coarse,1,0.0,6623
+token_t045,helmet_icl,trec_coarse,1,0.0,6411
+token_t045,helmet_icl,trec_coarse,1,1.0,6577
+token_t045,helmet_icl,trec_fine,1,0.0,6858
+token_t045,helmet_icl,trec_fine,1,1.0,6973
+token_t045,helmet_icl,trec_fine,1,0.0,6977
+token_t045,helmet_icl,trec_fine,1,1.0,6952
+token_t045,helmet_icl,trec_fine,1,0.0,6919
+token_t045,helmet_icl,trec_fine,1,1.0,6979
+token_t045,helmet_icl,trec_fine,1,1.0,6957
+token_t045,helmet_icl,trec_fine,1,0.0,7013
+token_t045,helmet_icl,trec_fine,1,1.0,6824
+token_t045,helmet_icl,trec_fine,1,1.0,6833
+token_t045,helmet_icl,trec_fine,1,0.0,6868
+token_t045,helmet_icl,trec_fine,1,1.0,6914
+token_t045,helmet_icl,trec_fine,1,1.0,6759
+token_t045,helmet_icl,trec_fine,1,1.0,6933
+token_t045,helmet_icl,trec_fine,1,0.0,6868
+token_t045,helmet_icl,trec_fine,1,0.0,6839
+token_t045,helmet_icl,trec_fine,1,0.0,6838
+token_t045,helmet_icl,trec_fine,1,0.0,7000
+token_t045,helmet_icl,trec_fine,1,0.0,6800
+token_t045,helmet_icl,trec_fine,1,0.0,6829
+token_t045,helmet_icl,trec_fine,1,0.0,6971
+token_t045,helmet_icl,trec_fine,1,1.0,6907
+token_t045,helmet_icl,trec_fine,1,0.0,6818
+token_t045,helmet_icl,trec_fine,1,1.0,6902
+token_t045,helmet_icl,trec_fine,1,1.0,6941
+token_t045,helmet_icl,trec_fine,1,0.0,6830
+token_t045,helmet_icl,trec_fine,1,0.0,6934
+token_t045,helmet_icl,trec_fine,1,1.0,6879
+token_t045,helmet_icl,trec_fine,1,1.0,6915
+token_t045,helmet_icl,trec_fine,1,0.0,6953
+token_t045,helmet_icl,trec_fine,1,0.0,6875
+token_t045,helmet_icl,trec_fine,1,1.0,6847
+token_t045,helmet_icl,trec_fine,1,0.0,6847
+token_t045,helmet_icl,trec_fine,1,0.0,6944
+token_t045,helmet_icl,trec_fine,1,0.0,6818
+token_t045,helmet_icl,trec_fine,1,0.0,6868
+token_t045,helmet_icl,trec_fine,1,0.0,7022
+token_t045,helmet_icl,trec_fine,1,0.0,6930
+token_t045,helmet_icl,trec_fine,1,0.0,6863
+token_t045,helmet_icl,trec_fine,1,0.0,6807
+token_t045,helmet_icl,trec_fine,1,0.0,7001
+token_t045,helmet_icl,trec_fine,1,0.0,6890
+token_t045,helmet_icl,trec_fine,1,0.0,6952
+token_t045,helmet_icl,trec_fine,1,1.0,6845
+token_t045,helmet_icl,trec_fine,1,0.0,6902
+token_t045,helmet_icl,trec_fine,1,0.0,6879
+token_t045,helmet_icl,trec_fine,1,1.0,6896
+token_t045,helmet_icl,trec_fine,1,0.0,6770
+token_t045,helmet_icl,trec_fine,1,0.0,6933
+token_t045,helmet_icl,trec_fine,1,1.0,6886
+token_t045,helmet_icl,banking77,1,1.0,6809
+token_t045,helmet_icl,banking77,1,0.0,7162
+token_t045,helmet_icl,banking77,1,1.0,6876
+token_t045,helmet_icl,banking77,1,1.0,7033
+token_t045,helmet_icl,banking77,1,1.0,6647
+token_t045,helmet_icl,banking77,1,1.0,6818
+token_t045,helmet_icl,banking77,1,1.0,6841
+token_t045,helmet_icl,banking77,1,1.0,6944
+token_t045,helmet_icl,banking77,1,1.0,6776
+token_t045,helmet_icl,banking77,1,1.0,7100
+token_t045,helmet_icl,banking77,1,0.0,6967
+token_t045,helmet_icl,banking77,1,1.0,6894
+token_t045,helmet_icl,banking77,1,1.0,6900
+token_t045,helmet_icl,banking77,1,0.0,6964
+token_t045,helmet_icl,banking77,1,1.0,7085
+token_t045,helmet_icl,banking77,1,1.0,6953
+token_t045,helmet_icl,banking77,1,0.0,7406
+token_t045,helmet_icl,banking77,1,0.0,6802
+token_t045,helmet_icl,banking77,1,1.0,7119
+token_t045,helmet_icl,banking77,1,0.0,7202
+token_t045,helmet_icl,banking77,1,0.0,7103
+token_t045,helmet_icl,banking77,1,0.0,7009
+token_t045,helmet_icl,banking77,1,1.0,6940
+token_t045,helmet_icl,banking77,1,0.0,6857
+token_t045,helmet_icl,banking77,1,0.0,6922
+token_t045,helmet_icl,banking77,1,1.0,6988
+token_t045,helmet_icl,banking77,1,1.0,6945
+token_t045,helmet_icl,banking77,1,1.0,6835
+token_t045,helmet_icl,banking77,1,0.0,7035
+token_t045,helmet_icl,banking77,1,1.0,6787
+token_t045,helmet_icl,banking77,1,0.0,6940
+token_t045,helmet_icl,banking77,1,0.0,6661
+token_t045,helmet_icl,banking77,1,1.0,6934
+token_t045,helmet_icl,banking77,1,0.0,7235
+token_t045,helmet_icl,banking77,1,1.0,6829
+token_t045,helmet_icl,banking77,1,1.0,7169
+token_t045,helmet_icl,banking77,1,0.0,7074
+token_t045,helmet_icl,banking77,1,0.0,6797
+token_t045,helmet_icl,banking77,1,0.0,7013
+token_t045,helmet_icl,banking77,1,1.0,6811
+token_t045,helmet_icl,banking77,1,0.0,6983
+token_t045,helmet_icl,banking77,1,0.0,7183
+token_t045,helmet_icl,banking77,1,1.0,6801
+token_t045,helmet_icl,banking77,1,0.0,7174
+token_t045,helmet_icl,banking77,1,1.0,6773
+token_t045,helmet_icl,banking77,1,1.0,7134
+token_t045,helmet_icl,banking77,1,1.0,6907
+token_t045,helmet_icl,banking77,1,1.0,7341
+token_t045,helmet_icl,banking77,1,1.0,7042
+token_t045,helmet_icl,banking77,1,1.0,7304
+token_t045,helmet_icl,clinic150,1,1.0,7241
+token_t045,helmet_icl,clinic150,1,0.0,7282
+token_t045,helmet_icl,clinic150,1,1.0,7261
+token_t045,helmet_icl,clinic150,1,0.0,7292
+token_t045,helmet_icl,clinic150,1,1.0,7217
+token_t045,helmet_icl,clinic150,1,0.0,7280
+token_t045,helmet_icl,clinic150,1,1.0,7293
+token_t045,helmet_icl,clinic150,1,1.0,7250
+token_t045,helmet_icl,clinic150,1,1.0,7264
+token_t045,helmet_icl,clinic150,1,1.0,7238
+token_t045,helmet_icl,clinic150,1,0.0,7243
+token_t045,helmet_icl,clinic150,1,1.0,7191
+token_t045,helmet_icl,clinic150,1,1.0,7333
+token_t045,helmet_icl,clinic150,1,1.0,7299
+token_t045,helmet_icl,clinic150,1,1.0,7225
+token_t045,helmet_icl,clinic150,1,1.0,7409
+token_t045,helmet_icl,clinic150,1,1.0,7277
+token_t045,helmet_icl,clinic150,1,1.0,7376
+token_t045,helmet_icl,clinic150,1,1.0,7383
+token_t045,helmet_icl,clinic150,1,0.0,7270
+token_t045,helmet_icl,clinic150,1,1.0,7173
+token_t045,helmet_icl,clinic150,1,1.0,7106
+token_t045,helmet_icl,clinic150,1,1.0,7413
+token_t045,helmet_icl,clinic150,1,1.0,7253
+token_t045,helmet_icl,clinic150,1,0.0,7224
+token_t045,helmet_icl,clinic150,1,1.0,7307
+token_t045,helmet_icl,clinic150,1,1.0,7272
+token_t045,helmet_icl,clinic150,1,0.0,7215
+token_t045,helmet_icl,clinic150,1,1.0,7344
+token_t045,helmet_icl,clinic150,1,1.0,7224
+token_t045,helmet_icl,clinic150,1,1.0,7310
+token_t045,helmet_icl,clinic150,1,1.0,7222
+token_t045,helmet_icl,clinic150,1,0.0,7260
+token_t045,helmet_icl,clinic150,1,1.0,7351
+token_t045,helmet_icl,clinic150,1,1.0,7301
+token_t045,helmet_icl,clinic150,1,1.0,7155
+token_t045,helmet_icl,clinic150,1,1.0,7253
+token_t045,helmet_icl,clinic150,1,1.0,7248
+token_t045,helmet_icl,clinic150,1,1.0,7128
+token_t045,helmet_icl,clinic150,1,0.0,7282
+token_t045,helmet_icl,clinic150,1,1.0,7301
+token_t045,helmet_icl,clinic150,1,1.0,7388
+token_t045,helmet_icl,clinic150,1,1.0,7244
+token_t045,helmet_icl,clinic150,1,1.0,7186
+token_t045,helmet_icl,clinic150,1,0.0,7309
+token_t045,helmet_icl,clinic150,1,1.0,7241
+token_t045,helmet_icl,clinic150,1,1.0,7214
+token_t045,helmet_icl,clinic150,1,0.0,7264
+token_t045,helmet_icl,clinic150,1,0.0,7219
+token_t045,helmet_icl,clinic150,1,1.0,7228
+token_t045,helmet_icl,nlu,1,1.0,7063
+token_t045,helmet_icl,nlu,1,1.0,6959
+token_t045,helmet_icl,nlu,1,0.0,6985
+token_t045,helmet_icl,nlu,1,1.0,6970
+token_t045,helmet_icl,nlu,1,1.0,7025
+token_t045,helmet_icl,nlu,1,0.0,7034
+token_t045,helmet_icl,nlu,1,1.0,7004
+token_t045,helmet_icl,nlu,1,0.0,7062
+token_t045,helmet_icl,nlu,1,0.0,7062
+token_t045,helmet_icl,nlu,1,0.0,7045
+token_t045,helmet_icl,nlu,1,1.0,7028
+token_t045,helmet_icl,nlu,1,1.0,7018
+token_t045,helmet_icl,nlu,1,1.0,7075
+token_t045,helmet_icl,nlu,1,1.0,7068
+token_t045,helmet_icl,nlu,1,1.0,7025
+token_t045,helmet_icl,nlu,1,1.0,7022
+token_t045,helmet_icl,nlu,1,1.0,7112
+token_t045,helmet_icl,nlu,1,1.0,6967
+token_t045,helmet_icl,nlu,1,0.0,6980
+token_t045,helmet_icl,nlu,1,1.0,7031
+token_t045,helmet_icl,nlu,1,0.0,7093
+token_t045,helmet_icl,nlu,1,0.0,6949
+token_t045,helmet_icl,nlu,1,1.0,7046
+token_t045,helmet_icl,nlu,1,1.0,7039
+token_t045,helmet_icl,nlu,1,0.0,7126
+token_t045,helmet_icl,nlu,1,1.0,7064
+token_t045,helmet_icl,nlu,1,1.0,7057
+token_t045,helmet_icl,nlu,1,0.0,6971
+token_t045,helmet_icl,nlu,1,0.0,7034
+token_t045,helmet_icl,nlu,1,1.0,7094
+token_t045,helmet_icl,nlu,1,1.0,6979
+token_t045,helmet_icl,nlu,1,1.0,7123
+token_t045,helmet_icl,nlu,1,1.0,7144
+token_t045,helmet_icl,nlu,1,1.0,7024
+token_t045,helmet_icl,nlu,1,1.0,6943
+token_t045,helmet_icl,nlu,1,0.0,6961
+token_t045,helmet_icl,nlu,1,1.0,7005
+token_t045,helmet_icl,nlu,1,1.0,7080
+token_t045,helmet_icl,nlu,1,1.0,7179
+token_t045,helmet_icl,nlu,1,1.0,7115
+token_t045,helmet_icl,nlu,1,1.0,6946
+token_t045,helmet_icl,nlu,1,1.0,6985
+token_t045,helmet_icl,nlu,1,1.0,7131
+token_t045,helmet_icl,nlu,1,1.0,7100
+token_t045,helmet_icl,nlu,1,1.0,7190
+token_t045,helmet_icl,nlu,1,1.0,7040
+token_t045,helmet_icl,nlu,1,1.0,7013
+token_t045,helmet_icl,nlu,1,0.0,7080
+token_t045,helmet_icl,nlu,1,0.0,7098
+token_t045,helmet_icl,nlu,1,1.0,6891

outputs/eval/token_t045/mrcr.DONE ADDED Viewed

File without changes

outputs/eval/token_t045/mrcr_8k_2_4_8needle_n10.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d55ec4e8a46dc3a01299c2b51889ade81ec65a00c6e87e003c45673703d90b0c
+size 1003949

outputs/eval/token_t045/mrcr_8k_2_4_8needle_n10.summary.csv ADDED Viewed

	@@ -0,0 +1,31 @@

+method,benchmark,config,n,score,prompt_tokens
+token_t045,mrcr,8k_2needle,1,0.9685194294146582,4852
+token_t045,mrcr,8k_2needle,1,0.983710407239819,7119
+token_t045,mrcr,8k_2needle,1,0.037745098039215684,5756
+token_t045,mrcr,8k_2needle,1,0.05052372150338879,5072
+token_t045,mrcr,8k_2needle,1,0.07633928571428572,5753
+token_t045,mrcr,8k_2needle,1,0.5607594936708861,5481
+token_t045,mrcr,8k_2needle,1,1.0,5143
+token_t045,mrcr,8k_2needle,1,1.0,6399
+token_t045,mrcr,8k_2needle,1,1.0,5987
+token_t045,mrcr,8k_2needle,1,0.0767583006069261,4380
+token_t045,mrcr,8k_4needle,1,0.0678657519006119,5123
+token_t045,mrcr,8k_4needle,1,0.03232242617717478,7908
+token_t045,mrcr,8k_4needle,1,0.02783400809716599,5814
+token_t045,mrcr,8k_4needle,1,0.9595443833464258,4856
+token_t045,mrcr,8k_4needle,1,0.0,7413
+token_t045,mrcr,8k_4needle,1,0.007665505226480836,7764
+token_t045,mrcr,8k_4needle,1,0.11432241490044959,4312
+token_t045,mrcr,8k_4needle,1,0.03953445409200074,4847
+token_t045,mrcr,8k_4needle,1,0.08090698273640815,6287
+token_t045,mrcr,8k_4needle,1,0.3413333333333333,7841
+token_t045,mrcr,8k_8needle,1,0.009378663540445486,4581
+token_t045,mrcr,8k_8needle,1,0.18287107589149648,6357
+token_t045,mrcr,8k_8needle,1,0.0962800875273523,6981
+token_t045,mrcr,8k_8needle,1,0.04247286455875413,6308
+token_t045,mrcr,8k_8needle,1,0.0,6901
+token_t045,mrcr,8k_8needle,1,0.02224694104560623,5407
+token_t045,mrcr,8k_8needle,1,0.1317365269461078,6351
+token_t045,mrcr,8k_8needle,1,0.9895,5657
+token_t045,mrcr,8k_8needle,1,0.9852774631936579,7330
+token_t045,mrcr,8k_8needle,1,0.076951811729548,8028

outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/results_2026-07-18T12-12-07.465325.json ADDED Viewed

	@@ -0,0 +1,821 @@

+{
+  "results": {
+    "niah_multikey_2": {
+      "alias": "niah_multikey_2",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_multiquery": {
+      "alias": "niah_multiquery",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.9875,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_single_1": {
+      "alias": "niah_single_1",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_single_3": {
+      "alias": "niah_single_3",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.95,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_fwe": {
+      "alias": "ruler_fwe",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.7666666666666666,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_qa_hotpot": {
+      "alias": "ruler_qa_hotpot",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.4,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_vt": {
+      "alias": "ruler_vt",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.8700000000000003,
+      "8192_stderr,none": "N/A"
+    }
+  },
+  "group_subtasks": {
+    "niah_single_1": [],
+    "niah_single_3": [],
+    "niah_multikey_2": [],
+    "niah_multiquery": [],
+    "ruler_vt": [],
+    "ruler_fwe": [],
+    "ruler_qa_hotpot": []
+  },
+  "configs": {
+    "niah_multikey_2": {
+      "task": "niah_multikey_2",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multikey_2(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"needle\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"needle\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_multiquery": {
+      "task": "niah_multiquery",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multiquery(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_needle_q=4,\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_single_1": {
+      "task": "niah_single_1",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_single_1(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"repeat\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"repeat\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_single_3": {
+      "task": "niah_single_3",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_single_3(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"uuids\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_fwe": {
+      "task": "ruler_fwe",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def fwe_download(**kwargs):\n    pretrained = kwargs.get(\"tokenizer\", kwargs.get(\"pretrained\", {}))\n    df = (\n        get_dataset(pretrained, max_seq_length=seq)\n        for seq in kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    )\n\n    return {\n        \"test\": datasets.Dataset.from_list(\n            list(itertools.chain.from_iterable(df)), split=datasets.Split.TEST\n        )\n    }\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 50,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_qa_hotpot": {
+      "task": "ruler_qa_hotpot",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def get_hotpotqa(**kwargs):\n    return get_qa_dataset(\"hotpotqa\", **kwargs)\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results_part(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_part(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 32,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_vt": {
+      "task": "ruler_vt",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def get_vt_dataset(**kwargs) -> dict[str, datasets.Dataset]:\n    pretrained = kwargs.get(\"tokenizer\", kwargs.get(\"pretrained\", \"\"))\n    df = (\n        get_dataset(tokenizer=get_tokenizer(pretrained), seq=seq)\n        for seq in kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    )\n\n    return {\n        \"test\": datasets.Dataset.from_list(\n            list(itertools.chain.from_iterable(df)), split=datasets.Split.TEST\n        )\n    }\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 30,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    }
+  },
+  "versions": {
+    "niah_multikey_2": 1.0,
+    "niah_multiquery": 1.0,
+    "niah_single_1": 1.0,
+    "niah_single_3": 1.0,
+    "ruler_fwe": 1.0,
+    "ruler_qa_hotpot": 1.0,
+    "ruler_vt": 1.0
+  },
+  "n-shot": {
+    "niah_multikey_2": 0,
+    "niah_multiquery": 0,
+    "niah_single_1": 0,
+    "niah_single_3": 0,
+    "ruler_fwe": 0,
+    "ruler_qa_hotpot": 0,
+    "ruler_vt": 0
+  },
+  "higher_is_better": {
+    "niah_multikey_2": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_multiquery": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_single_1": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_single_3": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_fwe": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_qa_hotpot": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_vt": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    }
+  },
+  "n-samples": {
+    "ruler_qa_hotpot": {
+      "original": 500,
+      "effective": 20
+    },
+    "ruler_fwe": {
+      "original": 500,
+      "effective": 20
+    },
+    "ruler_vt": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multiquery": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multikey_2": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_single_3": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_single_1": {
+      "original": 500,
+      "effective": 20
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+      "trust_remote_code": true,
+      "dtype": "bfloat16",
+      "max_length": 16384,
+      "attn_implementation": "sdpa"
+    },
+    "model_num_parameters": 2031969308,
+    "model_dtype": "torch.bfloat16",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "1",
+    "batch_sizes": [],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": 20.0,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1784375919.9357646,
+  "pretty_env_info": "PyTorch version: 2.9.1+cu128\nIs debug build: False\nCUDA used to build PyTorch: 12.8\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.5 LTS (x86_64)\nGCC version: Could not collect\nClang version: Could not collect\nCMake version: version 4.1.2\nLibc version: glibc-2.35\n\nPython version: 3.11.14 | packaged by conda-forge | (main, Oct 22 2025, 22:46:25) [GCC 14.3.0] (64-bit runtime)\nPython platform: Linux-6.12.90-120.164.amzn2023.x86_64-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: Could not collect\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB\nNvidia driver version: 580.159.03\ncuDNN version: Could not collect\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           46 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  96\nOn-line CPU(s) list:                     0-95\nVendor ID:                               GenuineIntel\nModel name:                              Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz\nCPU family:                              6\nModel:                                   85\nThread(s) per core:                      2\nCore(s) per socket:                      24\nSocket(s):                               2\nStepping:                                7\nBogoMIPS:                                6000.00\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\nHypervisor vendor:                       KVM\nVirtualization type:                     full\nL1d cache:                               1.5 MiB (48 instances)\nL1i cache:                               1.5 MiB (48 instances)\nL2 cache:                                48 MiB (48 instances)\nL3 cache:                                71.5 MiB (2 instances)\nNUMA node(s):                            2\nNUMA node0 CPU(s):                       0-23,48-71\nNUMA node1 CPU(s):                       24-47,72-95\nVulnerability Gather data sampling:      Unknown: Dependent on hypervisor status\nVulnerability Indirect target selection: Mitigation; Aligned branch/return thunks\nVulnerability Itlb multihit:             KVM: Mitigation: VMX unsupported\nVulnerability L1tf:                      Mitigation; PTE Inversion\nVulnerability Mds:                       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown:                  Mitigation; PTI\nVulnerability Mmio stale data:           Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Reg file data sampling:    Not affected\nVulnerability Retbleed:                  Vulnerable\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Vulnerable\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==2.3.4\n[pip3] nvidia-cublas-cu12==12.8.4.1\n[pip3] nvidia-cuda-cupti-cu12==12.8.90\n[pip3] nvidia-cuda-nvrtc-cu12==12.8.93\n[pip3] nvidia-cuda-runtime-cu12==12.8.90\n[pip3] nvidia-cudnn-cu12==9.10.2.21\n[pip3] nvidia-cufft-cu12==11.3.3.83\n[pip3] nvidia-curand-cu12==10.3.9.90\n[pip3] nvidia-cusolver-cu12==11.7.3.90\n[pip3] nvidia-cusparse-cu12==12.5.8.93\n[pip3] nvidia-cusparselt-cu12==0.7.1\n[pip3] nvidia-nccl-cu12==2.27.5\n[pip3] nvidia-nvjitlink-cu12==12.8.93\n[pip3] nvidia-nvtx-cu12==12.8.90\n[pip3] optree==0.17.0\n[pip3] torch==2.9.1+cu128\n[pip3] torchaudio==2.9.1+cu128\n[pip3] torchelastic==0.2.2\n[pip3] torchvision==0.24.1+cu128\n[pip3] triton==3.5.1\n[conda] numpy                        2.3.4            py311h2e04523_0       conda-forge\n[conda] nvidia-cublas-cu12           12.8.4.1         pypi_0                pypi\n[conda] nvidia-cuda-cupti-cu12       12.8.90          pypi_0                pypi\n[conda] nvidia-cuda-nvrtc-cu12       12.8.93          pypi_0                pypi\n[conda] nvidia-cuda-runtime-cu12     12.8.90          pypi_0                pypi\n[conda] nvidia-cudnn-cu12            9.10.2.21        pypi_0                pypi\n[conda] nvidia-cufft-cu12            11.3.3.83        pypi_0                pypi\n[conda] nvidia-curand-cu12           10.3.9.90        pypi_0                pypi\n[conda] nvidia-cusolver-cu12         11.7.3.90        pypi_0                pypi\n[conda] nvidia-cusparse-cu12         12.5.8.93        pypi_0                pypi\n[conda] nvidia-cusparselt-cu12       0.7.1            pypi_0                pypi\n[conda] nvidia-nccl-cu12             2.27.5           pypi_0                pypi\n[conda] nvidia-nvjitlink-cu12        12.8.93          pypi_0                pypi\n[conda] nvidia-nvtx-cu12             12.8.90          pypi_0                pypi\n[conda] optree                       0.17.0           pypi_0                pypi\n[conda] torch                        2.9.1+cu128      pypi_0                pypi\n[conda] torchaudio                   2.9.1+cu128      pypi_0                pypi\n[conda] torchelastic                 0.2.2            pypi_0                pypi\n[conda] torchvision                  0.24.1+cu128     pypi_0                pypi\n[conda] triton                       3.5.1            pypi_0                pypi",
+  "transformers_version": "4.54.0",
+  "lm_eval_version": "0.4.11",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|endoftext|>",
+    "151643"
+  ],
+  "tokenizer_eos_token": [
+    "<|im_end|>",
+    "151645"
+  ],
+  "tokenizer_bos_token": [
+    null,
+    "None"
+  ],
+  "eot_token_id": 151645,
+  "max_length": 16384,
+  "task_hashes": {
+    "ruler_qa_hotpot": "fd48cb605efc8c5adac7e3771533db4d4c354b71bf91ce00841c814604ba6404",
+    "ruler_fwe": "c4633bcd0acf3ba4b96f905b5ba269ba94cf45a4c8ca0c13aae1468227cc3a10",
+    "ruler_vt": "af011da859c8b86f63f57f6256cf59077d3d3d8536084ab67ccf39e18e9a615a",
+    "niah_multiquery": "f9a36d2854dfd2bb52ffc7a6bf3f53490fb5303ca9da0046256bc28b55c2adb2",
+    "niah_multikey_2": "86e49b8f875843454b2923143b9124ba82e8a2e23b145f792726cd82f4e49c24",
+    "niah_single_3": "905e40146a287e5f0c8e2a81f64f082108c45a23862949e6a99bfcbbf561cad2",
+    "niah_single_1": "426e8138e992480aa416d01e6b9cef5865a68c00f532400e448a0c20ee026cff"
+  },
+  "model_source": "hf",
+  "model_name": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+  "model_name_sanitized": "__workspace__outputs__l2a_style__stage2__checkpoint-25",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "810.8221090380102"
+}

outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multikey_2_2026-07-18T12-12-07.465325.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ec7ad717b95e4d5fa923a2627ae2eaf57f1f9db47d459d64eb1ce5f9d6647f8c
+size 987526

outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multiquery_2026-07-18T12-12-07.465325.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dac9a6008ea1a5889eaeb1f76cacfe65c4c4c5a7abd6f5608b3dadddaa221648
+size 1440697

outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_single_1_2026-07-18T12-12-07.465325.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b01ced4cc6722de3a956473b8ae0600525ca6e5485940b5a5e2267633cc3bd5d
+size 1231588

outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_single_3_2026-07-18T12-12-07.465325.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f20ebd346ce5932c6d4cbcdf5f0330574b9b13b9b52951d58b08d2b974886e0c
+size 1435128

outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_ruler_fwe_2026-07-18T12-12-07.465325.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:84bda7768fbe2ab66d514e01fa24e4dadff944d9c721e69274765df0bcdd8864
+size 827295

outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_ruler_qa_hotpot_2026-07-18T12-12-07.465325.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2541d5179a144eda9e0c39f423f5ec895fad28e7188eeadae9317754b88954f1
+size 1172252

outputs/eval/token_t045/ruler8k_splits/a/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_ruler_vt_2026-07-18T12-12-07.465325.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b7c6124651beb57d18609eccd0f1db78d94c9b96851d2e0759422a5477af8a06
+size 1228994

outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/results_2026-07-18T12-26-30.235088.json ADDED Viewed

	@@ -0,0 +1,714 @@

+{
+  "results": {
+    "niah_multikey_1": {
+      "alias": "niah_multikey_1",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_multikey_3": {
+      "alias": "niah_multikey_3",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.75,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_multivalue": {
+      "alias": "niah_multivalue",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "niah_single_2": {
+      "alias": "niah_single_2",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 1.0,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_cwe": {
+      "alias": "ruler_cwe",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.3949999999999999,
+      "8192_stderr,none": "N/A"
+    },
+    "ruler_qa_squad": {
+      "alias": "ruler_qa_squad",
+      "4096,none": -1,
+      "4096_stderr,none": "N/A",
+      "8192,none": 0.4208333333333334,
+      "8192_stderr,none": "N/A"
+    }
+  },
+  "group_subtasks": {
+    "niah_single_2": [],
+    "niah_multikey_1": [],
+    "niah_multikey_3": [],
+    "niah_multivalue": [],
+    "ruler_cwe": [],
+    "ruler_qa_squad": []
+  },
+  "configs": {
+    "niah_multikey_1": {
+      "task": "niah_multikey_1",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multikey_1(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_needle_k=4,\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_multikey_3": {
+      "task": "niah_multikey_3",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multikey_3(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"needle\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"needle\",\n            type_needle_k=\"uuids\",\n            type_needle_v=\"uuids\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_multivalue": {
+      "task": "niah_multivalue",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_multivalue(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_needle_v=4,\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "niah_single_2": {
+      "task": "niah_single_2",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def niah_single_2(**kwargs):\n    seq_lengths = kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    return download_dataset(\n        generate_samples(\n            get_haystack(type_haystack=\"essay\"),\n            max_seq_length=seq,\n            template=TEMPLATE,\n            type_haystack=\"essay\",\n            type_needle_k=\"words\",\n            type_needle_v=\"numbers\",\n            num_samples=500,\n            TOKENIZER=get_tokenizer(**kwargs),\n        )\n        for seq in seq_lengths\n    )\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 128,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_cwe": {
+      "task": "ruler_cwe",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def get_cw_dataset(**kwargs):\n    pretrained = kwargs.get(\"tokenizer\", kwargs.get(\"pretrained\", {}))\n    df = (\n        get_dataset(pretrained, seq=seq)\n        for seq in kwargs.pop(\"max_seq_lengths\", DEFAULT_SEQ_LENGTHS)\n    )\n\n    return {\n        \"test\": datasets.Dataset.from_list(\n            list(itertools.chain.from_iterable(df)), split=datasets.Split.TEST\n        )\n    }\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_all(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": "\n\n",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": "\n\n"
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 120,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    },
+    "ruler_qa_squad": {
+      "task": "ruler_qa_squad",
+      "tag": [
+        "longcxt"
+      ],
+      "custom_dataset": "def get_squad(**kwargs):\n    return get_qa_dataset(\"squad\", **kwargs)\n",
+      "dataset_path": "",
+      "dataset_name": "",
+      "test_split": "test",
+      "doc_to_text": "{{input}}",
+      "doc_to_target": "{{outputs}}",
+      "unsafe_code": false,
+      "process_results": "def process_results_part(doc: dict, results: list[str]) -> dict[str, float]:\n    # hacky: set all other lengths to -1\n    metrics = {str(length): -1.0 for length in DEFAULT_SEQ_LENGTHS}\n    input_len = doc[\"max_length\"]\n    pred = postprocess_pred(results)\n    score = string_match_part(pred, [doc[\"outputs\"]])\n    metrics[str(input_len)] = score\n    return metrics\n",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": null,
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{input}}",
+        "doc_to_choice": null,
+        "doc_to_target": "{{outputs}}",
+        "gen_prefix": "{{gen_prefix}}",
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "4096",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "8192",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "16384",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "32768",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "65536",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        },
+        {
+          "metric": "131072",
+          "aggregation": "def aggregate_metrics(metrics: list[float]) -> float:\n    res = [x for x in metrics if x != -1]\n    if not res:\n        # we don't have any samples with this length\n        return -1\n    return sum(res) / len(res)\n",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "generate_until",
+      "generation_kwargs": {
+        "do_sample": false,
+        "temperature": 0.0,
+        "max_gen_toks": 32,
+        "until": []
+      },
+      "repeats": 1,
+      "should_decontaminate": false,
+      "gen_prefix": "{{gen_prefix}}",
+      "metadata": {
+        "version": 1.0,
+        "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+        "trust_remote_code": true,
+        "dtype": "bfloat16",
+        "max_length": 16384,
+        "attn_implementation": "sdpa",
+        "max_seq_lengths": [
+          8192
+        ]
+      }
+    }
+  },
+  "versions": {
+    "niah_multikey_1": 1.0,
+    "niah_multikey_3": 1.0,
+    "niah_multivalue": 1.0,
+    "niah_single_2": 1.0,
+    "ruler_cwe": 1.0,
+    "ruler_qa_squad": 1.0
+  },
+  "n-shot": {
+    "niah_multikey_1": 0,
+    "niah_multikey_3": 0,
+    "niah_multivalue": 0,
+    "niah_single_2": 0,
+    "ruler_cwe": 0,
+    "ruler_qa_squad": 0
+  },
+  "higher_is_better": {
+    "niah_multikey_1": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_multikey_3": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_multivalue": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "niah_single_2": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_cwe": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    },
+    "ruler_qa_squad": {
+      "4096": true,
+      "8192": true,
+      "16384": true,
+      "32768": true,
+      "65536": true,
+      "131072": true
+    }
+  },
+  "n-samples": {
+    "ruler_qa_squad": {
+      "original": 500,
+      "effective": 20
+    },
+    "ruler_cwe": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multivalue": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multikey_3": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_multikey_1": {
+      "original": 500,
+      "effective": 20
+    },
+    "niah_single_2": {
+      "original": 500,
+      "effective": 20
+    }
+  },
+  "config": {
+    "model": "hf",
+    "model_args": {
+      "pretrained": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+      "trust_remote_code": true,
+      "dtype": "bfloat16",
+      "max_length": 16384,
+      "attn_implementation": "sdpa"
+    },
+    "model_num_parameters": 2031969308,
+    "model_dtype": "torch.bfloat16",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": "1",
+    "batch_sizes": [],
+    "device": "cuda:0",
+    "use_cache": null,
+    "limit": 20.0,
+    "bootstrap_iters": 100000,
+    "gen_kwargs": {},
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1784376734.1502712,
+  "pretty_env_info": "PyTorch version: 2.9.1+cu128\nIs debug build: False\nCUDA used to build PyTorch: 12.8\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.5 LTS (x86_64)\nGCC version: Could not collect\nClang version: Could not collect\nCMake version: version 4.1.2\nLibc version: glibc-2.35\n\nPython version: 3.11.14 | packaged by conda-forge | (main, Oct 22 2025, 22:46:25) [GCC 14.3.0] (64-bit runtime)\nPython platform: Linux-6.12.90-120.164.amzn2023.x86_64-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: Could not collect\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB\nNvidia driver version: 580.159.03\ncuDNN version: Could not collect\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           46 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  96\nOn-line CPU(s) list:                     0-95\nVendor ID:                               GenuineIntel\nModel name:                              Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz\nCPU family:                              6\nModel:                                   85\nThread(s) per core:                      2\nCore(s) per socket:                      24\nSocket(s):                               2\nStepping:                                7\nBogoMIPS:                                6000.00\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\nHypervisor vendor:                       KVM\nVirtualization type:                     full\nL1d cache:                               1.5 MiB (48 instances)\nL1i cache:                               1.5 MiB (48 instances)\nL2 cache:                                48 MiB (48 instances)\nL3 cache:                                71.5 MiB (2 instances)\nNUMA node(s):                            2\nNUMA node0 CPU(s):                       0-23,48-71\nNUMA node1 CPU(s):                       24-47,72-95\nVulnerability Gather data sampling:      Unknown: Dependent on hypervisor status\nVulnerability Indirect target selection: Mitigation; Aligned branch/return thunks\nVulnerability Itlb multihit:             KVM: Mitigation: VMX unsupported\nVulnerability L1tf:                      Mitigation; PTE Inversion\nVulnerability Mds:                       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown:                  Mitigation; PTI\nVulnerability Mmio stale data:           Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Reg file data sampling:    Not affected\nVulnerability Retbleed:                  Vulnerable\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Vulnerable\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Not affected\nVulnerability Vmscape:                   Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==2.3.4\n[pip3] nvidia-cublas-cu12==12.8.4.1\n[pip3] nvidia-cuda-cupti-cu12==12.8.90\n[pip3] nvidia-cuda-nvrtc-cu12==12.8.93\n[pip3] nvidia-cuda-runtime-cu12==12.8.90\n[pip3] nvidia-cudnn-cu12==9.10.2.21\n[pip3] nvidia-cufft-cu12==11.3.3.83\n[pip3] nvidia-curand-cu12==10.3.9.90\n[pip3] nvidia-cusolver-cu12==11.7.3.90\n[pip3] nvidia-cusparse-cu12==12.5.8.93\n[pip3] nvidia-cusparselt-cu12==0.7.1\n[pip3] nvidia-nccl-cu12==2.27.5\n[pip3] nvidia-nvjitlink-cu12==12.8.93\n[pip3] nvidia-nvtx-cu12==12.8.90\n[pip3] optree==0.17.0\n[pip3] torch==2.9.1+cu128\n[pip3] torchaudio==2.9.1+cu128\n[pip3] torchelastic==0.2.2\n[pip3] torchvision==0.24.1+cu128\n[pip3] triton==3.5.1\n[conda] numpy                        2.3.4            py311h2e04523_0       conda-forge\n[conda] nvidia-cublas-cu12           12.8.4.1         pypi_0                pypi\n[conda] nvidia-cuda-cupti-cu12       12.8.90          pypi_0                pypi\n[conda] nvidia-cuda-nvrtc-cu12       12.8.93          pypi_0                pypi\n[conda] nvidia-cuda-runtime-cu12     12.8.90          pypi_0                pypi\n[conda] nvidia-cudnn-cu12            9.10.2.21        pypi_0                pypi\n[conda] nvidia-cufft-cu12            11.3.3.83        pypi_0                pypi\n[conda] nvidia-curand-cu12           10.3.9.90        pypi_0                pypi\n[conda] nvidia-cusolver-cu12         11.7.3.90        pypi_0                pypi\n[conda] nvidia-cusparse-cu12         12.5.8.93        pypi_0                pypi\n[conda] nvidia-cusparselt-cu12       0.7.1            pypi_0                pypi\n[conda] nvidia-nccl-cu12             2.27.5           pypi_0                pypi\n[conda] nvidia-nvjitlink-cu12        12.8.93          pypi_0                pypi\n[conda] nvidia-nvtx-cu12             12.8.90          pypi_0                pypi\n[conda] optree                       0.17.0           pypi_0                pypi\n[conda] torch                        2.9.1+cu128      pypi_0                pypi\n[conda] torchaudio                   2.9.1+cu128      pypi_0                pypi\n[conda] torchelastic                 0.2.2            pypi_0                pypi\n[conda] torchvision                  0.24.1+cu128     pypi_0                pypi\n[conda] triton                       3.5.1            pypi_0                pypi",
+  "transformers_version": "4.54.0",
+  "lm_eval_version": "0.4.11",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|endoftext|>",
+    "151643"
+  ],
+  "tokenizer_eos_token": [
+    "<|im_end|>",
+    "151645"
+  ],
+  "tokenizer_bos_token": [
+    null,
+    "None"
+  ],
+  "eot_token_id": 151645,
+  "max_length": 16384,
+  "task_hashes": {
+    "ruler_qa_squad": "8c658a728ccd3d67ddd3f45e8e76527cf15a1ae4119033d8246f293ecc68cf11",
+    "ruler_cwe": "f68206129fc7d709951d550ec9269fed05e73bea2fc86ce9f2c758974db5db71",
+    "niah_multivalue": "29646655c230b4f155a8de1fbc7ddf22c49e43045bd8fdab5770e7cb45026c7c",
+    "niah_multikey_3": "de2cec52885e38d56e0bda0e344e989909969c71ab5f8048b9368a011dd3616b",
+    "niah_multikey_1": "338f81e9efa98dbe6979d4f178f7f06cdbd6d367072318667a932b43c5032ab8",
+    "niah_single_2": "88ba10e924c2fc6c47543eb646b9e6f7487c7ca026c9391d6cd7054fde59eaad"
+  },
+  "model_source": "hf",
+  "model_name": "/workspace/outputs/l2a_style/stage2/checkpoint-25",
+  "model_name_sanitized": "__workspace__outputs__l2a_style__stage2__checkpoint-25",
+  "system_instruction": null,
+  "system_instruction_sha": null,
+  "fewshot_as_multiturn": null,
+  "chat_template": null,
+  "chat_template_sha": null,
+  "total_evaluation_time_seconds": "859.353213556984"
+}

outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multikey_1_2026-07-18T12-26-30.235088.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a33e240ce70e10445a34fea2d42efc0519a02e24a140de317904411fc5f1ef03
+size 1445444

outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multikey_3_2026-07-18T12-26-30.235088.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6d2e7da44df1fc5f05d1a36a2576a831b9b7488a08a2a3749c885bbf1d34d6cd
+size 502030

outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_multivalue_2026-07-18T12-26-30.235088.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6bb6d9fdfc2c27b4b41ba425c1be458b8c3adcc4bf7d376447622c3772e7ffbb
+size 1433534

outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_niah_single_2_2026-07-18T12-26-30.235088.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c9f28fb7ee751ed53efff7df7ca83a1a5924c3ab6903c091ebc104c50ab765d5
+size 1438222

outputs/eval/token_t045/ruler8k_splits/b/lm_eval/__workspace__outputs__l2a_style__stage2__checkpoint-25/samples_ruler_cwe_2026-07-18T12-26-30.235088.jsonl ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6a5399a30e42235fe359c466f8b5a56f2ab4da5a81b78fadb4b9c8e353308fb1
+size 635553