Text Generation
Transformers
Safetensors
qwen3
rl
skyrl
agentic
swe
conversational
text-generation-inference
Instructions to use laion/ablation-pymethods2test-seqmean-arm0-30-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use laion/ablation-pymethods2test-seqmean-arm0-30-8B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="laion/ablation-pymethods2test-seqmean-arm0-30-8B") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("laion/ablation-pymethods2test-seqmean-arm0-30-8B") model = AutoModelForCausalLM.from_pretrained("laion/ablation-pymethods2test-seqmean-arm0-30-8B", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use laion/ablation-pymethods2test-seqmean-arm0-30-8B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "laion/ablation-pymethods2test-seqmean-arm0-30-8B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "laion/ablation-pymethods2test-seqmean-arm0-30-8B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/laion/ablation-pymethods2test-seqmean-arm0-30-8B
- SGLang
How to use laion/ablation-pymethods2test-seqmean-arm0-30-8B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "laion/ablation-pymethods2test-seqmean-arm0-30-8B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "laion/ablation-pymethods2test-seqmean-arm0-30-8B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "laion/ablation-pymethods2test-seqmean-arm0-30-8B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "laion/ablation-pymethods2test-seqmean-arm0-30-8B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use laion/ablation-pymethods2test-seqmean-arm0-30-8B with Docker Model Runner:
docker model run hf.co/laion/ablation-pymethods2test-seqmean-arm0-30-8B
Upload folder using huggingface_hub
Browse files- .gitattributes +4 -0
- training_logs/20260612_220124_metrics_job_630124.csv +29 -0
- training_logs/20260612_220124_metrics_job_630127.csv +29 -0
- training_logs/20260612_220124_metrics_job_630130.csv +15 -0
- training_logs/20260612_220124_metrics_job_661735.csv +9 -0
- training_logs/20260612_220124_metrics_table.csv +79 -0
- training_logs/20260612_220124_vllm_metrics_job_630124.csv +0 -0
- training_logs/20260612_220124_vllm_metrics_job_630127.csv +0 -0
- training_logs/20260612_220124_vllm_metrics_job_630130.csv +0 -0
- training_logs/20260612_220124_vllm_metrics_job_661734.csv +1153 -0
- training_logs/20260612_220124_vllm_metrics_job_661735.csv +0 -0
- training_logs/20260612_220124_vllm_metrics_table.csv +0 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_630124.out +3 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_630125.out +0 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_630126.out +0 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_630127.out +3 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_630128.out +370 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_630129.out +0 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_630130.out +3 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_657667.out +17 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_657668.out +17 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_657669.out +17 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_657670.out +17 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_657671.out +17 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_661733.out +364 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_661734.out +0 -0
- training_logs/ablation-pymethods2test-seqmean-arm0_661735.out +3 -0
.gitattributes
CHANGED
|
@@ -34,3 +34,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
training_logs/ablation-pymethods2test-seqmean-arm0_630124.out filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
training_logs/ablation-pymethods2test-seqmean-arm0_630127.out filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
training_logs/ablation-pymethods2test-seqmean-arm0_630130.out filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
training_logs/ablation-pymethods2test-seqmean-arm0_661735.out filter=lfs diff=lfs merge=lfs -text
|
training_logs/20260612_220124_metrics_job_630124.csv
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError
|
| 2 |
+
0.0,0,64,512,0,0.0,0,0.0,4169.1133,3891.7792,5095.1271,29264,1263,2914.3086,0.7695,-0.0069,0.0744,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1189,0.0,0.0,1.0,0.0,0.0105,0.8125,0.7695,11.6896,50.3563,578.7643,32.5,857.9675,279.2032,0.1941,4.3043,30.7211,146.6405,177.747,3371.9765,22.9409,146.8314,3166.9842,0,1,119,952,918,22,26,0.04201680672268908,5.0,0.025210084033613446,3.0,0.008403361344537815,1.0,0.10084033613445378,12,0.14285714285714285,17,,,,,,,,,
|
| 3 |
+
0.0,0,64,512,1,0.8594,0,0.8594,5688.084,5824.2419,5589.5185,31885,1,5477.7731,0.4199,0.0464,0.1255,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1361,-0.005,0.0,1.0,0.0,0.0161,0.5781,0.4199,12.6834,50.5541,574.2212,33.1,857.9675,283.7463,0.0641,4.8042,39.1437,260.9703,300.4542,1718.0337,24.4369,261.2461,1388.3356,0,2,63,504,454,24,37,0.15873015873015872,10.0,0.047619047619047616,3.0,0.06349206349206349,4.0,0.36507936507936506,23,0.15873015873015872,10,14.2681,24.8089,,,,,,,
|
| 4 |
+
0.0,0,64,512,2,1.5625,0,0.9375,5287.3867,5200.8475,5333.1104,31817,1,5758.917,0.3457,0.0073,0.0951,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,-0.0021,0.0,1.0,0.0,0.0171,0.4688,0.3457,12.9334,50.6717,572.7552,33.2,857.9675,285.2123,0.1003,4.7883,38.0045,267.5741,305.9524,1220.661,24.4652,267.8472,885.4504,0,3,62,496,468,14,23,0.08064516129032258,5.0,,,,,0.3064516129032258,19,0.0967741935483871,6,,,,,,,,,
|
| 5 |
+
0.0,0,64,512,3,1.8438,0,0.875,5684.4512,4907.0708,6233.8,31815,1,5008.2154,0.4141,-0.0061,0.0833,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1411,-0.004,0.0,1.0,0.0,0.0121,0.4844,0.4141,13.7453,50.795,563.5295,34.3,857.9675,294.438,0.0673,4.9105,37.4051,234.7096,272.4655,1357.8902,20.825,234.9927,1059.6856,0,4,62,496,451,21,36,0.0967741935483871,6.0,0.04838709677419355,3.0,,,0.3709677419354839,23,0.22580645161290322,14,0.008,12.7181,,,,,,,
|
| 6 |
+
0.0,0,64,512,4,1.875,0,0.9219,5039.3906,5546.9962,4499.0363,31714,1,4334.7369,0.5156,0.0318,0.1173,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1279,-0.0036,0.0,1.0,0.0,0.0141,0.625,0.5156,13.8122,50.8325,562.6003,34.4,857.9675,295.3672,0.0989,4.6928,39.1023,199.9551,239.4905,1224.4331,24.7234,200.2888,955.52,0,5,62,496,449,22,32,0.11290322580645161,7.0,0.12903225806451613,8.0,,,0.3387096774193548,21,0.1935483870967742,12,,,8.2216,,,,,,
|
| 7 |
+
0.0,0,64,512,4,2.1719,0,0.9375,5591.9941,5830.8641,5431.1863,31798,1,4994.2618,0.4023,0.0066,0.0843,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0004,0.0,1.0,0.0,0.0111,0.4844,0.4023,13.8712,50.9265,561.6264,34.5,857.9675,296.3411,0.0653,4.9236,37.4489,233.3771,271.219,1281.0159,22.6774,233.7045,982.1918,0,6,62,496,435,23,51,0.12903225806451613,8.0,0.03225806451612903,2.0,0.04838709677419355,3.0,0.5161290322580645,32,0.25806451612903225,16,0.0067,13.924,,0.016129032258064516,1.0,,,,
|
| 8 |
+
0.0,0,64,512,4,2.1094,0,0.9062,4851.3477,5205.5652,4562.4468,31817,1,4336.985,0.4492,0.0059,0.0777,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1271,-0.0014,0.0,1.0,0.0,0.0139,0.5469,0.4492,13.9109,50.9409,561.0771,34.6,857.9675,296.8904,0.0644,4.6898,33.9517,197.6421,231.9494,1559.2848,23.4677,197.9329,1299.1744,0,7,63,504,457,15,42,,,0.07936507936507936,5.0,0.031746031746031744,2.0,0.5555555555555556,35,0.07936507936507936,5,,,,,,,,,
|
| 9 |
+
0.0,0,64,512,4,2.1094,0,0.9219,5596.0137,5715.5123,5487.2164,31840,1,5416.4656,0.4766,0.0199,0.145,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,0.0,0.0,1.0,0.0,0.0136,0.625,0.4766,13.9023,50.9439,561.6647,34.5,857.9675,296.3028,0.0672,4.797,43.3436,268.6347,312.3633,1610.273,22.6954,268.9523,1270.4134,0,8,62,496,457,18,38,,,,,0.016129032258064516,1.0,0.4032258064516129,25,0.20967741935483872,13,0.0071,12.1975,,,,0.016129032258064516,1.0,,
|
| 10 |
+
0.0,0,64,512,4,1.9844,0,0.9531,5326.7715,5167.9342,5454.2887,31758,1,4691.8758,0.4453,0.0119,0.0971,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1268,0.0,0.0,1.0,0.0,0.0137,0.5469,0.4453,13.8936,50.9354,559.6754,34.8,857.9675,298.2921,0.0782,4.7117,38.0949,222.0535,260.5136,1321.2856,23.5801,222.3401,1032.476,0,9,61,488,449,18,38,,,0.01639344262295082,1.0,0.03278688524590164,2.0,0.4426229508196721,27,0.22950819672131148,14,,,,,,,,,
|
| 11 |
+
0.0,0,64,512,3,1.8906,0,0.9062,4968.5547,4853.8377,5036.8131,31768,1,4721.9898,0.373,0.0139,0.118,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1359,0.0,0.0,1.0,0.0,0.0144,0.5312,0.373,13.9257,51.0073,560.3441,34.7,857.9675,297.6235,0.0714,4.8344,33.5456,220.0195,253.8928,1294.919,24.3901,220.2753,1011.7949,0,10,62,496,442,18,46,,,0.12903225806451613,8.0,0.016129032258064516,1.0,0.6451612903225806,40,0.12903225806451613,8,0.0084,14.0711,8.8277,,,,,,
|
| 12 |
+
0.0,0,64,512,4,1.7812,0,0.9219,4462.5332,4532.996,4394.2385,31762,1,4298.9711,0.4922,0.0008,0.1662,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,-0.0056,0.0,1.0,0.0,0.0183,0.6562,0.4922,14.0333,51.1276,559.0244,34.8,857.9675,298.9431,0.0938,4.6549,32.2602,198.6454,231.2913,1475.0347,24.5599,198.9368,1214.5237,0,11,64,512,471,16,33,,,0.125,8.0,0.046875,3.0,0.328125,21,0.140625,9,,,,,,,,,
|
| 13 |
+
0.0,0,64,512,4,2.0938,0,0.8906,5849.1094,5035.9944,6282.4461,31797,1,4813.1067,0.3477,-0.0072,0.0606,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1327,0.0,0.0,1.0,0.0,0.013,0.4219,0.3477,14.0347,51.1288,558.5858,34.9,857.9675,299.3817,0.0732,5.2651,38.6448,224.9423,263.9456,1559.3505,25.6632,225.2273,1264.4721,0,12,57,456,426,14,30,,,,,0.07017543859649122,4.0,0.42105263157894735,24,0.14035087719298245,8,0.0084,14.0752,,,,,,,
|
| 14 |
+
0.0,0,64,512,4,1.8125,0,0.875,4866.082,5038.6537,4692.1569,31760,1,4066.2403,0.502,-0.0034,0.1269,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1255,-0.0,0.0,1.0,0.0,0.0186,0.625,0.502,14.0677,51.1943,557.1813,35.1,857.9675,300.7862,0.0704,4.6745,31.6351,183.7078,215.6617,1431.9056,23.0843,183.9558,1188.48,0,13,64,512,450,21,49,,,0.203125,13.0,0.25,16.0,0.46875,30,0.09375,6,,,,,,,,,
|
| 15 |
+
0.0,0,64,512,4,2.0469,0,0.9375,4926.5117,4839.1749,4993.9031,31751,1,4266.9039,0.4355,0.0028,0.0804,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1228,-0.0013,0.0,1.0,0.0,0.0162,0.5469,0.4355,14.322,51.506,557.3558,35.0,857.9675,300.6118,0.0972,4.7529,33.9188,193.7138,228.0319,1142.7788,24.0818,194.0154,885.9041,0,14,63,504,473,18,30,,,,,,,0.3333333333333333,21,0.14285714285714285,9,0.0084,13.3568,,,,,,0.015873015873015872,1.0
|
| 16 |
+
0.0,0,64,512,4,2.0312,0,0.9219,5025.9766,4558.8924,5475.1648,31707,1,4544.2113,0.4902,0.0047,0.108,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1336,-0.0,0.0,1.0,0.0,0.0154,0.6094,0.4902,14.6948,51.8791,556.9661,35.1,857.9675,301.0014,0.0749,4.7275,31.8001,218.695,250.8643,1313.9526,22.5691,218.989,1035.7868,0,15,64,512,477,18,28,,,0.109375,7.0,,,0.296875,19,0.1875,12,,,8.2163,,,,,,
|
| 17 |
+
0.0,0,64,512,4,1.9375,0,0.8906,5304.2988,4924.3203,5616.6655,31702,1,4477.752,0.4512,0.0092,0.1087,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1249,-0.0,0.0,1.0,0.0,0.0148,0.5781,0.4512,14.4315,51.6356,556.9907,35.1,857.9675,300.9769,0.1006,4.7795,36.8211,219.4484,256.6458,1308.8006,22.9796,219.7236,1024.3882,0,16,61,488,421,21,66,,,,,0.09836065573770492,6.0,0.8360655737704918,51,0.16393442622950818,10,0.0077,14.8915,,,,0.01639344262295082,1.0,,
|
| 18 |
+
0.0,0,64,512,4,2.3594,0,0.9375,5391.4434,6104.0653,4938.3706,31758,1,5320.6865,0.3887,0.0223,0.0896,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1221,-0.002,0.0,1.0,0.0,0.0108,0.5156,0.3887,14.5796,51.7618,556.1977,35.2,857.9675,301.7698,0.0974,5.3524,38.0588,253.8113,292.2721,1220.8525,24.0951,254.1154,899.1259,0,17,57,456,419,17,35,,,0.017543859649122806,1.0,0.03508771929824561,2.0,0.38596491228070173,22,0.24561403508771928,14,,,,,,0.017543859649122806,1.0,,
|
| 19 |
+
0.0,0,64,512,4,1.7188,0,0.875,5066.6953,5047.4764,5082.7455,31770,1,4672.0163,0.4551,0.0005,0.1098,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1223,-0.0028,0.0,1.0,0.0,0.0154,0.5625,0.4551,14.7728,51.9543,556.2234,35.2,857.9675,301.7441,0.0991,4.843,39.5878,215.8858,255.888,1582.6579,22.1364,216.2007,1299.7832,0,18,60,480,437,18,41,,,0.03333333333333333,2.0,0.016666666666666666,1.0,0.5333333333333333,32,0.16666666666666666,10,0.0075,16.3173,,,,,,,
|
| 20 |
+
0.0,0,64,512,4,1.8281,0,0.8906,5320.7246,4968.5291,5526.808,31825,1,4635.3488,0.3691,-0.0052,0.097,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1408,-0.0,0.0,1.0,0.0,0.0136,0.4531,0.3691,14.7764,51.9575,554.7101,35.3,857.9675,303.2574,0.0666,4.8474,33.322,217.0675,250.7206,1533.2708,23.3393,217.3317,1254.3583,0,19,62,496,453,17,43,,,,,,,0.6129032258064516,38,0.0967741935483871,6,,,,,,,,,
|
| 21 |
+
0.0,0,64,512,4,1.6875,0,0.8906,4981.0215,5122.7233,4885.6275,31734,1,4443.4633,0.4023,-0.0006,0.0906,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1347,0.0,0.0,1.0,0.0,0.0168,0.5156,0.4023,14.7832,52.0205,554.9844,35.3,857.9675,302.9831,0.0665,4.7337,34.0494,210.2643,244.6153,1501.26,23.3047,210.499,1228.6018,0,20,63,504,461,18,38,,,0.047619047619047616,3.0,0.06349206349206349,4.0,0.5396825396825397,34,0.06349206349206349,4,0.0078,13.3388,8.0322,,,0.031746031746031744,2.0,,
|
| 22 |
+
0.0,0,64,512,4,1.7969,0,0.8906,5089.5234,5337.2258,4948.1963,31834,1,4103.7515,0.3633,-0.0098,0.0985,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1382,-0.0055,0.0,1.0,0.0,0.0123,0.4688,0.3633,14.7559,51.9613,554.3621,35.4,857.9675,303.6055,0.0721,4.8151,32.0319,197.6414,230.0275,1384.6688,23.9344,197.9232,1125.887,0,21,61,488,432,21,46,,,0.13114754098360656,8.0,,,0.6229508196721312,38,0.13114754098360656,8,,,,,,0.03278688524590164,2.0,,
|
| 23 |
+
0.0,0,64,512,3,1.5156,0,0.8281,4888.4629,4445.7686,5246.6855,31774,1,4638.3152,0.4473,0.0144,0.0967,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1277,0.0,0.0,1.0,0.0,0.0135,0.5781,0.4473,15.1406,52.3359,554.3356,35.4,857.9675,303.6319,0.0672,4.8409,37.4714,213.0209,250.8158,1341.0965,22.4572,213.2768,1062.9785,0,22,64,512,452,21,60,,,,,0.046875,3.0,0.734375,47,0.15625,10,0.0085,14.204,,,,,,,
|
| 24 |
+
0.0,0,64,512,4,1.6094,0,0.8594,5010.1426,5339.5915,4775.4515,31738,1,5125.7203,0.416,0.0029,0.123,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1269,-0.0,0.0,1.0,0.0,0.0161,0.5781,0.416,15.1399,52.3352,554.2477,35.4,857.9675,303.7198,0.0666,4.7525,38.7094,241.3185,280.3783,1636.7829,23.2267,241.602,1328.421,0,23,61,488,423,17,65,,,,,0.04918032786885246,3.0,0.639344262295082,39,0.3770491803278688,23,,,,,,,,,
|
| 25 |
+
0.0,0,64,512,3,1.5156,0,0.8281,4957.0039,5145.7238,4853.8066,31744,1,4792.2542,0.3535,-0.005,0.139,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1197,0.0,0.0,1.0,0.0,0.0118,0.4531,0.3535,15.2222,52.591,552.66,35.6,857.9675,305.3075,0.0667,4.8036,34.8997,225.7814,261.0063,1813.0727,21.5017,226.0394,1525.7569,0,24,53,424,405,10,17,,,0.1509433962264151,8.0,0.09433962264150944,5.0,0.1320754716981132,7,0.24528301886792453,13,0.0085,13.7903,,,,,,,
|
| 26 |
+
0.0,0,64,512,4,1.8281,0,0.9062,4999.9961,4608.5735,5274.3821,31665,1,3727.2595,0.4121,0.0076,0.0996,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1278,-0.0,0.0,1.0,0.0,0.0129,0.4844,0.4121,15.4929,52.9026,551.8461,35.7,857.9675,306.1215,0.0644,4.6949,29.2144,181.8811,211.389,988.3901,25.1751,182.1098,747.1268,0,25,54,432,406,12,19,,,0.12962962962962962,7.0,0.07407407407407407,4.0,0.2777777777777778,15,0.1111111111111111,6,,,8.7088,,,,,,
|
| 27 |
+
0.0,0,64,512,4,1.7344,0,0.875,4801.6133,4766.3344,4852.3476,31766,1,3813.5277,0.5898,-0.0032,0.1052,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1273,-0.0035,0.0,1.0,0.0,0.0164,0.6875,0.5898,15.8889,53.344,552.8224,35.6,857.9675,305.1451,0.0946,4.7486,31.8278,188.1649,220.5231,1031.4949,22.9306,188.6001,783.2847,0,26,59,472,410,26,50,,,0.11864406779661017,7.0,0.3050847457627119,18.0,0.3220338983050847,19,0.23728813559322035,14,0.009,12.6671,,,,0.288135593220339,17.0,,
|
| 28 |
+
0.0,0,64,512,4,2.1875,0,0.9688,5055.6387,4951.9907,5130.67,31763,1,4942.405,0.4199,0.0045,0.1012,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1162,-0.01,0.0,1.0,0.0,0.0135,0.5469,0.4199,16.046,53.4671,552.5507,35.6,857.9675,305.4168,0.0766,4.8488,33.5052,219.951,253.8289,2130.9697,18.5315,220.2467,1853.7559,0,27,52,416,390,13,6,,,0.28846153846153844,15.0,,,0.2692307692307692,14,0.057692307692307696,3,,,,,,0.19230769230769232,10.0,0.019230769230769232,1.0
|
| 29 |
+
0.0,0,64,512,4,1.6719,0,0.8281,4799.998,4474.8537,5214.7378,21478,1,3336.6768,0.5605,0.0081,0.0474,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1169,-0.0193,0.0,1.0,0.0,0.0089,0.625,0.5605,15.7877,53.2084,552.8907,35.6,857.9675,305.0768,0.0486,3.4809,28.5303,161.7902,190.6488,998.4337,25.9964,162.0695,778.3023,0,28,44,352,335,11,15,,,0.13636363636363635,6.0,0.022727272727272728,1.0,0.4772727272727273,21,0.45454545454545453,20,0.0082,11.9058,,,,0.022727272727272728,1.0,,
|
training_logs/20260612_220124_metrics_job_630127.csv
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError
|
| 2 |
+
0.0,0,64,512,5,0.8125,0,0.2031,5219.4023,5279.0935,5148.4872,24019,1,3593.1869,0.543,0.0174,0.1093,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.127,-0.0059,0.0,1.0,0.0,0.0136,0.661,0.543,9.9914,50.291,594.6118,30.7,857.9675,263.3557,0.2443,3.7877,34.8635,181.2173,216.5936,3907.0864,25.2212,181.4851,3661.4836,0,29,54,432,397,20,35,0.18518518518518517,10,0.48148148148148145,26,0.05555555555555555,3.0,,,,,,,
|
| 3 |
+
0.0,0,64,512,1,0.875,0,0.875,6119.2617,5371.9608,6614.2273,31787,1,5360.732,0.3984,0.0066,0.11,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1391,-0.0,0.0,1.0,0.0,0.0133,0.5156,0.3984,12.6683,50.5745,589.7981,31.3,857.9675,268.1694,0.0668,4.853,45.3325,249.4957,295.1659,1561.0798,22.8392,249.7659,1238.2155,0,30,51,408,360,18,48,0.19607843137254902,10,0.9803921568627451,50,0.0784313725490196,4.0,17.6986,25.6657,6.5413,0.0784313725490196,4.0,,
|
| 4 |
+
0.0,0,64,512,2,1.7031,1,1.0,5416.0312,5191.9848,5493.8579,31782,1,5258.8542,0.2578,0.0188,0.1227,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1224,-0.0045,0.0,1.0,0.0,0.0132,0.3906,0.2578,13.0473,50.9899,588.6119,31.4,857.9675,269.3556,0.0915,4.9642,38.9391,242.4843,281.7563,558.6231,24.3077,242.7249,247.5902,0,31,62,496,450,16,38,0.03225806451612903,2,0.5806451612903226,36,,,,,,0.06451612903225806,4.0,,
|
| 5 |
+
0.0,0,64,512,3,1.8438,0,0.8438,4876.5957,4321.0181,5298.5292,31788,1,4349.0068,0.4316,0.008,0.1024,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1299,-0.0033,0.0,1.0,0.0,0.0147,0.5469,0.4316,13.3226,51.1669,587.694,31.5,857.9675,270.2736,0.0969,4.6902,32.9398,184.8866,218.1917,1881.6998,22.4991,185.1545,1636.313,0,32,61,488,435,20,50,0.26229508196721313,16,0.5245901639344263,32,0.04918032786885246,3.0,0.0086,8.4163,,0.04918032786885246,3.0,,
|
| 6 |
+
0.0,0,64,512,4,1.9688,0,0.8906,5456.2109,4808.7673,5878.0935,31806,1,4961.1718,0.3945,0.0083,0.1325,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,0.0,0.0,1.0,0.0,0.0126,0.5469,0.3945,13.4299,51.223,587.0558,31.6,857.9675,270.9117,0.067,4.9084,36.655,238.8678,275.8322,1516.0964,23.2669,239.1098,1212.085,0,33,62,496,436,21,55,0.27419354838709675,17,0.5806451612903226,36,0.03225806451612903,2.0,,,,0.08064516129032258,5.0,,
|
| 7 |
+
0.0,0,64,512,4,2.125,0,0.9844,5502.4434,5212.8122,5708.7692,31765,1,5396.2041,0.416,-0.004,0.0835,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0,0.0,1.0,0.0,0.0137,0.5156,0.416,13.5508,51.3198,587.3987,31.5,857.9675,270.5688,0.0691,5.493,44.4908,256.2057,301.0644,1068.5266,22.5088,256.5041,739.4567,0,34,63,504,462,14,41,0.23809523809523808,15,0.3968253968253968,25,0.015873015873015872,1.0,0.0088,8.3216,,0.015873015873015872,1.0,,
|
| 8 |
+
0.0,0,64,512,3,1.8594,0,0.9219,5384.9941,5343.019,5414.1821,31664,1,4412.7671,0.4102,0.0025,0.0519,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1331,-0.002,0.0,1.0,0.0,0.0142,0.5,0.4102,13.7969,51.3292,586.7628,31.6,857.9675,271.2047,0.0906,4.7764,33.1627,211.1102,244.5902,1384.0633,22.1442,211.3362,1112.5444,0,35,61,488,463,8,17,0.04918032786885246,3,0.26229508196721313,16,,,,,5.8995,0.13114754098360656,8.0,,
|
| 9 |
+
0.0,0,64,512,4,1.9219,0,0.9062,5825.4473,5333.0212,6246.5072,31794,1,4498.7568,0.4609,0.0027,0.1742,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1384,-0.0,0.0,1.0,0.0,0.0171,0.6406,0.4609,13.8943,51.4631,586.2767,31.7,857.9675,271.6909,0.5695,4.8349,37.7488,223.605,262.1992,1464.7075,20.515,223.8804,1177.1535,0,36,64,512,471,15,40,0.109375,7,0.515625,33,,,0.009,7.9437,,0.015625,1.0,,
|
| 10 |
+
0.0,0,64,512,4,1.8438,0,0.9062,5148.5137,4631.637,5834.55,31764,1,4648.1595,0.5703,-0.0029,0.1133,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,-0.0011,0.0,1.0,0.0,0.0167,0.6875,0.5703,14.5586,51.6496,584.9858,31.8,857.9675,272.9817,0.0919,4.8241,38.277,222.4597,261.0628,1077.7825,26.2516,222.6934,785.6361,0,37,61,488,451,15,35,0.14754098360655737,9,0.4262295081967213,26,0.09836065573770492,6.0,,,,0.03278688524590164,2.0,,
|
| 11 |
+
0.0,0,64,512,4,2.1875,0,0.9375,5691.4238,5458.8718,5887.1691,31763,1,4985.511,0.457,0.0025,0.1314,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1239,-0.0,0.0,1.0,0.0,0.0158,0.6094,0.457,14.4333,51.5906,584.4034,31.9,857.9675,273.5641,0.8661,4.8087,37.1183,229.9869,268.1954,1281.4976,24.0319,230.2106,984.4536,0,38,63,504,457,20,44,0.1111111111111111,7,0.5238095238095238,33,,,0.0082,8.3888,,0.047619047619047616,3.0,,
|
| 12 |
+
0.0,0,64,512,4,1.875,0,0.9219,4852.4375,4700.9213,4963.0034,31721,1,3852.6075,0.4219,-0.0118,0.1369,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1297,-0.0025,0.0,1.0,0.0,0.0151,0.5625,0.4219,14.8096,51.9002,584.34,31.9,857.9675,273.6276,0.1019,4.7148,35.3281,188.2665,223.9309,1220.1543,22.2426,188.5005,969.2585,0,39,63,504,446,22,48,0.09523809523809523,6,0.6666666666666666,42,0.031746031746031744,2.0,,,,0.15873015873015872,10.0,,
|
| 13 |
+
0.0,0,64,512,4,1.8906,0,0.8438,4717.9707,4944.8443,4511.4142,31770,1,4317.3591,0.4766,0.0072,0.086,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1232,0.0,0.0,1.0,0.0,0.0143,0.5938,0.4766,14.7138,51.7721,583.8667,31.9,857.9675,274.1008,0.0992,4.8622,31.9697,193.9941,226.3482,1608.4681,22.3175,194.2789,1354.9332,0,40,60,480,456,10,24,0.05,3,0.36666666666666664,22,0.03333333333333333,2.0,0.0074,8.3081,6.0239,,,,
|
| 14 |
+
0.0,0,64,512,4,1.6875,0,0.9531,5037.7695,5312.8784,4764.8016,27138,1,3679.8466,0.498,-0.0001,0.0849,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1349,-0.0,0.0,1.0,0.0,0.0122,0.5781,0.498,14.5892,51.6475,583.7137,32.0,857.9675,274.2538,0.0824,4.1919,29.9247,176.4747,206.7121,1037.8229,24.3689,176.7045,802.5427,0,41,60,480,441,19,37,0.18333333333333332,11,0.43333333333333335,26,,,,,,0.016666666666666666,1.0,0.016666666666666666,1.0
|
| 15 |
+
0.0,0,64,512,4,1.9844,0,0.8906,5756.3848,5835.0311,5708.8025,31761,1,4473.6908,0.377,0.0069,0.1087,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1346,0.0,0.0,1.0,0.0,0.013,0.4531,0.377,14.9636,52.1455,583.6249,32.0,857.9675,274.3426,0.0719,4.8088,36.9135,222.7902,260.0054,1385.1388,21.7016,223.0196,1098.6195,0,42,61,488,424,20,50,0.22950819672131148,14,0.5573770491803278,34,0.03278688524590164,2.0,0.0076,7.9965,,0.22950819672131148,14.0,,
|
| 16 |
+
0.0,0,64,512,4,1.5781,0,0.8281,4886.5996,4211.4396,5344.823,31794,1,4658.4948,0.4043,0.003,0.0477,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.119,-0.0014,0.0,1.0,0.0,0.0109,0.4844,0.4043,14.9648,52.146,583.4162,32.0,857.9675,274.5513,0.0932,4.8883,35.3553,210.4627,246.1689,1473.9582,23.9821,210.7199,1198.9132,0,43,61,488,441,16,45,0.16393442622950818,10,0.5081967213114754,31,0.06557377049180328,4.0,,,,0.03278688524590164,2.0,,
|
| 17 |
+
0.0,0,64,512,4,1.9375,0,0.9844,5006.252,5006.7137,5005.7938,31832,1,4198.283,0.498,0.0058,0.1318,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1302,-0.0013,0.0,1.0,0.0,0.0142,0.625,0.498,15.1405,52.1464,582.1508,32.1,857.9675,275.8167,0.0701,4.8376,37.1394,204.9028,242.3402,995.3073,21.6574,205.1302,726.4685,0,44,63,504,464,15,40,0.2698412698412698,17,0.3492063492063492,22,0.015873015873015872,1.0,0.0087,8.5306,,,,,
|
| 18 |
+
0.0,0,64,512,4,1.9688,0,0.9531,5307.5059,4945.0996,5629.7934,31811,1,4502.6994,0.4707,0.0034,0.0645,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0087,0.5156,0.4707,15.2917,52.3331,581.6784,32.2,857.9675,276.2891,0.0713,4.8688,35.6411,219.1314,255.1244,1030.1018,22.2906,219.4115,747.8149,0,45,57,456,419,11,20,0.07017543859649122,4,0.3508771929824561,20,0.03508771929824561,2.0,,,7.1117,0.2982456140350877,17.0,,
|
| 19 |
+
0.0,0,64,512,4,2.3281,0,0.9531,5162.7871,4802.6639,5490.6604,31759,1,4252.9044,0.4766,-0.0039,0.0939,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1296,0.0,0.0,1.0,0.0,0.0135,0.5625,0.4766,15.655,52.6444,580.4724,32.3,857.9675,277.4952,0.1001,4.9649,38.1605,200.0621,238.5392,1116.6096,24.6502,200.2781,848.4495,0,46,62,496,452,18,44,0.14516129032258066,9,0.5483870967741935,34,0.016129032258064516,1.0,0.0077,8.5628,,,,,
|
| 20 |
+
0.0,0,64,512,4,2.1094,0,0.9375,5453.8691,5448.0192,5459.9522,31779,1,4423.535,0.5098,0.01,0.1256,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0129,0.625,0.5098,15.5344,52.5839,580.0443,32.4,857.9675,277.9232,0.0896,4.8374,35.3367,219.7689,255.4572,1436.5679,22.4966,220.0305,1153.7719,0,47,61,488,451,18,36,0.16393442622950818,10,0.3770491803278688,23,0.04918032786885246,3.0,,,,0.01639344262295082,1.0,,
|
| 21 |
+
0.0,0,64,512,4,1.875,0,0.875,5230.4609,4627.1281,5626.8252,31716,1,4361.9157,0.3965,-0.0025,0.057,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1316,-0.0,0.0,1.0,0.0,0.0113,0.5,0.3965,15.6175,52.7105,579.3495,32.5,857.9675,278.618,0.0831,4.8056,35.7289,205.2303,241.3389,1388.9346,24.246,205.5265,1118.5394,0,48,63,504,448,25,55,0.1746031746031746,11,0.6666666666666666,42,0.031746031746031744,2.0,0.0078,8.0292,,0.015873015873015872,1.0,,
|
| 22 |
+
0.0,0,64,512,4,1.9688,0,0.9062,5567.3223,3973.2919,6469.1437,31784,1,5829.7172,0.3613,-0.0074,0.1077,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0168,0.4844,0.3613,15.5988,52.7747,578.7762,32.5,857.9675,279.1913,0.1107,4.8687,41.1551,269.7094,311.286,1674.2094,22.9805,270.0198,1335.0696,0,49,60,480,440,17,39,0.13333333333333333,8,0.5166666666666667,31,,,,,,0.016666666666666666,1.0,,
|
| 23 |
+
0.0,0,64,512,3,1.6875,0,0.9375,5274.1113,4738.0175,5707.9117,31829,1,4398.882,0.4473,0.008,0.0793,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0132,0.5469,0.4473,15.6793,52.8362,578.3136,32.6,857.9675,279.6539,0.0972,4.7852,34.4386,207.8657,242.6571,1060.2084,23.4244,208.1209,789.3372,0,50,65,520,431,21,76,0.12307692307692308,8,0.9538461538461539,62,0.09230769230769231,6.0,0.0081,9.1305,5.9495,0.18461538461538463,12.0,0.015384615384615385,1.0
|
| 24 |
+
0.0,0,64,512,4,1.9219,0,0.8906,5243.7461,5174.6447,5286.9619,31743,1,5188.684,0.3848,-0.0033,0.0945,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1207,0.0,0.0,1.0,0.0,0.0967,0.4531,0.3848,15.6188,52.7762,577.5959,32.7,857.9675,280.3716,0.076,4.7495,40.1813,238.4651,278.9729,1686.7408,24.1315,238.7151,1378.8824,0,51,62,496,446,22,49,0.20967741935483872,13,0.5645161290322581,35,0.016129032258064516,1.0,,,,0.016129032258064516,1.0,,
|
| 25 |
+
0.0,0,64,512,4,1.7812,0,0.8438,5252.4961,5411.8429,5141.6921,31823,1,4683.711,0.4102,0.0062,0.0796,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.133,-0.002,0.0,1.0,0.0,0.0124,0.5312,0.4102,15.7651,52.9658,577.1581,32.7,857.9675,280.8094,0.0944,4.7678,38.5332,214.205,253.0737,1600.439,23.268,214.4456,1319.3232,0,52,61,488,451,14,37,0.08196721311475409,5,0.4918032786885246,30,0.03278688524590164,2.0,0.0082,8.7241,,,,,
|
| 26 |
+
0.0,0,64,512,3,1.8125,0,0.9531,5107.9043,4684.3886,5450.6078,31689,1,4035.9699,0.4473,-0.0001,0.0667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.132,0.0,0.0,1.0,0.0,0.0132,0.5312,0.4473,15.7054,52.9061,577.4001,32.7,857.9675,280.5674,0.097,4.8708,33.9418,188.7929,223.0849,1082.9166,25.281,189.0458,829.6739,0,53,63,504,454,17,47,0.015873015873015872,1,0.6984126984126984,44,0.031746031746031744,2.0,,,,0.047619047619047616,3.0,,
|
| 27 |
+
0.0,0,64,512,3,1.7969,0,0.875,5634.0664,5882.4221,5476.1661,31801,1,5198.4542,0.3887,0.0101,0.0932,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1305,0.0,0.0,1.0,0.0,0.0144,0.5156,0.3887,15.8018,52.9791,576.4988,32.8,857.9675,281.4687,0.0964,4.9065,40.1938,233.8941,274.5002,1827.1928,24.3101,234.2096,1523.4687,0,54,61,488,434,19,52,0.13114754098360656,8,0.6557377049180327,40,0.08196721311475409,5.0,0.0084,8.6128,,0.03278688524590164,2.0,,
|
| 28 |
+
0.0,0,64,512,4,1.9375,0,0.9531,5371.4082,5022.761,5604.2182,31785,1,4782.3049,0.4004,-0.0082,0.1177,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1222,-0.0007,0.0,1.0,0.0,0.0128,0.5156,0.4004,15.8948,53.0398,576.6206,32.8,857.9675,281.3469,0.0653,4.8129,41.537,228.2535,270.096,1483.9509,23.783,228.4932,1185.2547,0,55,62,496,429,19,64,0.0967741935483871,6,0.9032258064516129,56,0.04838709677419355,3.0,,,6.0259,0.04838709677419355,3.0,,
|
| 29 |
+
0.0,0,64,512,4,1.6094,0,0.9062,4940.75,4904.8341,4965.9269,31836,1,4570.0348,0.4121,-0.0024,0.1498,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.124,-0.0,0.0,1.0,0.0,0.0148,0.5156,0.4121,16.2064,53.3552,575.7253,32.9,857.9675,282.2422,0.0989,4.7645,34.0253,209.3135,243.7093,1375.7686,22.7461,209.5845,1104.5439,0,56,59,472,390,33,69,0.288135593220339,17,0.5084745762711864,30,0.3728813559322034,22.0,0.0075,15.5646,,0.22033898305084745,13.0,,
|
training_logs/20260612_220124_metrics_job_630130.csv
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError
|
| 2 |
+
0.0,0,64,512,4,0.3438,0,0.1406,2745.916,4536.7524,2283.9066,31842,1,3652.8031,0.2051,0.0705,0.1449,-0.0007,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0683,-0.0442,0.0,1.0,0.0,0.0137,0.3898,0.2051,12.7084,50.5586,566.71,33.9,857.9675,291.2575,0.1351,4.4486,104.946,152.1059,257.396,10105.4336,18.2174,152.3142,9825.3711,0,57,63,504,281,58,154,1.8253968253968254,115,0.19047619047619047,12.0,0.42857142857142855,27,0.8095238095238095,51.0,0.25396825396825395,16.0,0.031746031746031744,2.0,,,,,
|
| 3 |
+
0.0,0,64,512,1,0.9844,0,0.9844,2990.8008,4339.4828,2595.7323,31794,1,4262.9787,0.2266,0.0705,0.1484,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0829,-0.0481,0.0,1.0,0.0,0.0181,0.5,0.2266,12.7156,50.6524,565.6756,34.1,857.9675,292.2919,0.0685,4.3634,31.3815,172.8736,204.58,3407.9472,17.5583,173.1294,3181.4385,0,58,59,472,191,58,157,2.3389830508474576,138,0.22033898305084745,13.0,0.3050847457627119,18,1.9491525423728813,115.0,0.2711864406779661,16.0,,,113.7448,123.1255,,,
|
| 4 |
+
0.0,0,64,512,2,1.4531,0,0.9219,1842.3164,3830.7287,1172.5901,31730,1,3083.3057,0.252,0.1959,0.2355,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0529,-0.0935,0.0,1.0,0.0,0.0296,0.5,0.252,12.8444,50.7361,566.7198,33.9,857.9675,291.2477,0.0904,4.2018,17.8872,126.8601,145.0002,3636.1329,18.2801,127.022,3468.6454,0,59,59,472,191,58,160,2.559322033898305,151,0.23728813559322035,14.0,0.3728813559322034,22,1.8305084745762712,108.0,0.3728813559322034,22.0,,,,,,,
|
| 5 |
+
0.0,0,64,512,3,1.5469,0,0.9531,1687.9023,3823.3853,1110.3151,19335,1,2744.5105,0.2129,0.1122,0.1362,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0511,-0.0532,0.0,1.0,0.0,0.0138,0.4531,0.2129,12.8585,50.7328,566.4686,34.0,857.9675,291.4989,0.0597,2.7178,15.1498,108.5325,123.9181,3855.0493,17.9434,108.7081,3710.4636,0,60,63,504,312,48,79,0.7936507936507936,50,0.14285714285714285,9.0,0.31746031746031744,20,1.746031746031746,110.0,0.047619047619047616,3.0,,,0.0099,8.8255,275.4291,,
|
| 6 |
+
0.0,0,64,512,3,1.4844,0,0.875,2705.2773,3926.8809,1668.8989,31646,1,3012.6819,0.459,0.1462,0.1798,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0697,-0.0959,0.0,1.0,0.0,0.0179,0.7188,0.459,13.0881,50.9166,566.6464,34.0,857.9675,291.3212,0.0697,4.435,19.2157,123.2323,142.7038,1882.4268,20.2012,123.418,1715.083,0,61,62,496,397,29,58,0.3387096774193548,21,0.1935483870967742,12.0,0.4032258064516129,25,0.6290322580645161,39.0,0.03225806451612903,2.0,,,,,,0.016129032258064516,1.0
|
| 7 |
+
0.0,0,64,512,3,1.5625,0,0.875,4372.2871,4678.6603,4051.208,31765,1,4031.5953,0.5117,0.0301,0.1117,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1006,-0.022,0.0,1.0,0.0,0.0107,0.6094,0.5117,13.2397,51.094,568.5527,33.7,857.9675,289.4149,0.0919,4.5973,32.0721,189.1022,221.4727,1318.3369,21.5119,189.3083,1070.747,0,62,62,496,467,17,21,0.03225806451612903,2,0.11290322580645161,7.0,0.24193548387096775,15,0.11290322580645161,7.0,0.016129032258064516,1.0,,,0.0085,8.9973,,0.016129032258064516,1.0
|
| 8 |
+
0.0,0,64,512,4,1.8906,0,0.9688,5146.4805,4434.7366,5789.4312,31600,1,4146.9309,0.4746,0.0104,0.0669,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1238,-0.0056,0.0,1.0,0.0,0.0102,0.5469,0.4746,13.2111,51.0682,568.2429,33.8,857.9675,289.7247,0.0709,4.7406,35.0463,194.8197,230.1324,1359.0594,21.4195,195.0147,1102.761,0,63,63,504,448,20,51,0.07936507936507936,5,0.09523809523809523,6.0,0.746031746031746,47,0.07936507936507936,5.0,,,,,,,,,
|
| 9 |
+
0.0,0,64,512,7,1.8906,0,0.875,5587.1855,5587.4615,5587.0158,31766,1,4900.2073,0.3809,0.0053,0.1109,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1128,0.0,0.0,1.0,0.0,0.0114,0.4844,0.3809,13.5775,51.5483,569.7428,33.6,857.9675,288.2247,0.1075,4.8015,38.5633,225.6816,264.5674,1636.5882,22.1274,225.8961,1345.0836,0,64,62,496,422,26,68,0.04838709677419355,3,0.04838709677419355,3.0,1.0,62,0.0967741935483871,6.0,,,,,0.0081,8.5496,,0.06451612903225806,4.0
|
| 10 |
+
0.0,0,64,512,4,1.9531,0,0.9375,4798.2617,4883.6444,4723.5128,31812,1,4722.6084,0.4668,0.0183,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1087,0.0,0.0,1.0,0.0,0.0168,0.6094,0.4668,13.718,51.5807,568.653,33.7,857.9675,289.3145,0.075,4.7303,35.8252,213.9135,250.0471,1536.8344,21.852,214.1464,1260.1995,0,65,61,488,434,20,44,0.06557377049180328,4,0.04918032786885246,3.0,0.7213114754098361,44,0.16393442622950818,10.0,,,,,,,5.9602,,
|
| 11 |
+
0.0,0,64,512,3,1.6875,0,0.9062,4990.3906,4683.8621,5191.767,31741,1,4339.1846,0.3965,0.0113,0.1192,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1176,-0.0005,0.0,1.0,0.0,0.0138,0.5781,0.3965,13.9927,51.9113,567.4896,33.9,857.9675,290.4779,0.0707,4.8411,34.6825,205.3311,240.3191,1507.0204,22.8738,205.5654,1238.9803,0,66,64,512,451,19,50,0.03125,2,0.046875,3.0,0.75,48,0.171875,11.0,,,,,0.0085,8.6432,,,
|
| 12 |
+
0.0,0,64,512,3,1.5938,0,0.8906,4687.6602,4780.5265,4614.2762,31720,1,3991.8123,0.4414,0.0147,0.1094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1043,-0.0031,0.0,1.0,0.0,0.0126,0.5625,0.4414,14.1397,52.1146,567.5696,33.8,857.9675,290.3979,0.0726,4.6609,30.5811,185.1371,216.0168,1571.263,21.9126,185.3626,1328.6676,0,67,58,464,422,16,42,0.034482758620689655,2,0.08620689655172414,5.0,0.6896551724137931,40,,,,,,,,,,,
|
| 13 |
+
0.0,0,64,512,4,1.5469,0,0.8281,4902.5645,4448.6776,5228.5101,31770,1,4445.7836,0.418,-0.0081,0.0909,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1147,-0.0,0.0,1.0,0.0,0.0128,0.5312,0.418,14.1629,52.2518,569.6768,33.6,857.9675,288.2908,0.0962,4.7131,33.8447,209.0269,243.2635,1442.218,21.5536,209.322,1172.6792,0,68,64,512,471,16,37,0.046875,3,0.0625,4.0,0.46875,30,0.046875,3.0,0.015625,1.0,,,0.0082,8.6812,,,
|
| 14 |
+
0.0,0,64,512,4,1.7344,0,0.875,4912.1777,4638.6631,5239.691,31749,1,4027.0232,0.5449,0.0247,0.126,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1083,-0.002,0.0,1.0,0.0,0.0111,0.6562,0.5449,14.5627,52.6307,568.1539,33.8,857.9675,289.8137,0.0922,4.7384,33.01,189.5539,222.8763,1413.2169,23.2675,189.7737,1162.3272,0,69,62,496,442,18,54,0.03225806451612903,2,,,0.8387096774193549,52,,,,,,,,,,,
|
| 15 |
+
0.0,0,64,512,3,1.9219,0,0.9062,4989.8223,4749.7429,5156.7649,31793,1,4613.5261,0.4102,-0.0096,0.0915,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1134,-0.0,0.0,1.0,0.0,0.0151,0.5312,0.4102,14.5696,52.6356,567.6034,33.8,857.9675,290.3641,0.0859,4.6794,34.9319,213.77,248.9891,1779.8183,22.218,213.9708,1503.9267,0,70,62,496,433,16,61,0.12903225806451613,8,0.03225806451612903,2.0,0.8387096774193549,52,0.03225806451612903,2.0,,,,,0.0081,8.4344,5.9241,,
|
training_logs/20260612_220124_metrics_job_661735.csv
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,timing/save_hf_model,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError
|
| 2 |
+
0.0,0,64,512,3,0.3281,0,0.2188,4521.9473,4333.5193,4758.5198,31663,1,3365.2785,0.5566,0.0074,0.094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0,0.0,1.0,0.0,0.0131,0.5965,0.5566,12.5978,50.5919,579.345,32.5,857.9676,278.6226,0.1569,4.5133,32.1738,167.8055,200.36,3675.9345,21.9439,168.0287,3449.1169,0,71,58,464,375,29,88,1.4137931034482758,82,0.20689655172413793,12,0.017241379310344827,1.0,,,,,,,,,
|
| 3 |
+
0.0,0,64,512,1,0.9375,0,0.9375,5361.2598,5767.993,5205.1622,31699,1,5207.4533,0.2773,-0.013,0.1323,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1151,0.0,0.0,1.0,0.0,0.017,0.4219,0.2773,12.8662,50.7994,576.8029,32.8,857.9676,281.1647,0.0648,4.6437,37.3367,229.2593,266.8776,1642.3033,25.4393,229.4755,1345.3366,0,72,56,448,364,24,80,1.3392857142857142,75,0.16071428571428573,9,0.07142857142857142,4.0,11.424,21.2849,0.16071428571428573,9.0,0.05357142857142857,3.0,,,
|
| 4 |
+
0.0,0,64,512,2,1.6406,1,1.0,4303.7402,4841.9941,4036.1871,31778,1,4623.5614,0.332,0.0023,0.0825,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0977,0.0,0.0,1.0,0.0,0.0095,0.4062,0.332,13.201,51.0858,576.4416,32.8,857.9676,281.526,0.0649,4.6475,32.7604,195.3408,228.4104,875.421,23.4598,195.5846,618.8995,0,73,63,504,471,12,33,0.47619047619047616,30,0.07936507936507936,5,,,,,0.047619047619047616,3.0,,,,,
|
| 5 |
+
0.0,0,64,512,3,1.6406,0,0.8906,5242.7773,4926.2055,5552.0154,31761,1,4678.9952,0.4941,0.0138,0.1243,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1009,-0.0,0.0,1.0,0.0,0.0145,0.5938,0.4941,13.4431,51.3293,575.8802,32.9,857.9676,282.0874,0.062,4.6498,37.0456,231.3918,268.7201,1733.9771,23.6881,231.6122,1436.9161,0,74,59,472,401,23,56,0.847457627118644,50,0.1864406779661017,11,0.2542372881355932,15.0,0.007,10.013,0.03389830508474576,2.0,0.05084745762711865,3.0,,,
|
| 6 |
+
0.0,0,64,512,4,1.9844,1,1.0,4678.0137,4390.89,4862.0673,31720,1,4446.3943,0.3906,-0.0162,0.1412,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1014,-0.002,0.0,1.0,0.0,0.0143,0.5625,0.3906,13.5915,51.5202,575.6154,32.9,857.9676,282.3522,0.0636,4.5241,34.8828,208.4871,243.6766,1567.8023,22.6625,208.7299,1296.9223,0,75,61,488,422,25,66,0.7704918032786885,47,0.21311475409836064,13,,,,,0.06557377049180328,4.0,0.09836065573770492,6.0,6.424,,
|
| 7 |
+
0.0,0,64,512,4,2.4219,2,1.0,4572.6895,4582.679,4568.0657,31799,1,4302.6602,0.3164,-0.0023,0.112,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.103,0.0,0.0,1.0,0.0,0.0129,0.4531,0.3164,13.6858,51.5748,574.5665,33.0,857.9676,283.4011,0.0663,4.6913,30.5968,200.2044,231.0996,1415.5476,24.2091,200.4362,1155.5266,0,76,61,488,415,22,64,0.7868852459016393,48,0.21311475409836064,13,0.14754098360655737,9.0,0.0074,9.8798,0.04918032786885246,3.0,0.11475409836065574,7.0,,,
|
| 8 |
+
0.0,0,64,512,4,3.0312,3,1.0,4597.7812,4086.6667,4944.6689,31759,1,4559.7375,0.4043,0.0013,0.0431,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0033,0.0,1.0,0.0,0.0068,0.4688,0.4043,14.3994,51.5634,573.7055,33.1,857.9676,284.2621,0.0642,4.5039,32.7244,196.2376,229.2847,1161.9454,21.4738,196.4958,906.6629,0,77,63,504,414,32,86,0.8095238095238095,51,0.4603174603174603,29,0.15873015873015872,10.0,,,,,0.09523809523809523,6.0,,0.015873015873015872,1.0
|
| 9 |
+
0.0,0,64,512,4,4.0,4,1.0,5291.9141,6195.0086,5027.3712,28898,1,4117.8095,0.2266,0.0072,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1143,0.0,0.0,1.0,0.0,0.0105,0.375,0.2266,14.2961,51.5109,573.1121,33.2,857.9676,284.8555,0.1485,4.4428,33.4904,188.476,222.3217,1837.7946,17.1266,188.6825,1593.8676,0,78,67,536,466,24,64,0.6567164179104478,44,0.23880597014925373,16,0.08955223880597014,6.0,0.0588,10.9374,0.029850746268656716,2.0,0.029850746268656716,2.0,,,
|
training_logs/20260612_220124_metrics_table.csv
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
log_file,async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,global_step
|
| 2 |
+
job_630124,0.0,0,64,512,0,0.0,0,0.0,4169.1133,3891.7792,5095.1271,29264,1263,2914.3086,0.7695,-0.0069,0.0744,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1189,0.0,0.0,1.0,0.0,0.0105,0.8125,0.7695,11.6896,50.3563,578.7643,32.5,857.9675,279.2032,0.1941,4.3043,30.7211,146.6405,177.747,3371.9765,22.9409,146.8314,3166.9842,0,1,119,952,918,22,26,0.04201680672268908,5.0,0.025210084033613446,3.0,0.008403361344537815,1.0,0.10084033613445378,12,0.14285714285714285,17.0,,,,,,,,,,1
|
| 3 |
+
job_630124,0.0,0,64,512,1,0.8594,0,0.8594,5688.084,5824.2419,5589.5185,31885,1,5477.7731,0.4199,0.0464,0.1255,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1361,-0.005,0.0,1.0,0.0,0.0161,0.5781,0.4199,12.6834,50.5541,574.2212,33.1,857.9675,283.7463,0.0641,4.8042,39.1437,260.9703,300.4542,1718.0337,24.4369,261.2461,1388.3356,0,2,63,504,454,24,37,0.15873015873015872,10.0,0.047619047619047616,3.0,0.06349206349206349,4.0,0.36507936507936506,23,0.15873015873015872,10.0,14.2681,24.8089,,,,,,,,2
|
| 4 |
+
job_630124,0.0,0,64,512,2,1.5625,0,0.9375,5287.3867,5200.8475,5333.1104,31817,1,5758.917,0.3457,0.0073,0.0951,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,-0.0021,0.0,1.0,0.0,0.0171,0.4688,0.3457,12.9334,50.6717,572.7552,33.2,857.9675,285.2123,0.1003,4.7883,38.0045,267.5741,305.9524,1220.661,24.4652,267.8472,885.4504,0,3,62,496,468,14,23,0.08064516129032258,5.0,,,,,0.3064516129032258,19,0.0967741935483871,6.0,,,,,,,,,,3
|
| 5 |
+
job_630124,0.0,0,64,512,3,1.8438,0,0.875,5684.4512,4907.0708,6233.8,31815,1,5008.2154,0.4141,-0.0061,0.0833,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1411,-0.004,0.0,1.0,0.0,0.0121,0.4844,0.4141,13.7453,50.795,563.5295,34.3,857.9675,294.438,0.0673,4.9105,37.4051,234.7096,272.4655,1357.8902,20.825,234.9927,1059.6856,0,4,62,496,451,21,36,0.0967741935483871,6.0,0.04838709677419355,3.0,,,0.3709677419354839,23,0.22580645161290322,14.0,0.008,12.7181,,,,,,,,4
|
| 6 |
+
job_630124,0.0,0,64,512,4,1.875,0,0.9219,5039.3906,5546.9962,4499.0363,31714,1,4334.7369,0.5156,0.0318,0.1173,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1279,-0.0036,0.0,1.0,0.0,0.0141,0.625,0.5156,13.8122,50.8325,562.6003,34.4,857.9675,295.3672,0.0989,4.6928,39.1023,199.9551,239.4905,1224.4331,24.7234,200.2888,955.52,0,5,62,496,449,22,32,0.11290322580645161,7.0,0.12903225806451613,8.0,,,0.3387096774193548,21,0.1935483870967742,12.0,,,8.2216,,,,,,,5
|
| 7 |
+
job_630124,0.0,0,64,512,4,2.1719,0,0.9375,5591.9941,5830.8641,5431.1863,31798,1,4994.2618,0.4023,0.0066,0.0843,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0004,0.0,1.0,0.0,0.0111,0.4844,0.4023,13.8712,50.9265,561.6264,34.5,857.9675,296.3411,0.0653,4.9236,37.4489,233.3771,271.219,1281.0159,22.6774,233.7045,982.1918,0,6,62,496,435,23,51,0.12903225806451613,8.0,0.03225806451612903,2.0,0.04838709677419355,3.0,0.5161290322580645,32,0.25806451612903225,16.0,0.0067,13.924,,0.016129032258064516,1.0,,,,,6
|
| 8 |
+
job_630124,0.0,0,64,512,4,2.1094,0,0.9062,4851.3477,5205.5652,4562.4468,31817,1,4336.985,0.4492,0.0059,0.0777,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1271,-0.0014,0.0,1.0,0.0,0.0139,0.5469,0.4492,13.9109,50.9409,561.0771,34.6,857.9675,296.8904,0.0644,4.6898,33.9517,197.6421,231.9494,1559.2848,23.4677,197.9329,1299.1744,0,7,63,504,457,15,42,,,0.07936507936507936,5.0,0.031746031746031744,2.0,0.5555555555555556,35,0.07936507936507936,5.0,,,,,,,,,,7
|
| 9 |
+
job_630124,0.0,0,64,512,4,2.1094,0,0.9219,5596.0137,5715.5123,5487.2164,31840,1,5416.4656,0.4766,0.0199,0.145,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,0.0,0.0,1.0,0.0,0.0136,0.625,0.4766,13.9023,50.9439,561.6647,34.5,857.9675,296.3028,0.0672,4.797,43.3436,268.6347,312.3633,1610.273,22.6954,268.9523,1270.4134,0,8,62,496,457,18,38,,,,,0.016129032258064516,1.0,0.4032258064516129,25,0.20967741935483872,13.0,0.0071,12.1975,,,,0.016129032258064516,1.0,,,8
|
| 10 |
+
job_630124,0.0,0,64,512,4,1.9844,0,0.9531,5326.7715,5167.9342,5454.2887,31758,1,4691.8758,0.4453,0.0119,0.0971,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1268,0.0,0.0,1.0,0.0,0.0137,0.5469,0.4453,13.8936,50.9354,559.6754,34.8,857.9675,298.2921,0.0782,4.7117,38.0949,222.0535,260.5136,1321.2856,23.5801,222.3401,1032.476,0,9,61,488,449,18,38,,,0.01639344262295082,1.0,0.03278688524590164,2.0,0.4426229508196721,27,0.22950819672131148,14.0,,,,,,,,,,9
|
| 11 |
+
job_630124,0.0,0,64,512,3,1.8906,0,0.9062,4968.5547,4853.8377,5036.8131,31768,1,4721.9898,0.373,0.0139,0.118,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1359,0.0,0.0,1.0,0.0,0.0144,0.5312,0.373,13.9257,51.0073,560.3441,34.7,857.9675,297.6235,0.0714,4.8344,33.5456,220.0195,253.8928,1294.919,24.3901,220.2753,1011.7949,0,10,62,496,442,18,46,,,0.12903225806451613,8.0,0.016129032258064516,1.0,0.6451612903225806,40,0.12903225806451613,8.0,0.0084,14.0711,8.8277,,,,,,,10
|
| 12 |
+
job_630124,0.0,0,64,512,4,1.7812,0,0.9219,4462.5332,4532.996,4394.2385,31762,1,4298.9711,0.4922,0.0008,0.1662,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,-0.0056,0.0,1.0,0.0,0.0183,0.6562,0.4922,14.0333,51.1276,559.0244,34.8,857.9675,298.9431,0.0938,4.6549,32.2602,198.6454,231.2913,1475.0347,24.5599,198.9368,1214.5237,0,11,64,512,471,16,33,,,0.125,8.0,0.046875,3.0,0.328125,21,0.140625,9.0,,,,,,,,,,11
|
| 13 |
+
job_630124,0.0,0,64,512,4,2.0938,0,0.8906,5849.1094,5035.9944,6282.4461,31797,1,4813.1067,0.3477,-0.0072,0.0606,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1327,0.0,0.0,1.0,0.0,0.013,0.4219,0.3477,14.0347,51.1288,558.5858,34.9,857.9675,299.3817,0.0732,5.2651,38.6448,224.9423,263.9456,1559.3505,25.6632,225.2273,1264.4721,0,12,57,456,426,14,30,,,,,0.07017543859649122,4.0,0.42105263157894735,24,0.14035087719298245,8.0,0.0084,14.0752,,,,,,,,12
|
| 14 |
+
job_630124,0.0,0,64,512,4,1.8125,0,0.875,4866.082,5038.6537,4692.1569,31760,1,4066.2403,0.502,-0.0034,0.1269,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1255,-0.0,0.0,1.0,0.0,0.0186,0.625,0.502,14.0677,51.1943,557.1813,35.1,857.9675,300.7862,0.0704,4.6745,31.6351,183.7078,215.6617,1431.9056,23.0843,183.9558,1188.48,0,13,64,512,450,21,49,,,0.203125,13.0,0.25,16.0,0.46875,30,0.09375,6.0,,,,,,,,,,13
|
| 15 |
+
job_630124,0.0,0,64,512,4,2.0469,0,0.9375,4926.5117,4839.1749,4993.9031,31751,1,4266.9039,0.4355,0.0028,0.0804,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1228,-0.0013,0.0,1.0,0.0,0.0162,0.5469,0.4355,14.322,51.506,557.3558,35.0,857.9675,300.6118,0.0972,4.7529,33.9188,193.7138,228.0319,1142.7788,24.0818,194.0154,885.9041,0,14,63,504,473,18,30,,,,,,,0.3333333333333333,21,0.14285714285714285,9.0,0.0084,13.3568,,,,,,0.015873015873015872,1.0,14
|
| 16 |
+
job_630124,0.0,0,64,512,4,2.0312,0,0.9219,5025.9766,4558.8924,5475.1648,31707,1,4544.2113,0.4902,0.0047,0.108,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1336,-0.0,0.0,1.0,0.0,0.0154,0.6094,0.4902,14.6948,51.8791,556.9661,35.1,857.9675,301.0014,0.0749,4.7275,31.8001,218.695,250.8643,1313.9526,22.5691,218.989,1035.7868,0,15,64,512,477,18,28,,,0.109375,7.0,,,0.296875,19,0.1875,12.0,,,8.2163,,,,,,,15
|
| 17 |
+
job_630124,0.0,0,64,512,4,1.9375,0,0.8906,5304.2988,4924.3203,5616.6655,31702,1,4477.752,0.4512,0.0092,0.1087,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1249,-0.0,0.0,1.0,0.0,0.0148,0.5781,0.4512,14.4315,51.6356,556.9907,35.1,857.9675,300.9769,0.1006,4.7795,36.8211,219.4484,256.6458,1308.8006,22.9796,219.7236,1024.3882,0,16,61,488,421,21,66,,,,,0.09836065573770492,6.0,0.8360655737704918,51,0.16393442622950818,10.0,0.0077,14.8915,,,,0.01639344262295082,1.0,,,16
|
| 18 |
+
job_630124,0.0,0,64,512,4,2.3594,0,0.9375,5391.4434,6104.0653,4938.3706,31758,1,5320.6865,0.3887,0.0223,0.0896,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1221,-0.002,0.0,1.0,0.0,0.0108,0.5156,0.3887,14.5796,51.7618,556.1977,35.2,857.9675,301.7698,0.0974,5.3524,38.0588,253.8113,292.2721,1220.8525,24.0951,254.1154,899.1259,0,17,57,456,419,17,35,,,0.017543859649122806,1.0,0.03508771929824561,2.0,0.38596491228070173,22,0.24561403508771928,14.0,,,,,,0.017543859649122806,1.0,,,17
|
| 19 |
+
job_630124,0.0,0,64,512,4,1.7188,0,0.875,5066.6953,5047.4764,5082.7455,31770,1,4672.0163,0.4551,0.0005,0.1098,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1223,-0.0028,0.0,1.0,0.0,0.0154,0.5625,0.4551,14.7728,51.9543,556.2234,35.2,857.9675,301.7441,0.0991,4.843,39.5878,215.8858,255.888,1582.6579,22.1364,216.2007,1299.7832,0,18,60,480,437,18,41,,,0.03333333333333333,2.0,0.016666666666666666,1.0,0.5333333333333333,32,0.16666666666666666,10.0,0.0075,16.3173,,,,,,,,18
|
| 20 |
+
job_630124,0.0,0,64,512,4,1.8281,0,0.8906,5320.7246,4968.5291,5526.808,31825,1,4635.3488,0.3691,-0.0052,0.097,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1408,-0.0,0.0,1.0,0.0,0.0136,0.4531,0.3691,14.7764,51.9575,554.7101,35.3,857.9675,303.2574,0.0666,4.8474,33.322,217.0675,250.7206,1533.2708,23.3393,217.3317,1254.3583,0,19,62,496,453,17,43,,,,,,,0.6129032258064516,38,0.0967741935483871,6.0,,,,,,,,,,19
|
| 21 |
+
job_630124,0.0,0,64,512,4,1.6875,0,0.8906,4981.0215,5122.7233,4885.6275,31734,1,4443.4633,0.4023,-0.0006,0.0906,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1347,0.0,0.0,1.0,0.0,0.0168,0.5156,0.4023,14.7832,52.0205,554.9844,35.3,857.9675,302.9831,0.0665,4.7337,34.0494,210.2643,244.6153,1501.26,23.3047,210.499,1228.6018,0,20,63,504,461,18,38,,,0.047619047619047616,3.0,0.06349206349206349,4.0,0.5396825396825397,34,0.06349206349206349,4.0,0.0078,13.3388,8.0322,,,0.031746031746031744,2.0,,,20
|
| 22 |
+
job_630124,0.0,0,64,512,4,1.7969,0,0.8906,5089.5234,5337.2258,4948.1963,31834,1,4103.7515,0.3633,-0.0098,0.0985,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1382,-0.0055,0.0,1.0,0.0,0.0123,0.4688,0.3633,14.7559,51.9613,554.3621,35.4,857.9675,303.6055,0.0721,4.8151,32.0319,197.6414,230.0275,1384.6688,23.9344,197.9232,1125.887,0,21,61,488,432,21,46,,,0.13114754098360656,8.0,,,0.6229508196721312,38,0.13114754098360656,8.0,,,,,,0.03278688524590164,2.0,,,21
|
| 23 |
+
job_630124,0.0,0,64,512,3,1.5156,0,0.8281,4888.4629,4445.7686,5246.6855,31774,1,4638.3152,0.4473,0.0144,0.0967,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1277,0.0,0.0,1.0,0.0,0.0135,0.5781,0.4473,15.1406,52.3359,554.3356,35.4,857.9675,303.6319,0.0672,4.8409,37.4714,213.0209,250.8158,1341.0965,22.4572,213.2768,1062.9785,0,22,64,512,452,21,60,,,,,0.046875,3.0,0.734375,47,0.15625,10.0,0.0085,14.204,,,,,,,,22
|
| 24 |
+
job_630124,0.0,0,64,512,4,1.6094,0,0.8594,5010.1426,5339.5915,4775.4515,31738,1,5125.7203,0.416,0.0029,0.123,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1269,-0.0,0.0,1.0,0.0,0.0161,0.5781,0.416,15.1399,52.3352,554.2477,35.4,857.9675,303.7198,0.0666,4.7525,38.7094,241.3185,280.3783,1636.7829,23.2267,241.602,1328.421,0,23,61,488,423,17,65,,,,,0.04918032786885246,3.0,0.639344262295082,39,0.3770491803278688,23.0,,,,,,,,,,23
|
| 25 |
+
job_630124,0.0,0,64,512,3,1.5156,0,0.8281,4957.0039,5145.7238,4853.8066,31744,1,4792.2542,0.3535,-0.005,0.139,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1197,0.0,0.0,1.0,0.0,0.0118,0.4531,0.3535,15.2222,52.591,552.66,35.6,857.9675,305.3075,0.0667,4.8036,34.8997,225.7814,261.0063,1813.0727,21.5017,226.0394,1525.7569,0,24,53,424,405,10,17,,,0.1509433962264151,8.0,0.09433962264150944,5.0,0.1320754716981132,7,0.24528301886792453,13.0,0.0085,13.7903,,,,,,,,24
|
| 26 |
+
job_630124,0.0,0,64,512,4,1.8281,0,0.9062,4999.9961,4608.5735,5274.3821,31665,1,3727.2595,0.4121,0.0076,0.0996,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1278,-0.0,0.0,1.0,0.0,0.0129,0.4844,0.4121,15.4929,52.9026,551.8461,35.7,857.9675,306.1215,0.0644,4.6949,29.2144,181.8811,211.389,988.3901,25.1751,182.1098,747.1268,0,25,54,432,406,12,19,,,0.12962962962962962,7.0,0.07407407407407407,4.0,0.2777777777777778,15,0.1111111111111111,6.0,,,8.7088,,,,,,,25
|
| 27 |
+
job_630124,0.0,0,64,512,4,1.7344,0,0.875,4801.6133,4766.3344,4852.3476,31766,1,3813.5277,0.5898,-0.0032,0.1052,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1273,-0.0035,0.0,1.0,0.0,0.0164,0.6875,0.5898,15.8889,53.344,552.8224,35.6,857.9675,305.1451,0.0946,4.7486,31.8278,188.1649,220.5231,1031.4949,22.9306,188.6001,783.2847,0,26,59,472,410,26,50,,,0.11864406779661017,7.0,0.3050847457627119,18.0,0.3220338983050847,19,0.23728813559322035,14.0,0.009,12.6671,,,,0.288135593220339,17.0,,,26
|
| 28 |
+
job_630124,0.0,0,64,512,4,2.1875,0,0.9688,5055.6387,4951.9907,5130.67,31763,1,4942.405,0.4199,0.0045,0.1012,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1162,-0.01,0.0,1.0,0.0,0.0135,0.5469,0.4199,16.046,53.4671,552.5507,35.6,857.9675,305.4168,0.0766,4.8488,33.5052,219.951,253.8289,2130.9697,18.5315,220.2467,1853.7559,0,27,52,416,390,13,6,,,0.28846153846153844,15.0,,,0.2692307692307692,14,0.057692307692307696,3.0,,,,,,0.19230769230769232,10.0,0.019230769230769232,1.0,27
|
| 29 |
+
job_630124,0.0,0,64,512,4,1.6719,0,0.8281,4799.998,4474.8537,5214.7378,21478,1,3336.6768,0.5605,0.0081,0.0474,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1169,-0.0193,0.0,1.0,0.0,0.0089,0.625,0.5605,15.7877,53.2084,552.8907,35.6,857.9675,305.0768,0.0486,3.4809,28.5303,161.7902,190.6488,998.4337,25.9964,162.0695,778.3023,0,28,44,352,335,11,15,,,0.13636363636363635,6.0,0.022727272727272728,1.0,0.4772727272727273,21,0.45454545454545453,20.0,0.0082,11.9058,,,,0.022727272727272728,1.0,,,28
|
| 30 |
+
job_630127,0.0,0,64,512,5,0.8125,0,0.2031,5219.4023,5279.0935,5148.4872,24019,1,3593.1869,0.543,0.0174,0.1093,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.127,-0.0059,0.0,1.0,0.0,0.0136,0.661,0.543,9.9914,50.291,594.6118,30.7,857.9675,263.3557,0.2443,3.7877,34.8635,181.2173,216.5936,3907.0864,25.2212,181.4851,3661.4836,0,29,54,432,397,20,35,,,,,0.05555555555555555,3.0,0.48148148148148145,26,0.18518518518518517,10.0,,,,,,,,,,29
|
| 31 |
+
job_630127,0.0,0,64,512,1,0.875,0,0.875,6119.2617,5371.9608,6614.2273,31787,1,5360.732,0.3984,0.0066,0.11,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1391,-0.0,0.0,1.0,0.0,0.0133,0.5156,0.3984,12.6683,50.5745,589.7981,31.3,857.9675,268.1694,0.0668,4.853,45.3325,249.4957,295.1659,1561.0798,22.8392,249.7659,1238.2155,0,30,51,408,360,18,48,,,0.0784313725490196,4.0,0.0784313725490196,4.0,0.9803921568627451,50,0.19607843137254902,10.0,17.6986,25.6657,6.5413,,,,,,,30
|
| 32 |
+
job_630127,0.0,0,64,512,2,1.7031,1,1.0,5416.0312,5191.9848,5493.8579,31782,1,5258.8542,0.2578,0.0188,0.1227,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1224,-0.0045,0.0,1.0,0.0,0.0132,0.3906,0.2578,13.0473,50.9899,588.6119,31.4,857.9675,269.3556,0.0915,4.9642,38.9391,242.4843,281.7563,558.6231,24.3077,242.7249,247.5902,0,31,62,496,450,16,38,,,0.06451612903225806,4.0,,,0.5806451612903226,36,0.03225806451612903,2.0,,,,,,,,,,31
|
| 33 |
+
job_630127,0.0,0,64,512,3,1.8438,0,0.8438,4876.5957,4321.0181,5298.5292,31788,1,4349.0068,0.4316,0.008,0.1024,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1299,-0.0033,0.0,1.0,0.0,0.0147,0.5469,0.4316,13.3226,51.1669,587.694,31.5,857.9675,270.2736,0.0969,4.6902,32.9398,184.8866,218.1917,1881.6998,22.4991,185.1545,1636.313,0,32,61,488,435,20,50,,,0.04918032786885246,3.0,0.04918032786885246,3.0,0.5245901639344263,32,0.26229508196721313,16.0,0.0086,8.4163,,,,,,,,32
|
| 34 |
+
job_630127,0.0,0,64,512,4,1.9688,0,0.8906,5456.2109,4808.7673,5878.0935,31806,1,4961.1718,0.3945,0.0083,0.1325,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,0.0,0.0,1.0,0.0,0.0126,0.5469,0.3945,13.4299,51.223,587.0558,31.6,857.9675,270.9117,0.067,4.9084,36.655,238.8678,275.8322,1516.0964,23.2669,239.1098,1212.085,0,33,62,496,436,21,55,,,0.08064516129032258,5.0,0.03225806451612903,2.0,0.5806451612903226,36,0.27419354838709675,17.0,,,,,,,,,,33
|
| 35 |
+
job_630127,0.0,0,64,512,4,2.125,0,0.9844,5502.4434,5212.8122,5708.7692,31765,1,5396.2041,0.416,-0.004,0.0835,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0,0.0,1.0,0.0,0.0137,0.5156,0.416,13.5508,51.3198,587.3987,31.5,857.9675,270.5688,0.0691,5.493,44.4908,256.2057,301.0644,1068.5266,22.5088,256.5041,739.4567,0,34,63,504,462,14,41,,,0.015873015873015872,1.0,0.015873015873015872,1.0,0.3968253968253968,25,0.23809523809523808,15.0,0.0088,8.3216,,,,,,,,34
|
| 36 |
+
job_630127,0.0,0,64,512,3,1.8594,0,0.9219,5384.9941,5343.019,5414.1821,31664,1,4412.7671,0.4102,0.0025,0.0519,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1331,-0.002,0.0,1.0,0.0,0.0142,0.5,0.4102,13.7969,51.3292,586.7628,31.6,857.9675,271.2047,0.0906,4.7764,33.1627,211.1102,244.5902,1384.0633,22.1442,211.3362,1112.5444,0,35,61,488,463,8,17,,,0.13114754098360656,8.0,,,0.26229508196721313,16,0.04918032786885246,3.0,,,5.8995,,,,,,,35
|
| 37 |
+
job_630127,0.0,0,64,512,4,1.9219,0,0.9062,5825.4473,5333.0212,6246.5072,31794,1,4498.7568,0.4609,0.0027,0.1742,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1384,-0.0,0.0,1.0,0.0,0.0171,0.6406,0.4609,13.8943,51.4631,586.2767,31.7,857.9675,271.6909,0.5695,4.8349,37.7488,223.605,262.1992,1464.7075,20.515,223.8804,1177.1535,0,36,64,512,471,15,40,,,0.015625,1.0,,,0.515625,33,0.109375,7.0,0.009,7.9437,,,,,,,,36
|
| 38 |
+
job_630127,0.0,0,64,512,4,1.8438,0,0.9062,5148.5137,4631.637,5834.55,31764,1,4648.1595,0.5703,-0.0029,0.1133,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,-0.0011,0.0,1.0,0.0,0.0167,0.6875,0.5703,14.5586,51.6496,584.9858,31.8,857.9675,272.9817,0.0919,4.8241,38.277,222.4597,261.0628,1077.7825,26.2516,222.6934,785.6361,0,37,61,488,451,15,35,,,0.03278688524590164,2.0,0.09836065573770492,6.0,0.4262295081967213,26,0.14754098360655737,9.0,,,,,,,,,,37
|
| 39 |
+
job_630127,0.0,0,64,512,4,2.1875,0,0.9375,5691.4238,5458.8718,5887.1691,31763,1,4985.511,0.457,0.0025,0.1314,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1239,-0.0,0.0,1.0,0.0,0.0158,0.6094,0.457,14.4333,51.5906,584.4034,31.9,857.9675,273.5641,0.8661,4.8087,37.1183,229.9869,268.1954,1281.4976,24.0319,230.2106,984.4536,0,38,63,504,457,20,44,,,0.047619047619047616,3.0,,,0.5238095238095238,33,0.1111111111111111,7.0,0.0082,8.3888,,,,,,,,38
|
| 40 |
+
job_630127,0.0,0,64,512,4,1.875,0,0.9219,4852.4375,4700.9213,4963.0034,31721,1,3852.6075,0.4219,-0.0118,0.1369,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1297,-0.0025,0.0,1.0,0.0,0.0151,0.5625,0.4219,14.8096,51.9002,584.34,31.9,857.9675,273.6276,0.1019,4.7148,35.3281,188.2665,223.9309,1220.1543,22.2426,188.5005,969.2585,0,39,63,504,446,22,48,,,0.15873015873015872,10.0,0.031746031746031744,2.0,0.6666666666666666,42,0.09523809523809523,6.0,,,,,,,,,,39
|
| 41 |
+
job_630127,0.0,0,64,512,4,1.8906,0,0.8438,4717.9707,4944.8443,4511.4142,31770,1,4317.3591,0.4766,0.0072,0.086,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1232,0.0,0.0,1.0,0.0,0.0143,0.5938,0.4766,14.7138,51.7721,583.8667,31.9,857.9675,274.1008,0.0992,4.8622,31.9697,193.9941,226.3482,1608.4681,22.3175,194.2789,1354.9332,0,40,60,480,456,10,24,,,,,0.03333333333333333,2.0,0.36666666666666664,22,0.05,3.0,0.0074,8.3081,6.0239,,,,,,,40
|
| 42 |
+
job_630127,0.0,0,64,512,4,1.6875,0,0.9531,5037.7695,5312.8784,4764.8016,27138,1,3679.8466,0.498,-0.0001,0.0849,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1349,-0.0,0.0,1.0,0.0,0.0122,0.5781,0.498,14.5892,51.6475,583.7137,32.0,857.9675,274.2538,0.0824,4.1919,29.9247,176.4747,206.7121,1037.8229,24.3689,176.7045,802.5427,0,41,60,480,441,19,37,,,0.016666666666666666,1.0,,,0.43333333333333335,26,0.18333333333333332,11.0,,,,,,0.016666666666666666,1.0,,,41
|
| 43 |
+
job_630127,0.0,0,64,512,4,1.9844,0,0.8906,5756.3848,5835.0311,5708.8025,31761,1,4473.6908,0.377,0.0069,0.1087,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1346,0.0,0.0,1.0,0.0,0.013,0.4531,0.377,14.9636,52.1455,583.6249,32.0,857.9675,274.3426,0.0719,4.8088,36.9135,222.7902,260.0054,1385.1388,21.7016,223.0196,1098.6195,0,42,61,488,424,20,50,,,0.22950819672131148,14.0,0.03278688524590164,2.0,0.5573770491803278,34,0.22950819672131148,14.0,0.0076,7.9965,,,,,,,,42
|
| 44 |
+
job_630127,0.0,0,64,512,4,1.5781,0,0.8281,4886.5996,4211.4396,5344.823,31794,1,4658.4948,0.4043,0.003,0.0477,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.119,-0.0014,0.0,1.0,0.0,0.0109,0.4844,0.4043,14.9648,52.146,583.4162,32.0,857.9675,274.5513,0.0932,4.8883,35.3553,210.4627,246.1689,1473.9582,23.9821,210.7199,1198.9132,0,43,61,488,441,16,45,,,0.03278688524590164,2.0,0.06557377049180328,4.0,0.5081967213114754,31,0.16393442622950818,10.0,,,,,,,,,,43
|
| 45 |
+
job_630127,0.0,0,64,512,4,1.9375,0,0.9844,5006.252,5006.7137,5005.7938,31832,1,4198.283,0.498,0.0058,0.1318,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1302,-0.0013,0.0,1.0,0.0,0.0142,0.625,0.498,15.1405,52.1464,582.1508,32.1,857.9675,275.8167,0.0701,4.8376,37.1394,204.9028,242.3402,995.3073,21.6574,205.1302,726.4685,0,44,63,504,464,15,40,,,,,0.015873015873015872,1.0,0.3492063492063492,22,0.2698412698412698,17.0,0.0087,8.5306,,,,,,,,44
|
| 46 |
+
job_630127,0.0,0,64,512,4,1.9688,0,0.9531,5307.5059,4945.0996,5629.7934,31811,1,4502.6994,0.4707,0.0034,0.0645,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0087,0.5156,0.4707,15.2917,52.3331,581.6784,32.2,857.9675,276.2891,0.0713,4.8688,35.6411,219.1314,255.1244,1030.1018,22.2906,219.4115,747.8149,0,45,57,456,419,11,20,,,0.2982456140350877,17.0,0.03508771929824561,2.0,0.3508771929824561,20,0.07017543859649122,4.0,,,7.1117,,,,,,,45
|
| 47 |
+
job_630127,0.0,0,64,512,4,2.3281,0,0.9531,5162.7871,4802.6639,5490.6604,31759,1,4252.9044,0.4766,-0.0039,0.0939,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1296,0.0,0.0,1.0,0.0,0.0135,0.5625,0.4766,15.655,52.6444,580.4724,32.3,857.9675,277.4952,0.1001,4.9649,38.1605,200.0621,238.5392,1116.6096,24.6502,200.2781,848.4495,0,46,62,496,452,18,44,,,,,0.016129032258064516,1.0,0.5483870967741935,34,0.14516129032258066,9.0,0.0077,8.5628,,,,,,,,46
|
| 48 |
+
job_630127,0.0,0,64,512,4,2.1094,0,0.9375,5453.8691,5448.0192,5459.9522,31779,1,4423.535,0.5098,0.01,0.1256,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0129,0.625,0.5098,15.5344,52.5839,580.0443,32.4,857.9675,277.9232,0.0896,4.8374,35.3367,219.7689,255.4572,1436.5679,22.4966,220.0305,1153.7719,0,47,61,488,451,18,36,,,0.01639344262295082,1.0,0.04918032786885246,3.0,0.3770491803278688,23,0.16393442622950818,10.0,,,,,,,,,,47
|
| 49 |
+
job_630127,0.0,0,64,512,4,1.875,0,0.875,5230.4609,4627.1281,5626.8252,31716,1,4361.9157,0.3965,-0.0025,0.057,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1316,-0.0,0.0,1.0,0.0,0.0113,0.5,0.3965,15.6175,52.7105,579.3495,32.5,857.9675,278.618,0.0831,4.8056,35.7289,205.2303,241.3389,1388.9346,24.246,205.5265,1118.5394,0,48,63,504,448,25,55,,,0.015873015873015872,1.0,0.031746031746031744,2.0,0.6666666666666666,42,0.1746031746031746,11.0,0.0078,8.0292,,,,,,,,48
|
| 50 |
+
job_630127,0.0,0,64,512,4,1.9688,0,0.9062,5567.3223,3973.2919,6469.1437,31784,1,5829.7172,0.3613,-0.0074,0.1077,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0168,0.4844,0.3613,15.5988,52.7747,578.7762,32.5,857.9675,279.1913,0.1107,4.8687,41.1551,269.7094,311.286,1674.2094,22.9805,270.0198,1335.0696,0,49,60,480,440,17,39,,,0.016666666666666666,1.0,,,0.5166666666666667,31,0.13333333333333333,8.0,,,,,,,,,,49
|
| 51 |
+
job_630127,0.0,0,64,512,3,1.6875,0,0.9375,5274.1113,4738.0175,5707.9117,31829,1,4398.882,0.4473,0.008,0.0793,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0132,0.5469,0.4473,15.6793,52.8362,578.3136,32.6,857.9675,279.6539,0.0972,4.7852,34.4386,207.8657,242.6571,1060.2084,23.4244,208.1209,789.3372,0,50,65,520,431,21,76,,,0.18461538461538463,12.0,0.09230769230769231,6.0,0.9538461538461539,62,0.12307692307692308,8.0,0.0081,9.1305,5.9495,,,0.015384615384615385,1.0,,,50
|
| 52 |
+
job_630127,0.0,0,64,512,4,1.9219,0,0.8906,5243.7461,5174.6447,5286.9619,31743,1,5188.684,0.3848,-0.0033,0.0945,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1207,0.0,0.0,1.0,0.0,0.0967,0.4531,0.3848,15.6188,52.7762,577.5959,32.7,857.9675,280.3716,0.076,4.7495,40.1813,238.4651,278.9729,1686.7408,24.1315,238.7151,1378.8824,0,51,62,496,446,22,49,,,0.016129032258064516,1.0,0.016129032258064516,1.0,0.5645161290322581,35,0.20967741935483872,13.0,,,,,,,,,,51
|
| 53 |
+
job_630127,0.0,0,64,512,4,1.7812,0,0.8438,5252.4961,5411.8429,5141.6921,31823,1,4683.711,0.4102,0.0062,0.0796,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.133,-0.002,0.0,1.0,0.0,0.0124,0.5312,0.4102,15.7651,52.9658,577.1581,32.7,857.9675,280.8094,0.0944,4.7678,38.5332,214.205,253.0737,1600.439,23.268,214.4456,1319.3232,0,52,61,488,451,14,37,,,,,0.03278688524590164,2.0,0.4918032786885246,30,0.08196721311475409,5.0,0.0082,8.7241,,,,,,,,52
|
| 54 |
+
job_630127,0.0,0,64,512,3,1.8125,0,0.9531,5107.9043,4684.3886,5450.6078,31689,1,4035.9699,0.4473,-0.0001,0.0667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.132,0.0,0.0,1.0,0.0,0.0132,0.5312,0.4473,15.7054,52.9061,577.4001,32.7,857.9675,280.5674,0.097,4.8708,33.9418,188.7929,223.0849,1082.9166,25.281,189.0458,829.6739,0,53,63,504,454,17,47,,,0.047619047619047616,3.0,0.031746031746031744,2.0,0.6984126984126984,44,0.015873015873015872,1.0,,,,,,,,,,53
|
| 55 |
+
job_630127,0.0,0,64,512,3,1.7969,0,0.875,5634.0664,5882.4221,5476.1661,31801,1,5198.4542,0.3887,0.0101,0.0932,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1305,0.0,0.0,1.0,0.0,0.0144,0.5156,0.3887,15.8018,52.9791,576.4988,32.8,857.9675,281.4687,0.0964,4.9065,40.1938,233.8941,274.5002,1827.1928,24.3101,234.2096,1523.4687,0,54,61,488,434,19,52,,,0.03278688524590164,2.0,0.08196721311475409,5.0,0.6557377049180327,40,0.13114754098360656,8.0,0.0084,8.6128,,,,,,,,54
|
| 56 |
+
job_630127,0.0,0,64,512,4,1.9375,0,0.9531,5371.4082,5022.761,5604.2182,31785,1,4782.3049,0.4004,-0.0082,0.1177,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1222,-0.0007,0.0,1.0,0.0,0.0128,0.5156,0.4004,15.8948,53.0398,576.6206,32.8,857.9675,281.3469,0.0653,4.8129,41.537,228.2535,270.096,1483.9509,23.783,228.4932,1185.2547,0,55,62,496,429,19,64,,,0.04838709677419355,3.0,0.04838709677419355,3.0,0.9032258064516129,56,0.0967741935483871,6.0,,,6.0259,,,,,,,55
|
| 57 |
+
job_630127,0.0,0,64,512,4,1.6094,0,0.9062,4940.75,4904.8341,4965.9269,31836,1,4570.0348,0.4121,-0.0024,0.1498,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.124,-0.0,0.0,1.0,0.0,0.0148,0.5156,0.4121,16.2064,53.3552,575.7253,32.9,857.9675,282.2422,0.0989,4.7645,34.0253,209.3135,243.7093,1375.7686,22.7461,209.5845,1104.5439,0,56,59,472,390,33,69,,,0.22033898305084745,13.0,0.3728813559322034,22.0,0.5084745762711864,30,0.288135593220339,17.0,0.0075,15.5646,,,,,,,,56
|
| 58 |
+
job_630130,0.0,0,64,512,4,0.3438,0,0.1406,2745.916,4536.7524,2283.9066,31842,1,3652.8031,0.2051,0.0705,0.1449,-0.0007,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0683,-0.0442,0.0,1.0,0.0,0.0137,0.3898,0.2051,12.7084,50.5586,566.71,33.9,857.9675,291.2575,0.1351,4.4486,104.946,152.1059,257.396,10105.4336,18.2174,152.3142,9825.3711,0,57,63,504,281,58,154,,,0.8095238095238095,51.0,1.8253968253968254,115.0,0.42857142857142855,27,0.19047619047619047,12.0,,,,,,0.25396825396825395,16.0,0.031746031746031744,2.0,57
|
| 59 |
+
job_630130,0.0,0,64,512,1,0.9844,0,0.9844,2990.8008,4339.4828,2595.7323,31794,1,4262.9787,0.2266,0.0705,0.1484,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0829,-0.0481,0.0,1.0,0.0,0.0181,0.5,0.2266,12.7156,50.6524,565.6756,34.1,857.9675,292.2919,0.0685,4.3634,31.3815,172.8736,204.58,3407.9472,17.5583,173.1294,3181.4385,0,58,59,472,191,58,157,,,1.9491525423728813,115.0,2.3389830508474576,138.0,0.3050847457627119,18,0.22033898305084745,13.0,113.7448,123.1255,,,,0.2711864406779661,16.0,,,58
|
| 60 |
+
job_630130,0.0,0,64,512,2,1.4531,0,0.9219,1842.3164,3830.7287,1172.5901,31730,1,3083.3057,0.252,0.1959,0.2355,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0529,-0.0935,0.0,1.0,0.0,0.0296,0.5,0.252,12.8444,50.7361,566.7198,33.9,857.9675,291.2477,0.0904,4.2018,17.8872,126.8601,145.0002,3636.1329,18.2801,127.022,3468.6454,0,59,59,472,191,58,160,,,1.8305084745762712,108.0,2.559322033898305,151.0,0.3728813559322034,22,0.23728813559322035,14.0,,,,,,0.3728813559322034,22.0,,,59
|
| 61 |
+
job_630130,0.0,0,64,512,3,1.5469,0,0.9531,1687.9023,3823.3853,1110.3151,19335,1,2744.5105,0.2129,0.1122,0.1362,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0511,-0.0532,0.0,1.0,0.0,0.0138,0.4531,0.2129,12.8585,50.7328,566.4686,34.0,857.9675,291.4989,0.0597,2.7178,15.1498,108.5325,123.9181,3855.0493,17.9434,108.7081,3710.4636,0,60,63,504,312,48,79,,,1.746031746031746,110.0,0.7936507936507936,50.0,0.31746031746031744,20,0.14285714285714285,9.0,0.0099,8.8255,275.4291,,,0.047619047619047616,3.0,,,60
|
| 62 |
+
job_630130,0.0,0,64,512,3,1.4844,0,0.875,2705.2773,3926.8809,1668.8989,31646,1,3012.6819,0.459,0.1462,0.1798,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0697,-0.0959,0.0,1.0,0.0,0.0179,0.7188,0.459,13.0881,50.9166,566.6464,34.0,857.9675,291.3212,0.0697,4.435,19.2157,123.2323,142.7038,1882.4268,20.2012,123.418,1715.083,0,61,62,496,397,29,58,,,0.6290322580645161,39.0,0.3387096774193548,21.0,0.4032258064516129,25,0.1935483870967742,12.0,,,,0.016129032258064516,1.0,0.03225806451612903,2.0,,,61
|
| 63 |
+
job_630130,0.0,0,64,512,3,1.5625,0,0.875,4372.2871,4678.6603,4051.208,31765,1,4031.5953,0.5117,0.0301,0.1117,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1006,-0.022,0.0,1.0,0.0,0.0107,0.6094,0.5117,13.2397,51.094,568.5527,33.7,857.9675,289.4149,0.0919,4.5973,32.0721,189.1022,221.4727,1318.3369,21.5119,189.3083,1070.747,0,62,62,496,467,17,21,,,0.11290322580645161,7.0,0.03225806451612903,2.0,0.24193548387096775,15,0.11290322580645161,7.0,0.0085,8.9973,,0.016129032258064516,1.0,0.016129032258064516,1.0,,,62
|
| 64 |
+
job_630130,0.0,0,64,512,4,1.8906,0,0.9688,5146.4805,4434.7366,5789.4312,31600,1,4146.9309,0.4746,0.0104,0.0669,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1238,-0.0056,0.0,1.0,0.0,0.0102,0.5469,0.4746,13.2111,51.0682,568.2429,33.8,857.9675,289.7247,0.0709,4.7406,35.0463,194.8197,230.1324,1359.0594,21.4195,195.0147,1102.761,0,63,63,504,448,20,51,,,0.07936507936507936,5.0,0.07936507936507936,5.0,0.746031746031746,47,0.09523809523809523,6.0,,,,,,,,,,63
|
| 65 |
+
job_630130,0.0,0,64,512,7,1.8906,0,0.875,5587.1855,5587.4615,5587.0158,31766,1,4900.2073,0.3809,0.0053,0.1109,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1128,0.0,0.0,1.0,0.0,0.0114,0.4844,0.3809,13.5775,51.5483,569.7428,33.6,857.9675,288.2247,0.1075,4.8015,38.5633,225.6816,264.5674,1636.5882,22.1274,225.8961,1345.0836,0,64,62,496,422,26,68,,,0.0967741935483871,6.0,0.04838709677419355,3.0,1.0,62,0.04838709677419355,3.0,0.0081,8.5496,,0.06451612903225806,4.0,,,,,64
|
| 66 |
+
job_630130,0.0,0,64,512,4,1.9531,0,0.9375,4798.2617,4883.6444,4723.5128,31812,1,4722.6084,0.4668,0.0183,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1087,0.0,0.0,1.0,0.0,0.0168,0.6094,0.4668,13.718,51.5807,568.653,33.7,857.9675,289.3145,0.075,4.7303,35.8252,213.9135,250.0471,1536.8344,21.852,214.1464,1260.1995,0,65,61,488,434,20,44,,,0.16393442622950818,10.0,0.06557377049180328,4.0,0.7213114754098361,44,0.04918032786885246,3.0,,,5.9602,,,,,,,65
|
| 67 |
+
job_630130,0.0,0,64,512,3,1.6875,0,0.9062,4990.3906,4683.8621,5191.767,31741,1,4339.1846,0.3965,0.0113,0.1192,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1176,-0.0005,0.0,1.0,0.0,0.0138,0.5781,0.3965,13.9927,51.9113,567.4896,33.9,857.9675,290.4779,0.0707,4.8411,34.6825,205.3311,240.3191,1507.0204,22.8738,205.5654,1238.9803,0,66,64,512,451,19,50,,,0.171875,11.0,0.03125,2.0,0.75,48,0.046875,3.0,0.0085,8.6432,,,,,,,,66
|
| 68 |
+
job_630130,0.0,0,64,512,3,1.5938,0,0.8906,4687.6602,4780.5265,4614.2762,31720,1,3991.8123,0.4414,0.0147,0.1094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1043,-0.0031,0.0,1.0,0.0,0.0126,0.5625,0.4414,14.1397,52.1146,567.5696,33.8,857.9675,290.3979,0.0726,4.6609,30.5811,185.1371,216.0168,1571.263,21.9126,185.3626,1328.6676,0,67,58,464,422,16,42,,,,,0.034482758620689655,2.0,0.6896551724137931,40,0.08620689655172414,5.0,,,,,,,,,,67
|
| 69 |
+
job_630130,0.0,0,64,512,4,1.5469,0,0.8281,4902.5645,4448.6776,5228.5101,31770,1,4445.7836,0.418,-0.0081,0.0909,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1147,-0.0,0.0,1.0,0.0,0.0128,0.5312,0.418,14.1629,52.2518,569.6768,33.6,857.9675,288.2908,0.0962,4.7131,33.8447,209.0269,243.2635,1442.218,21.5536,209.322,1172.6792,0,68,64,512,471,16,37,,,0.046875,3.0,0.046875,3.0,0.46875,30,0.0625,4.0,0.0082,8.6812,,,,0.015625,1.0,,,68
|
| 70 |
+
job_630130,0.0,0,64,512,4,1.7344,0,0.875,4912.1777,4638.6631,5239.691,31749,1,4027.0232,0.5449,0.0247,0.126,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1083,-0.002,0.0,1.0,0.0,0.0111,0.6562,0.5449,14.5627,52.6307,568.1539,33.8,857.9675,289.8137,0.0922,4.7384,33.01,189.5539,222.8763,1413.2169,23.2675,189.7737,1162.3272,0,69,62,496,442,18,54,,,,,0.03225806451612903,2.0,0.8387096774193549,52,,,,,,,,,,,,69
|
| 71 |
+
job_630130,0.0,0,64,512,3,1.9219,0,0.9062,4989.8223,4749.7429,5156.7649,31793,1,4613.5261,0.4102,-0.0096,0.0915,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1134,-0.0,0.0,1.0,0.0,0.0151,0.5312,0.4102,14.5696,52.6356,567.6034,33.8,857.9675,290.3641,0.0859,4.6794,34.9319,213.77,248.9891,1779.8183,22.218,213.9708,1503.9267,0,70,62,496,433,16,61,,,0.03225806451612903,2.0,0.12903225806451613,8.0,0.8387096774193549,52,0.03225806451612903,2.0,0.0081,8.4344,5.9241,,,,,,,70
|
| 72 |
+
job_661735,0.0,0,64,512,3,0.3281,0,0.2188,4521.9473,4333.5193,4758.5198,31663,1,3365.2785,0.5566,0.0074,0.094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0,0.0,1.0,0.0,0.0131,0.5965,0.5566,12.5978,50.5919,579.345,32.5,857.9676,278.6226,0.1569,4.5133,32.1738,167.8055,200.36,3675.9345,21.9439,168.0287,3449.1169,0,71,58,464,375,29,88,,,0.017241379310344827,1.0,,,1.4137931034482758,82,0.20689655172413793,12.0,,,,,,,,,,71
|
| 73 |
+
job_661735,0.0,0,64,512,1,0.9375,0,0.9375,5361.2598,5767.993,5205.1622,31699,1,5207.4533,0.2773,-0.013,0.1323,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1151,0.0,0.0,1.0,0.0,0.017,0.4219,0.2773,12.8662,50.7994,576.8029,32.8,857.9676,281.1647,0.0648,4.6437,37.3367,229.2593,266.8776,1642.3033,25.4393,229.4755,1345.3366,0,72,56,448,364,24,80,,,0.07142857142857142,4.0,0.16071428571428573,9.0,1.3392857142857142,75,0.16071428571428573,9.0,11.424,21.2849,,0.05357142857142857,3.0,,,,,72
|
| 74 |
+
job_661735,0.0,0,64,512,2,1.6406,1,1.0,4303.7402,4841.9941,4036.1871,31778,1,4623.5614,0.332,0.0023,0.0825,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0977,0.0,0.0,1.0,0.0,0.0095,0.4062,0.332,13.201,51.0858,576.4416,32.8,857.9676,281.526,0.0649,4.6475,32.7604,195.3408,228.4104,875.421,23.4598,195.5846,618.8995,0,73,63,504,471,12,33,,,,,0.047619047619047616,3.0,0.47619047619047616,30,0.07936507936507936,5.0,,,,,,,,,,73
|
| 75 |
+
job_661735,0.0,0,64,512,3,1.6406,0,0.8906,5242.7773,4926.2055,5552.0154,31761,1,4678.9952,0.4941,0.0138,0.1243,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1009,-0.0,0.0,1.0,0.0,0.0145,0.5938,0.4941,13.4431,51.3293,575.8802,32.9,857.9676,282.0874,0.062,4.6498,37.0456,231.3918,268.7201,1733.9771,23.6881,231.6122,1436.9161,0,74,59,472,401,23,56,,,0.2542372881355932,15.0,0.03389830508474576,2.0,0.847457627118644,50,0.1864406779661017,11.0,0.007,10.013,,0.05084745762711865,3.0,,,,,74
|
| 76 |
+
job_661735,0.0,0,64,512,4,1.9844,1,1.0,4678.0137,4390.89,4862.0673,31720,1,4446.3943,0.3906,-0.0162,0.1412,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1014,-0.002,0.0,1.0,0.0,0.0143,0.5625,0.3906,13.5915,51.5202,575.6154,32.9,857.9676,282.3522,0.0636,4.5241,34.8828,208.4871,243.6766,1567.8023,22.6625,208.7299,1296.9223,0,75,61,488,422,25,66,,,,,0.06557377049180328,4.0,0.7704918032786885,47,0.21311475409836064,13.0,,,6.424,0.09836065573770492,6.0,,,,,75
|
| 77 |
+
job_661735,0.0,0,64,512,4,2.4219,2,1.0,4572.6895,4582.679,4568.0657,31799,1,4302.6602,0.3164,-0.0023,0.112,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.103,0.0,0.0,1.0,0.0,0.0129,0.4531,0.3164,13.6858,51.5748,574.5665,33.0,857.9676,283.4011,0.0663,4.6913,30.5968,200.2044,231.0996,1415.5476,24.2091,200.4362,1155.5266,0,76,61,488,415,22,64,,,0.14754098360655737,9.0,0.04918032786885246,3.0,0.7868852459016393,48,0.21311475409836064,13.0,0.0074,9.8798,,0.11475409836065574,7.0,,,,,76
|
| 78 |
+
job_661735,0.0,0,64,512,4,3.0312,3,1.0,4597.7812,4086.6667,4944.6689,31759,1,4559.7375,0.4043,0.0013,0.0431,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0033,0.0,1.0,0.0,0.0068,0.4688,0.4043,14.3994,51.5634,573.7055,33.1,857.9676,284.2621,0.0642,4.5039,32.7244,196.2376,229.2847,1161.9454,21.4738,196.4958,906.6629,0,77,63,504,414,32,86,,,0.15873015873015872,10.0,,,0.8095238095238095,51,0.4603174603174603,29.0,,,,0.09523809523809523,6.0,0.015873015873015872,1.0,,,77
|
| 79 |
+
job_661735,0.0,0,64,512,4,4.0,4,1.0,5291.9141,6195.0086,5027.3712,28898,1,4117.8095,0.2266,0.0072,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1143,0.0,0.0,1.0,0.0,0.0105,0.375,0.2266,14.2961,51.5109,573.1121,33.2,857.9676,284.8555,0.1485,4.4428,33.4904,188.476,222.3217,1837.7946,17.1266,188.6825,1593.8676,0,78,67,536,466,24,64,,,0.08955223880597014,6.0,0.029850746268656716,2.0,0.6567164179104478,44,0.23880597014925373,16.0,0.0588,10.9374,,0.029850746268656716,2.0,,,,,78
|
training_logs/20260612_220124_vllm_metrics_job_630124.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/20260612_220124_vllm_metrics_job_630127.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/20260612_220124_vllm_metrics_job_630130.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/20260612_220124_vllm_metrics_job_661734.csv
ADDED
|
@@ -0,0 +1,1153 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
datetime_str,n_engines_reporting,unique_ips,total_prompt_throughput_tokens_per_sec,total_generation_throughput_tokens_per_sec,total_running_requests,total_waiting_requests,avg_prompt_throughput_per_engine,avg_generation_throughput_per_engine,avg_running_requests_per_engine,avg_waiting_requests_per_engine,avg_gpu_kv_cache_usage_pct,avg_prefix_cache_hit_rate_pct,min_running_requests,max_running_requests,min_generation_throughput,max_generation_throughput
|
| 2 |
+
06-08 14:43:32,1,1,1.5,0.0,1,0,1.5,0.0,1.0,0.0,0.3,0.0,1,1,0.0,0.0
|
| 3 |
+
06-08 14:43:37,1,1,0.0,44.6,1,0,0.0,44.6,1.0,0.0,0.4,0.0,1,1,44.6,44.6
|
| 4 |
+
06-08 14:43:42,1,1,0.0,41.0,1,0,0.0,41.0,1.0,0.0,0.4,0.0,1,1,41.0,41.0
|
| 5 |
+
06-08 14:43:47,1,1,74.4,46.7,2,0,74.4,46.7,2.0,0.0,0.6,30.9,2,2,46.7,46.7
|
| 6 |
+
06-08 14:43:52,1,1,0.0,81.2,2,0,0.0,81.2,2.0,0.0,0.7,30.9,2,2,81.2,81.2
|
| 7 |
+
06-08 14:43:57,1,1,0.0,81.5,2,0,0.0,81.5,2.0,0.0,0.8,30.9,2,2,81.5,81.5
|
| 8 |
+
06-08 14:44:02,1,1,0.0,81.2,2,0,0.0,81.2,2.0,0.0,0.9,30.9,2,2,81.2,81.2
|
| 9 |
+
06-08 14:44:07,1,1,0.0,45.3,1,0,0.0,45.3,1.0,0.0,0.5,30.9,1,1,45.3,45.3
|
| 10 |
+
06-08 14:44:12,1,1,111.1,59.3,2,0,111.1,59.3,2.0,0.0,1.2,59.6,2,2,59.3,59.3
|
| 11 |
+
06-08 14:44:17,1,1,0.0,80.8,2,0,0.0,80.8,2.0,0.0,1.3,59.6,2,2,80.8,80.8
|
| 12 |
+
06-08 14:44:22,1,1,0.0,81.0,2,0,0.0,81.0,2.0,0.0,1.5,59.6,2,2,81.0,81.0
|
| 13 |
+
06-08 14:44:27,1,1,0.0,81.0,2,0,0.0,81.0,2.0,0.0,1.6,59.6,2,2,81.0,81.0
|
| 14 |
+
06-08 14:44:32,1,1,0.0,80.8,2,0,0.0,80.8,2.0,0.0,1.7,59.6,2,2,80.8,80.8
|
| 15 |
+
06-08 14:44:37,1,1,0.0,80.9,2,0,0.0,80.9,2.0,0.0,1.8,59.6,2,2,80.9,80.9
|
| 16 |
+
06-08 14:44:42,1,1,0.0,72.2,2,0,0.0,72.2,2.0,0.0,1.9,59.6,2,2,72.2,72.2
|
| 17 |
+
06-08 14:44:47,1,1,0.0,70.3,2,0,0.0,70.3,2.0,0.0,2.0,59.6,2,2,70.3,70.3
|
| 18 |
+
06-08 14:44:52,1,1,0.0,69.2,2,0,0.0,69.2,2.0,0.0,2.1,59.6,2,2,69.2,69.2
|
| 19 |
+
06-08 14:44:57,1,1,0.0,85.7,2,0,0.0,85.7,2.0,0.0,2.2,59.6,2,2,85.7,85.7
|
| 20 |
+
06-08 14:45:02,1,1,0.0,50.9,1,0,0.0,50.9,1.0,0.0,1.1,59.6,1,1,50.9,50.9
|
| 21 |
+
06-08 14:45:07,1,1,0.0,45.7,1,0,0.0,45.7,1.0,0.0,1.2,59.6,1,1,45.7,45.7
|
| 22 |
+
06-08 14:45:12,1,1,0.0,45.6,1,0,0.0,45.6,1.0,0.0,1.3,59.6,1,1,45.6,45.6
|
| 23 |
+
06-08 14:45:17,1,1,151.7,112.6,3,0,151.7,112.6,3.0,0.0,2.8,75.1,3,3,112.6,112.6
|
| 24 |
+
06-08 14:45:22,1,1,0.0,136.0,3,0,0.0,136.0,3.0,0.0,3.0,75.1,3,3,136.0,136.0
|
| 25 |
+
06-08 14:45:27,1,1,0.0,132.8,3,0,0.0,132.8,3.0,0.0,4.6,68.0,3,3,132.8,132.8
|
| 26 |
+
06-08 14:45:32,1,1,0.0,132.7,3,0,0.0,132.7,3.0,0.0,4.8,68.0,3,3,132.7,132.7
|
| 27 |
+
06-08 14:45:37,1,1,0.0,132.3,3,0,0.0,132.3,3.0,0.0,5.0,68.0,3,3,132.3,132.3
|
| 28 |
+
06-08 14:45:42,1,1,0.0,119.1,2,0,0.0,119.1,2.0,0.0,2.1,75.1,2,2,119.1,119.1
|
| 29 |
+
06-08 14:45:47,1,1,0.0,50.9,1,0,0.0,50.9,1.0,0.0,1.7,75.1,1,1,50.9,50.9
|
| 30 |
+
06-08 14:45:52,1,1,0.0,45.5,1,0,0.0,45.5,1.0,0.0,1.8,75.1,1,1,45.5,45.5
|
| 31 |
+
06-08 14:45:57,1,1,0.0,177.1,4,0,0.0,177.1,4.0,0.0,6.9,69.4,4,4,177.1,177.1
|
| 32 |
+
06-08 14:46:02,1,1,0.0,170.9,3,0,0.0,170.9,3.0,0.0,5.9,69.4,3,3,170.9,170.9
|
| 33 |
+
06-08 14:46:07,1,1,0.0,132.7,3,0,0.0,132.7,3.0,0.0,6.1,69.4,3,3,132.7,132.7
|
| 34 |
+
06-08 14:46:12,1,1,0.0,89.4,2,0,0.0,89.4,2.0,0.0,2.2,30.4,2,2,89.4,89.4
|
| 35 |
+
06-08 14:46:17,1,1,0.0,89.9,2,0,0.0,89.9,2.0,0.0,2.6,82.0,2,2,89.9,89.9
|
| 36 |
+
06-08 14:46:22,1,1,21.3,98.0,3,0,21.3,98.0,3.0,0.0,4.3,84.2,3,3,98.0,98.0
|
| 37 |
+
06-08 14:46:27,1,1,152.6,164.6,4,0,152.6,164.6,4.0,0.0,6.5,85.1,4,4,164.6,164.6
|
| 38 |
+
06-08 14:46:32,1,1,0.0,179.3,4,0,0.0,179.3,4.0,0.0,6.8,85.1,4,4,179.3,179.3
|
| 39 |
+
06-08 14:46:37,1,1,0.0,152.4,3,0,0.0,152.4,3.0,0.0,5.3,85.1,3,3,152.4,152.4
|
| 40 |
+
06-08 14:46:42,1,1,241.9,151.0,4,0,241.9,151.0,4.0,0.0,6.9,84.5,4,4,151.0,151.0
|
| 41 |
+
06-08 14:46:47,1,1,0.0,180.4,4,0,0.0,180.4,4.0,0.0,7.1,84.5,4,4,180.4,180.4
|
| 42 |
+
06-08 14:46:52,1,1,0.0,179.0,4,0,0.0,179.0,4.0,0.0,7.4,84.5,4,4,179.0,179.0
|
| 43 |
+
06-08 14:46:57,1,1,0.0,179.4,4,0,0.0,179.4,4.0,0.0,7.6,84.5,4,4,179.4,179.4
|
| 44 |
+
06-08 14:47:02,1,1,0.0,179.7,4,0,0.0,179.7,4.0,0.0,7.9,84.5,4,4,179.7,179.7
|
| 45 |
+
06-08 14:47:07,1,1,0.0,179.8,4,0,0.0,179.8,4.0,0.0,8.1,84.5,4,4,179.8,179.8
|
| 46 |
+
06-08 14:47:12,1,1,0.0,179.3,4,0,0.0,179.3,4.0,0.0,8.4,84.5,4,4,179.3,179.3
|
| 47 |
+
06-08 14:47:17,1,1,0.0,180.9,4,0,0.0,180.9,4.0,0.0,8.6,84.5,4,4,180.9,180.9
|
| 48 |
+
06-08 14:47:23,1,1,0.0,180.9,4,0,0.0,180.9,4.0,0.0,8.9,84.5,4,4,180.9,180.9
|
| 49 |
+
06-08 14:47:28,1,1,0.0,181.1,4,0,0.0,181.1,4.0,0.0,9.1,84.5,4,4,181.1,181.1
|
| 50 |
+
06-08 14:47:33,1,1,0.0,180.9,4,0,0.0,180.9,4.0,0.0,9.4,84.5,4,4,180.9,180.9
|
| 51 |
+
06-08 14:47:38,1,1,0.0,181.0,4,0,0.0,181.0,4.0,0.0,9.6,84.5,4,4,181.0,181.0
|
| 52 |
+
06-08 14:47:43,1,1,0.0,180.7,4,0,0.0,180.7,4.0,0.0,9.9,84.5,4,4,180.7,180.7
|
| 53 |
+
06-08 14:47:48,1,1,0.0,180.4,4,0,0.0,180.4,4.0,0.0,10.1,84.5,4,4,180.4,180.4
|
| 54 |
+
06-08 14:47:53,1,1,0.0,180.9,4,0,0.0,180.9,4.0,0.0,10.4,84.5,4,4,180.9,180.9
|
| 55 |
+
06-08 14:47:58,1,1,0.0,98.1,2,0,0.0,98.1,2.0,0.0,5.8,86.2,2,2,98.1,98.1
|
| 56 |
+
06-08 14:48:03,1,1,0.0,98.5,2,0,0.0,98.5,2.0,0.0,5.9,86.2,2,2,98.5,98.5
|
| 57 |
+
06-08 14:48:08,1,1,22.2,105.6,3,0,22.2,105.6,3.0,0.0,7.8,87.7,3,3,105.6,105.6
|
| 58 |
+
06-08 14:48:13,1,1,0.0,146.7,3,0,0.0,146.7,3.0,0.0,8.0,87.7,3,3,146.7,146.7
|
| 59 |
+
06-08 14:48:18,1,1,109.1,124.7,3,0,109.1,124.7,3.0,0.0,6.5,87.0,3,3,124.7,124.7
|
| 60 |
+
06-08 14:48:23,1,1,0.0,147.4,3,0,0.0,147.4,3.0,0.0,6.7,87.0,3,3,147.4,147.4
|
| 61 |
+
06-08 14:48:28,1,1,20.8,165.7,4,0,20.8,165.7,4.0,0.0,9.2,88.6,4,4,165.7,165.7
|
| 62 |
+
06-08 14:48:33,1,1,0.0,195.8,4,0,0.0,195.8,4.0,0.0,9.4,88.6,4,4,195.8,195.8
|
| 63 |
+
06-08 14:48:38,1,1,72.3,190.7,4,0,72.3,190.7,4.0,0.0,9.2,89.3,4,4,190.7,190.7
|
| 64 |
+
06-08 14:48:43,1,1,0.0,195.6,4,0,0.0,195.6,4.0,0.0,9.5,89.3,4,4,195.6,195.6
|
| 65 |
+
06-08 14:48:48,1,1,0.0,196.4,4,0,0.0,196.4,4.0,0.0,9.8,89.3,4,4,196.4,196.4
|
| 66 |
+
06-08 14:48:53,1,1,0.0,195.7,4,0,0.0,195.7,4.0,0.0,10.0,89.3,4,4,195.7,195.7
|
| 67 |
+
06-08 14:48:58,1,1,0.0,196.0,4,0,0.0,196.0,4.0,0.0,10.3,89.3,4,4,196.0,196.0
|
| 68 |
+
06-08 14:49:03,1,1,0.0,196.3,4,0,0.0,196.3,4.0,0.0,10.6,89.3,4,4,196.3,196.3
|
| 69 |
+
06-08 14:49:08,1,1,0.0,196.0,4,0,0.0,196.0,4.0,0.0,10.9,89.3,4,4,196.0,196.0
|
| 70 |
+
06-08 14:49:13,1,1,0.0,195.9,4,0,0.0,195.9,4.0,0.0,11.1,89.3,4,4,195.9,195.9
|
| 71 |
+
06-08 14:49:18,1,1,0.0,195.8,4,0,0.0,195.8,4.0,0.0,11.4,89.3,4,4,195.8,195.8
|
| 72 |
+
06-08 14:49:23,1,1,0.0,196.1,4,0,0.0,196.1,4.0,0.0,11.7,89.3,4,4,196.1,196.1
|
| 73 |
+
06-08 14:49:28,1,1,0.0,138.9,3,0,0.0,138.9,3.0,0.0,8.9,56.4,3,3,138.9,138.9
|
| 74 |
+
06-08 14:49:33,1,1,0.0,139.2,3,0,0.0,139.2,3.0,0.0,9.1,56.4,3,3,139.2,139.2
|
| 75 |
+
06-08 14:49:38,1,1,0.0,139.0,3,0,0.0,139.0,3.0,0.0,9.3,56.4,3,3,139.0,139.0
|
| 76 |
+
06-08 14:49:43,1,1,0.0,139.3,3,0,0.0,139.3,3.0,0.0,9.5,56.4,3,3,139.3,139.3
|
| 77 |
+
06-08 14:49:48,1,1,0.0,139.2,3,0,0.0,139.2,3.0,0.0,9.6,56.4,3,3,139.2,139.2
|
| 78 |
+
06-08 14:49:53,1,1,0.0,138.8,3,0,0.0,138.8,3.0,0.0,9.8,56.4,3,3,138.8,138.8
|
| 79 |
+
06-08 14:49:58,1,1,0.0,138.9,3,0,0.0,138.9,3.0,0.0,10.0,56.4,3,3,138.9,138.9
|
| 80 |
+
06-08 14:50:03,1,1,0.0,139.4,3,0,0.0,139.4,3.0,0.0,10.2,56.4,3,3,139.4,139.4
|
| 81 |
+
06-08 14:50:08,1,1,0.0,139.5,3,0,0.0,139.5,3.0,0.0,10.4,56.4,3,3,139.5,139.5
|
| 82 |
+
06-08 14:50:13,1,1,0.0,139.0,3,0,0.0,139.0,3.0,0.0,10.6,56.4,3,3,139.0,139.0
|
| 83 |
+
06-08 14:50:18,1,1,0.0,212.7,5,0,0.0,212.7,5.0,0.0,16.6,88.9,5,5,212.7,212.7
|
| 84 |
+
06-08 14:50:23,1,1,0.0,154.6,3,0,0.0,154.6,3.0,0.0,12.9,88.9,3,3,154.6,154.6
|
| 85 |
+
06-08 14:50:28,1,1,0.0,128.0,3,0,0.0,128.0,3.0,0.0,13.1,88.9,3,3,128.0,128.0
|
| 86 |
+
06-08 14:50:33,1,1,0.0,128.3,3,0,0.0,128.3,3.0,0.0,13.2,88.9,3,3,128.3,128.3
|
| 87 |
+
06-08 14:50:38,1,1,0.0,128.1,3,0,0.0,128.1,3.0,0.0,13.4,88.9,3,3,128.1,128.1
|
| 88 |
+
06-08 14:50:43,1,1,0.0,127.9,3,0,0.0,127.9,3.0,0.0,13.6,88.9,3,3,127.9,127.9
|
| 89 |
+
06-08 14:50:48,1,1,0.0,128.8,3,0,0.0,128.8,3.0,0.0,13.8,88.9,3,3,128.8,128.8
|
| 90 |
+
06-08 14:50:53,1,1,0.0,128.7,3,0,0.0,128.7,3.0,0.0,13.9,88.9,3,3,128.7,128.7
|
| 91 |
+
06-08 14:50:59,1,1,44.3,178.5,4,0,44.3,178.5,4.0,0.0,10.1,75.7,4,4,178.5,178.5
|
| 92 |
+
06-08 14:51:04,1,1,111.8,190.4,5,0,111.8,190.4,5.0,0.0,10.9,75.8,5,5,190.4,190.4
|
| 93 |
+
06-08 14:51:09,1,1,208.0,261.9,7,0,208.0,261.9,7.0,0.0,12.2,76.0,7,7,261.9,261.9
|
| 94 |
+
06-08 14:51:14,1,1,0.0,327.2,7,0,0.0,327.2,7.0,0.0,12.6,76.0,7,7,327.2,327.2
|
| 95 |
+
06-08 14:51:19,1,1,0.0,326.6,7,0,0.0,326.6,7.0,0.0,13.1,76.0,7,7,326.6,326.6
|
| 96 |
+
06-08 14:51:24,1,1,0.0,303.7,6,0,0.0,303.7,6.0,0.0,12.5,76.0,6,6,303.7,303.7
|
| 97 |
+
06-08 14:51:29,1,1,0.0,281.2,6,0,0.0,281.2,6.0,0.0,12.9,76.0,6,6,281.2,281.2
|
| 98 |
+
06-08 14:51:34,1,1,75.7,287.9,7,0,75.7,287.9,7.0,0.0,13.4,75.6,7,7,287.9,287.9
|
| 99 |
+
06-08 14:51:39,1,1,98.7,382.7,10,0,98.7,382.7,10.0,0.0,15.1,78.3,10,10,382.7,382.7
|
| 100 |
+
06-08 14:51:44,1,1,0.0,466.9,10,0,0.0,466.9,10.0,0.0,15.7,78.3,10,10,466.9,466.9
|
| 101 |
+
06-08 14:51:49,1,1,0.0,435.8,9,0,0.0,435.8,9.0,0.0,15.2,78.3,9,9,435.8,435.8
|
| 102 |
+
06-08 14:51:54,1,1,217.6,434.8,12,0,217.6,434.8,12.0,0.0,16.1,77.3,12,12,434.8,434.8
|
| 103 |
+
06-08 14:51:59,1,1,0.0,563.2,12,0,0.0,563.2,12.0,0.0,16.9,77.3,12,12,563.2,563.2
|
| 104 |
+
06-08 14:52:04,1,1,0.0,562.0,12,0,0.0,562.0,12.0,0.0,17.7,77.3,12,12,562.0,562.0
|
| 105 |
+
06-08 14:52:09,1,1,83.5,569.2,13,0,83.5,569.2,13.0,0.0,18.6,76.9,13,13,569.2,569.2
|
| 106 |
+
06-08 14:52:14,1,1,0.0,604.5,13,0,0.0,604.5,13.0,0.0,19.4,76.9,13,13,604.5,604.5
|
| 107 |
+
06-08 14:52:19,1,1,0.0,604.4,13,0,0.0,604.4,13.0,0.0,20.3,76.9,13,13,604.4,604.4
|
| 108 |
+
06-08 14:52:22,1,1,117.4,198.7,4,0,117.4,198.7,4.0,0.0,11.5,81.3,4,4,198.7,198.7
|
| 109 |
+
06-08 14:52:29,1,1,297.3,454.5,11,0,297.3,454.5,11.0,0.0,22.6,81.3,11,11,454.5,454.5
|
| 110 |
+
06-08 14:52:34,1,1,0.0,321.2,7,0,0.0,321.2,7.0,0.0,27.8,89.6,7,7,321.2,321.2
|
| 111 |
+
06-08 14:52:39,1,1,0.0,409.9,9,0,0.0,409.9,9.0,0.0,21.8,81.3,9,9,409.9,409.9
|
| 112 |
+
06-08 14:52:44,1,1,0.0,385.2,9,0,0.0,385.2,9.0,0.0,22.4,81.3,9,9,385.2,385.2
|
| 113 |
+
06-08 14:52:49,1,1,0.0,385.0,9,0,0.0,385.0,9.0,0.0,22.9,81.3,9,9,385.0,385.0
|
| 114 |
+
06-08 14:52:54,1,1,200.7,378.9,10,0,200.7,378.9,10.0,0.0,23.6,81.1,10,10,378.9,378.9
|
| 115 |
+
06-08 14:52:59,1,1,385.9,450.4,11,0,385.9,450.4,11.0,0.0,26.2,80.6,11,11,450.4,450.4
|
| 116 |
+
06-08 14:53:04,1,1,0.0,460.0,11,0,0.0,460.0,11.0,0.0,26.9,80.6,11,11,460.0,460.0
|
| 117 |
+
06-08 14:53:09,1,1,0.0,355.4,7,0,0.0,355.4,7.0,0.0,25.0,81.5,7,7,355.4,355.4
|
| 118 |
+
06-08 14:53:14,1,1,0.0,322.8,7,0,0.0,322.8,7.0,0.0,25.4,81.5,7,7,322.8,322.8
|
| 119 |
+
06-08 14:53:19,1,1,0.0,323.5,7,0,0.0,323.5,7.0,0.0,25.9,81.5,7,7,323.5,323.5
|
| 120 |
+
06-08 14:53:24,1,1,0.0,304.0,6,0,0.0,304.0,6.0,0.0,25.7,81.5,6,6,304.0,304.0
|
| 121 |
+
06-08 14:53:29,1,1,0.0,279.2,6,0,0.0,279.2,6.0,0.0,26.1,81.5,6,6,279.2,279.2
|
| 122 |
+
06-08 14:53:34,1,1,0.0,402.9,9,0,0.0,402.9,9.0,0.0,27.1,80.6,9,9,402.9,402.9
|
| 123 |
+
06-08 14:53:39,1,1,0.0,380.9,9,0,0.0,380.9,9.0,0.0,27.6,80.6,9,9,380.9,380.9
|
| 124 |
+
06-08 14:53:44,1,1,0.0,380.8,9,0,0.0,380.8,9.0,0.0,28.2,80.6,9,9,380.8,380.8
|
| 125 |
+
06-08 14:53:49,1,1,0.0,381.8,9,0,0.0,381.8,9.0,0.0,28.7,80.6,9,9,381.8,381.8
|
| 126 |
+
06-08 14:53:54,1,1,0.0,380.6,9,0,0.0,380.6,9.0,0.0,29.2,80.6,9,9,380.6,380.6
|
| 127 |
+
06-08 14:53:59,1,1,0.0,380.1,9,0,0.0,380.1,9.0,0.0,29.8,80.6,9,9,380.1,380.1
|
| 128 |
+
06-08 14:54:04,1,1,0.0,383.8,9,0,0.0,383.8,9.0,0.0,38.8,86.9,9,9,383.8,383.8
|
| 129 |
+
06-08 14:54:09,1,1,0.0,384.5,9,0,0.0,384.5,9.0,0.0,39.3,86.9,9,9,384.5,384.5
|
| 130 |
+
06-08 14:54:14,1,1,0.0,383.7,9,0,0.0,383.7,9.0,0.0,39.9,86.9,9,9,383.7,383.7
|
| 131 |
+
06-08 14:54:19,1,1,0.0,383.0,9,0,0.0,383.0,9.0,0.0,40.4,86.9,9,9,383.0,383.0
|
| 132 |
+
06-08 14:54:24,1,1,179.5,408.4,11,0,179.5,408.4,11.0,0.0,41.6,86.6,11,11,408.4,408.4
|
| 133 |
+
06-08 14:54:30,1,1,8.8,61.0,2,0,8.8,61.0,2.0,0.0,2.3,86.5,2,2,61.0,61.0
|
| 134 |
+
06-08 14:54:35,1,1,0.0,59.5,1,0,0.0,59.5,1.0,0.0,0.4,86.5,1,1,59.5,59.5
|
| 135 |
+
06-08 14:54:40,1,1,0.0,49.4,1,0,0.0,49.4,1.0,0.0,0.5,86.5,1,1,49.4,49.4
|
| 136 |
+
06-08 14:54:45,1,1,0.0,49.4,1,0,0.0,49.4,1.0,0.0,0.6,86.5,1,1,49.4,49.4
|
| 137 |
+
06-08 14:54:50,1,1,0.0,49.4,1,0,0.0,49.4,1.0,0.0,0.6,86.5,1,1,49.4,49.4
|
| 138 |
+
06-08 14:54:55,1,1,10.0,92.4,2,0,10.0,92.4,2.0,0.0,0.8,86.7,2,2,92.4,92.4
|
| 139 |
+
06-08 14:55:00,1,1,0.0,97.9,2,0,0.0,97.9,2.0,0.0,0.9,86.7,2,2,97.9,97.9
|
| 140 |
+
06-08 14:55:05,1,1,95.1,134.0,3,0,95.1,134.0,3.0,0.0,1.8,86.6,3,3,134.0,134.0
|
| 141 |
+
06-08 14:55:10,1,1,0.0,147.1,3,0,0.0,147.1,3.0,0.0,2.0,86.6,3,3,147.1,147.1
|
| 142 |
+
06-08 14:55:15,1,1,0.0,147.3,3,0,0.0,147.3,3.0,0.0,2.2,86.6,3,3,147.3,147.3
|
| 143 |
+
06-08 14:55:20,1,1,36.8,149.2,4,0,36.8,149.2,4.0,0.0,3.1,87.0,4,4,149.2,149.2
|
| 144 |
+
06-08 14:55:25,1,1,0.0,194.9,4,0,0.0,194.9,4.0,0.0,3.4,87.0,4,4,194.9,194.9
|
| 145 |
+
06-08 14:55:30,1,1,0.0,162.9,2,0,0.0,162.9,2.0,0.0,1.7,87.0,2,2,162.9,162.9
|
| 146 |
+
06-08 14:55:35,1,1,0.0,463.9,9,0,0.0,463.9,9.0,0.0,36.4,89.6,9,9,463.9,463.9
|
| 147 |
+
06-08 14:55:40,1,1,0.0,419.8,9,0,0.0,419.8,9.0,0.0,36.9,89.6,9,9,419.8,419.8
|
| 148 |
+
06-08 14:55:45,1,1,0.0,421.0,9,0,0.0,421.0,9.0,0.0,37.5,89.6,9,9,421.0,421.0
|
| 149 |
+
06-08 14:55:50,1,1,0.0,421.9,9,0,0.0,421.9,9.0,0.0,38.1,89.6,9,9,421.9,421.9
|
| 150 |
+
06-08 14:55:55,1,1,0.0,395.7,8,0,0.0,395.7,8.0,0.0,37.7,89.6,8,8,395.7,395.7
|
| 151 |
+
06-08 14:56:00,1,1,211.0,390.2,8,0,211.0,390.2,8.0,0.0,30.5,89.2,8,8,390.2,390.2
|
| 152 |
+
06-08 14:56:05,1,1,0.0,458.5,10,0,0.0,458.5,10.0,0.0,44.9,82.0,10,10,458.5,458.5
|
| 153 |
+
06-08 14:56:10,1,1,0.0,455.2,9,0,0.0,455.2,9.0,0.0,36.7,82.0,9,9,455.2,455.2
|
| 154 |
+
06-08 14:56:15,1,1,0.0,397.7,8,0,0.0,397.7,8.0,0.0,28.3,82.0,8,8,397.7,397.7
|
| 155 |
+
06-08 14:56:20,1,1,0.0,368.6,8,0,0.0,368.6,8.0,0.0,28.8,82.0,8,8,368.6,368.6
|
| 156 |
+
06-08 14:56:25,1,1,0.0,367.6,8,0,0.0,367.6,8.0,0.0,29.4,82.0,8,8,367.6,367.6
|
| 157 |
+
06-08 14:56:30,1,1,0.0,325.5,7,0,0.0,325.5,7.0,0.0,28.2,82.0,7,7,325.5,325.5
|
| 158 |
+
06-08 14:56:35,1,1,0.0,321.7,7,0,0.0,321.7,7.0,0.0,28.7,82.0,7,7,321.7,321.7
|
| 159 |
+
06-08 14:56:40,1,1,0.0,321.6,7,0,0.0,321.6,7.0,0.0,29.2,82.0,7,7,321.6,321.6
|
| 160 |
+
06-08 14:56:45,1,1,0.0,320.5,7,0,0.0,320.5,7.0,0.0,29.6,82.0,7,7,320.5,320.5
|
| 161 |
+
06-08 14:56:50,1,1,0.0,291.7,6,0,0.0,291.7,6.0,0.0,21.1,82.0,6,6,291.7,291.7
|
| 162 |
+
06-08 14:56:55,1,1,0.0,294.4,7,0,0.0,294.4,7.0,0.0,24.6,87.0,7,7,294.4,294.4
|
| 163 |
+
06-08 14:57:01,1,1,68.1,302.2,7,0,68.1,302.2,7.0,0.0,22.7,82.5,7,7,302.2,302.2
|
| 164 |
+
06-08 14:57:06,1,1,0.0,321.3,7,0,0.0,321.3,7.0,0.0,23.2,82.5,7,7,321.3,321.3
|
| 165 |
+
06-08 14:57:11,1,1,0.0,316.5,6,0,0.0,316.5,6.0,0.0,22.6,82.5,6,6,316.5,316.5
|
| 166 |
+
06-08 14:57:16,1,1,87.1,279.0,7,0,87.1,279.0,7.0,0.0,23.1,82.2,7,7,279.0,279.0
|
| 167 |
+
06-08 14:57:21,1,1,0.0,435.6,10,0,0.0,435.6,10.0,0.0,41.0,84.2,10,10,435.6,435.6
|
| 168 |
+
06-08 14:57:26,1,1,121.7,479.3,12,0,121.7,479.3,12.0,0.0,42.8,84.4,12,12,479.3,479.3
|
| 169 |
+
06-08 14:57:31,1,1,0.0,506.9,10,0,0.0,506.9,10.0,0.0,34.5,84.4,10,10,506.9,506.9
|
| 170 |
+
06-08 14:57:36,1,1,0.0,424.1,10,0,0.0,424.1,10.0,0.0,35.1,84.4,10,10,424.1,424.1
|
| 171 |
+
06-08 14:57:41,1,1,0.0,422.5,10,0,0.0,422.5,10.0,0.0,35.6,84.4,10,10,422.5,422.5
|
| 172 |
+
06-08 14:57:46,1,1,153.3,431.6,10,0,153.3,431.6,10.0,0.0,40.0,86.8,10,10,431.6,431.6
|
| 173 |
+
06-08 14:57:51,1,1,51.8,508.4,11,0,51.8,508.4,11.0,0.0,41.7,86.9,11,11,508.4,508.4
|
| 174 |
+
06-08 14:57:56,1,1,0.0,468.6,11,0,0.0,468.6,11.0,0.0,37.6,84.5,11,11,468.6,468.6
|
| 175 |
+
06-08 14:58:01,1,1,0.0,483.1,10,0,0.0,483.1,10.0,0.0,42.4,86.9,10,10,483.1,483.1
|
| 176 |
+
06-08 14:58:06,1,1,91.4,479.1,10,0,91.4,479.1,10.0,0.0,42.8,86.9,10,10,479.1,479.1
|
| 177 |
+
06-08 14:58:11,1,1,0.0,465.2,10,0,0.0,465.2,10.0,0.0,43.5,86.9,10,10,465.2,465.2
|
| 178 |
+
06-08 14:58:16,1,1,0.0,464.0,10,0,0.0,464.0,10.0,0.0,44.1,86.9,10,10,464.0,464.0
|
| 179 |
+
06-08 14:58:21,1,1,51.5,474.6,8,0,51.5,474.6,8.0,0.0,34.9,86.8,8,8,474.6,474.6
|
| 180 |
+
06-08 14:58:26,1,1,0.0,372.2,8,0,0.0,372.2,8.0,0.0,35.4,86.8,8,8,372.2,372.2
|
| 181 |
+
06-08 14:58:31,1,1,0.0,372.4,8,0,0.0,372.4,8.0,0.0,35.9,86.8,8,8,372.4,372.4
|
| 182 |
+
06-08 14:58:36,1,1,0.0,372.7,8,0,0.0,372.7,8.0,0.0,36.5,86.8,8,8,372.7,372.7
|
| 183 |
+
06-08 14:58:41,1,1,0.0,372.0,8,0,0.0,372.0,8.0,0.0,37.0,86.8,8,8,372.0,372.0
|
| 184 |
+
06-08 14:58:46,1,1,1.4,386.9,9,0,1.4,386.9,9.0,0.0,37.6,86.9,9,9,386.9,386.9
|
| 185 |
+
06-08 14:58:51,1,1,0.0,400.4,8,0,0.0,400.4,8.0,0.0,36.4,86.9,8,8,400.4,400.4
|
| 186 |
+
06-08 14:58:56,1,1,0.0,371.7,8,0,0.0,371.7,8.0,0.0,36.9,86.9,8,8,371.7,371.7
|
| 187 |
+
06-08 14:59:01,1,1,0.0,372.7,8,0,0.0,372.7,8.0,0.0,37.4,86.9,8,8,372.7,372.7
|
| 188 |
+
06-08 14:59:06,1,1,0.0,372.9,8,0,0.0,372.9,8.0,0.0,38.0,86.9,8,8,372.9,372.9
|
| 189 |
+
06-08 14:59:11,1,1,0.0,371.7,8,0,0.0,371.7,8.0,0.0,38.5,86.9,8,8,371.7,371.7
|
| 190 |
+
06-08 14:59:16,1,1,0.0,318.5,7,0,0.0,318.5,7.0,0.0,28.5,85.1,7,7,318.5,318.5
|
| 191 |
+
06-08 14:59:21,1,1,0.0,319.2,7,0,0.0,319.2,7.0,0.0,28.9,85.1,7,7,319.2,319.2
|
| 192 |
+
06-08 14:59:26,1,1,0.0,319.1,7,0,0.0,319.1,7.0,0.0,29.4,85.1,7,7,319.1,319.1
|
| 193 |
+
06-08 14:59:32,1,1,0.0,320.0,7,0,0.0,320.0,7.0,0.0,29.8,85.1,7,7,320.0,320.0
|
| 194 |
+
06-08 14:59:37,1,1,0.0,318.8,7,0,0.0,318.8,7.0,0.0,30.3,85.1,7,7,318.8,318.8
|
| 195 |
+
06-08 14:59:42,1,1,112.6,327.0,8,0,112.6,327.0,8.0,0.0,32.7,85.4,8,8,327.0,327.0
|
| 196 |
+
06-08 14:59:47,1,1,68.4,386.8,9,0,68.4,386.8,9.0,0.0,33.7,85.4,9,9,386.8,386.8
|
| 197 |
+
06-08 14:59:52,1,1,57.4,450.1,10,0,57.4,450.1,10.0,0.0,34.5,85.3,10,10,450.1,450.1
|
| 198 |
+
06-08 14:59:57,1,1,0.0,360.1,7,0,0.0,360.1,7.0,0.0,34.0,82.3,7,7,360.1,360.1
|
| 199 |
+
06-08 15:00:02,1,1,0.0,322.7,7,0,0.0,322.7,7.0,0.0,34.4,82.3,7,7,322.7,322.7
|
| 200 |
+
06-08 15:00:07,1,1,0.0,322.4,7,0,0.0,322.4,7.0,0.0,34.9,82.3,7,7,322.4,322.4
|
| 201 |
+
06-08 15:00:12,1,1,0.0,282.7,6,0,0.0,282.7,6.0,0.0,33.8,82.3,6,6,282.7,282.7
|
| 202 |
+
06-08 15:00:17,1,1,0.0,276.0,6,0,0.0,276.0,6.0,0.0,34.2,82.3,6,6,276.0,276.0
|
| 203 |
+
06-08 15:00:22,1,1,0.0,276.4,6,0,0.0,276.4,6.0,0.0,34.6,82.3,6,6,276.4,276.4
|
| 204 |
+
06-08 15:00:27,1,1,0.0,276.6,6,0,0.0,276.6,6.0,0.0,34.9,82.3,6,6,276.6,276.6
|
| 205 |
+
06-08 15:00:32,1,1,0.0,276.1,6,0,0.0,276.1,6.0,0.0,35.3,82.3,6,6,276.1,276.1
|
| 206 |
+
06-08 15:00:37,1,1,0.0,275.7,6,0,0.0,275.7,6.0,0.0,35.7,82.3,6,6,275.7,275.7
|
| 207 |
+
06-08 15:00:42,1,1,0.0,276.3,6,0,0.0,276.3,6.0,0.0,36.1,82.3,6,6,276.3,276.3
|
| 208 |
+
06-08 15:00:47,1,1,0.0,277.0,6,0,0.0,277.0,6.0,0.0,36.5,82.3,6,6,277.0,277.0
|
| 209 |
+
06-08 15:00:52,1,1,0.0,404.3,9,0,0.0,404.3,9.0,0.0,52.1,86.6,9,9,404.3,404.3
|
| 210 |
+
06-08 15:00:57,1,1,0.0,404.4,9,0,0.0,404.4,9.0,0.0,52.7,86.6,9,9,404.4,404.4
|
| 211 |
+
06-08 15:01:02,1,1,0.0,406.0,9,0,0.0,406.0,9.0,0.0,53.2,86.6,9,9,406.0,406.0
|
| 212 |
+
06-08 15:01:07,1,1,0.0,404.9,9,0,0.0,404.9,9.0,0.0,53.8,86.6,9,9,404.9,404.9
|
| 213 |
+
06-08 15:01:12,1,1,0.0,404.8,9,0,0.0,404.8,9.0,0.0,54.4,86.6,9,9,404.8,404.8
|
| 214 |
+
06-08 15:01:17,1,1,0.0,406.0,9,0,0.0,406.0,9.0,0.0,54.9,86.6,9,9,406.0,406.0
|
| 215 |
+
06-08 15:01:22,1,1,2.4,426.6,10,0,2.4,426.6,10.0,0.0,55.6,86.7,10,10,426.6,426.6
|
| 216 |
+
06-08 15:01:27,1,1,0.0,313.9,7,0,0.0,313.9,7.0,0.0,39.1,86.1,7,7,313.9,313.9
|
| 217 |
+
06-08 15:01:32,1,1,1.8,339.2,8,0,1.8,339.2,8.0,0.0,39.7,86.2,8,8,339.2,339.2
|
| 218 |
+
06-08 15:01:37,1,1,2.8,376.7,9,0,2.8,376.7,9.0,0.0,40.4,86.3,9,9,376.7,376.7
|
| 219 |
+
06-08 15:01:42,1,1,0.0,403.4,9,0,0.0,403.4,9.0,0.0,41.0,86.3,9,9,403.4,403.4
|
| 220 |
+
06-08 15:01:47,1,1,0.0,372.6,8,0,0.0,372.6,8.0,0.0,41.1,86.3,8,8,372.6,372.6
|
| 221 |
+
06-08 15:01:52,1,1,0.0,359.4,8,0,0.0,359.4,8.0,0.0,41.6,86.3,8,8,359.4,359.4
|
| 222 |
+
06-08 15:01:57,1,1,0.0,357.7,8,0,0.0,357.7,8.0,0.0,42.1,86.3,8,8,357.7,357.7
|
| 223 |
+
06-08 15:02:02,1,1,2.8,373.0,9,0,2.8,373.0,9.0,0.0,42.7,86.4,9,9,373.0,373.0
|
| 224 |
+
06-08 15:02:07,1,1,0.0,402.3,9,0,0.0,402.3,9.0,0.0,43.2,86.4,9,9,402.3,402.3
|
| 225 |
+
06-08 15:02:12,1,1,0.0,402.2,9,0,0.0,402.2,9.0,0.0,43.8,86.4,9,9,402.2,402.2
|
| 226 |
+
06-08 15:02:17,1,1,0.0,401.7,9,0,0.0,401.7,9.0,0.0,44.4,86.4,9,9,401.7,401.7
|
| 227 |
+
06-08 15:02:22,1,1,0.0,351.5,8,0,0.0,351.5,8.0,0.0,31.8,82.7,8,8,351.5,351.5
|
| 228 |
+
06-08 15:02:27,1,1,0.0,350.0,8,0,0.0,350.0,8.0,0.0,32.3,82.7,8,8,350.0,350.0
|
| 229 |
+
06-08 15:02:32,1,1,0.0,349.8,8,0,0.0,349.8,8.0,0.0,32.8,82.7,8,8,349.8,349.8
|
| 230 |
+
06-08 15:02:37,1,1,23.2,390.1,9,0,23.2,390.1,9.0,0.0,34.9,83.9,9,9,390.1,390.1
|
| 231 |
+
06-08 15:02:42,1,1,0.0,391.7,9,0,0.0,391.7,9.0,0.0,35.5,83.9,9,9,391.7,391.7
|
| 232 |
+
06-08 15:02:48,1,1,0.0,355.4,8,0,0.0,355.4,8.0,0.0,34.2,83.9,8,8,355.4,355.4
|
| 233 |
+
06-08 15:02:53,1,1,101.8,265.0,6,0,101.8,265.0,6.0,0.0,35.0,87.2,6,6,265.0,265.0
|
| 234 |
+
06-08 15:02:58,1,1,0.0,275.7,6,0,0.0,275.7,6.0,0.0,35.4,87.2,6,6,275.7,275.7
|
| 235 |
+
06-08 15:03:03,1,1,91.9,296.2,7,0,91.9,296.2,7.0,0.0,38.4,87.6,7,7,296.2,296.2
|
| 236 |
+
06-08 15:03:08,1,1,97.9,324.4,8,0,97.9,324.4,8.0,0.0,39.4,87.5,8,8,324.4,324.4
|
| 237 |
+
06-08 15:03:13,1,1,0.0,368.8,8,0,0.0,368.8,8.0,0.0,40.0,87.5,8,8,368.8,368.8
|
| 238 |
+
06-08 15:03:18,1,1,119.4,387.8,8,0,119.4,387.8,8.0,0.0,38.3,87.4,8,8,387.8,387.8
|
| 239 |
+
06-08 15:03:23,1,1,0.0,336.8,7,0,0.0,336.8,7.0,0.0,29.8,87.4,7,7,336.8,336.8
|
| 240 |
+
06-08 15:03:28,1,1,0.0,317.7,5,0,0.0,317.7,5.0,0.0,26.6,87.4,5,5,317.7,317.7
|
| 241 |
+
06-08 15:03:33,1,1,0.0,231.2,5,0,0.0,231.2,5.0,0.0,26.9,87.4,5,5,231.2,231.2
|
| 242 |
+
06-08 15:03:38,1,1,0.0,307.5,7,0,0.0,307.5,7.0,0.0,29.5,89.4,7,7,307.5,307.5
|
| 243 |
+
06-08 15:03:43,1,1,0.0,537.3,12,0,0.0,537.3,12.0,0.0,63.2,78.4,12,12,537.3,537.3
|
| 244 |
+
06-08 15:03:48,1,1,0.0,534.3,12,0,0.0,534.3,12.0,0.0,64.0,78.4,12,12,534.3,534.3
|
| 245 |
+
06-08 15:03:53,1,1,0.0,507.8,11,0,0.0,507.8,11.0,0.0,62.8,78.4,11,11,507.8,507.8
|
| 246 |
+
06-08 15:03:58,1,1,0.0,474.1,10,0,0.0,474.1,10.0,0.0,54.5,78.4,10,10,474.1,474.1
|
| 247 |
+
06-08 15:04:03,1,1,0.0,447.2,10,0,0.0,447.2,10.0,0.0,55.2,78.4,10,10,447.2,447.2
|
| 248 |
+
06-08 15:04:08,1,1,0.0,447.5,10,0,0.0,447.5,10.0,0.0,55.8,78.4,10,10,447.5,447.5
|
| 249 |
+
06-08 15:04:13,1,1,0.0,263.5,6,0,0.0,263.5,6.0,0.0,23.5,89.4,6,6,263.5,263.5
|
| 250 |
+
06-08 15:04:18,1,1,0.0,263.8,6,0,0.0,263.8,6.0,0.0,23.9,89.4,6,6,263.8,263.8
|
| 251 |
+
06-08 15:04:23,1,1,0.0,263.3,6,0,0.0,263.3,6.0,0.0,24.2,89.4,6,6,263.3,263.3
|
| 252 |
+
06-08 15:04:28,1,1,0.0,244.7,5,0,0.0,244.7,5.0,0.0,21.6,89.4,5,5,244.7,244.7
|
| 253 |
+
06-08 15:04:33,1,1,0.0,221.3,5,0,0.0,221.3,5.0,0.0,21.9,89.4,5,5,221.3,221.3
|
| 254 |
+
06-08 15:04:38,1,1,0.0,220.5,5,0,0.0,220.5,5.0,0.0,22.2,89.4,5,5,220.5,220.5
|
| 255 |
+
06-08 15:04:43,1,1,0.0,491.2,11,0,0.0,491.2,11.0,0.0,61.3,78.8,11,11,491.2,491.2
|
| 256 |
+
06-08 15:04:48,1,1,0.0,488.0,11,0,0.0,488.0,11.0,0.0,61.9,78.8,11,11,488.0,488.0
|
| 257 |
+
06-08 15:04:53,1,1,0.0,485.0,11,0,0.0,485.0,11.0,0.0,62.6,78.8,11,11,485.0,485.0
|
| 258 |
+
06-08 15:04:58,1,1,0.0,492.2,11,0,0.0,492.2,11.0,0.0,63.3,78.8,11,11,492.2,492.2
|
| 259 |
+
06-08 15:05:03,1,1,0.0,483.6,11,0,0.0,483.6,11.0,0.0,64.0,78.8,11,11,483.6,483.6
|
| 260 |
+
06-08 15:05:08,1,1,0.0,275.3,6,0,0.0,275.3,6.0,0.0,30.1,88.3,6,6,275.3,275.3
|
| 261 |
+
06-08 15:05:13,1,1,0.0,245.1,6,0,0.0,245.1,6.0,0.0,30.5,88.3,6,6,245.1,245.1
|
| 262 |
+
06-08 15:05:18,1,1,0.0,244.9,6,0,0.0,244.9,6.0,0.0,30.8,88.3,6,6,244.9,244.9
|
| 263 |
+
06-08 15:05:23,1,1,0.0,244.7,6,0,0.0,244.7,6.0,0.0,31.1,88.3,6,6,244.7,244.7
|
| 264 |
+
06-08 15:05:28,1,1,0.0,244.7,6,0,0.0,244.7,6.0,0.0,31.5,88.3,6,6,244.7,244.7
|
| 265 |
+
06-08 15:05:33,1,1,0.0,244.5,6,0,0.0,244.5,6.0,0.0,31.8,88.3,6,6,244.5,244.5
|
| 266 |
+
06-08 15:05:38,1,1,0.0,245.3,6,0,0.0,245.3,6.0,0.0,32.2,88.3,6,6,245.3,245.3
|
| 267 |
+
06-08 15:05:43,1,1,0.0,245.1,6,0,0.0,245.1,6.0,0.0,32.5,88.3,6,6,245.1,245.1
|
| 268 |
+
06-08 15:05:48,1,1,0.0,244.9,6,0,0.0,244.9,6.0,0.0,32.9,88.3,6,6,244.9,244.9
|
| 269 |
+
06-08 15:05:54,1,1,0.0,232.6,5,0,0.0,232.6,5.0,0.0,28.7,87.6,5,5,232.6,232.6
|
| 270 |
+
06-08 15:05:59,1,1,0.0,233.5,5,0,0.0,233.5,5.0,0.0,29.1,87.6,5,5,233.5,233.5
|
| 271 |
+
06-08 15:06:04,1,1,0.0,233.3,5,0,0.0,233.3,5.0,0.0,29.4,87.6,5,5,233.3,233.3
|
| 272 |
+
06-08 15:06:09,1,1,0.0,233.0,5,0,0.0,233.0,5.0,0.0,29.7,87.6,5,5,233.0,233.0
|
| 273 |
+
06-08 15:06:14,1,1,0.0,233.1,5,0,0.0,233.1,5.0,0.0,30.1,87.6,5,5,233.1,233.1
|
| 274 |
+
06-08 15:06:19,1,1,0.0,232.9,5,0,0.0,232.9,5.0,0.0,30.4,87.6,5,5,232.9,232.9
|
| 275 |
+
06-08 15:06:24,1,1,0.0,233.0,5,0,0.0,233.0,5.0,0.0,30.7,87.6,5,5,233.0,233.0
|
| 276 |
+
06-08 15:06:29,1,1,0.0,232.6,5,0,0.0,232.6,5.0,0.0,31.0,87.6,5,5,232.6,232.6
|
| 277 |
+
06-08 15:06:34,1,1,0.0,232.3,5,0,0.0,232.3,5.0,0.0,31.3,87.6,5,5,232.3,232.3
|
| 278 |
+
06-08 15:06:39,1,1,0.0,232.4,5,0,0.0,232.4,5.0,0.0,31.7,87.6,5,5,232.4,232.4
|
| 279 |
+
06-08 15:06:44,1,1,0.0,232.8,5,0,0.0,232.8,5.0,0.0,32.0,87.6,5,5,232.8,232.8
|
| 280 |
+
06-08 15:06:49,1,1,0.0,231.9,5,0,0.0,231.9,5.0,0.0,32.3,87.6,5,5,231.9,231.9
|
| 281 |
+
06-08 15:06:54,1,1,0.0,233.2,5,0,0.0,233.2,5.0,0.0,32.6,87.6,5,5,233.2,233.2
|
| 282 |
+
06-08 15:06:59,1,1,0.0,232.0,5,0,0.0,232.0,5.0,0.0,33.0,87.6,5,5,232.0,232.0
|
| 283 |
+
06-08 15:07:04,1,1,0.0,232.5,5,0,0.0,232.5,5.0,0.0,33.3,87.6,5,5,232.5,232.5
|
| 284 |
+
06-08 15:07:09,1,1,0.0,233.2,5,0,0.0,233.2,5.0,0.0,33.6,87.6,5,5,233.2,233.2
|
| 285 |
+
06-08 15:07:14,1,1,0.0,232.9,5,0,0.0,232.9,5.0,0.0,34.0,87.6,5,5,232.9,232.9
|
| 286 |
+
06-08 15:07:19,1,1,0.0,232.9,5,0,0.0,232.9,5.0,0.0,34.3,87.6,5,5,232.9,232.9
|
| 287 |
+
06-08 15:07:24,1,1,0.0,189.8,4,0,0.0,189.8,4.0,0.0,25.6,87.6,4,4,189.8,189.8
|
| 288 |
+
06-08 15:07:29,1,1,139.4,408.5,9,0,139.4,408.5,9.0,0.0,33.5,77.2,9,9,408.5,408.5
|
| 289 |
+
06-08 15:07:34,1,1,0.0,404.5,9,0,0.0,404.5,9.0,0.0,34.1,77.2,9,9,404.5,404.5
|
| 290 |
+
06-08 15:07:39,1,1,0.0,404.6,9,0,0.0,404.6,9.0,0.0,34.6,77.2,9,9,404.6,404.6
|
| 291 |
+
06-08 15:07:44,1,1,0.0,314.6,7,0,0.0,314.6,7.0,0.0,33.0,82.2,7,7,314.6,314.6
|
| 292 |
+
06-08 15:07:49,1,1,0.0,315.4,7,0,0.0,315.4,7.0,0.0,33.4,82.2,7,7,315.4,315.4
|
| 293 |
+
06-08 15:07:54,1,1,0.0,314.7,7,0,0.0,314.7,7.0,0.0,33.9,82.2,7,7,314.7,314.7
|
| 294 |
+
06-08 15:07:59,1,1,0.0,314.7,7,0,0.0,314.7,7.0,0.0,34.3,82.2,7,7,314.7,314.7
|
| 295 |
+
06-08 15:08:04,1,1,0.0,314.6,7,0,0.0,314.6,7.0,0.0,34.7,82.2,7,7,314.6,314.6
|
| 296 |
+
06-08 15:08:09,1,1,0.0,315.2,7,0,0.0,315.2,7.0,0.0,35.2,82.2,7,7,315.2,315.2
|
| 297 |
+
06-08 15:08:14,1,1,0.0,316.1,7,0,0.0,316.1,7.0,0.0,35.6,82.2,7,7,316.1,316.1
|
| 298 |
+
06-08 15:08:19,1,1,0.0,315.8,7,0,0.0,315.8,7.0,0.0,36.1,82.2,7,7,315.8,315.8
|
| 299 |
+
06-08 15:08:24,1,1,0.0,315.9,7,0,0.0,315.9,7.0,0.0,36.5,82.2,7,7,315.9,315.9
|
| 300 |
+
06-08 15:08:29,1,1,0.0,147.0,3,0,0.0,147.0,3.0,0.0,13.2,88.5,3,3,147.0,147.0
|
| 301 |
+
06-08 15:08:34,1,1,0.0,138.2,3,0,0.0,138.2,3.0,0.0,13.4,88.5,3,3,138.2,138.2
|
| 302 |
+
06-08 15:08:39,1,1,0.0,138.3,3,0,0.0,138.3,3.0,0.0,13.6,88.5,3,3,138.3,138.3
|
| 303 |
+
06-08 15:08:44,1,1,0.0,137.7,3,0,0.0,137.7,3.0,0.0,13.8,88.5,3,3,137.7,137.7
|
| 304 |
+
06-08 15:08:49,1,1,0.0,137.7,3,0,0.0,137.7,3.0,0.0,14.0,88.5,3,3,137.7,137.7
|
| 305 |
+
06-08 15:08:54,1,1,0.0,137.9,3,0,0.0,137.9,3.0,0.0,14.2,88.5,3,3,137.9,137.9
|
| 306 |
+
06-08 15:08:59,1,1,0.0,137.8,3,0,0.0,137.8,3.0,0.0,14.4,88.5,3,3,137.8,137.8
|
| 307 |
+
06-08 15:09:04,1,1,0.0,137.7,3,0,0.0,137.7,3.0,0.0,14.6,88.5,3,3,137.7,137.7
|
| 308 |
+
06-08 15:09:09,1,1,154.9,156.5,4,0,154.9,156.5,4.0,0.0,16.0,88.3,4,4,156.5,156.5
|
| 309 |
+
06-08 15:09:14,1,1,81.7,193.0,5,0,81.7,193.0,5.0,0.0,17.0,88.3,5,5,193.0,193.0
|
| 310 |
+
06-08 15:09:19,1,1,0.0,220.7,4,0,0.0,220.7,4.0,0.0,16.0,88.3,4,4,220.7,220.7
|
| 311 |
+
06-08 15:09:24,1,1,0.0,161.8,3,0,0.0,161.8,3.0,0.0,15.3,88.3,3,3,161.8,161.8
|
| 312 |
+
06-08 15:09:29,1,1,0.0,138.1,3,0,0.0,138.1,3.0,0.0,15.5,88.3,3,3,138.1,138.1
|
| 313 |
+
06-08 15:09:34,1,1,0.0,137.6,3,0,0.0,137.6,3.0,0.0,15.7,88.3,3,3,137.6,137.6
|
| 314 |
+
06-08 15:09:39,1,1,0.0,375.7,8,0,0.0,375.7,8.0,0.0,45.1,84.1,8,8,375.7,375.7
|
| 315 |
+
06-08 15:09:44,1,1,0.0,357.3,7,0,0.0,357.3,7.0,0.0,41.0,84.1,7,7,357.3,357.3
|
| 316 |
+
06-08 15:09:49,1,1,0.0,329.6,7,0,0.0,329.6,7.0,0.0,41.5,84.1,7,7,329.6,329.6
|
| 317 |
+
06-08 15:09:54,1,1,0.0,329.9,7,0,0.0,329.9,7.0,0.0,42.0,84.1,7,7,329.9,329.9
|
| 318 |
+
06-08 15:09:59,1,1,0.0,330.3,7,0,0.0,330.3,7.0,0.0,42.4,84.1,7,7,330.3,330.3
|
| 319 |
+
06-08 15:10:05,1,1,0.0,300.8,7,0,0.0,300.8,7.0,0.0,32.6,88.6,7,7,300.8,300.8
|
| 320 |
+
06-08 15:10:10,1,1,0.0,300.9,7,0,0.0,300.9,7.0,0.0,33.0,88.6,7,7,300.9,300.9
|
| 321 |
+
06-08 15:10:15,1,1,0.0,282.4,6,0,0.0,282.4,6.0,0.0,34.7,84.1,6,6,282.4,282.4
|
| 322 |
+
06-08 15:10:20,1,1,0.0,271.2,6,0,0.0,271.2,6.0,0.0,35.0,86.7,6,6,271.2,271.2
|
| 323 |
+
06-08 15:10:25,1,1,0.0,271.2,6,0,0.0,271.2,6.0,0.0,35.3,86.7,6,6,271.2,271.2
|
| 324 |
+
06-08 15:10:30,1,1,0.0,271.2,6,0,0.0,271.2,6.0,0.0,35.7,86.7,6,6,271.2,271.2
|
| 325 |
+
06-08 15:10:35,1,1,0.0,271.2,6,0,0.0,271.2,6.0,0.0,36.1,86.7,6,6,271.2,271.2
|
| 326 |
+
06-08 15:10:40,1,1,0.0,271.2,6,0,0.0,271.2,6.0,0.0,36.5,86.7,6,6,271.2,271.2
|
| 327 |
+
06-08 15:10:45,1,1,0.0,271.3,6,0,0.0,271.3,6.0,0.0,36.8,86.7,6,6,271.3,271.3
|
| 328 |
+
06-08 15:10:50,1,1,0.0,271.1,6,0,0.0,271.1,6.0,0.0,37.2,86.7,6,6,271.1,271.1
|
| 329 |
+
06-08 15:10:55,1,1,0.0,270.9,6,0,0.0,270.9,6.0,0.0,37.6,86.7,6,6,270.9,270.9
|
| 330 |
+
06-08 15:11:00,1,1,0.0,270.5,6,0,0.0,270.5,6.0,0.0,38.0,86.7,6,6,270.5,270.5
|
| 331 |
+
06-08 15:11:05,1,1,0.0,272.0,6,0,0.0,272.0,6.0,0.0,38.4,86.7,6,6,272.0,272.0
|
| 332 |
+
06-08 15:11:10,1,1,0.0,271.2,6,0,0.0,271.2,6.0,0.0,38.7,86.7,6,6,271.2,271.2
|
| 333 |
+
06-08 15:11:15,1,1,0.0,270.9,6,0,0.0,270.9,6.0,0.0,39.1,86.7,6,6,270.9,270.9
|
| 334 |
+
06-08 15:11:20,1,1,0.0,181.4,4,0,0.0,181.4,4.0,0.0,21.2,84.7,4,4,181.4,181.4
|
| 335 |
+
06-08 15:11:25,1,1,0.0,271.4,6,0,0.0,271.4,6.0,0.0,39.9,86.7,6,6,271.4,271.4
|
| 336 |
+
06-08 15:11:30,1,1,0.0,93.5,2,0,0.0,93.5,2.0,0.0,4.8,86.8,2,2,93.5,93.5
|
| 337 |
+
06-08 15:11:35,1,1,0.0,93.6,2,0,0.0,93.6,2.0,0.0,4.9,86.8,2,2,93.6,93.6
|
| 338 |
+
06-08 15:11:40,1,1,0.0,93.7,2,0,0.0,93.7,2.0,0.0,5.0,86.8,2,2,93.7,93.7
|
| 339 |
+
06-08 15:11:45,1,1,0.0,266.7,6,0,0.0,266.7,6.0,0.0,12.3,85.3,6,6,266.7,266.7
|
| 340 |
+
06-08 15:11:50,1,1,0.0,253.0,6,0,0.0,253.0,6.0,0.0,29.1,63.8,6,6,253.0,253.0
|
| 341 |
+
06-08 15:11:55,1,1,0.0,253.4,6,0,0.0,253.4,6.0,0.0,29.5,63.8,6,6,253.4,253.4
|
| 342 |
+
06-08 15:12:00,1,1,0.0,253.5,6,0,0.0,253.5,6.0,0.0,29.8,63.8,6,6,253.5,253.5
|
| 343 |
+
06-08 15:12:06,1,1,0.0,253.3,6,0,0.0,253.3,6.0,0.0,30.2,63.8,6,6,253.3,253.3
|
| 344 |
+
06-08 15:12:11,1,1,0.0,253.1,6,0,0.0,253.1,6.0,0.0,30.5,63.8,6,6,253.1,253.1
|
| 345 |
+
06-08 15:12:16,1,1,0.0,253.7,6,0,0.0,253.7,6.0,0.0,30.9,63.8,6,6,253.7,253.7
|
| 346 |
+
06-08 15:12:21,1,1,0.0,252.9,6,0,0.0,252.9,6.0,0.0,31.3,63.8,6,6,252.9,252.9
|
| 347 |
+
06-08 15:12:26,1,1,0.0,252.6,6,0,0.0,252.6,6.0,0.0,31.6,63.8,6,6,252.6,252.6
|
| 348 |
+
06-08 15:12:31,1,1,0.0,253.2,6,0,0.0,253.2,6.0,0.0,32.0,63.8,6,6,253.2,253.2
|
| 349 |
+
06-08 15:12:36,1,1,0.0,252.3,6,0,0.0,252.3,6.0,0.0,32.3,63.8,6,6,252.3,252.3
|
| 350 |
+
06-08 15:12:41,1,1,0.0,252.4,6,0,0.0,252.4,6.0,0.0,32.7,63.8,6,6,252.4,252.4
|
| 351 |
+
06-08 15:12:46,1,1,0.0,252.8,6,0,0.0,252.8,6.0,0.0,33.0,63.8,6,6,252.8,252.8
|
| 352 |
+
06-08 15:12:51,1,1,0.0,177.9,4,0,0.0,177.9,4.0,0.0,14.9,85.3,4,4,177.9,177.9
|
| 353 |
+
06-08 15:12:56,1,1,0.0,177.9,4,0,0.0,177.9,4.0,0.0,15.2,85.3,4,4,177.9,177.9
|
| 354 |
+
06-08 15:13:01,1,1,0.0,177.9,4,0,0.0,177.9,4.0,0.0,15.4,85.3,4,4,177.9,177.9
|
| 355 |
+
06-08 15:13:06,1,1,0.0,178.0,4,0,0.0,178.0,4.0,0.0,15.7,85.3,4,4,178.0,178.0
|
| 356 |
+
06-08 15:13:11,1,1,0.0,258.8,6,0,0.0,258.8,6.0,0.0,34.8,63.8,6,6,258.8,258.8
|
| 357 |
+
06-08 15:13:16,1,1,311.4,212.1,7,0,311.4,212.1,7.0,0.0,17.9,85.1,7,7,212.1,212.1
|
| 358 |
+
06-08 15:13:21,1,1,0.0,242.1,5,0,0.0,242.1,5.0,0.0,16.3,87.9,5,5,242.1,242.1
|
| 359 |
+
06-08 15:13:26,1,1,73.7,286.9,6,0,73.7,286.9,6.0,0.0,16.8,87.8,6,6,286.9,286.9
|
| 360 |
+
06-08 15:13:31,1,1,0.0,288.5,6,0,0.0,288.5,6.0,0.0,17.2,87.8,6,6,288.5,288.5
|
| 361 |
+
06-08 15:13:36,1,1,0.0,288.8,6,0,0.0,288.8,6.0,0.0,17.6,87.8,6,6,288.8,288.8
|
| 362 |
+
06-08 15:13:41,1,1,0.0,289.1,6,0,0.0,289.1,6.0,0.0,18.0,87.8,6,6,289.1,289.1
|
| 363 |
+
06-08 15:13:46,1,1,0.0,289.3,6,0,0.0,289.3,6.0,0.0,18.4,87.8,6,6,289.3,289.3
|
| 364 |
+
06-08 15:13:51,1,1,0.0,266.8,5,0,0.0,266.8,5.0,0.0,18.3,87.8,5,5,266.8,266.8
|
| 365 |
+
06-08 15:13:56,1,1,0.0,241.8,5,0,0.0,241.8,5.0,0.0,18.6,87.8,5,5,241.8,241.8
|
| 366 |
+
06-08 15:14:01,1,1,0.0,241.3,5,0,0.0,241.3,5.0,0.0,18.9,87.8,5,5,241.3,241.3
|
| 367 |
+
06-08 15:14:06,1,1,0.0,242.4,5,0,0.0,242.4,5.0,0.0,19.3,87.8,5,5,242.4,242.4
|
| 368 |
+
06-08 15:14:11,1,1,0.0,92.1,2,0,0.0,92.1,2.0,0.0,7.1,85.8,2,2,92.1,92.1
|
| 369 |
+
06-08 15:14:16,1,1,0.0,91.7,2,0,0.0,91.7,2.0,0.0,7.3,85.8,2,2,91.7,91.7
|
| 370 |
+
06-08 15:14:21,1,1,0.0,91.7,2,0,0.0,91.7,2.0,0.0,7.4,85.8,2,2,91.7,91.7
|
| 371 |
+
06-08 15:14:26,1,1,0.0,222.8,5,0,0.0,222.8,5.0,0.0,19.4,89.7,5,5,222.8,222.8
|
| 372 |
+
06-08 15:14:31,1,1,0.0,222.6,5,0,0.0,222.6,5.0,0.0,19.7,89.7,5,5,222.6,222.6
|
| 373 |
+
06-08 15:14:36,1,1,0.0,190.7,4,0,0.0,190.7,4.0,0.0,11.1,89.7,4,4,190.7,190.7
|
| 374 |
+
06-08 15:14:41,1,1,0.0,178.2,4,0,0.0,178.2,4.0,0.0,11.3,89.7,4,4,178.2,178.2
|
| 375 |
+
06-08 15:14:46,1,1,0.0,178.3,4,0,0.0,178.3,4.0,0.0,11.6,89.7,4,4,178.3,178.3
|
| 376 |
+
06-08 15:14:51,1,1,64.2,103.6,3,0,64.2,103.6,3.0,0.0,8.9,86.0,3,3,103.6,103.6
|
| 377 |
+
06-08 15:14:56,1,1,8.8,140.1,4,0,8.8,140.1,4.0,0.0,9.2,86.0,4,4,140.1,140.1
|
| 378 |
+
06-08 15:15:01,1,1,217.0,263.4,5,0,217.0,263.4,5.0,0.0,9.6,85.8,5,5,263.4,263.4
|
| 379 |
+
06-08 15:15:06,1,1,0.0,230.6,5,0,0.0,230.6,5.0,0.0,10.0,85.8,5,5,230.6,230.6
|
| 380 |
+
06-08 15:15:11,1,1,0.0,230.6,5,0,0.0,230.6,5.0,0.0,10.3,85.8,5,5,230.6,230.6
|
| 381 |
+
06-08 15:15:16,1,1,0.0,230.6,5,0,0.0,230.6,5.0,0.0,10.6,85.8,5,5,230.6,230.6
|
| 382 |
+
06-08 15:15:21,1,1,0.0,230.1,5,0,0.0,230.1,5.0,0.0,10.9,85.8,5,5,230.1,230.1
|
| 383 |
+
06-08 15:15:26,1,1,0.0,196.6,4,0,0.0,196.6,4.0,0.0,10.1,85.8,4,4,196.6,196.6
|
| 384 |
+
06-08 15:15:31,1,1,0.0,183.6,4,0,0.0,183.6,4.0,0.0,10.3,85.8,4,4,183.6,183.6
|
| 385 |
+
06-08 15:15:36,1,1,0.0,183.8,4,0,0.0,183.8,4.0,0.0,10.6,85.8,4,4,183.8,183.8
|
| 386 |
+
06-08 15:15:41,1,1,0.0,183.5,4,0,0.0,183.5,4.0,0.0,10.9,85.8,4,4,183.5,183.5
|
| 387 |
+
06-08 15:15:46,1,1,0.0,183.1,4,0,0.0,183.1,4.0,0.0,11.1,85.8,4,4,183.1,183.1
|
| 388 |
+
06-08 15:15:51,1,1,0.0,133.0,3,0,0.0,133.0,3.0,0.0,7.5,86.3,3,3,133.0,133.0
|
| 389 |
+
06-08 15:15:56,1,1,0.0,133.3,3,0,0.0,133.3,3.0,0.0,7.7,86.3,3,3,133.3,133.3
|
| 390 |
+
06-08 15:16:01,1,1,0.0,133.0,3,0,0.0,133.0,3.0,0.0,7.9,86.3,3,3,133.0,133.0
|
| 391 |
+
06-08 15:16:07,1,1,8.8,144.1,4,0,8.8,144.1,4.0,0.0,8.3,86.4,4,4,144.1,144.1
|
| 392 |
+
06-08 15:16:12,1,1,0.0,176.9,4,0,0.0,176.9,4.0,0.0,8.5,86.4,4,4,176.9,176.9
|
| 393 |
+
06-08 15:16:17,1,1,0.0,167.0,3,0,0.0,167.0,3.0,0.0,8.2,86.4,3,3,167.0,167.0
|
| 394 |
+
06-08 15:16:22,1,1,0.0,133.3,3,0,0.0,133.3,3.0,0.0,8.4,86.4,3,3,133.3,133.3
|
| 395 |
+
06-08 15:16:27,1,1,0.0,133.3,3,0,0.0,133.3,3.0,0.0,8.6,86.4,3,3,133.3,133.3
|
| 396 |
+
06-08 15:16:32,1,1,0.0,133.1,3,0,0.0,133.1,3.0,0.0,8.8,86.4,3,3,133.1,133.1
|
| 397 |
+
06-08 15:16:37,1,1,80.9,170.8,4,0,80.9,170.8,4.0,0.0,9.1,86.2,4,4,170.8,170.8
|
| 398 |
+
06-08 15:16:42,1,1,0.0,177.4,4,0,0.0,177.4,4.0,0.0,9.4,86.2,4,4,177.4,177.4
|
| 399 |
+
06-08 15:16:47,1,1,11.6,210.0,5,0,11.6,210.0,5.0,0.0,9.8,86.3,5,5,210.0,210.0
|
| 400 |
+
06-08 15:16:52,1,1,0.0,332.7,6,0,0.0,332.7,6.0,0.0,7.3,86.9,6,6,332.7,332.7
|
| 401 |
+
06-08 15:16:57,1,1,0.0,232.8,5,0,0.0,232.8,5.0,0.0,10.0,87.2,5,5,232.8,232.8
|
| 402 |
+
06-08 15:17:02,1,1,67.0,235.8,6,0,67.0,235.8,6.0,0.0,10.6,87.2,6,6,235.8,235.8
|
| 403 |
+
06-08 15:17:07,1,1,0.0,279.6,6,0,0.0,279.6,6.0,0.0,11.0,87.2,6,6,279.6,279.6
|
| 404 |
+
06-08 15:17:12,1,1,0.0,266.7,5,0,0.0,266.7,5.0,0.0,10.9,87.2,5,5,266.7,266.7
|
| 405 |
+
06-08 15:17:17,1,1,0.0,233.0,5,0,0.0,233.0,5.0,0.0,11.3,87.2,5,5,233.0,233.0
|
| 406 |
+
06-08 15:17:22,1,1,258.2,473.0,13,0,258.2,473.0,13.0,0.0,35.9,88.8,13,13,473.0,473.0
|
| 407 |
+
06-08 15:17:27,1,1,0.0,506.4,12,0,0.0,506.4,12.0,0.0,36.3,88.8,12,12,506.4,506.4
|
| 408 |
+
06-08 15:17:32,1,1,123.0,542.0,14,0,123.0,542.0,14.0,0.0,38.4,88.8,14,14,542.0,542.0
|
| 409 |
+
06-08 15:17:37,1,1,9.2,618.7,15,0,9.2,618.7,15.0,0.0,39.4,88.8,15,15,618.7,618.7
|
| 410 |
+
06-08 15:17:42,1,1,0.0,585.4,13,0,0.0,585.4,13.0,0.0,30.1,88.8,13,13,585.4,585.4
|
| 411 |
+
06-08 15:17:47,1,1,0.0,547.9,13,0,0.0,547.9,13.0,0.0,30.9,88.8,13,13,547.9,547.9
|
| 412 |
+
06-08 15:17:52,1,1,0.0,547.7,13,0,0.0,547.7,13.0,0.0,31.6,88.8,13,13,547.7,547.7
|
| 413 |
+
06-08 15:17:57,1,1,117.5,562.2,15,0,117.5,562.2,15.0,0.0,32.9,88.8,15,15,562.2,562.2
|
| 414 |
+
06-08 15:18:02,1,1,0.0,627.2,14,0,0.0,627.2,14.0,0.0,33.3,88.8,14,14,627.2,627.2
|
| 415 |
+
06-08 15:18:07,1,1,0.0,559.9,13,0,0.0,559.9,13.0,0.0,33.1,88.8,13,13,559.9,559.9
|
| 416 |
+
06-08 15:18:12,1,1,0.0,371.7,7,0,0.0,371.7,7.0,0.0,32.2,85.7,7,7,371.7,371.7
|
| 417 |
+
06-08 15:18:17,1,1,9.2,351.0,8,0,9.2,351.0,8.0,0.0,32.8,85.7,8,8,351.0,351.0
|
| 418 |
+
06-08 15:18:22,1,1,0.0,381.1,8,0,0.0,381.1,8.0,0.0,33.3,85.7,8,8,381.1,381.1
|
| 419 |
+
06-08 15:18:27,1,1,0.0,381.9,8,0,0.0,381.9,8.0,0.0,33.9,85.7,8,8,381.9,381.9
|
| 420 |
+
06-08 15:18:32,1,1,0.0,381.1,8,0,0.0,381.1,8.0,0.0,34.4,85.7,8,8,381.1,381.1
|
| 421 |
+
06-08 15:18:37,1,1,0.0,381.6,8,0,0.0,381.6,8.0,0.0,34.9,85.7,8,8,381.6,381.6
|
| 422 |
+
06-08 15:18:42,1,1,0.0,381.6,8,0,0.0,381.6,8.0,0.0,35.5,85.7,8,8,381.6,381.6
|
| 423 |
+
06-08 15:18:47,1,1,0.0,343.4,6,0,0.0,343.4,6.0,0.0,26.2,85.7,6,6,343.4,343.4
|
| 424 |
+
06-08 15:18:52,1,1,0.0,287.4,5,0,0.0,287.4,5.0,0.0,26.0,85.7,5,5,287.4,287.4
|
| 425 |
+
06-08 15:18:57,1,1,0.0,240.1,5,0,0.0,240.1,5.0,0.0,26.4,85.7,5,5,240.1,240.1
|
| 426 |
+
06-08 15:19:02,1,1,0.0,258.0,6,0,0.0,258.0,6.0,0.0,20.8,88.2,6,6,258.0,258.0
|
| 427 |
+
06-08 15:19:07,1,1,0.0,258.1,6,0,0.0,258.1,6.0,0.0,21.2,88.2,6,6,258.1,258.1
|
| 428 |
+
06-08 15:19:12,1,1,22.3,259.1,7,0,22.3,259.1,7.0,0.0,22.1,88.3,7,7,259.1,259.1
|
| 429 |
+
06-08 15:19:17,1,1,0.0,300.6,7,0,0.0,300.6,7.0,0.0,22.5,88.3,7,7,300.6,300.6
|
| 430 |
+
06-08 15:19:23,1,1,0.0,268.6,6,0,0.0,268.6,6.0,0.0,22.3,88.3,6,6,268.6,268.6
|
| 431 |
+
06-08 15:19:28,1,1,0.0,257.8,6,0,0.0,257.8,6.0,0.0,22.6,88.3,6,6,257.8,257.8
|
| 432 |
+
06-08 15:19:33,1,1,0.0,258.3,6,0,0.0,258.3,6.0,0.0,23.0,88.3,6,6,258.3,258.3
|
| 433 |
+
06-08 15:19:38,1,1,0.0,257.3,6,0,0.0,257.3,6.0,0.0,23.4,88.3,6,6,257.3,257.3
|
| 434 |
+
06-08 15:19:43,1,1,0.0,257.8,6,0,0.0,257.8,6.0,0.0,23.7,88.3,6,6,257.8,257.8
|
| 435 |
+
06-08 15:19:48,1,1,0.0,257.8,6,0,0.0,257.8,6.0,0.0,24.1,88.3,6,6,257.8,257.8
|
| 436 |
+
06-08 15:19:53,1,1,0.0,257.8,6,0,0.0,257.8,6.0,0.0,24.4,88.3,6,6,257.8,257.8
|
| 437 |
+
06-08 15:19:58,1,1,0.0,318.1,7,0,0.0,318.1,7.0,0.0,23.7,89.9,7,7,318.1,318.1
|
| 438 |
+
06-08 15:20:03,1,1,0.0,319.8,7,0,0.0,319.8,7.0,0.0,24.2,89.9,7,7,319.8,319.8
|
| 439 |
+
06-08 15:20:09,1,1,0.0,411.6,9,0,0.0,411.6,9.0,0.0,20.2,86.2,9,9,411.6,411.6
|
| 440 |
+
06-08 15:20:15,1,1,0.0,325.2,7,0,0.0,325.2,7.0,0.0,21.6,88.6,7,7,325.2,325.2
|
| 441 |
+
06-08 15:20:20,1,1,0.0,322.4,7,0,0.0,322.4,7.0,0.0,22.1,88.6,7,7,322.4,322.4
|
| 442 |
+
06-08 15:20:25,1,1,0.0,322.3,7,0,0.0,322.3,7.0,0.0,22.5,88.6,7,7,322.3,322.3
|
| 443 |
+
06-08 15:20:30,1,1,0.0,321.3,7,0,0.0,321.3,7.0,0.0,23.0,88.6,7,7,321.3,321.3
|
| 444 |
+
06-08 15:20:35,1,1,0.0,323.8,7,0,0.0,323.8,7.0,0.0,23.4,88.6,7,7,323.8,323.8
|
| 445 |
+
06-08 15:20:40,1,1,0.0,323.6,7,0,0.0,323.6,7.0,0.0,23.9,88.6,7,7,323.6,323.6
|
| 446 |
+
06-08 15:20:45,1,1,0.0,324.4,7,0,0.0,324.4,7.0,0.0,24.3,88.6,7,7,324.4,324.4
|
| 447 |
+
06-08 15:20:51,1,1,0.0,407.1,10,0,0.0,407.1,10.0,0.0,36.7,87.9,10,10,407.1,407.1
|
| 448 |
+
06-08 15:20:56,1,1,0.0,408.1,10,0,0.0,408.1,10.0,0.0,37.3,87.9,10,10,408.1,408.1
|
| 449 |
+
06-08 15:21:01,1,1,0.0,407.9,10,0,0.0,407.9,10.0,0.0,37.8,87.9,10,10,407.9,407.9
|
| 450 |
+
06-08 15:21:06,1,1,0.0,407.4,10,0,0.0,407.4,10.0,0.0,38.4,87.9,10,10,407.4,407.4
|
| 451 |
+
06-08 15:21:11,1,1,0.0,408.5,10,0,0.0,408.5,10.0,0.0,39.0,87.9,10,10,408.5,408.5
|
| 452 |
+
06-08 15:21:16,1,1,0.0,408.1,10,0,0.0,408.1,10.0,0.0,39.5,87.9,10,10,408.1,408.1
|
| 453 |
+
06-08 15:21:21,1,1,0.0,406.6,10,0,0.0,406.6,10.0,0.0,40.1,87.9,10,10,406.6,406.6
|
| 454 |
+
06-08 15:21:26,1,1,0.0,639.5,14,0,0.0,639.5,14.0,0.0,72.8,80.6,14,14,639.5,639.5
|
| 455 |
+
06-08 15:21:31,1,1,0.0,624.5,14,0,0.0,624.5,14.0,0.0,73.7,80.6,14,14,624.5,624.5
|
| 456 |
+
06-08 15:21:36,1,1,3.0,641.7,15,0,3.0,641.7,15.0,0.0,74.6,80.6,15,15,641.7,641.7
|
| 457 |
+
06-08 15:21:41,1,1,0.0,650.8,14,0,0.0,650.8,14.0,0.0,66.6,80.6,14,14,650.8,650.8
|
| 458 |
+
06-08 15:21:46,1,1,0.0,624.6,14,0,0.0,624.6,14.0,0.0,67.5,80.6,14,14,624.6,624.6
|
| 459 |
+
06-08 15:21:51,1,1,0.0,626.3,14,0,0.0,626.3,14.0,0.0,68.4,80.6,14,14,626.3,626.3
|
| 460 |
+
06-08 15:21:56,1,1,61.9,358.5,9,0,61.9,358.5,9.0,0.0,30.0,83.1,9,9,358.5,358.5
|
| 461 |
+
06-08 15:22:01,1,1,0.0,447.4,11,0,0.0,447.4,11.0,0.0,44.9,88.0,11,11,447.4,447.4
|
| 462 |
+
06-08 15:22:06,1,1,0.0,287.8,6,0,0.0,287.8,6.0,0.0,21.7,86.7,6,6,287.8,287.8
|
| 463 |
+
06-08 15:22:11,1,1,0.0,261.5,6,0,0.0,261.5,6.0,0.0,22.1,86.7,6,6,261.5,261.5
|
| 464 |
+
06-08 15:22:16,1,1,0.0,261.3,6,0,0.0,261.3,6.0,0.0,22.5,86.7,6,6,261.3,261.3
|
| 465 |
+
06-08 15:22:21,1,1,0.0,261.5,6,0,0.0,261.5,6.0,0.0,22.8,86.7,6,6,261.5,261.5
|
| 466 |
+
06-08 15:22:26,1,1,0.0,261.3,6,0,0.0,261.3,6.0,0.0,23.2,86.7,6,6,261.3,261.3
|
| 467 |
+
06-08 15:22:31,1,1,0.0,380.8,8,0,0.0,380.8,8.0,0.0,33.2,88.6,8,8,380.8,380.8
|
| 468 |
+
06-08 15:22:36,1,1,0.0,367.6,8,0,0.0,367.6,8.0,0.0,33.7,88.6,8,8,367.6,367.6
|
| 469 |
+
06-08 15:22:41,1,1,0.0,366.2,8,0,0.0,366.2,8.0,0.0,34.2,88.6,8,8,366.2,366.2
|
| 470 |
+
06-08 15:22:46,1,1,67.8,286.8,7,0,67.8,286.8,7.0,0.0,25.5,86.8,7,7,286.8,286.8
|
| 471 |
+
06-08 15:22:51,1,1,368.4,348.0,9,0,368.4,348.0,9.0,0.0,29.3,86.8,9,9,348.0,348.0
|
| 472 |
+
06-08 15:22:56,1,1,0.0,346.0,8,0,0.0,346.0,8.0,0.0,28.9,86.8,8,8,346.0,346.0
|
| 473 |
+
06-08 15:23:01,1,1,0.0,366.4,8,0,0.0,366.4,8.0,0.0,36.2,88.7,8,8,366.4,366.4
|
| 474 |
+
06-08 15:23:06,1,1,0.0,365.5,8,0,0.0,365.5,8.0,0.0,36.8,88.7,8,8,365.5,365.5
|
| 475 |
+
06-08 15:23:11,1,1,0.0,401.1,9,0,0.0,401.1,9.0,0.0,38.4,83.7,9,9,401.1,401.1
|
| 476 |
+
06-08 15:23:17,1,1,0.0,317.3,7,0,0.0,317.3,7.0,0.0,30.5,86.7,7,7,317.3,317.3
|
| 477 |
+
06-08 15:23:22,1,1,0.0,317.0,7,0,0.0,317.0,7.0,0.0,31.0,86.7,7,7,317.0,317.0
|
| 478 |
+
06-08 15:23:27,1,1,0.0,317.4,7,0,0.0,317.4,7.0,0.0,31.4,86.7,7,7,317.4,317.4
|
| 479 |
+
06-08 15:23:32,1,1,0.0,317.4,7,0,0.0,317.4,7.0,0.0,31.9,86.7,7,7,317.4,317.4
|
| 480 |
+
06-08 15:23:37,1,1,0.0,317.3,7,0,0.0,317.3,7.0,0.0,32.3,86.7,7,7,317.3,317.3
|
| 481 |
+
06-08 15:23:42,1,1,0.0,316.9,7,0,0.0,316.9,7.0,0.0,32.7,86.7,7,7,316.9,316.9
|
| 482 |
+
06-08 15:23:47,1,1,0.0,317.6,7,0,0.0,317.6,7.0,0.0,33.2,86.7,7,7,317.6,317.6
|
| 483 |
+
06-08 15:23:52,1,1,0.0,317.7,7,0,0.0,317.7,7.0,0.0,33.6,86.7,7,7,317.7,317.7
|
| 484 |
+
06-08 15:23:57,1,1,108.6,334.8,8,0,108.6,334.8,8.0,0.0,35.7,86.9,8,8,334.8,334.8
|
| 485 |
+
06-08 15:24:02,1,1,0.0,712.3,16,0,0.0,712.3,16.0,0.0,68.6,86.9,16,16,712.3,712.3
|
| 486 |
+
06-08 15:24:07,1,1,0.0,715.0,16,0,0.0,715.0,16.0,0.0,69.6,86.9,16,16,715.0,715.0
|
| 487 |
+
06-08 15:24:12,1,1,0.0,141.1,3,0,0.0,141.1,3.0,0.0,17.6,86.9,3,3,141.1,141.1
|
| 488 |
+
06-08 15:24:17,1,1,0.0,141.0,3,0,0.0,141.0,3.0,0.0,17.8,86.9,3,3,141.0,141.0
|
| 489 |
+
06-08 15:24:22,1,1,0.0,141.1,3,0,0.0,141.1,3.0,0.0,18.0,86.9,3,3,141.1,141.1
|
| 490 |
+
06-08 15:24:27,1,1,0.0,140.9,3,0,0.0,140.9,3.0,0.0,18.2,86.9,3,3,140.9,140.9
|
| 491 |
+
06-08 15:24:32,1,1,0.0,141.3,3,0,0.0,141.3,3.0,0.0,18.4,86.9,3,3,141.3,141.3
|
| 492 |
+
06-08 15:24:37,1,1,0.2,184.5,4,0,0.2,184.5,4.0,0.0,18.7,87.0,4,4,184.5,184.5
|
| 493 |
+
06-08 15:24:42,1,1,0.0,187.1,4,0,0.0,187.1,4.0,0.0,18.9,87.0,4,4,187.1,187.1
|
| 494 |
+
06-08 15:24:47,1,1,0.0,345.7,8,0,0.0,345.7,8.0,0.0,35.6,87.6,8,8,345.7,345.7
|
| 495 |
+
06-08 15:24:52,1,1,0.0,345.3,8,0,0.0,345.3,8.0,0.0,36.1,87.6,8,8,345.3,345.3
|
| 496 |
+
06-08 15:24:57,1,1,9.8,352.8,9,0,9.8,352.8,9.0,0.0,38.4,87.9,9,9,352.8,352.8
|
| 497 |
+
06-08 15:25:02,1,1,0.0,430.1,9,0,0.0,430.1,9.0,0.0,41.1,87.9,9,9,430.1,430.1
|
| 498 |
+
06-08 15:25:07,1,1,106.4,436.2,9,0,106.4,436.2,9.0,0.0,33.0,87.8,9,9,436.2,436.2
|
| 499 |
+
06-08 15:25:12,1,1,0.0,433.1,10,0,0.0,433.1,10.0,0.0,40.2,88.0,10,10,433.1,433.1
|
| 500 |
+
06-08 15:25:17,1,1,0.0,385.5,8,0,0.0,385.5,8.0,0.0,32.8,87.8,8,8,385.5,385.5
|
| 501 |
+
06-08 15:25:22,1,1,0.0,384.8,8,0,0.0,384.8,8.0,0.0,33.4,87.8,8,8,384.8,384.8
|
| 502 |
+
06-08 15:25:27,1,1,0.0,385.0,8,0,0.0,385.0,8.0,0.0,33.9,87.8,8,8,385.0,385.0
|
| 503 |
+
06-08 15:25:32,1,1,0.0,384.7,8,0,0.0,384.7,8.0,0.0,34.4,87.8,8,8,384.7,384.7
|
| 504 |
+
06-08 15:25:37,1,1,0.0,384.6,8,0,0.0,384.6,8.0,0.0,35.0,87.8,8,8,384.6,384.6
|
| 505 |
+
06-08 15:25:42,1,1,52.5,423.5,9,0,52.5,423.5,9.0,0.0,36.3,87.8,9,9,423.5,423.5
|
| 506 |
+
06-08 15:25:47,1,1,87.5,447.0,9,0,87.5,447.0,9.0,0.0,36.4,87.8,9,9,447.0,447.0
|
| 507 |
+
06-08 15:25:52,1,1,0.0,432.2,9,0,0.0,432.2,9.0,0.0,37.0,87.8,9,9,432.2,432.2
|
| 508 |
+
06-08 15:25:57,1,1,0.0,416.7,7,0,0.0,416.7,7.0,0.0,28.2,87.8,7,7,416.7,416.7
|
| 509 |
+
06-08 15:26:02,1,1,132.3,374.2,8,0,132.3,374.2,8.0,0.0,31.8,88.0,8,8,374.2,374.2
|
| 510 |
+
06-08 15:26:07,1,1,0.0,385.1,8,0,0.0,385.1,8.0,0.0,32.3,88.0,8,8,385.1,385.1
|
| 511 |
+
06-08 15:26:12,1,1,76.1,415.4,9,0,76.1,415.4,9.0,0.0,34.2,88.1,9,9,415.4,415.4
|
| 512 |
+
06-08 15:26:17,1,1,0.0,410.6,8,0,0.0,410.6,8.0,0.0,31.4,88.1,8,8,410.6,410.6
|
| 513 |
+
06-08 15:26:22,1,1,0.0,384.3,8,0,0.0,384.3,8.0,0.0,32.0,88.1,8,8,384.3,384.3
|
| 514 |
+
06-08 15:26:27,1,1,0.0,385.1,8,0,0.0,385.1,8.0,0.0,32.5,88.1,8,8,385.1,385.1
|
| 515 |
+
06-08 15:26:33,1,1,0.0,184.1,4,0,0.0,184.1,4.0,0.0,24.5,87.7,4,4,184.1,184.1
|
| 516 |
+
06-08 15:26:38,1,1,0.0,183.5,4,0,0.0,183.5,4.0,0.0,24.8,87.7,4,4,183.5,183.5
|
| 517 |
+
06-08 15:26:43,1,1,0.0,184.2,4,0,0.0,184.2,4.0,0.0,25.0,87.7,4,4,184.2,184.2
|
| 518 |
+
06-08 15:26:48,1,1,0.0,184.0,4,0,0.0,184.0,4.0,0.0,25.3,87.7,4,4,184.0,184.0
|
| 519 |
+
06-08 15:26:53,1,1,0.0,158.7,3,0,0.0,158.7,3.0,0.0,16.5,87.7,3,3,158.7,158.7
|
| 520 |
+
06-08 15:26:58,1,1,0.0,137.3,3,0,0.0,137.3,3.0,0.0,16.7,87.7,3,3,137.3,137.3
|
| 521 |
+
06-08 15:27:03,1,1,0.0,137.6,3,0,0.0,137.6,3.0,0.0,16.9,87.7,3,3,137.6,137.6
|
| 522 |
+
06-08 15:27:08,1,1,0.0,137.6,3,0,0.0,137.6,3.0,0.0,17.1,87.7,3,3,137.6,137.6
|
| 523 |
+
06-08 15:27:13,1,1,8.8,168.4,3,0,8.8,168.4,3.0,0.0,17.3,88.0,3,3,168.4,168.4
|
| 524 |
+
06-08 15:27:18,1,1,0.0,137.2,3,0,0.0,137.2,3.0,0.0,17.5,88.0,3,3,137.2,137.2
|
| 525 |
+
06-08 15:27:23,1,1,0.0,137.5,3,0,0.0,137.5,3.0,0.0,17.7,88.0,3,3,137.5,137.5
|
| 526 |
+
06-08 15:27:28,1,1,0.0,137.4,3,0,0.0,137.4,3.0,0.0,17.9,88.0,3,3,137.4,137.4
|
| 527 |
+
06-08 15:27:33,1,1,0.0,138.0,3,0,0.0,138.0,3.0,0.0,18.1,88.0,3,3,138.0,138.0
|
| 528 |
+
06-08 15:27:38,1,1,0.0,138.2,3,0,0.0,138.2,3.0,0.0,18.3,88.0,3,3,138.2,138.2
|
| 529 |
+
06-08 15:27:43,1,1,0.0,138.2,3,0,0.0,138.2,3.0,0.0,18.5,88.0,3,3,138.2,138.2
|
| 530 |
+
06-08 15:27:48,1,1,2.6,181.0,4,0,2.6,181.0,4.0,0.0,18.7,88.0,4,4,181.0,181.0
|
| 531 |
+
06-08 15:27:53,1,1,262.8,208.1,5,0,262.8,208.1,5.0,0.0,21.2,87.9,5,5,208.1,208.1
|
| 532 |
+
06-08 15:27:58,1,1,0.0,205.7,4,0,0.0,205.7,4.0,0.0,19.2,87.9,4,4,205.7,205.7
|
| 533 |
+
06-08 15:28:03,1,1,0.0,182.9,4,0,0.0,182.9,4.0,0.0,19.5,87.9,4,4,182.9,182.9
|
| 534 |
+
06-08 15:28:08,1,1,82.7,204.0,5,0,82.7,204.0,5.0,0.0,21.1,88.0,5,5,204.0,204.0
|
| 535 |
+
06-08 15:28:13,1,1,0.0,229.1,5,0,0.0,229.1,5.0,0.0,21.4,88.0,5,5,229.1,229.1
|
| 536 |
+
06-08 15:28:18,1,1,0.0,206.2,4,0,0.0,206.2,4.0,0.0,20.3,88.0,4,4,206.2,206.2
|
| 537 |
+
06-08 15:28:23,1,1,0.0,184.0,4,0,0.0,184.0,4.0,0.0,20.5,88.0,4,4,184.0,184.0
|
| 538 |
+
06-08 15:28:28,1,1,0.0,177.1,3,0,0.0,177.1,3.0,0.0,11.9,88.0,3,3,177.1,177.1
|
| 539 |
+
06-08 15:28:33,1,1,21.1,148.6,4,0,21.1,148.6,4.0,0.0,14.5,88.3,4,4,148.6,148.6
|
| 540 |
+
06-08 15:28:38,1,1,9.8,206.0,4,0,9.8,206.0,4.0,0.0,14.7,88.5,4,4,206.0,206.0
|
| 541 |
+
06-08 15:28:43,1,1,0.0,141.5,3,0,0.0,141.5,3.0,0.0,12.5,88.5,3,3,141.5,141.5
|
| 542 |
+
06-08 15:28:48,1,1,23.2,152.3,4,0,23.2,152.3,4.0,0.0,14.2,88.7,4,4,152.3,152.3
|
| 543 |
+
06-08 15:28:53,1,1,0.0,400.7,8,0,0.0,400.7,8.0,0.0,20.6,91.0,8,8,400.7,400.7
|
| 544 |
+
06-08 15:28:58,1,1,0.0,825.3,23,0,0.0,825.3,23.0,0.0,93.3,65.9,23,23,825.3,825.3
|
| 545 |
+
06-08 15:29:03,1,1,0.0,815.2,22,0,0.0,815.2,22.0,0.0,85.6,65.9,22,22,815.2,815.2
|
| 546 |
+
06-08 15:29:08,1,1,0.0,787.8,22,0,0.0,787.8,22.0,0.0,86.7,65.9,22,22,787.8,787.8
|
| 547 |
+
06-08 15:29:13,1,1,0.0,789.2,22,0,0.0,789.2,22.0,0.0,87.8,65.9,22,22,789.2,789.2
|
| 548 |
+
06-08 15:29:18,1,1,0.0,790.2,22,0,0.0,790.2,22.0,0.0,88.9,65.9,22,22,790.2,790.2
|
| 549 |
+
06-08 15:29:23,1,1,3313.9,729.6,23,0,3313.9,729.6,23.0,0.0,90.8,63.7,23,23,729.6,729.6
|
| 550 |
+
06-08 15:29:28,1,1,0.0,828.4,23,0,0.0,828.4,23.0,0.0,92.0,63.7,23,23,828.4,828.4
|
| 551 |
+
06-08 15:29:33,1,1,1.6,849.8,24,0,1.6,849.8,24.0,0.0,93.2,63.8,24,24,849.8,849.8
|
| 552 |
+
06-08 15:29:38,1,1,0.0,861.1,24,0,0.0,861.1,24.0,0.0,94.4,63.8,24,24,861.1,861.1
|
| 553 |
+
06-08 15:29:43,1,1,0.0,862.0,24,0,0.0,862.0,24.0,0.0,95.6,63.8,24,24,862.0,862.0
|
| 554 |
+
06-08 15:29:48,1,1,0.0,859.9,24,0,0.0,859.9,24.0,0.0,96.8,63.8,24,24,859.9,859.9
|
| 555 |
+
06-08 15:29:53,1,1,0.0,840.3,23,0,0.0,840.3,23.0,0.0,95.1,63.8,23,23,840.3,840.3
|
| 556 |
+
06-08 15:29:58,1,1,0.0,137.5,3,0,0.0,137.5,3.0,0.0,15.4,88.9,3,3,137.5,137.5
|
| 557 |
+
06-08 15:30:03,1,1,0.0,861.4,24,0,0.0,861.4,24.0,0.0,98.4,63.4,24,24,861.4,861.4
|
| 558 |
+
06-08 15:30:08,1,1,0.0,860.9,24,0,0.0,860.9,24.0,0.0,99.6,63.4,24,24,860.9,860.9
|
| 559 |
+
06-08 15:30:13,1,1,0.0,834.1,23,2,0.0,834.1,23.0,2.0,99.7,63.4,23,23,834.1,834.1
|
| 560 |
+
06-08 15:30:18,1,1,0.0,777.8,21,4,0.0,777.8,21.0,4.0,97.7,63.4,21,21,777.8,777.8
|
| 561 |
+
06-08 15:30:23,1,1,3055.9,712.7,23,0,3055.9,712.7,23.0,0.0,89.4,61.5,23,23,712.7,712.7
|
| 562 |
+
06-08 15:30:28,1,1,0.0,825.8,23,0,0.0,825.8,23.0,0.0,90.6,61.5,23,23,825.8,825.8
|
| 563 |
+
06-08 15:30:33,1,1,0.0,824.0,23,0,0.0,824.0,23.0,0.0,91.7,61.5,23,23,824.0,824.0
|
| 564 |
+
06-08 15:30:38,1,1,0.0,824.7,23,0,0.0,824.7,23.0,0.0,92.9,61.5,23,23,824.7,824.7
|
| 565 |
+
06-08 15:30:43,1,1,0.0,824.0,23,0,0.0,824.0,23.0,0.0,94.0,61.5,23,23,824.0,824.0
|
| 566 |
+
06-08 15:30:48,1,1,0.0,806.2,22,0,0.0,806.2,22.0,0.0,86.3,61.5,22,22,806.2,806.2
|
| 567 |
+
06-08 15:30:53,1,1,0.0,769.5,21,0,0.0,769.5,21.0,0.0,85.9,61.5,21,21,769.5,769.5
|
| 568 |
+
06-08 15:30:58,1,1,0.0,753.7,21,0,0.0,753.7,21.0,0.0,87.0,61.5,21,21,753.7,753.7
|
| 569 |
+
06-08 15:31:03,1,1,0.0,729.7,20,0,0.0,729.7,20.0,0.0,87.2,61.5,20,20,729.7,729.7
|
| 570 |
+
06-08 15:31:08,1,1,0.0,702.0,19,0,0.0,702.0,19.0,0.0,79.3,61.5,19,19,702.0,702.0
|
| 571 |
+
06-08 15:31:13,1,1,0.0,681.8,19,0,0.0,681.8,19.0,0.0,80.3,61.5,19,19,681.8,681.8
|
| 572 |
+
06-08 15:31:18,1,1,0.0,681.0,19,0,0.0,681.0,19.0,0.0,81.2,61.5,19,19,681.0,681.0
|
| 573 |
+
06-08 15:31:24,1,1,0.0,179.9,4,0,0.0,179.9,4.0,0.0,16.2,89.4,4,4,179.9,179.9
|
| 574 |
+
06-08 15:31:29,1,1,0.0,179.8,4,0,0.0,179.8,4.0,0.0,16.5,89.4,4,4,179.8,179.8
|
| 575 |
+
06-08 15:31:34,1,1,0.0,179.4,4,0,0.0,179.4,4.0,0.0,16.7,89.4,4,4,179.4,179.4
|
| 576 |
+
06-08 15:31:39,1,1,0.0,179.4,4,0,0.0,179.4,4.0,0.0,17.0,89.4,4,4,179.4,179.4
|
| 577 |
+
06-08 15:31:44,1,1,0.0,179.4,4,0,0.0,179.4,4.0,0.0,17.2,89.4,4,4,179.4,179.4
|
| 578 |
+
06-08 15:31:49,1,1,0.0,179.6,4,0,0.0,179.6,4.0,0.0,17.5,89.4,4,4,179.6,179.6
|
| 579 |
+
06-08 15:31:54,1,1,0.0,180.0,4,0,0.0,180.0,4.0,0.0,17.7,89.4,4,4,180.0,180.0
|
| 580 |
+
06-08 15:31:59,1,1,0.0,179.5,4,0,0.0,179.5,4.0,0.0,18.0,89.4,4,4,179.5,179.5
|
| 581 |
+
06-08 15:32:04,1,1,0.0,179.7,4,0,0.0,179.7,4.0,0.0,18.2,89.4,4,4,179.7,179.7
|
| 582 |
+
06-08 15:32:09,1,1,0.0,180.1,4,0,0.0,180.1,4.0,0.0,18.5,89.4,4,4,180.1,180.1
|
| 583 |
+
06-08 15:32:14,1,1,0.0,179.5,4,0,0.0,179.5,4.0,0.0,18.7,89.4,4,4,179.5,179.5
|
| 584 |
+
06-08 15:32:19,1,1,0.0,179.5,4,0,0.0,179.5,4.0,0.0,19.0,89.4,4,4,179.5,179.5
|
| 585 |
+
06-08 15:32:24,1,1,0.0,179.1,4,0,0.0,179.1,4.0,0.0,19.2,89.4,4,4,179.1,179.1
|
| 586 |
+
06-08 15:32:29,1,1,111.4,185.9,5,0,111.4,185.9,5.0,0.0,20.0,89.3,5,5,185.9,185.9
|
| 587 |
+
06-08 15:32:34,1,1,0.0,224.4,5,0,0.0,224.4,5.0,0.0,20.4,89.3,5,5,224.4,224.4
|
| 588 |
+
06-08 15:32:39,1,1,0.0,621.0,17,0,0.0,621.0,17.0,0.0,76.4,60.1,17,17,621.0,621.0
|
| 589 |
+
06-08 15:32:44,1,1,0.0,620.8,17,0,0.0,620.8,17.0,0.0,77.2,60.1,17,17,620.8,620.8
|
| 590 |
+
06-08 15:32:49,1,1,0.0,618.9,17,0,0.0,618.9,17.0,0.0,78.1,60.1,17,17,618.9,618.9
|
| 591 |
+
06-08 15:32:54,1,1,0.0,662.9,16,0,0.0,662.9,16.0,0.0,70.1,60.1,16,16,662.9,662.9
|
| 592 |
+
06-08 15:32:59,1,1,0.0,91.6,2,0,0.0,91.6,2.0,0.0,11.1,88.9,2,2,91.6,91.6
|
| 593 |
+
06-08 15:33:04,1,1,0.0,142.1,3,0,0.0,142.1,3.0,0.0,15.1,89.2,3,3,142.1,142.1
|
| 594 |
+
06-08 15:33:09,1,1,0.0,141.8,3,0,0.0,141.8,3.0,0.0,15.3,89.2,3,3,141.8,141.8
|
| 595 |
+
06-08 15:33:14,1,1,0.0,141.4,3,0,0.0,141.4,3.0,0.0,15.5,89.2,3,3,141.4,141.4
|
| 596 |
+
06-08 15:33:19,1,1,0.0,141.1,3,0,0.0,141.1,3.0,0.0,15.7,89.2,3,3,141.1,141.1
|
| 597 |
+
06-08 15:33:24,1,1,0.0,142.0,3,0,0.0,142.0,3.0,0.0,15.8,89.2,3,3,142.0,142.0
|
| 598 |
+
06-08 15:33:29,1,1,0.0,141.5,3,0,0.0,141.5,3.0,0.0,16.0,89.2,3,3,141.5,141.5
|
| 599 |
+
06-08 15:33:34,1,1,0.0,141.6,3,0,0.0,141.6,3.0,0.0,16.2,89.2,3,3,141.6,141.6
|
| 600 |
+
06-08 15:33:39,1,1,0.0,141.0,3,0,0.0,141.0,3.0,0.0,16.4,89.2,3,3,141.0,141.0
|
| 601 |
+
06-08 15:33:44,1,1,0.0,237.2,5,0,0.0,237.2,5.0,0.0,22.8,88.8,5,5,237.2,237.2
|
| 602 |
+
06-08 15:33:49,1,1,0.0,378.9,9,0,0.0,378.9,9.0,0.0,47.6,90.7,9,9,378.9,378.9
|
| 603 |
+
06-08 15:33:54,1,1,0.0,378.8,9,0,0.0,378.8,9.0,0.0,48.1,90.7,9,9,378.8,378.8
|
| 604 |
+
06-08 15:33:59,1,1,0.0,379.1,9,0,0.0,379.1,9.0,0.0,48.7,90.7,9,9,379.1,379.1
|
| 605 |
+
06-08 15:34:04,1,1,0.0,378.2,9,0,0.0,378.2,9.0,0.0,49.2,90.7,9,9,378.2,378.2
|
| 606 |
+
06-08 15:34:09,1,1,0.0,429.6,10,0,0.0,429.6,10.0,0.0,55.4,87.5,10,10,429.6,429.6
|
| 607 |
+
06-08 15:34:14,1,1,0.0,377.6,9,0,0.0,377.6,9.0,0.0,50.2,90.7,9,9,377.6,377.6
|
| 608 |
+
06-08 15:34:19,1,1,0.0,376.9,9,0,0.0,376.9,9.0,0.0,50.8,90.7,9,9,376.9,376.9
|
| 609 |
+
06-08 15:34:24,1,1,0.0,377.1,9,0,0.0,377.1,9.0,0.0,51.3,90.7,9,9,377.1,377.1
|
| 610 |
+
06-08 15:34:29,1,1,0.0,376.1,9,0,0.0,376.1,9.0,0.0,51.8,90.7,9,9,376.1,376.1
|
| 611 |
+
06-08 15:34:34,1,1,0.0,375.7,9,0,0.0,375.7,9.0,0.0,52.3,90.7,9,9,375.7,375.7
|
| 612 |
+
06-08 15:34:39,1,1,0.0,376.5,9,0,0.0,376.5,9.0,0.0,52.9,90.7,9,9,376.5,376.5
|
| 613 |
+
06-08 15:34:44,1,1,0.0,375.1,9,0,0.0,375.1,9.0,0.0,53.4,90.7,9,9,375.1,375.1
|
| 614 |
+
06-08 15:34:49,1,1,0.0,375.3,9,0,0.0,375.3,9.0,0.0,53.9,90.7,9,9,375.3,375.3
|
| 615 |
+
06-08 15:34:54,1,1,0.0,375.5,9,0,0.0,375.5,9.0,0.0,54.4,90.7,9,9,375.5,375.5
|
| 616 |
+
06-08 15:34:59,1,1,0.0,375.1,9,0,0.0,375.1,9.0,0.0,55.0,90.7,9,9,375.1,375.1
|
| 617 |
+
06-08 15:35:04,1,1,0.0,428.4,10,0,0.0,428.4,10.0,0.0,62.0,87.5,10,10,428.4,428.4
|
| 618 |
+
06-08 15:35:09,1,1,0.0,428.5,10,0,0.0,428.5,10.0,0.0,62.6,87.5,10,10,428.5,428.5
|
| 619 |
+
06-08 15:35:14,1,1,0.0,428.6,10,0,0.0,428.6,10.0,0.0,63.2,87.5,10,10,428.6,428.6
|
| 620 |
+
06-08 15:35:19,1,1,0.0,399.1,9,0,0.0,399.1,9.0,0.0,54.9,87.5,9,9,399.1,399.1
|
| 621 |
+
06-08 15:35:25,1,1,2.0,408.8,10,0,2.0,408.8,10.0,0.0,56.3,87.6,10,10,408.8,408.8
|
| 622 |
+
06-08 15:35:30,1,1,0.0,426.7,10,0,0.0,426.7,10.0,0.0,56.9,87.6,10,10,426.7,426.7
|
| 623 |
+
06-08 15:35:35,1,1,0.0,236.8,5,0,0.0,236.8,5.0,0.0,18.7,87.9,5,5,236.8,236.8
|
| 624 |
+
06-08 15:35:40,1,1,0.0,236.4,5,0,0.0,236.4,5.0,0.0,19.0,87.9,5,5,236.4,236.4
|
| 625 |
+
06-08 15:35:45,1,1,0.0,197.4,4,0,0.0,197.4,4.0,0.0,18.6,87.9,4,4,197.4,197.4
|
| 626 |
+
06-08 15:35:50,1,1,0.0,182.8,4,0,0.0,182.8,4.0,0.0,25.1,90.9,4,4,182.8,182.8
|
| 627 |
+
06-08 15:35:55,1,1,0.0,183.1,4,0,0.0,183.1,4.0,0.0,25.3,90.9,4,4,183.1,183.1
|
| 628 |
+
06-08 15:36:00,1,1,0.0,183.2,4,0,0.0,183.2,4.0,0.0,25.6,90.9,4,4,183.2,183.2
|
| 629 |
+
06-08 15:36:05,1,1,0.0,182.4,4,0,0.0,182.4,4.0,0.0,25.9,90.9,4,4,182.4,182.4
|
| 630 |
+
06-08 15:36:10,1,1,0.0,182.5,4,0,0.0,182.5,4.0,0.0,26.1,90.9,4,4,182.5,182.5
|
| 631 |
+
06-08 15:36:15,1,1,0.0,183.2,4,0,0.0,183.2,4.0,0.0,26.4,90.9,4,4,183.2,183.2
|
| 632 |
+
06-08 15:36:19,1,1,0.0,174.3,4,0,0.0,174.3,4.0,0.0,25.9,87.3,4,4,174.3,174.3
|
| 633 |
+
06-08 15:36:26,1,1,0.0,171.4,4,0,0.0,171.4,4.0,0.0,21.6,88.6,4,4,171.4,171.4
|
| 634 |
+
06-08 15:36:31,1,1,128.4,210.3,5,0,128.4,210.3,5.0,0.0,23.8,88.7,5,5,210.3,210.3
|
| 635 |
+
06-08 15:36:36,1,1,0.0,215.3,5,0,0.0,215.3,5.0,0.0,24.1,88.7,5,5,215.3,215.3
|
| 636 |
+
06-08 15:36:41,1,1,0.0,326.1,7,0,0.0,326.1,7.0,0.0,38.3,89.9,7,7,326.1,326.1
|
| 637 |
+
06-08 15:36:46,1,1,0.0,324.9,7,0,0.0,324.9,7.0,0.0,38.7,89.9,7,7,324.9,324.9
|
| 638 |
+
06-08 15:36:51,1,1,0.0,325.2,7,0,0.0,325.2,7.0,0.0,39.2,89.9,7,7,325.2,325.2
|
| 639 |
+
06-08 15:36:56,1,1,0.0,324.5,7,0,0.0,324.5,7.0,0.0,39.6,89.9,7,7,324.5,324.5
|
| 640 |
+
06-08 15:37:01,1,1,0.0,325.6,7,0,0.0,325.6,7.0,0.0,40.1,89.9,7,7,325.6,325.6
|
| 641 |
+
06-08 15:37:07,1,1,0.0,230.2,5,0,0.0,230.2,5.0,0.0,31.9,89.9,5,5,230.2,230.2
|
| 642 |
+
06-08 15:37:12,1,1,0.0,64.2,1,0,0.0,64.2,1.0,0.0,8.5,88.3,1,1,64.2,64.2
|
| 643 |
+
06-08 15:37:17,1,1,0.0,44.3,1,0,0.0,44.3,1.0,0.0,8.6,88.3,1,1,44.3,44.3
|
| 644 |
+
06-08 15:37:22,1,1,0.0,44.3,1,0,0.0,44.3,1.0,0.0,8.6,88.3,1,1,44.3,44.3
|
| 645 |
+
06-08 15:37:27,1,1,0.0,44.4,1,0,0.0,44.4,1.0,0.0,8.7,88.3,1,1,44.4,44.4
|
| 646 |
+
06-08 15:37:32,1,1,0.0,44.3,1,0,0.0,44.3,1.0,0.0,8.8,88.3,1,1,44.3,44.3
|
| 647 |
+
06-08 15:37:37,1,1,0.0,43.9,1,0,0.0,43.9,1.0,0.0,8.8,88.3,1,1,43.9,43.9
|
| 648 |
+
06-08 15:37:42,1,1,46.8,86.4,2,0,46.8,86.4,2.0,0.0,9.0,88.3,2,2,86.4,86.4
|
| 649 |
+
06-08 15:37:48,1,1,0.0,231.6,5,0,0.0,231.6,5.0,0.0,32.7,90.7,5,5,231.6,231.6
|
| 650 |
+
06-08 15:37:53,1,1,0.0,231.4,5,0,0.0,231.4,5.0,0.0,33.0,90.7,5,5,231.4,231.4
|
| 651 |
+
06-08 15:37:58,1,1,0.0,231.2,5,0,0.0,231.2,5.0,0.0,33.3,90.7,5,5,231.2,231.2
|
| 652 |
+
06-08 15:38:03,1,1,0.0,231.4,5,0,0.0,231.4,5.0,0.0,33.6,90.7,5,5,231.4,231.4
|
| 653 |
+
06-08 15:38:08,1,1,0.0,231.1,5,0,0.0,231.1,5.0,0.0,34.0,90.7,5,5,231.1,231.1
|
| 654 |
+
06-08 15:38:13,1,1,0.0,232.2,5,0,0.0,232.2,5.0,0.0,34.3,90.7,5,5,232.2,232.2
|
| 655 |
+
06-08 15:38:18,1,1,0.0,232.1,5,0,0.0,232.1,5.0,0.0,34.6,90.7,5,5,232.1,232.1
|
| 656 |
+
06-08 15:38:23,1,1,76.5,238.0,6,0,76.5,238.0,6.0,0.0,35.0,90.7,6,6,238.0,238.0
|
| 657 |
+
06-08 15:38:28,1,1,0.0,276.9,6,0,0.0,276.9,6.0,0.0,35.4,90.7,6,6,276.9,276.9
|
| 658 |
+
06-08 15:38:33,1,1,0.0,278.2,6,0,0.0,278.2,6.0,0.0,35.8,90.7,6,6,278.2,278.2
|
| 659 |
+
06-08 15:38:38,1,1,0.0,278.6,6,0,0.0,278.6,6.0,0.0,36.2,90.7,6,6,278.6,278.6
|
| 660 |
+
06-08 15:38:43,1,1,0.0,278.8,6,0,0.0,278.8,6.0,0.0,36.6,90.7,6,6,278.8,278.8
|
| 661 |
+
06-08 15:38:48,1,1,0.0,278.5,6,0,0.0,278.5,6.0,0.0,37.0,90.7,6,6,278.5,278.5
|
| 662 |
+
06-08 15:38:53,1,1,7.6,303.9,7,0,7.6,303.9,7.0,0.0,37.5,90.7,7,7,303.9,303.9
|
| 663 |
+
06-08 15:38:58,1,1,0.0,324.0,7,0,0.0,324.0,7.0,0.0,38.0,90.7,7,7,324.0,324.0
|
| 664 |
+
06-08 15:39:03,1,1,8.8,338.6,8,0,8.8,338.6,8.0,0.0,38.6,90.7,8,8,338.6,338.6
|
| 665 |
+
06-08 15:39:08,1,1,0.0,369.6,8,0,0.0,369.6,8.0,0.0,39.1,90.7,8,8,369.6,369.6
|
| 666 |
+
06-08 15:39:13,1,1,0.0,593.5,13,0,0.0,593.5,13.0,0.0,44.9,81.0,13,13,593.5,593.5
|
| 667 |
+
06-08 15:39:18,1,1,0.0,539.2,12,0,0.0,539.2,12.0,0.0,43.7,81.0,12,12,539.2,539.2
|
| 668 |
+
06-08 15:39:23,1,1,0.0,320.9,7,0,0.0,320.9,7.0,0.0,36.0,89.3,7,7,320.9,320.9
|
| 669 |
+
06-08 15:39:28,1,1,0.0,229.8,5,0,0.0,229.8,5.0,0.0,21.5,87.0,5,5,229.8,229.8
|
| 670 |
+
06-08 15:39:33,1,1,0.0,223.9,5,0,0.0,223.9,5.0,0.0,21.8,87.0,5,5,223.9,223.9
|
| 671 |
+
06-08 15:39:38,1,1,0.0,224.0,5,0,0.0,224.0,5.0,0.0,22.1,87.0,5,5,224.0,224.0
|
| 672 |
+
06-08 15:39:43,1,1,0.0,224.4,5,0,0.0,224.4,5.0,0.0,22.5,87.0,5,5,224.4,224.4
|
| 673 |
+
06-08 15:39:48,1,1,0.0,224.2,5,0,0.0,224.2,5.0,0.0,22.8,87.0,5,5,224.2,224.2
|
| 674 |
+
06-08 15:39:53,1,1,0.0,224.7,5,0,0.0,224.7,5.0,0.0,23.1,87.0,5,5,224.7,224.7
|
| 675 |
+
06-08 15:39:58,1,1,0.0,224.5,5,0,0.0,224.5,5.0,0.0,23.4,87.0,5,5,224.5,224.5
|
| 676 |
+
06-08 15:40:03,1,1,0.0,224.6,5,0,0.0,224.6,5.0,0.0,23.7,87.0,5,5,224.6,224.6
|
| 677 |
+
06-08 15:40:08,1,1,0.0,224.8,5,0,0.0,224.8,5.0,0.0,24.0,87.0,5,5,224.8,224.8
|
| 678 |
+
06-08 15:40:13,1,1,0.0,225.2,5,0,0.0,225.2,5.0,0.0,24.3,87.0,5,5,225.2,225.2
|
| 679 |
+
06-08 15:40:18,1,1,0.0,225.3,5,0,0.0,225.3,5.0,0.0,24.7,87.0,5,5,225.3,225.3
|
| 680 |
+
06-08 15:40:23,1,1,114.5,242.8,6,0,114.5,242.8,6.0,0.0,25.7,86.9,6,6,242.8,242.8
|
| 681 |
+
06-08 15:40:28,1,1,0.0,268.7,6,0,0.0,268.7,6.0,0.0,26.1,86.9,6,6,268.7,268.7
|
| 682 |
+
06-08 15:40:33,1,1,0.0,221.0,4,0,0.0,221.0,4.0,0.0,12.9,86.7,4,4,221.0,221.0
|
| 683 |
+
06-08 15:40:38,1,1,0.0,189.1,4,0,0.0,189.1,4.0,0.0,13.2,86.7,4,4,189.1,189.1
|
| 684 |
+
06-08 15:40:44,1,1,0.0,190.6,4,0,0.0,190.6,4.0,0.0,13.5,86.7,4,4,190.6,190.6
|
| 685 |
+
06-08 15:40:48,1,1,0.0,206.6,4,0,0.0,206.6,4.0,0.0,17.7,86.9,4,4,206.6,206.6
|
| 686 |
+
06-08 15:40:53,1,1,0.0,179.4,4,0,0.0,179.4,4.0,0.0,17.9,86.9,4,4,179.4,179.4
|
| 687 |
+
06-08 15:40:58,1,1,0.0,179.8,4,0,0.0,179.8,4.0,0.0,18.2,86.9,4,4,179.8,179.8
|
| 688 |
+
06-08 15:41:04,1,1,0.0,179.6,4,0,0.0,179.6,4.0,0.0,18.4,86.9,4,4,179.6,179.6
|
| 689 |
+
06-08 15:41:09,1,1,0.0,179.0,4,0,0.0,179.0,4.0,0.0,18.7,86.9,4,4,179.0,179.0
|
| 690 |
+
06-08 15:41:14,1,1,0.0,189.8,4,0,0.0,189.8,4.0,0.0,15.1,86.7,4,4,189.8,189.8
|
| 691 |
+
06-08 15:41:19,1,1,0.0,189.7,4,0,0.0,189.7,4.0,0.0,15.3,86.7,4,4,189.7,189.7
|
| 692 |
+
06-08 15:41:24,1,1,0.0,189.7,4,0,0.0,189.7,4.0,0.0,15.6,86.7,4,4,189.7,189.7
|
| 693 |
+
06-08 15:41:29,1,1,0.0,189.7,4,0,0.0,189.7,4.0,0.0,15.8,86.7,4,4,189.7,189.7
|
| 694 |
+
06-08 15:41:34,1,1,0.0,189.6,4,0,0.0,189.6,4.0,0.0,16.1,86.7,4,4,189.6,189.6
|
| 695 |
+
06-08 15:41:39,1,1,155.7,300.1,9,0,155.7,300.1,9.0,0.0,23.3,88.3,9,9,300.1,300.1
|
| 696 |
+
06-08 15:41:44,1,1,8.0,390.7,10,0,8.0,390.7,10.0,0.0,23.9,88.3,10,10,390.7,390.7
|
| 697 |
+
06-08 15:41:49,1,1,0.0,379.4,9,0,0.0,379.4,9.0,0.0,23.8,88.3,9,9,379.4,379.4
|
| 698 |
+
06-08 15:41:54,1,1,0.0,335.7,8,0,0.0,335.7,8.0,0.0,23.9,88.3,8,8,335.7,335.7
|
| 699 |
+
06-08 15:41:59,1,1,0.0,130.1,3,0,0.0,130.1,3.0,0.0,2.2,87.4,3,3,130.1,130.1
|
| 700 |
+
06-08 15:42:04,1,1,8.6,164.7,4,0,8.6,164.7,4.0,0.0,2.6,87.4,4,4,164.7,164.7
|
| 701 |
+
06-08 15:42:09,1,1,169.1,211.3,6,0,169.1,211.3,6.0,0.0,4.4,87.4,6,6,211.3,211.3
|
| 702 |
+
06-08 15:42:14,1,1,159.5,273.9,5,0,159.5,273.9,5.0,0.0,5.4,87.5,5,5,273.9,273.9
|
| 703 |
+
06-08 15:42:19,1,1,263.0,253.1,6,0,263.0,253.1,6.0,0.0,5.4,87.4,6,6,253.1,253.1
|
| 704 |
+
06-08 15:42:24,1,1,0.0,236.2,5,0,0.0,236.2,5.0,0.0,4.4,87.4,5,5,236.2,236.2
|
| 705 |
+
06-08 15:42:29,1,1,0.0,216.0,5,0,0.0,216.0,5.0,0.0,4.7,87.4,5,5,216.0,216.0
|
| 706 |
+
06-08 15:42:34,1,1,0.0,201.7,4,0,0.0,201.7,4.0,0.0,3.3,87.4,4,4,201.7,201.7
|
| 707 |
+
06-08 15:42:39,1,1,0.0,172.2,4,0,0.0,172.2,4.0,0.0,3.6,87.4,4,4,172.2,172.2
|
| 708 |
+
06-08 15:42:44,1,1,212.7,251.3,6,0,212.7,251.3,6.0,0.0,5.0,87.4,6,6,251.3,251.3
|
| 709 |
+
06-08 15:42:49,1,1,0.0,259.4,6,0,0.0,259.4,6.0,0.0,5.3,87.4,6,6,259.4,259.4
|
| 710 |
+
06-08 15:42:54,1,1,0.0,259.6,6,0,0.0,259.6,6.0,0.0,5.7,87.4,6,6,259.6,259.6
|
| 711 |
+
06-08 15:42:59,1,1,0.0,239.7,5,0,0.0,239.7,5.0,0.0,5.0,87.4,5,5,239.7,239.7
|
| 712 |
+
06-08 15:43:04,1,1,30.7,245.9,6,0,30.7,245.9,6.0,0.0,5.8,87.4,6,6,245.9,245.9
|
| 713 |
+
06-08 15:43:09,1,1,0.0,249.3,5,0,0.0,249.3,5.0,0.0,5.7,87.4,5,5,249.3,249.3
|
| 714 |
+
06-08 15:43:14,1,1,0.0,185.4,4,0,0.0,185.4,4.0,0.0,5.3,87.4,4,4,185.4,185.4
|
| 715 |
+
06-08 15:43:19,1,1,189.8,209.3,5,0,189.8,209.3,5.0,0.0,8.8,87.6,5,5,209.3,209.3
|
| 716 |
+
06-08 15:43:24,1,1,0.0,216.1,5,0,0.0,216.1,5.0,0.0,9.1,87.6,5,5,216.1,216.1
|
| 717 |
+
06-08 15:43:29,1,1,0.0,191.4,4,0,0.0,191.4,4.0,0.0,6.8,87.6,4,4,191.4,191.4
|
| 718 |
+
06-08 15:43:34,1,1,168.0,221.4,6,0,168.0,221.4,6.0,0.0,8.6,87.6,6,6,221.4,221.4
|
| 719 |
+
06-08 15:43:39,1,1,0.0,259.4,6,0,0.0,259.4,6.0,0.0,9.0,87.6,6,6,259.4,259.4
|
| 720 |
+
06-08 15:43:44,1,1,0.0,259.6,6,0,0.0,259.6,6.0,0.0,9.4,87.6,6,6,259.6,259.6
|
| 721 |
+
06-08 15:43:49,1,1,0.0,259.7,6,0,0.0,259.7,6.0,0.0,9.7,87.6,6,6,259.7,259.7
|
| 722 |
+
06-08 15:43:54,1,1,149.3,323.5,8,0,149.3,323.5,8.0,0.0,10.4,87.5,8,8,323.5,323.5
|
| 723 |
+
06-08 15:43:59,1,1,9.8,319.0,8,0,9.8,319.0,8.0,0.0,8.5,87.5,8,8,319.0,319.0
|
| 724 |
+
06-08 15:44:04,1,1,380.0,352.5,9,0,380.0,352.5,9.0,0.0,10.6,87.3,9,9,352.5,352.5
|
| 725 |
+
06-08 15:44:09,1,1,0.0,387.2,9,0,0.0,387.2,9.0,0.0,11.1,87.3,9,9,387.2,387.2
|
| 726 |
+
06-08 15:44:14,1,1,69.1,407.2,10,0,69.1,407.2,10.0,0.0,12.4,87.4,10,10,407.2,407.2
|
| 727 |
+
06-08 15:44:19,1,1,9.0,432.7,11,0,9.0,432.7,11.0,0.0,13.2,87.4,11,11,432.7,432.7
|
| 728 |
+
06-08 15:44:24,1,1,0.0,226.6,5,0,0.0,226.6,5.0,0.0,10.9,88.7,5,5,226.6,226.6
|
| 729 |
+
06-08 15:44:29,1,1,0.0,226.5,5,0,0.0,226.5,5.0,0.0,11.2,88.7,5,5,226.5,226.5
|
| 730 |
+
06-08 15:44:34,1,1,17.9,199.4,5,0,17.9,199.4,5.0,0.0,12.0,88.8,5,5,199.4,199.4
|
| 731 |
+
06-08 15:44:39,1,1,0.0,226.2,5,0,0.0,226.2,5.0,0.0,12.3,88.8,5,5,226.2,226.2
|
| 732 |
+
06-08 15:44:44,1,1,0.0,226.6,5,0,0.0,226.6,5.0,0.0,12.6,88.8,5,5,226.6,226.6
|
| 733 |
+
06-08 15:44:50,1,1,0.0,432.7,9,0,0.0,432.7,9.0,0.0,30.2,88.5,9,9,432.7,432.7
|
| 734 |
+
06-08 15:44:55,1,1,0.0,430.7,9,0,0.0,430.7,9.0,0.0,30.8,88.5,9,9,430.7,430.7
|
| 735 |
+
06-08 15:45:00,1,1,0.0,431.3,9,0,0.0,431.3,9.0,0.0,31.4,88.5,9,9,431.3,431.3
|
| 736 |
+
06-08 15:45:05,1,1,0.0,430.8,9,0,0.0,430.8,9.0,0.0,32.0,88.5,9,9,430.8,430.8
|
| 737 |
+
06-08 15:45:10,1,1,0.0,388.9,8,0,0.0,388.9,8.0,0.0,31.6,88.5,8,8,388.9,388.9
|
| 738 |
+
06-08 15:45:15,1,1,0.0,384.7,8,0,0.0,384.7,8.0,0.0,32.2,88.5,8,8,384.7,384.7
|
| 739 |
+
06-08 15:45:20,1,1,0.0,383.4,8,0,0.0,383.4,8.0,0.0,32.7,88.5,8,8,383.4,383.4
|
| 740 |
+
06-08 15:45:25,1,1,0.0,342.3,8,0,0.0,342.3,8.0,0.0,30.8,88.4,8,8,342.3,342.3
|
| 741 |
+
06-08 15:45:30,1,1,91.8,384.8,8,0,91.8,384.8,8.0,0.0,31.3,88.4,8,8,384.8,384.8
|
| 742 |
+
06-08 15:45:35,1,1,0.0,384.9,8,0,0.0,384.9,8.0,0.0,31.9,88.4,8,8,384.9,384.9
|
| 743 |
+
06-08 15:45:40,1,1,0.0,502.0,11,0,0.0,502.0,11.0,0.0,26.6,86.0,11,11,502.0,502.0
|
| 744 |
+
06-08 15:45:45,1,1,0.0,494.1,11,0,0.0,494.1,11.0,0.0,27.3,86.0,11,11,494.1,494.1
|
| 745 |
+
06-08 15:45:50,1,1,174.0,505.2,12,0,174.0,505.2,12.0,0.0,28.8,85.9,12,12,505.2,505.2
|
| 746 |
+
06-08 15:45:55,1,1,326.9,572.7,13,0,326.9,572.7,13.0,0.0,31.7,85.8,13,13,572.7,572.7
|
| 747 |
+
06-08 15:46:00,1,1,0.0,580.1,13,0,0.0,580.1,13.0,0.0,32.5,85.8,13,13,580.1,580.1
|
| 748 |
+
06-08 15:46:05,1,1,0.0,580.9,13,0,0.0,580.9,13.0,0.0,33.3,85.8,13,13,580.9,580.9
|
| 749 |
+
06-08 15:46:10,1,1,0.0,552.5,12,0,0.0,552.5,12.0,0.0,33.0,85.8,12,12,552.5,552.5
|
| 750 |
+
06-08 15:46:15,1,1,0.0,537.5,12,0,0.0,537.5,12.0,0.0,33.7,85.8,12,12,537.5,537.5
|
| 751 |
+
06-08 15:46:20,1,1,0.0,138.0,3,0,0.0,138.0,3.0,0.0,11.4,88.2,3,3,138.0,138.0
|
| 752 |
+
06-08 15:46:25,1,1,0.0,137.8,3,0,0.0,137.8,3.0,0.0,11.6,88.2,3,3,137.8,137.8
|
| 753 |
+
06-08 15:46:30,1,1,0.0,137.7,3,0,0.0,137.7,3.0,0.0,11.8,88.2,3,3,137.7,137.7
|
| 754 |
+
06-08 15:46:35,1,1,10.0,162.1,4,0,10.0,162.1,4.0,0.0,12.1,88.3,4,4,162.1,162.1
|
| 755 |
+
06-08 15:46:40,1,1,97.8,202.9,5,0,97.8,202.9,5.0,0.0,13.0,88.2,5,5,202.9,202.9
|
| 756 |
+
06-08 15:46:45,1,1,0.0,228.8,5,0,0.0,228.8,5.0,0.0,13.3,88.2,5,5,228.8,228.8
|
| 757 |
+
06-08 15:46:50,1,1,0.0,229.5,5,0,0.0,229.5,5.0,0.0,13.6,88.2,5,5,229.5,229.5
|
| 758 |
+
06-08 15:46:55,1,1,0.0,193.0,4,0,0.0,193.0,4.0,0.0,13.2,88.2,4,4,193.0,193.0
|
| 759 |
+
06-08 15:47:00,1,1,96.2,199.0,5,0,96.2,199.0,5.0,0.0,15.1,88.3,5,5,199.0,199.0
|
| 760 |
+
06-08 15:47:05,1,1,0.0,230.8,5,0,0.0,230.8,5.0,0.0,15.4,88.3,5,5,230.8,230.8
|
| 761 |
+
06-08 15:47:10,1,1,0.0,374.3,8,0,0.0,374.3,8.0,0.0,17.6,90.7,8,8,374.3,374.3
|
| 762 |
+
06-08 15:47:15,1,1,74.1,412.9,9,0,74.1,412.9,9.0,0.0,19.9,90.7,9,9,412.9,412.9
|
| 763 |
+
06-08 15:47:20,1,1,0.0,411.6,8,0,0.0,411.6,8.0,0.0,18.7,90.7,8,8,411.6,411.6
|
| 764 |
+
06-08 15:47:25,1,1,0.0,374.6,8,0,0.0,374.6,8.0,0.0,19.2,90.7,8,8,374.6,374.6
|
| 765 |
+
06-08 15:47:30,1,1,0.0,375.9,8,0,0.0,375.9,8.0,0.0,19.7,90.7,8,8,375.9,375.9
|
| 766 |
+
06-08 15:47:35,1,1,0.0,375.9,8,0,0.0,375.9,8.0,0.0,20.3,90.7,8,8,375.9,375.9
|
| 767 |
+
06-08 15:47:40,1,1,0.0,375.6,8,0,0.0,375.6,8.0,0.0,20.8,90.7,8,8,375.6,375.6
|
| 768 |
+
06-08 15:47:45,1,1,0.0,375.1,8,0,0.0,375.1,8.0,0.0,21.3,90.7,8,8,375.1,375.1
|
| 769 |
+
06-08 15:47:50,1,1,0.0,182.5,4,0,0.0,182.5,4.0,0.0,16.1,88.2,4,4,182.5,182.5
|
| 770 |
+
06-08 15:47:55,1,1,0.0,376.1,8,0,0.0,376.1,8.0,0.0,22.4,90.7,8,8,376.1,376.1
|
| 771 |
+
06-08 15:48:00,1,1,0.0,398.4,9,0,0.0,398.4,9.0,0.0,38.7,87.8,9,9,398.4,398.4
|
| 772 |
+
06-08 15:48:05,1,1,0.0,398.2,9,0,0.0,398.2,9.0,0.0,39.3,87.8,9,9,398.2,398.2
|
| 773 |
+
06-08 15:48:10,1,1,0.0,397.2,9,0,0.0,397.2,9.0,0.0,39.8,87.8,9,9,397.2,397.2
|
| 774 |
+
06-08 15:48:15,1,1,0.0,396.9,9,0,0.0,396.9,9.0,0.0,40.3,87.8,9,9,396.9,396.9
|
| 775 |
+
06-08 15:48:20,1,1,0.0,487.8,12,0,0.0,487.8,12.0,0.0,50.8,88.2,12,12,487.8,487.8
|
| 776 |
+
06-08 15:48:27,1,1,227.1,405.4,10,0,227.1,405.4,10.0,0.0,42.0,88.2,10,10,405.4,405.4
|
| 777 |
+
06-08 15:48:32,1,1,0.0,423.9,10,0,0.0,423.9,10.0,0.0,42.6,88.2,10,10,423.9,423.9
|
| 778 |
+
06-08 15:48:37,1,1,0.0,411.6,9,0,0.0,411.6,9.0,0.0,40.3,88.2,9,9,411.6,411.6
|
| 779 |
+
06-08 15:48:42,1,1,0.0,381.4,9,0,0.0,381.4,9.0,0.0,40.8,88.2,9,9,381.4,381.4
|
| 780 |
+
06-08 15:48:47,1,1,0.0,382.9,9,0,0.0,382.9,9.0,0.0,41.3,88.2,9,9,382.9,382.9
|
| 781 |
+
06-08 15:48:52,1,1,0.0,383.6,9,0,0.0,383.6,9.0,0.0,41.9,88.2,9,9,383.6,383.6
|
| 782 |
+
06-08 15:48:57,1,1,0.0,382.6,9,0,0.0,382.6,9.0,0.0,42.4,88.2,9,9,382.6,382.6
|
| 783 |
+
06-08 15:49:02,1,1,0.0,382.0,9,0,0.0,382.0,9.0,0.0,42.9,88.2,9,9,382.0,382.0
|
| 784 |
+
06-08 15:49:07,1,1,0.0,348.8,8,0,0.0,348.8,8.0,0.0,33.2,90.1,8,8,348.8,348.8
|
| 785 |
+
06-08 15:49:12,1,1,0.0,316.7,7,0,0.0,316.7,7.0,0.0,33.2,90.1,7,7,316.7,316.7
|
| 786 |
+
06-08 15:49:17,1,1,0.0,304.2,7,0,0.0,304.2,7.0,0.0,33.7,90.1,7,7,304.2,304.2
|
| 787 |
+
06-08 15:49:22,1,1,0.0,417.5,9,0,0.0,417.5,9.0,0.0,45.1,88.4,9,9,417.5,417.5
|
| 788 |
+
06-08 15:49:27,1,1,0.0,255.5,6,0,0.0,255.5,6.0,0.0,29.8,89.6,6,6,255.5,255.5
|
| 789 |
+
06-08 15:49:32,1,1,0.0,255.0,6,0,0.0,255.0,6.0,0.0,30.2,89.6,6,6,255.0,255.0
|
| 790 |
+
06-08 15:49:37,1,1,0.0,458.0,10,0,0.0,458.0,10.0,0.0,51.7,88.0,10,10,458.0,458.0
|
| 791 |
+
06-08 15:49:42,1,1,0.0,437.7,9,0,0.0,437.7,9.0,0.0,51.8,88.0,9,9,437.7,437.7
|
| 792 |
+
06-08 15:49:47,1,1,0.0,412.7,9,0,0.0,412.7,9.0,0.0,52.4,88.0,9,9,412.7,412.7
|
| 793 |
+
06-08 15:49:52,1,1,0.0,413.2,9,0,0.0,413.2,9.0,0.0,53.0,88.0,9,9,413.2,413.2
|
| 794 |
+
06-08 15:49:58,1,1,0.0,134.1,3,0,0.0,134.1,3.0,0.0,17.2,90.2,3,3,134.1,134.1
|
| 795 |
+
06-08 15:50:03,1,1,0.0,133.9,3,0,0.0,133.9,3.0,0.0,17.4,90.2,3,3,133.9,133.9
|
| 796 |
+
06-08 15:50:08,1,1,0.0,134.2,3,0,0.0,134.2,3.0,0.0,17.6,90.2,3,3,134.2,134.2
|
| 797 |
+
06-08 15:50:13,1,1,0.0,133.5,3,0,0.0,133.5,3.0,0.0,17.8,90.2,3,3,133.5,133.5
|
| 798 |
+
06-08 15:50:18,1,1,0.0,133.8,3,0,0.0,133.8,3.0,0.0,18.0,90.2,3,3,133.8,133.8
|
| 799 |
+
06-08 15:50:23,1,1,0.0,133.8,3,0,0.0,133.8,3.0,0.0,18.1,90.2,3,3,133.8,133.8
|
| 800 |
+
06-08 15:50:28,1,1,0.0,134.0,3,0,0.0,134.0,3.0,0.0,18.3,90.2,3,3,134.0,134.0
|
| 801 |
+
06-08 15:50:33,1,1,0.0,133.8,3,0,0.0,133.8,3.0,0.0,18.5,90.2,3,3,133.8,133.8
|
| 802 |
+
06-08 15:50:38,1,1,2.2,149.8,4,0,2.2,149.8,4.0,0.0,18.7,90.3,4,4,149.8,149.8
|
| 803 |
+
06-08 15:50:44,1,1,0.0,498.6,10,0,0.0,498.6,10.0,0.0,51.2,90.7,10,10,498.6,498.6
|
| 804 |
+
06-08 15:50:50,1,1,0.0,362.1,8,0,0.0,362.1,8.0,0.0,42.1,88.7,8,8,362.1,362.1
|
| 805 |
+
06-08 15:50:55,1,1,0.0,361.2,8,0,0.0,361.2,8.0,0.0,42.6,88.7,8,8,361.2,361.2
|
| 806 |
+
06-08 15:51:00,1,1,0.0,352.6,7,0,0.0,352.6,7.0,0.0,41.6,88.7,7,7,352.6,352.6
|
| 807 |
+
06-08 15:51:05,1,1,0.0,315.5,7,0,0.0,315.5,7.0,0.0,30.6,88.1,7,7,315.5,315.5
|
| 808 |
+
06-08 15:51:10,1,1,0.0,280.7,6,0,0.0,280.7,6.0,0.0,29.0,88.1,6,6,280.7,280.7
|
| 809 |
+
06-08 15:51:15,1,1,0.0,270.1,6,0,0.0,270.1,6.0,0.0,29.4,88.1,6,6,270.1,270.1
|
| 810 |
+
06-08 15:51:20,1,1,0.0,269.5,6,0,0.0,269.5,6.0,0.0,29.8,88.1,6,6,269.5,269.5
|
| 811 |
+
06-08 15:51:25,1,1,0.0,269.2,6,0,0.0,269.2,6.0,0.0,30.1,88.1,6,6,269.2,269.2
|
| 812 |
+
06-08 15:51:30,1,1,1.6,469.6,11,0,1.6,469.6,11.0,0.0,57.0,90.8,11,11,469.6,469.6
|
| 813 |
+
06-08 15:51:35,1,1,0.0,315.4,7,0,0.0,315.4,7.0,0.0,36.5,88.2,7,7,315.4,315.4
|
| 814 |
+
06-08 15:51:41,1,1,0.0,706.4,18,0,0.0,706.4,18.0,0.0,97.8,86.5,18,18,706.4,706.4
|
| 815 |
+
06-08 15:51:46,1,1,0.0,680.1,18,0,0.0,680.1,18.0,0.0,98.7,86.5,18,18,680.1,680.1
|
| 816 |
+
06-08 15:51:51,1,1,0.0,659.1,17,0,0.0,659.1,17.0,0.0,90.8,86.5,17,17,659.1,659.1
|
| 817 |
+
06-08 15:51:56,1,1,312.6,674.6,18,0,312.6,674.6,18.0,0.0,92.2,86.3,18,18,674.6,674.6
|
| 818 |
+
06-08 15:52:01,1,1,0.0,686.6,18,0,0.0,686.6,18.0,0.0,93.1,86.3,18,18,686.6,686.6
|
| 819 |
+
06-08 15:52:06,1,1,0.0,685.6,18,0,0.0,685.6,18.0,0.0,94.1,86.3,18,18,685.6,685.6
|
| 820 |
+
06-08 15:52:11,1,1,0.0,683.7,18,0,0.0,683.7,18.0,0.0,95.0,86.3,18,18,683.7,683.7
|
| 821 |
+
06-08 15:52:16,1,1,2.4,712.9,20,0,2.4,712.9,20.0,0.0,96.0,86.4,20,20,712.9,712.9
|
| 822 |
+
06-08 15:52:21,1,1,0.0,729.4,19,0,0.0,729.4,19.0,0.0,96.4,86.4,19,19,729.4,729.4
|
| 823 |
+
06-08 15:52:26,1,1,0.0,705.4,18,0,0.0,705.4,18.0,0.0,92.4,86.4,18,18,705.4,705.4
|
| 824 |
+
06-08 15:52:31,1,1,0.0,653.2,17,0,0.0,653.2,17.0,0.0,85.4,86.4,17,17,653.2,653.2
|
| 825 |
+
06-08 15:52:36,1,1,0.0,647.1,17,0,0.0,647.1,17.0,0.0,86.3,86.4,17,17,647.1,647.1
|
| 826 |
+
06-08 15:52:41,1,1,0.0,644.5,17,0,0.0,644.5,17.0,0.0,87.2,86.4,17,17,644.5,644.5
|
| 827 |
+
06-08 15:52:46,1,1,70.1,672.8,18,0,70.1,672.8,18.0,0.0,88.2,86.3,18,18,672.8,672.8
|
| 828 |
+
06-08 15:52:51,1,1,0.0,684.5,18,0,0.0,684.5,18.0,0.0,89.2,86.3,18,18,684.5,684.5
|
| 829 |
+
06-08 15:52:56,1,1,0.0,684.6,18,0,0.0,684.6,18.0,0.0,90.1,86.3,18,18,684.6,684.6
|
| 830 |
+
06-08 15:53:01,1,1,1.0,690.7,18,0,1.0,690.7,18.0,0.0,90.6,86.4,18,18,690.7,690.7
|
| 831 |
+
06-08 15:53:06,1,1,0.0,684.5,18,0,0.0,684.5,18.0,0.0,91.6,86.4,18,18,684.5,684.5
|
| 832 |
+
06-08 15:53:11,1,1,0.0,682.6,18,0,0.0,682.6,18.0,0.0,92.5,86.4,18,18,682.6,682.6
|
| 833 |
+
06-08 15:53:16,1,1,0.0,683.6,18,0,0.0,683.6,18.0,0.0,93.5,86.4,18,18,683.6,683.6
|
| 834 |
+
06-08 15:53:21,1,1,1545.4,689.4,21,0,1545.4,689.4,21.0,0.0,96.6,85.4,21,21,689.4,689.4
|
| 835 |
+
06-08 15:53:26,1,1,0.0,789.7,21,0,0.0,789.7,21.0,0.0,97.7,85.4,21,21,789.7,789.7
|
| 836 |
+
06-08 15:53:31,1,1,0.0,785.8,21,0,0.0,785.8,21.0,0.0,98.8,85.4,21,21,785.8,785.8
|
| 837 |
+
06-08 15:53:36,1,1,0.0,780.4,21,0,0.0,780.4,21.0,0.0,99.9,85.4,21,21,780.4,780.4
|
| 838 |
+
06-08 15:53:41,1,1,0.0,730.0,19,3,0.0,730.0,19.0,3.0,99.9,62.3,19,19,730.0,730.0
|
| 839 |
+
06-08 15:53:46,1,1,0.0,674.5,18,4,0.0,674.5,18.0,4.0,99.1,62.3,18,18,674.5,674.5
|
| 840 |
+
06-08 15:53:51,1,1,4798.1,554.2,22,0,4798.1,554.2,22.0,0.0,99.8,61.0,22,22,554.2,554.2
|
| 841 |
+
06-08 15:53:56,1,1,0.0,771.7,20,2,0.0,771.7,20.0,2.0,94.9,61.0,20,20,771.7,771.7
|
| 842 |
+
06-08 15:54:01,1,1,0.0,683.6,21,0,0.0,683.6,21.0,0.0,93.0,61.0,21,21,683.6,683.6
|
| 843 |
+
06-08 15:54:06,1,1,0.0,789.5,21,0,0.0,789.5,21.0,0.0,94.1,61.0,21,21,789.5,789.5
|
| 844 |
+
06-08 15:54:11,1,1,0.0,773.7,19,0,0.0,773.7,19.0,0.0,93.5,61.0,19,19,773.7,773.7
|
| 845 |
+
06-08 15:54:16,1,1,0.0,717.5,19,0,0.0,717.5,19.0,0.0,94.5,61.0,19,19,717.5,717.5
|
| 846 |
+
06-08 15:54:21,1,1,0.0,684.4,18,0,0.0,684.4,18.0,0.0,94.6,61.0,18,18,684.4,684.4
|
| 847 |
+
06-08 15:54:26,1,1,0.0,681.0,18,0,0.0,681.0,18.0,0.0,95.6,61.0,18,18,681.0,681.0
|
| 848 |
+
06-08 15:54:31,1,1,0.0,655.9,17,0,0.0,655.9,17.0,0.0,95.0,61.0,17,17,655.9,655.9
|
| 849 |
+
06-08 15:54:36,1,1,0.0,640.9,16,0,0.0,640.9,16.0,0.0,94.9,61.0,16,16,640.9,640.9
|
| 850 |
+
06-08 15:54:41,1,1,0.0,625.5,16,0,0.0,625.5,16.0,0.0,95.7,61.0,16,16,625.5,625.5
|
| 851 |
+
06-08 15:54:46,1,1,0.0,621.0,16,0,0.0,621.0,16.0,0.0,96.6,61.0,16,16,621.0,621.0
|
| 852 |
+
06-08 15:54:51,1,1,1.4,628.7,17,0,1.4,628.7,17.0,0.0,97.5,61.0,17,17,628.7,628.7
|
| 853 |
+
06-08 15:54:56,1,1,0.0,637.1,17,0,0.0,637.1,17.0,0.0,98.4,61.0,17,17,637.1,637.1
|
| 854 |
+
06-08 15:55:01,1,1,0.0,633.0,17,0,0.0,633.0,17.0,0.0,99.3,61.0,17,17,633.0,633.0
|
| 855 |
+
06-08 15:55:06,1,1,0.0,628.6,16,1,0.0,628.6,16.0,1.0,100.0,61.0,16,16,628.6,628.6
|
| 856 |
+
06-08 15:55:11,1,1,0.0,605.6,15,2,0.0,605.6,15.0,2.0,94.8,61.0,15,15,605.6,605.6
|
| 857 |
+
06-08 15:55:16,1,1,0.0,577.9,16,0,0.0,577.9,16.0,0.0,92.9,61.0,16,16,577.9,577.9
|
| 858 |
+
06-08 15:55:21,1,1,648.3,632.8,18,0,648.3,632.8,18.0,0.0,94.6,60.9,18,18,632.8,632.8
|
| 859 |
+
06-08 15:55:26,1,1,0.8,666.9,17,0,0.8,666.9,17.0,0.0,85.8,61.1,17,17,666.9,666.9
|
| 860 |
+
06-08 15:55:31,1,1,996.1,645.6,18,0,996.1,645.6,18.0,0.0,88.1,60.8,18,18,645.6,645.6
|
| 861 |
+
06-08 15:55:36,1,1,0.0,683.8,18,0,0.0,683.8,18.0,0.0,89.0,60.8,18,18,683.8,683.8
|
| 862 |
+
06-08 15:55:41,1,1,0.0,683.7,18,0,0.0,683.7,18.0,0.0,90.0,60.8,18,18,683.7,683.7
|
| 863 |
+
06-08 15:55:46,1,1,703.1,698.7,19,0,703.1,698.7,19.0,0.0,91.9,60.7,19,19,698.7,698.7
|
| 864 |
+
06-08 15:55:51,1,1,0.0,689.7,18,0,0.0,689.7,18.0,0.0,91.9,60.7,18,18,689.7,689.7
|
| 865 |
+
06-08 15:55:56,1,1,938.5,666.2,20,0,938.5,666.2,20.0,0.0,95.1,60.6,20,20,666.2,666.2
|
| 866 |
+
06-08 15:56:01,1,1,0.0,739.5,20,0,0.0,739.5,20.0,0.0,96.1,60.6,20,20,739.5,739.5
|
| 867 |
+
06-08 15:56:06,1,1,1176.4,699.0,20,0,1176.4,699.0,20.0,0.0,91.3,60.3,20,20,699.0,699.0
|
| 868 |
+
06-08 15:56:11,1,1,0.0,722.0,18,0,0.0,722.0,18.0,0.0,82.2,60.3,18,18,722.0,722.0
|
| 869 |
+
06-08 15:56:16,1,1,699.9,696.7,19,0,699.9,696.7,19.0,0.0,84.3,60.1,19,19,696.7,696.7
|
| 870 |
+
06-08 15:56:21,1,1,3.8,751.1,21,0,3.8,751.1,21.0,0.0,87.3,60.6,21,21,751.1,751.1
|
| 871 |
+
06-08 15:56:26,1,1,536.7,764.8,21,0,536.7,764.8,21.0,0.0,88.0,60.4,21,21,764.8,764.8
|
| 872 |
+
06-08 15:56:32,1,1,1.8,327.0,8,0,1.8,327.0,8.0,0.0,51.0,91.4,8,8,327.0,327.0
|
| 873 |
+
06-08 15:56:37,1,1,0.0,370.4,7,0,0.0,370.4,7.0,0.0,42.6,91.4,7,7,370.4,370.4
|
| 874 |
+
06-08 15:56:42,1,1,0.0,309.3,6,0,0.0,309.3,6.0,0.0,34.2,91.4,6,6,309.3,309.3
|
| 875 |
+
06-08 15:56:47,1,1,3.6,309.3,7,0,3.6,309.3,7.0,0.0,26.4,91.5,7,7,309.3,309.3
|
| 876 |
+
06-08 15:56:52,1,1,0.0,328.1,7,0,0.0,328.1,7.0,0.0,26.9,91.5,7,7,328.1,328.1
|
| 877 |
+
06-08 15:56:57,1,1,3.2,338.7,8,0,3.2,338.7,8.0,0.0,27.4,91.5,8,8,338.7,338.7
|
| 878 |
+
06-08 15:57:02,1,1,0.0,373.5,8,0,0.0,373.5,8.0,0.0,27.9,91.5,8,8,373.5,373.5
|
| 879 |
+
06-08 15:57:07,1,1,0.0,374.3,8,0,0.0,374.3,8.0,0.0,28.4,91.5,8,8,374.3,374.3
|
| 880 |
+
06-08 15:57:12,1,1,0.0,356.2,6,0,0.0,356.2,6.0,0.0,12.3,91.5,6,6,356.2,356.2
|
| 881 |
+
06-08 15:57:17,1,1,0.0,280.0,6,0,0.0,280.0,6.0,0.0,12.7,91.5,6,6,280.0,280.0
|
| 882 |
+
06-08 15:57:22,1,1,0.0,281.3,6,0,0.0,281.3,6.0,0.0,13.1,91.5,6,6,281.3,281.3
|
| 883 |
+
06-08 15:57:27,1,1,0.0,295.5,6,0,0.0,295.5,6.0,0.0,15.9,89.7,6,6,295.5,295.5
|
| 884 |
+
06-08 15:57:32,1,1,0.0,273.1,5,0,0.0,273.1,5.0,0.0,15.3,89.7,5,5,273.1,273.1
|
| 885 |
+
06-08 15:57:37,1,1,0.0,236.4,5,0,0.0,236.4,5.0,0.0,15.6,89.7,5,5,236.4,236.4
|
| 886 |
+
06-08 15:57:42,1,1,0.0,180.4,3,0,0.0,180.4,3.0,0.0,16.4,89.1,3,3,180.4,180.4
|
| 887 |
+
06-08 15:57:47,1,1,0.0,138.4,3,0,0.0,138.4,3.0,0.0,16.6,89.1,3,3,138.4,138.4
|
| 888 |
+
06-08 15:57:52,1,1,0.0,137.8,3,0,0.0,137.8,3.0,0.0,16.8,89.1,3,3,137.8,137.8
|
| 889 |
+
06-08 15:57:57,1,1,0.0,138.1,3,0,0.0,138.1,3.0,0.0,17.0,89.1,3,3,138.1,138.1
|
| 890 |
+
06-08 15:58:02,1,1,0.0,137.9,3,0,0.0,137.9,3.0,0.0,17.2,89.1,3,3,137.9,137.9
|
| 891 |
+
06-08 15:58:07,1,1,0.0,137.8,3,0,0.0,137.8,3.0,0.0,17.3,89.1,3,3,137.8,137.8
|
| 892 |
+
06-08 15:58:12,1,1,0.0,137.4,3,0,0.0,137.4,3.0,0.0,17.5,89.1,3,3,137.4,137.4
|
| 893 |
+
06-08 15:58:17,1,1,0.0,137.7,3,0,0.0,137.7,3.0,0.0,17.7,89.1,3,3,137.7,137.7
|
| 894 |
+
06-08 15:58:22,1,1,0.0,137.7,3,0,0.0,137.7,3.0,0.0,17.9,89.1,3,3,137.7,137.7
|
| 895 |
+
06-08 15:58:27,1,1,0.0,441.9,10,0,0.0,441.9,10.0,0.0,38.8,90.0,10,10,441.9,441.9
|
| 896 |
+
06-08 15:58:32,1,1,0.0,441.1,10,0,0.0,441.1,10.0,0.0,39.4,90.0,10,10,441.1,441.1
|
| 897 |
+
06-08 15:58:37,1,1,0.0,137.6,3,0,0.0,137.6,3.0,0.0,18.5,89.1,3,3,137.6,137.6
|
| 898 |
+
06-08 15:58:42,1,1,0.0,137.6,3,0,0.0,137.6,3.0,0.0,18.7,89.1,3,3,137.6,137.6
|
| 899 |
+
06-08 15:58:47,1,1,0.0,137.5,3,0,0.0,137.5,3.0,0.0,18.9,89.1,3,3,137.5,137.5
|
| 900 |
+
06-08 15:58:52,1,1,0.0,137.9,3,0,0.0,137.9,3.0,0.0,19.1,89.1,3,3,137.9,137.9
|
| 901 |
+
06-08 15:58:57,1,1,0.0,440.6,10,0,0.0,440.6,10.0,0.0,42.5,90.0,10,10,440.6,440.6
|
| 902 |
+
06-08 15:59:02,1,1,0.0,440.8,10,0,0.0,440.8,10.0,0.0,43.1,90.0,10,10,440.8,440.8
|
| 903 |
+
06-08 15:59:07,1,1,0.0,440.6,10,0,0.0,440.6,10.0,0.0,43.7,90.0,10,10,440.6,440.6
|
| 904 |
+
06-08 15:59:12,1,1,0.0,439.7,10,0,0.0,439.7,10.0,0.0,44.3,90.0,10,10,439.7,439.7
|
| 905 |
+
06-08 15:59:17,1,1,0.0,410.2,9,0,0.0,410.2,9.0,0.0,41.8,90.0,9,9,410.2,410.2
|
| 906 |
+
06-08 15:59:22,1,1,0.0,397.6,9,0,0.0,397.6,9.0,0.0,42.3,90.0,9,9,397.6,397.6
|
| 907 |
+
06-08 15:59:27,1,1,2.8,425.8,10,0,2.8,425.8,10.0,0.0,42.9,90.0,10,10,425.8,425.8
|
| 908 |
+
06-08 15:59:32,1,1,0.0,440.3,10,0,0.0,440.3,10.0,0.0,43.5,90.0,10,10,440.3,440.3
|
| 909 |
+
06-08 15:59:37,1,1,0.0,440.3,10,0,0.0,440.3,10.0,0.0,44.2,90.0,10,10,440.3,440.3
|
| 910 |
+
06-08 15:59:42,1,1,0.0,441.4,10,0,0.0,441.4,10.0,0.0,44.8,90.0,10,10,441.4,441.4
|
| 911 |
+
06-08 15:59:47,1,1,0.0,440.3,10,0,0.0,440.3,10.0,0.0,45.4,90.0,10,10,440.3,440.3
|
| 912 |
+
06-08 15:59:52,1,1,1.0,453.2,11,0,1.0,453.2,11.0,0.0,46.8,90.0,11,11,453.2,453.2
|
| 913 |
+
06-08 15:59:57,1,1,2.0,519.5,12,0,2.0,519.5,12.0,0.0,47.6,90.0,12,12,519.5,519.5
|
| 914 |
+
06-08 16:00:02,1,1,0.0,528.7,12,0,0.0,528.7,12.0,0.0,48.3,90.0,12,12,528.7,528.7
|
| 915 |
+
06-08 16:00:07,1,1,92.9,538.3,13,0,92.9,538.3,13.0,0.0,51.3,90.1,13,13,538.3,538.3
|
| 916 |
+
06-08 16:00:12,1,1,0.0,572.4,13,0,0.0,572.4,13.0,0.0,52.1,90.1,13,13,572.4,572.4
|
| 917 |
+
06-08 16:00:17,1,1,0.0,570.1,13,0,0.0,570.1,13.0,0.0,52.9,90.1,13,13,570.1,570.1
|
| 918 |
+
06-08 16:00:22,1,1,0.0,570.0,13,0,0.0,570.0,13.0,0.0,53.7,90.1,13,13,570.0,570.0
|
| 919 |
+
06-08 16:00:27,1,1,0.0,570.0,13,0,0.0,570.0,13.0,0.0,54.5,90.1,13,13,570.0,570.0
|
| 920 |
+
06-08 16:00:33,1,1,0.0,607.6,14,0,0.0,607.6,14.0,0.0,78.9,84.8,14,14,607.6,607.6
|
| 921 |
+
06-08 16:00:38,1,1,0.0,609.6,14,0,0.0,609.6,14.0,0.0,79.7,84.8,14,14,609.6,609.6
|
| 922 |
+
06-08 16:00:43,1,1,0.0,589.4,13,0,0.0,589.4,13.0,0.0,80.3,84.8,13,13,589.4,589.4
|
| 923 |
+
06-08 16:00:48,1,1,9.4,573.3,14,0,9.4,573.3,14.0,0.0,81.2,84.8,14,14,573.3,573.3
|
| 924 |
+
06-08 16:00:53,1,1,59.0,613.4,15,0,59.0,613.4,15.0,0.0,82.3,84.8,15,15,613.4,613.4
|
| 925 |
+
06-08 16:00:58,1,1,0.0,629.9,15,0,0.0,629.9,15.0,0.0,83.2,84.8,15,15,629.9,629.9
|
| 926 |
+
06-08 16:01:03,1,1,117.3,604.9,14,0,117.3,604.9,14.0,0.0,83.6,84.7,14,14,604.9,604.9
|
| 927 |
+
06-08 16:01:08,1,1,53.6,602.1,15,0,53.6,602.1,15.0,0.0,84.9,84.8,15,15,602.1,602.1
|
| 928 |
+
06-08 16:01:13,1,1,61.3,615.3,16,0,61.3,615.3,16.0,0.0,86.0,84.8,16,16,615.3,615.3
|
| 929 |
+
06-08 16:01:18,1,1,1.4,620.0,16,0,1.4,620.0,16.0,0.0,86.4,84.9,16,16,620.0,620.0
|
| 930 |
+
06-08 16:01:23,1,1,76.3,649.0,16,0,76.3,649.0,16.0,0.0,87.5,84.9,16,16,649.0,649.0
|
| 931 |
+
06-08 16:01:28,1,1,84.5,646.5,17,0,84.5,646.5,17.0,0.0,88.9,84.9,17,17,646.5,646.5
|
| 932 |
+
06-08 16:01:33,1,1,0.0,631.1,15,0,0.0,631.1,15.0,0.0,88.5,84.9,15,15,631.1,631.1
|
| 933 |
+
06-08 16:01:38,1,1,44.6,637.0,17,0,44.6,637.0,17.0,0.0,90.8,85.1,17,17,637.0,637.0
|
| 934 |
+
06-08 16:01:43,1,1,17.6,430.8,10,0,17.6,430.8,10.0,0.0,42.3,89.3,10,10,430.8,430.8
|
| 935 |
+
06-08 16:01:48,1,1,0.0,337.8,7,0,0.0,337.8,7.0,0.0,22.6,88.1,7,7,337.8,337.8
|
| 936 |
+
06-08 16:01:53,1,1,0.0,327.7,6,0,0.0,327.7,6.0,0.0,22.7,88.1,6,6,327.7,327.7
|
| 937 |
+
06-08 16:01:58,1,1,0.0,268.4,5,0,0.0,268.4,5.0,0.0,21.9,88.1,5,5,268.4,268.4
|
| 938 |
+
06-08 16:02:03,1,1,13.0,274.9,6,0,13.0,274.9,6.0,0.0,22.7,88.1,6,6,274.9,274.9
|
| 939 |
+
06-08 16:02:08,1,1,0.0,311.9,7,0,0.0,311.9,7.0,0.0,27.2,86.5,7,7,311.9,311.9
|
| 940 |
+
06-08 16:02:13,1,1,0.0,312.1,7,0,0.0,312.1,7.0,0.0,27.7,86.5,7,7,312.1,312.1
|
| 941 |
+
06-08 16:02:18,1,1,0.0,312.5,7,0,0.0,312.5,7.0,0.0,28.1,86.5,7,7,312.5,312.5
|
| 942 |
+
06-08 16:02:23,1,1,98.2,348.2,8,0,98.2,348.2,8.0,0.0,31.4,86.6,8,8,348.2,348.2
|
| 943 |
+
06-08 16:02:28,1,1,0.0,356.2,8,0,0.0,356.2,8.0,0.0,31.9,86.6,8,8,356.2,356.2
|
| 944 |
+
06-08 16:02:33,1,1,0.0,244.7,5,0,0.0,244.7,5.0,0.0,31.2,88.6,5,5,244.7,244.7
|
| 945 |
+
06-08 16:02:38,1,1,0.0,243.9,5,0,0.0,243.9,5.0,0.0,31.6,88.6,5,5,243.9,243.9
|
| 946 |
+
06-08 16:02:43,1,1,0.0,244.7,5,0,0.0,244.7,5.0,0.0,31.9,88.6,5,5,244.7,244.7
|
| 947 |
+
06-08 16:02:48,1,1,0.0,244.1,5,0,0.0,244.1,5.0,0.0,32.3,88.6,5,5,244.1,244.1
|
| 948 |
+
06-08 16:02:53,1,1,0.0,244.0,5,0,0.0,244.0,5.0,0.0,32.6,88.6,5,5,244.0,244.0
|
| 949 |
+
06-08 16:02:58,1,1,0.0,243.9,5,0,0.0,243.9,5.0,0.0,32.9,88.6,5,5,243.9,243.9
|
| 950 |
+
06-08 16:03:03,1,1,0.0,244.5,5,0,0.0,244.5,5.0,0.0,33.3,88.6,5,5,244.5,244.5
|
| 951 |
+
06-08 16:03:08,1,1,0.0,232.0,4,0,0.0,232.0,4.0,0.0,24.7,88.6,4,4,232.0,232.0
|
| 952 |
+
06-08 16:03:13,1,1,0.0,195.6,4,0,0.0,195.6,4.0,0.0,24.9,88.6,4,4,195.6,195.6
|
| 953 |
+
06-08 16:03:18,1,1,0.0,194.8,4,0,0.0,194.8,4.0,0.0,25.2,88.6,4,4,194.8,194.8
|
| 954 |
+
06-08 16:03:23,1,1,0.0,194.8,4,0,0.0,194.8,4.0,0.0,25.5,88.6,4,4,194.8,194.8
|
| 955 |
+
06-08 16:03:28,1,1,0.0,194.9,4,0,0.0,194.9,4.0,0.0,25.7,88.6,4,4,194.9,194.9
|
| 956 |
+
06-08 16:03:33,1,1,0.0,421.2,10,0,0.0,421.2,10.0,0.0,52.9,89.1,10,10,421.2,421.2
|
| 957 |
+
06-08 16:03:38,1,1,0.0,412.9,9,0,0.0,412.9,9.0,0.0,44.5,89.1,9,9,412.9,412.9
|
| 958 |
+
06-08 16:03:43,1,1,0.0,380.3,9,0,0.0,380.3,9.0,0.0,45.0,89.1,9,9,380.3,380.3
|
| 959 |
+
06-08 16:03:48,1,1,7.6,421.2,10,0,7.6,421.2,10.0,0.0,45.7,89.1,10,10,421.2,421.2
|
| 960 |
+
06-08 16:03:53,1,1,0.0,423.1,10,0,0.0,423.1,10.0,0.0,46.3,89.1,10,10,423.1,423.1
|
| 961 |
+
06-08 16:03:58,1,1,0.0,422.9,10,0,0.0,422.9,10.0,0.0,46.9,89.1,10,10,422.9,422.9
|
| 962 |
+
06-08 16:04:03,1,1,9.8,438.5,11,0,9.8,438.5,11.0,0.0,47.6,89.1,11,11,438.5,438.5
|
| 963 |
+
06-08 16:04:08,1,1,0.0,195.8,4,0,0.0,195.8,4.0,0.0,27.9,88.6,4,4,195.8,195.8
|
| 964 |
+
06-08 16:04:14,1,1,0.0,423.8,10,0,0.0,423.8,10.0,0.0,48.4,89.1,10,10,423.8,423.8
|
| 965 |
+
06-08 16:04:19,1,1,0.0,423.0,10,0,0.0,423.0,10.0,0.0,49.0,89.1,10,10,423.0,423.0
|
| 966 |
+
06-08 16:04:24,1,1,0.0,423.4,10,0,0.0,423.4,10.0,0.0,49.6,89.1,10,10,423.4,423.4
|
| 967 |
+
06-08 16:04:29,1,1,128.0,435.2,11,0,128.0,435.2,11.0,0.0,50.8,89.0,11,11,435.2,435.2
|
| 968 |
+
06-08 16:04:34,1,1,0.0,470.8,11,0,0.0,470.8,11.0,0.0,56.7,88.3,11,11,470.8,470.8
|
| 969 |
+
06-08 16:04:39,1,1,0.0,471.0,11,0,0.0,471.0,11.0,0.0,57.4,88.3,11,11,471.0,471.0
|
| 970 |
+
06-08 16:04:44,1,1,415.4,490.3,12,0,415.4,490.3,12.0,0.0,58.8,88.1,12,12,490.3,490.3
|
| 971 |
+
06-08 16:04:49,1,1,0.0,513.8,12,0,0.0,513.8,12.0,0.0,59.5,88.1,12,12,513.8,513.8
|
| 972 |
+
06-08 16:04:54,1,1,0.0,643.8,18,0,0.0,643.8,18.0,0.0,80.5,62.7,18,18,643.8,643.8
|
| 973 |
+
06-08 16:04:59,1,1,1.2,669.1,19,0,1.2,669.1,19.0,0.0,81.4,62.8,19,19,669.1,669.1
|
| 974 |
+
06-08 16:05:04,1,1,0.0,657.2,18,0,0.0,657.2,18.0,0.0,81.4,62.8,18,18,657.2,657.2
|
| 975 |
+
06-08 16:05:09,1,1,0.0,645.1,18,0,0.0,645.1,18.0,0.0,82.3,62.8,18,18,645.1,645.1
|
| 976 |
+
06-08 16:05:14,1,1,0.0,644.1,18,0,0.0,644.1,18.0,0.0,83.2,62.8,18,18,644.1,644.1
|
| 977 |
+
06-08 16:05:19,1,1,0.0,644.2,18,0,0.0,644.2,18.0,0.0,84.1,62.8,18,18,644.2,644.2
|
| 978 |
+
06-08 16:05:24,1,1,0.0,644.2,18,0,0.0,644.2,18.0,0.0,85.0,62.8,18,18,644.2,644.2
|
| 979 |
+
06-08 16:05:29,1,1,0.0,644.0,18,0,0.0,644.0,18.0,0.0,85.9,62.8,18,18,644.0,644.0
|
| 980 |
+
06-08 16:05:34,1,1,0.0,645.0,18,0,0.0,645.0,18.0,0.0,86.8,62.8,18,18,645.0,645.0
|
| 981 |
+
06-08 16:05:39,1,1,0.0,645.4,18,0,0.0,645.4,18.0,0.0,87.7,62.8,18,18,645.4,645.4
|
| 982 |
+
06-08 16:05:44,1,1,0.0,645.1,18,0,0.0,645.1,18.0,0.0,88.6,62.8,18,18,645.1,645.1
|
| 983 |
+
06-08 16:05:49,1,1,0.0,426.8,10,0,0.0,426.8,10.0,0.0,48.2,88.1,10,10,426.8,426.8
|
| 984 |
+
06-08 16:05:54,1,1,9.8,661.0,19,0,9.8,661.0,19.0,0.0,90.4,62.8,19,19,661.0,661.0
|
| 985 |
+
06-08 16:05:59,1,1,194.9,690.5,20,0,194.9,690.5,20.0,0.0,92.6,63.0,20,20,690.5,690.5
|
| 986 |
+
06-08 16:06:04,1,1,0.0,711.2,20,0,0.0,711.2,20.0,0.0,93.6,63.0,20,20,711.2,711.2
|
| 987 |
+
06-08 16:06:09,1,1,9.0,722.2,21,0,9.0,722.2,21.0,0.0,94.6,63.0,21,21,722.2,722.2
|
| 988 |
+
06-08 16:06:14,1,1,21.9,802.4,23,0,21.9,802.4,23.0,0.0,95.8,63.1,23,23,802.4,802.4
|
| 989 |
+
06-08 16:06:19,1,1,67.4,844.0,24,0,67.4,844.0,24.0,0.0,97.0,63.1,24,24,844.0,844.0
|
| 990 |
+
06-08 16:06:24,1,1,0.0,852.7,24,1,0.0,852.7,24.0,1.0,98.2,63.1,24,24,852.7,852.7
|
| 991 |
+
06-08 16:06:29,1,1,9.2,847.9,24,0,9.2,847.9,24.0,0.0,90.6,63.2,24,24,847.9,847.9
|
| 992 |
+
06-08 16:06:34,1,1,0.0,849.7,24,2,0.0,849.7,24.0,2.0,91.8,63.2,24,24,849.7,849.7
|
| 993 |
+
06-08 16:06:39,1,1,0.0,849.5,24,2,0.0,849.5,24.0,2.0,93.0,63.2,24,24,849.5,849.5
|
| 994 |
+
06-08 16:06:44,1,1,9.6,847.9,24,2,9.6,847.9,24.0,2.0,92.5,63.2,24,24,847.9,847.9
|
| 995 |
+
06-08 16:06:49,1,1,0.0,339.1,8,0,0.0,339.1,8.0,0.0,22.0,89.8,8,8,339.1,339.1
|
| 996 |
+
06-08 16:06:54,1,1,0.0,338.4,8,0,0.0,338.4,8.0,0.0,22.5,89.8,8,8,338.4,338.4
|
| 997 |
+
06-08 16:06:59,1,1,18.9,845.6,24,0,18.9,845.6,24.0,0.0,95.0,63.4,24,24,845.6,845.6
|
| 998 |
+
06-08 16:07:04,1,1,0.0,295.9,7,0,0.0,295.9,7.0,0.0,19.3,88.1,7,7,295.9,295.9
|
| 999 |
+
06-08 16:07:09,1,1,0.0,295.7,7,0,0.0,295.7,7.0,0.0,19.7,88.1,7,7,295.7,295.7
|
| 1000 |
+
06-08 16:07:14,1,1,0.0,289.8,6,0,0.0,289.8,6.0,0.0,19.7,88.1,6,6,289.8,289.8
|
| 1001 |
+
06-08 16:07:19,1,1,149.2,266.1,6,0,149.2,266.1,6.0,0.0,12.7,88.1,6,6,266.1,266.1
|
| 1002 |
+
06-08 16:07:24,1,1,0.0,255.4,6,0,0.0,255.4,6.0,0.0,13.1,88.1,6,6,255.4,255.4
|
| 1003 |
+
06-08 16:07:29,1,1,8.2,296.2,7,0,8.2,296.2,7.0,0.0,13.6,88.1,7,7,296.2,296.2
|
| 1004 |
+
06-08 16:07:34,1,1,0.0,281.3,6,0,0.0,281.3,6.0,0.0,12.4,88.1,6,6,281.3,281.3
|
| 1005 |
+
06-08 16:07:39,1,1,0.0,255.1,6,0,0.0,255.1,6.0,0.0,12.7,88.1,6,6,255.1,255.1
|
| 1006 |
+
06-08 16:07:44,1,1,0.0,254.6,6,0,0.0,254.6,6.0,0.0,13.1,88.1,6,6,254.6,254.6
|
| 1007 |
+
06-08 16:07:49,1,1,337.1,327.2,8,0,337.1,327.2,8.0,0.0,15.1,88.0,8,8,327.2,327.2
|
| 1008 |
+
06-08 16:07:54,1,1,0.0,314.0,7,0,0.0,314.0,7.0,0.0,14.8,88.0,7,7,314.0,314.0
|
| 1009 |
+
06-08 16:07:59,1,1,0.0,294.7,7,0,0.0,294.7,7.0,0.0,15.2,88.0,7,7,294.7,294.7
|
| 1010 |
+
06-08 16:08:04,1,1,0.0,294.5,7,0,0.0,294.5,7.0,0.0,15.7,88.0,7,7,294.5,294.5
|
| 1011 |
+
06-08 16:08:09,1,1,0.0,295.8,7,0,0.0,295.8,7.0,0.0,16.1,88.0,7,7,295.8,295.8
|
| 1012 |
+
06-08 16:08:14,1,1,0.0,295.4,7,0,0.0,295.4,7.0,0.0,16.5,88.0,7,7,295.4,295.4
|
| 1013 |
+
06-08 16:08:19,1,1,0.0,295.0,7,0,0.0,295.0,7.0,0.0,16.9,88.0,7,7,295.0,295.0
|
| 1014 |
+
06-08 16:08:25,1,1,0.0,294.6,7,0,0.0,294.6,7.0,0.0,17.3,88.0,7,7,294.6,294.6
|
| 1015 |
+
06-08 16:08:30,1,1,0.0,294.8,7,0,0.0,294.8,7.0,0.0,17.7,88.0,7,7,294.8,294.8
|
| 1016 |
+
06-08 16:08:35,1,1,0.0,295.3,7,0,0.0,295.3,7.0,0.0,18.1,88.0,7,7,295.3,295.3
|
| 1017 |
+
06-08 16:08:40,1,1,0.0,295.0,7,0,0.0,295.0,7.0,0.0,18.5,88.0,7,7,295.0,295.0
|
| 1018 |
+
06-08 16:08:45,1,1,42.8,308.6,8,0,42.8,308.6,8.0,0.0,19.7,88.1,8,8,308.6,308.6
|
| 1019 |
+
06-08 16:08:50,1,1,0.0,337.8,8,0,0.0,337.8,8.0,0.0,20.2,88.1,8,8,337.8,337.8
|
| 1020 |
+
06-08 16:08:55,1,1,0.0,337.3,8,0,0.0,337.3,8.0,0.0,20.7,88.1,8,8,337.3,337.3
|
| 1021 |
+
06-08 16:09:00,1,1,0.0,337.9,8,0,0.0,337.9,8.0,0.0,21.1,88.1,8,8,337.9,337.9
|
| 1022 |
+
06-08 16:09:05,1,1,0.0,299.3,7,0,0.0,299.3,7.0,0.0,20.6,88.1,7,7,299.3,299.3
|
| 1023 |
+
06-08 16:09:10,1,1,0.0,295.2,7,0,0.0,295.2,7.0,0.0,21.0,88.1,7,7,295.2,295.2
|
| 1024 |
+
06-08 16:09:15,1,1,0.0,295.4,7,0,0.0,295.4,7.0,0.0,21.4,88.1,7,7,295.4,295.4
|
| 1025 |
+
06-08 16:09:20,1,1,211.5,659.0,14,0,211.5,659.0,14.0,0.0,50.3,88.0,14,14,659.0,659.0
|
| 1026 |
+
06-08 16:09:25,1,1,0.0,600.1,13,0,0.0,600.1,13.0,0.0,50.6,88.0,13,13,600.1,600.1
|
| 1027 |
+
06-08 16:09:30,1,1,101.0,579.7,13,0,101.0,579.7,13.0,0.0,50.8,88.0,13,13,579.7,579.7
|
| 1028 |
+
06-08 16:09:35,1,1,0.0,569.9,12,0,0.0,569.9,12.0,0.0,51.0,88.0,12,12,569.9,569.9
|
| 1029 |
+
06-08 16:09:40,1,1,0.0,548.2,12,0,0.0,548.2,12.0,0.0,51.8,88.0,12,12,548.2,548.2
|
| 1030 |
+
06-08 16:09:45,1,1,0.0,397.2,8,0,0.0,397.2,8.0,0.0,35.1,90.5,8,8,397.2,397.2
|
| 1031 |
+
06-08 16:09:50,1,1,0.0,373.6,8,0,0.0,373.6,8.0,0.0,35.6,90.5,8,8,373.6,373.6
|
| 1032 |
+
06-08 16:09:55,1,1,0.0,374.2,8,0,0.0,374.2,8.0,0.0,36.1,90.5,8,8,374.2,374.2
|
| 1033 |
+
06-08 16:10:00,1,1,0.0,342.8,8,0,0.0,342.8,8.0,0.0,25.0,89.4,8,8,342.8,342.8
|
| 1034 |
+
06-08 16:10:05,1,1,0.0,342.3,8,0,0.0,342.3,8.0,0.0,25.5,89.4,8,8,342.3,342.3
|
| 1035 |
+
06-08 16:10:10,1,1,0.0,342.5,8,0,0.0,342.5,8.0,0.0,25.9,89.4,8,8,342.5,342.5
|
| 1036 |
+
06-08 16:10:15,1,1,0.0,341.8,8,0,0.0,341.8,8.0,0.0,26.4,89.4,8,8,341.8,341.8
|
| 1037 |
+
06-08 16:10:20,1,1,0.0,340.5,8,0,0.0,340.5,8.0,0.0,26.9,89.4,8,8,340.5,340.5
|
| 1038 |
+
06-08 16:10:25,1,1,0.0,341.5,8,0,0.0,341.5,8.0,0.0,27.4,89.4,8,8,341.5,341.5
|
| 1039 |
+
06-08 16:10:30,1,1,59.2,247.1,6,0,59.2,247.1,6.0,0.0,24.6,91.6,6,6,247.1,247.1
|
| 1040 |
+
06-08 16:10:35,1,1,0.0,212.6,4,0,0.0,212.6,4.0,0.0,15.6,91.6,4,4,212.6,212.6
|
| 1041 |
+
06-08 16:10:40,1,1,0.0,179.0,4,0,0.0,179.0,4.0,0.0,15.8,91.6,4,4,179.0,179.0
|
| 1042 |
+
06-08 16:10:45,1,1,0.0,179.2,4,0,0.0,179.2,4.0,0.0,16.1,91.6,4,4,179.2,179.2
|
| 1043 |
+
06-08 16:10:50,1,1,0.0,179.1,4,0,0.0,179.1,4.0,0.0,16.3,91.6,4,4,179.1,179.1
|
| 1044 |
+
06-08 16:10:56,1,1,0.0,415.9,8,0,0.0,415.9,8.0,0.0,45.3,90.1,8,8,415.9,415.9
|
| 1045 |
+
06-08 16:11:01,1,1,0.0,327.4,6,0,0.0,327.4,6.0,0.0,34.1,90.1,6,6,327.4,327.4
|
| 1046 |
+
06-08 16:11:06,1,1,0.0,287.1,6,0,0.0,287.1,6.0,0.0,34.5,90.1,6,6,287.1,287.1
|
| 1047 |
+
06-08 16:11:11,1,1,0.0,250.2,5,0,0.0,250.2,5.0,0.0,25.9,90.1,5,5,250.2,250.2
|
| 1048 |
+
06-08 16:11:16,1,1,0.0,239.6,5,0,0.0,239.6,5.0,0.0,26.2,90.1,5,5,239.6,239.6
|
| 1049 |
+
06-08 16:11:21,1,1,0.0,239.7,5,0,0.0,239.7,5.0,0.0,26.5,90.1,5,5,239.7,239.7
|
| 1050 |
+
06-08 16:11:26,1,1,83.3,241.3,6,0,83.3,241.3,6.0,0.0,30.0,90.2,6,6,241.3,241.3
|
| 1051 |
+
06-08 16:11:31,1,1,0.0,288.7,6,0,0.0,288.7,6.0,0.0,30.4,90.2,6,6,288.7,288.7
|
| 1052 |
+
06-08 16:11:36,1,1,0.0,292.4,6,0,0.0,292.4,6.0,0.0,30.8,90.2,6,6,292.4,292.4
|
| 1053 |
+
06-08 16:11:41,1,1,57.3,441.8,10,0,57.3,441.8,10.0,0.0,43.4,88.2,10,10,441.8,441.8
|
| 1054 |
+
06-08 16:11:46,1,1,0.0,423.6,9,0,0.0,423.6,9.0,0.0,43.4,88.2,9,9,423.6,423.6
|
| 1055 |
+
06-08 16:11:51,1,1,0.0,200.1,4,0,0.0,200.1,4.0,0.0,17.6,89.7,4,4,200.1,200.1
|
| 1056 |
+
06-08 16:11:56,1,1,0.0,170.1,4,0,0.0,170.1,4.0,0.0,17.8,89.7,4,4,170.1,170.1
|
| 1057 |
+
06-08 16:12:01,1,1,32.7,183.4,5,0,32.7,183.4,5.0,0.0,19.2,89.7,5,5,183.4,183.4
|
| 1058 |
+
06-08 16:12:06,1,1,0.0,206.8,4,0,0.0,206.8,4.0,0.0,18.3,89.7,4,4,206.8,206.8
|
| 1059 |
+
06-08 16:12:11,1,1,0.0,460.8,11,0,0.0,460.8,11.0,0.0,44.2,89.7,11,11,460.8,460.8
|
| 1060 |
+
06-08 16:12:16,1,1,0.0,460.2,11,0,0.0,460.2,11.0,0.0,44.8,89.7,11,11,460.2,460.2
|
| 1061 |
+
06-08 16:12:21,1,1,204.0,471.9,12,0,204.0,471.9,12.0,0.0,46.4,89.6,12,12,471.9,471.9
|
| 1062 |
+
06-08 16:12:26,1,1,0.0,502.0,12,0,0.0,502.0,12.0,0.0,47.1,89.6,12,12,502.0,502.0
|
| 1063 |
+
06-08 16:12:31,1,1,0.0,460.8,11,0,0.0,460.8,11.0,0.0,38.8,89.6,11,11,460.8,460.8
|
| 1064 |
+
06-08 16:12:36,1,1,0.0,458.7,11,0,0.0,458.7,11.0,0.0,39.5,89.6,11,11,458.7,458.7
|
| 1065 |
+
06-08 16:12:41,1,1,0.0,458.0,11,0,0.0,458.0,11.0,0.0,40.1,89.6,11,11,458.0,458.0
|
| 1066 |
+
06-08 16:12:46,1,1,0.0,458.8,11,0,0.0,458.8,11.0,0.0,40.8,89.6,11,11,458.8,458.8
|
| 1067 |
+
06-08 16:12:51,1,1,0.0,459.4,11,0,0.0,459.4,11.0,0.0,41.4,89.6,11,11,459.4,459.4
|
| 1068 |
+
06-08 16:12:56,1,1,0.0,459.0,11,0,0.0,459.0,11.0,0.0,42.0,89.6,11,11,459.0,459.0
|
| 1069 |
+
06-08 16:13:02,1,1,57.9,286.3,7,0,57.9,286.3,7.0,0.0,38.8,89.1,7,7,286.3,286.3
|
| 1070 |
+
06-08 16:13:07,1,1,1.8,350.5,8,0,1.8,350.5,8.0,0.0,39.4,89.1,8,8,350.5,350.5
|
| 1071 |
+
06-08 16:13:12,1,1,117.6,392.4,9,0,117.6,392.4,9.0,0.0,40.3,89.0,9,9,392.4,392.4
|
| 1072 |
+
06-08 16:13:17,1,1,0.0,377.2,7,0,0.0,377.2,7.0,0.0,31.4,89.0,7,7,377.2,377.2
|
| 1073 |
+
06-08 16:13:22,1,1,0.0,318.1,7,0,0.0,318.1,7.0,0.0,31.8,89.0,7,7,318.1,318.1
|
| 1074 |
+
06-08 16:13:27,1,1,0.0,317.9,7,0,0.0,317.9,7.0,0.0,32.3,89.0,7,7,317.9,317.9
|
| 1075 |
+
06-08 16:13:32,1,1,0.0,317.7,7,0,0.0,317.7,7.0,0.0,32.7,89.0,7,7,317.7,317.7
|
| 1076 |
+
06-08 16:13:37,1,1,0.0,318.2,7,0,0.0,318.2,7.0,0.0,33.2,89.0,7,7,318.2,318.2
|
| 1077 |
+
06-08 16:13:42,1,1,143.1,318.8,8,0,143.1,318.8,8.0,0.0,35.7,89.1,8,8,318.8,318.8
|
| 1078 |
+
06-08 16:13:47,1,1,0.0,362.6,8,0,0.0,362.6,8.0,0.0,36.2,89.1,8,8,362.6,362.6
|
| 1079 |
+
06-08 16:13:52,1,1,0.0,364.4,8,0,0.0,364.4,8.0,0.0,36.7,89.1,8,8,364.4,364.4
|
| 1080 |
+
06-08 16:13:57,1,1,0.0,358.7,7,0,0.0,358.7,7.0,0.0,34.9,89.1,7,7,358.7,358.7
|
| 1081 |
+
06-08 16:14:02,1,1,0.0,318.0,7,0,0.0,318.0,7.0,0.0,35.4,89.1,7,7,318.0,318.0
|
| 1082 |
+
06-08 16:14:07,1,1,3.2,333.4,8,0,3.2,333.4,8.0,0.0,36.0,89.1,8,8,333.4,333.4
|
| 1083 |
+
06-08 16:14:12,1,1,0.0,363.2,8,0,0.0,363.2,8.0,0.0,36.5,89.1,8,8,363.2,363.2
|
| 1084 |
+
06-08 16:14:17,1,1,0.0,362.7,8,0,0.0,362.7,8.0,0.0,37.0,89.1,8,8,362.7,362.7
|
| 1085 |
+
06-08 16:14:22,1,1,0.0,361.8,8,0,0.0,361.8,8.0,0.0,37.5,89.1,8,8,361.8,361.8
|
| 1086 |
+
06-08 16:14:27,1,1,0.0,366.4,8,0,0.0,366.4,8.0,0.0,42.0,90.7,8,8,366.4,366.4
|
| 1087 |
+
06-08 16:14:32,1,1,0.0,366.4,8,0,0.0,366.4,8.0,0.0,42.5,90.7,8,8,366.4,366.4
|
| 1088 |
+
06-08 16:14:37,1,1,0.0,324.1,7,0,0.0,324.1,7.0,0.0,40.3,90.7,7,7,324.1,324.1
|
| 1089 |
+
06-08 16:14:42,1,1,0.0,321.9,7,0,0.0,321.9,7.0,0.0,40.7,90.7,7,7,321.9,321.9
|
| 1090 |
+
06-08 16:14:47,1,1,0.0,321.4,7,0,0.0,321.4,7.0,0.0,41.2,90.7,7,7,321.4,321.4
|
| 1091 |
+
06-08 16:14:52,1,1,0.0,321.3,7,0,0.0,321.3,7.0,0.0,41.6,90.7,7,7,321.3,321.3
|
| 1092 |
+
06-08 16:14:57,1,1,0.0,321.3,7,0,0.0,321.3,7.0,0.0,42.1,90.7,7,7,321.3,321.3
|
| 1093 |
+
06-08 16:15:02,1,1,0.0,321.5,7,0,0.0,321.5,7.0,0.0,42.5,90.7,7,7,321.5,321.5
|
| 1094 |
+
06-08 16:15:07,1,1,0.0,322.4,7,0,0.0,322.4,7.0,0.0,43.0,90.7,7,7,322.4,322.4
|
| 1095 |
+
06-08 16:15:12,1,1,0.0,321.2,7,0,0.0,321.2,7.0,0.0,43.4,90.7,7,7,321.2,321.2
|
| 1096 |
+
06-08 16:15:17,1,1,0.0,350.9,8,0,0.0,350.9,8.0,0.0,32.9,89.1,8,8,350.9,350.9
|
| 1097 |
+
06-08 16:15:22,1,1,0.0,351.0,8,0,0.0,351.0,8.0,0.0,33.4,89.1,8,8,351.0,351.0
|
| 1098 |
+
06-08 16:15:27,1,1,0.0,350.2,8,0,0.0,350.2,8.0,0.0,33.9,89.1,8,8,350.2,350.2
|
| 1099 |
+
06-08 16:15:32,1,1,0.0,348.8,8,0,0.0,348.8,8.0,0.0,34.4,89.1,8,8,348.8,348.8
|
| 1100 |
+
06-08 16:15:37,1,1,0.0,350.4,8,0,0.0,350.4,8.0,0.0,34.9,89.1,8,8,350.4,350.4
|
| 1101 |
+
06-08 16:15:42,1,1,0.0,331.9,7,0,0.0,331.9,7.0,0.0,34.6,89.1,7,7,331.9,331.9
|
| 1102 |
+
06-08 16:15:47,1,1,20.4,346.9,8,0,20.4,346.9,8.0,0.0,36.2,89.2,8,8,346.9,346.9
|
| 1103 |
+
06-08 16:15:52,1,1,0.0,315.1,6,0,0.0,315.1,6.0,0.0,33.9,89.2,6,6,315.1,315.1
|
| 1104 |
+
06-08 16:15:57,1,1,0.0,262.9,6,0,0.0,262.9,6.0,0.0,34.3,89.2,6,6,262.9,262.9
|
| 1105 |
+
06-08 16:16:02,1,1,68.4,281.1,7,0,68.4,281.1,7.0,0.0,35.4,89.2,7,7,281.1,281.1
|
| 1106 |
+
06-08 16:16:07,1,1,0.0,424.0,10,0,0.0,424.0,10.0,0.0,51.3,89.6,10,10,424.0,424.0
|
| 1107 |
+
06-08 16:16:12,1,1,0.0,612.4,13,0,0.0,612.4,13.0,0.0,60.0,83.5,13,13,612.4,612.4
|
| 1108 |
+
06-08 16:16:17,1,1,1.8,591.3,14,0,1.8,591.3,14.0,0.0,61.0,83.5,14,14,591.3,591.3
|
| 1109 |
+
06-08 16:16:22,1,1,69.0,625.9,15,0,69.0,625.9,15.0,0.0,62.4,83.5,15,15,625.9,625.9
|
| 1110 |
+
06-08 16:16:27,1,1,0.0,660.9,15,0,0.0,660.9,15.0,0.0,63.4,83.5,15,15,660.9,660.9
|
| 1111 |
+
06-08 16:16:32,1,1,0.0,667.3,15,0,0.0,667.3,15.0,0.0,64.3,83.5,15,15,667.3,667.3
|
| 1112 |
+
06-08 16:16:37,1,1,0.0,305.9,7,0,0.0,305.9,7.0,0.0,38.3,89.2,7,7,305.9,305.9
|
| 1113 |
+
06-08 16:16:42,1,1,0.6,340.5,8,0,0.6,340.5,8.0,0.0,38.8,89.2,8,8,340.5,340.5
|
| 1114 |
+
06-08 16:16:47,1,1,0.0,349.1,8,0,0.0,349.1,8.0,0.0,39.3,89.2,8,8,349.1,349.1
|
| 1115 |
+
06-08 16:16:52,1,1,165.2,396.9,10,0,165.2,396.9,10.0,0.0,42.1,89.2,10,10,396.9,396.9
|
| 1116 |
+
06-08 16:16:58,1,1,0.0,608.5,13,0,0.0,608.5,13.0,0.0,50.3,89.5,13,13,608.5,608.5
|
| 1117 |
+
06-08 16:17:03,1,1,0.0,565.0,12,0,0.0,565.0,12.0,0.0,50.0,89.5,12,12,565.0,565.0
|
| 1118 |
+
06-08 16:17:08,1,1,83.6,610.7,14,0,83.6,610.7,14.0,0.0,51.8,89.5,14,14,610.7,610.7
|
| 1119 |
+
06-08 16:17:13,1,1,0.0,666.9,15,0,0.0,666.9,15.0,0.0,62.1,83.6,15,15,666.9,666.9
|
| 1120 |
+
06-08 16:17:18,1,1,234.7,634.5,14,0,234.7,634.5,14.0,0.0,55.7,89.5,14,14,634.5,634.5
|
| 1121 |
+
06-08 16:17:23,1,1,0.0,654.5,14,0,0.0,654.5,14.0,0.0,56.6,89.5,14,14,654.5,654.5
|
| 1122 |
+
06-08 16:17:28,1,1,0.0,612.8,12,0,0.0,612.8,12.0,0.0,39.6,89.5,12,12,612.8,612.8
|
| 1123 |
+
06-08 16:17:33,1,1,0.0,561.1,12,0,0.0,561.1,12.0,0.0,40.4,89.5,12,12,561.1,561.1
|
| 1124 |
+
06-08 16:17:38,1,1,0.0,553.6,11,0,0.0,553.6,11.0,0.0,40.3,89.5,11,11,553.6,553.6
|
| 1125 |
+
06-08 16:17:43,1,1,0.0,514.8,11,0,0.0,514.8,11.0,0.0,41.0,89.5,11,11,514.8,514.8
|
| 1126 |
+
06-08 16:17:48,1,1,0.0,513.6,11,0,0.0,513.6,11.0,0.0,41.7,89.5,11,11,513.6,513.6
|
| 1127 |
+
06-08 16:17:53,1,1,0.0,510.8,10,0,0.0,510.8,10.0,0.0,39.0,89.5,10,10,510.8,510.8
|
| 1128 |
+
06-08 16:17:58,1,1,0.0,468.5,10,0,0.0,468.5,10.0,0.0,39.7,89.5,10,10,468.5,468.5
|
| 1129 |
+
06-08 16:18:03,1,1,0.0,701.0,16,0,0.0,701.0,16.0,0.0,67.3,86.5,16,16,701.0,701.0
|
| 1130 |
+
06-08 16:18:08,1,1,0.0,694.9,16,0,0.0,694.9,16.0,0.0,68.3,86.5,16,16,694.9,694.9
|
| 1131 |
+
06-08 16:18:13,1,1,45.6,668.9,18,0,45.6,668.9,18.0,0.0,70.7,86.6,18,18,668.9,668.9
|
| 1132 |
+
06-08 16:18:18,1,1,0.0,673.2,18,0,0.0,673.2,18.0,0.0,71.6,86.6,18,18,673.2,673.2
|
| 1133 |
+
06-08 16:18:23,1,1,0.0,659.5,17,0,0.0,659.5,17.0,0.0,71.7,86.6,17,17,659.5,659.5
|
| 1134 |
+
06-08 16:18:28,1,1,21.2,655.0,18,0,21.2,655.0,18.0,0.0,76.0,86.7,18,18,655.0,655.0
|
| 1135 |
+
06-08 16:18:33,1,1,0.0,673.0,16,0,0.0,673.0,16.0,0.0,67.1,86.7,16,16,673.0,673.0
|
| 1136 |
+
06-08 16:18:38,1,1,105.1,661.8,17,0,105.1,661.8,17.0,0.0,69.0,86.8,17,17,661.8,661.8
|
| 1137 |
+
06-08 16:18:43,1,1,0.0,405.6,9,0,0.0,405.6,9.0,0.0,48.8,89.6,9,9,405.6,405.6
|
| 1138 |
+
06-08 16:18:48,1,1,0.0,404.3,9,0,0.0,404.3,9.0,0.0,49.4,89.6,9,9,404.3,404.3
|
| 1139 |
+
06-08 16:18:53,1,1,0.0,225.2,5,0,0.0,225.2,5.0,0.0,24.0,89.8,5,5,225.2,225.2
|
| 1140 |
+
06-08 16:18:58,1,1,0.0,404.7,9,0,0.0,404.7,9.0,0.0,50.5,89.6,9,9,404.7,404.7
|
| 1141 |
+
06-08 16:19:04,1,1,132.4,204.1,5,0,132.4,204.1,5.0,0.0,25.4,89.9,5,5,204.1,204.1
|
| 1142 |
+
06-08 16:19:09,1,1,0.0,225.3,5,0,0.0,225.3,5.0,0.0,25.7,89.9,5,5,225.3,225.3
|
| 1143 |
+
06-08 16:19:14,1,1,0.0,225.7,5,0,0.0,225.7,5.0,0.0,26.1,89.9,5,5,225.7,225.7
|
| 1144 |
+
06-08 16:19:19,1,1,0.0,225.2,5,0,0.0,225.2,5.0,0.0,26.4,89.9,5,5,225.2,225.2
|
| 1145 |
+
06-08 16:19:24,1,1,0.0,224.4,5,0,0.0,224.4,5.0,0.0,26.7,89.9,5,5,224.4,224.4
|
| 1146 |
+
06-08 16:19:29,1,1,0.0,225.0,5,0,0.0,225.0,5.0,0.0,27.0,89.9,5,5,225.0,225.0
|
| 1147 |
+
06-08 16:19:34,1,1,0.0,207.9,4,0,0.0,207.9,4.0,0.0,24.7,89.9,4,4,207.9,207.9
|
| 1148 |
+
06-08 16:19:39,1,1,0.0,180.9,4,0,0.0,180.9,4.0,0.0,24.9,89.9,4,4,180.9,180.9
|
| 1149 |
+
06-08 16:19:44,1,1,0.0,179.7,4,0,0.0,179.7,4.0,0.0,25.2,89.9,4,4,179.7,179.7
|
| 1150 |
+
06-08 16:19:49,1,1,0.0,180.0,4,0,0.0,180.0,4.0,0.0,25.4,89.9,4,4,180.0,180.0
|
| 1151 |
+
06-08 16:19:54,1,1,0.0,180.0,4,0,0.0,180.0,4.0,0.0,25.7,89.9,4,4,180.0,180.0
|
| 1152 |
+
06-08 16:19:59,1,1,0.0,180.2,4,0,0.0,180.2,4.0,0.0,25.9,89.9,4,4,180.2,180.2
|
| 1153 |
+
06-08 16:20:04,1,1,0.0,300.7,7,0,0.0,300.7,7.0,0.0,21.9,89.6,7,7,300.7,300.7
|
training_logs/20260612_220124_vllm_metrics_job_661735.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/20260612_220124_vllm_metrics_table.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/ablation-pymethods2test-seqmean-arm0_630124.out
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:57544589430cce1ff125aad5db8ab73a85bdbdec7dcaa61c6c4e48114e44a561
|
| 3 |
+
size 44390800
|
training_logs/ablation-pymethods2test-seqmean-arm0_630125.out
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/ablation-pymethods2test-seqmean-arm0_630126.out
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/ablation-pymethods2test-seqmean-arm0_630127.out
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:acba1fb03567f31a572dea981ac325fe2b4df0ecb5e9b3c49722f70599c5311e
|
| 3 |
+
size 42935463
|
training_logs/ablation-pymethods2test-seqmean-arm0_630128.out
ADDED
|
@@ -0,0 +1,370 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
The following have been reloaded with a version change:
|
| 3 |
+
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
Lmod is automatically replacing "GCC/14.3.0" with
|
| 7 |
+
"nvidia-compilers/25.9-CUDA-13".
|
| 8 |
+
|
| 9 |
+
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
| 10 |
+
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
| 11 |
+
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
| 12 |
+
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
| 13 |
+
[ray] RAY_TMPDIR=/tmp/ray/ray_630128
|
| 14 |
+
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_630128
|
| 15 |
+
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_630128
|
| 16 |
+
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
| 17 |
+
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
| 18 |
+
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
| 19 |
+
[proxy] Tunnel port: 7003
|
| 20 |
+
[proxy] Node IP: 10.128.32.33 (workers will connect here)
|
| 21 |
+
[proxy] ✓ SSH tunnel started successfully
|
| 22 |
+
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630128.conf
|
| 23 |
+
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
| 24 |
+
[proxy] - External traffic (internet) → PROXY via tunnel
|
| 25 |
+
[proxy] ✓ Daytona timeout settings configured
|
| 26 |
+
[proxy] Testing proxy connectivity...
|
| 27 |
+
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630128.conf
|
| 28 |
+
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
| 29 |
+
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 30 |
+
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
| 31 |
+
[proxy] ⚠ Tunnel not accessible at 10.128.32.33:7003 (workers may fail)
|
| 32 |
+
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
| 33 |
+
[container_runtime] Using cloud backend: daytona (no local container setup)
|
| 34 |
+
=== Universal RL Training Runner ===
|
| 35 |
+
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
|
| 36 |
+
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
| 37 |
+
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
| 38 |
+
Python version: Python 3.12.12
|
| 39 |
+
UV_USE_IO_URING: 0
|
| 40 |
+
Proxy: DISABLED (direct internet or not configured)
|
| 41 |
+
========================================
|
| 42 |
+
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
|
| 43 |
+
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
|
| 44 |
+
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
|
| 45 |
+
HF_TOKEN=****pDbg
|
| 46 |
+
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
| 47 |
+
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
| 48 |
+
Environment configured:
|
| 49 |
+
TENSOR_PARALLEL_SIZE=1
|
| 50 |
+
NUM_INFERENCE_ENGINES=56
|
| 51 |
+
POLICY_NUM_NODES=14
|
| 52 |
+
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
|
| 53 |
+
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
| 54 |
+
Cleaning up existing Ray instances...
|
| 55 |
+
=== Starting Ray Cluster ===
|
| 56 |
+
Nodes: 14
|
| 57 |
+
GPUs per node: 4
|
| 58 |
+
CPUs per node: 288
|
| 59 |
+
Head node: jpbo-041-33 (10.128.32.33)
|
| 60 |
+
Ray port: 6379
|
| 61 |
+
============================
|
| 62 |
+
Starting Ray head on jpbo-041-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-33.log)...
|
| 63 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.33 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.33 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 64 |
+
Started Ray head on jpbo-041-33
|
| 65 |
+
Starting Ray worker on jpbo-041-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-34.log)...
|
| 66 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.34 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 67 |
+
Started Ray worker 1 on jpbo-041-34
|
| 68 |
+
Starting Ray worker on jpbo-041-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-35.log)...
|
| 69 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.35 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 70 |
+
Started Ray worker 2 on jpbo-041-35
|
| 71 |
+
Starting Ray worker on jpbo-041-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-36.log)...
|
| 72 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 73 |
+
Started Ray worker 3 on jpbo-041-36
|
| 74 |
+
Starting Ray worker on jpbo-041-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-37.log)...
|
| 75 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 76 |
+
Started Ray worker 4 on jpbo-041-37
|
| 77 |
+
Starting Ray worker on jpbo-041-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-38.log)...
|
| 78 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 79 |
+
Started Ray worker 5 on jpbo-041-38
|
| 80 |
+
Starting Ray worker on jpbo-041-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-39.log)...
|
| 81 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 82 |
+
Started Ray worker 6 on jpbo-041-39
|
| 83 |
+
Starting Ray worker on jpbo-041-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-40.log)...
|
| 84 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 85 |
+
Started Ray worker 7 on jpbo-041-40
|
| 86 |
+
Starting Ray worker on jpbo-041-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-41.log)...
|
| 87 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 88 |
+
Started Ray worker 8 on jpbo-041-41
|
| 89 |
+
Starting Ray worker on jpbo-041-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-42.log)...
|
| 90 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 91 |
+
Started Ray worker 9 on jpbo-041-42
|
| 92 |
+
Starting Ray worker on jpbo-041-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-43.log)...
|
| 93 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 94 |
+
Started Ray worker 10 on jpbo-041-43
|
| 95 |
+
Starting Ray worker on jpbo-041-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-44.log)...
|
| 96 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 97 |
+
Started Ray worker 11 on jpbo-041-44
|
| 98 |
+
Starting Ray worker on jpbo-041-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-45.log)...
|
| 99 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 100 |
+
Started Ray worker 12 on jpbo-041-45
|
| 101 |
+
Starting Ray worker on jpbo-041-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-46.log)...
|
| 102 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 103 |
+
Started Ray worker 13 on jpbo-041-46
|
| 104 |
+
Waiting for cluster (56 GPUs, 14 nodes)...
|
| 105 |
+
Connecting to Ray at 10.128.32.33:6379 (expecting 14 nodes, 56.0 GPUs)
|
| 106 |
+
Ray connection established, polling for resources...
|
| 107 |
+
[Ray wait] nodes=2/14 GPUs=8.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 1511604617216.0, 'accelerator_type:GH200': 2.0, 'CPU': 576.0, 'GPU': 8.0, 'object_store_memory': 85899345920.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
| 108 |
+
[Ray wait] nodes=2/14 GPUs=8.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 1511604617216.0, 'accelerator_type:GH200': 2.0, 'CPU': 576.0, 'GPU': 8.0, 'object_store_memory': 85899345920.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0}
|
| 109 |
+
[Ray wait] nodes=3/14 GPUs=12.0/56.0 resources={'GPU': 12.0, 'CPU': 864.0, 'accelerator_type:GH200': 3.0, 'memory': 2273135755264.0, 'node:10.128.32.44': 1.0, 'object_store_memory': 128849018880.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
| 110 |
+
[Ray wait] nodes=4/14 GPUs=16.0/56.0 resources={'node:10.128.32.44': 1.0, 'CPU': 1152.0, 'accelerator_type:GH200': 4.0, 'memory': 3033402310656.0, 'GPU': 16.0, 'object_store_memory': 171798691840.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
| 111 |
+
[Ray wait] nodes=4/14 GPUs=16.0/56.0 resources={'node:10.128.32.44': 1.0, 'memory': 3033402310656.0, 'accelerator_type:GH200': 4.0, 'CPU': 1152.0, 'GPU': 16.0, 'object_store_memory': 171798691840.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0}
|
| 112 |
+
[Ray wait] nodes=5/14 GPUs=20.0/56.0 resources={'GPU': 20.0, 'memory': 3809847214080.0, 'accelerator_type:GH200': 5.0, 'CPU': 1440.0, 'node:10.128.32.44': 1.0, 'object_store_memory': 214748364800.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
| 113 |
+
[Ray wait] nodes=6/14 GPUs=24.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 4562133647360.0, 'accelerator_type:GH200': 6.0, 'CPU': 1728.0, 'GPU': 24.0, 'object_store_memory': 257698037760.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
|
| 114 |
+
[Ray wait] nodes=7/14 GPUs=28.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2016.0, 'accelerator_type:GH200': 7.0, 'memory': 5307486961664.0, 'GPU': 28.0, 'object_store_memory': 300647710720.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 115 |
+
[Ray wait] nodes=7/14 GPUs=28.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2016.0, 'accelerator_type:GH200': 7.0, 'memory': 5307486961664.0, 'GPU': 28.0, 'object_store_memory': 300647710720.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 116 |
+
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 117 |
+
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 118 |
+
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 119 |
+
[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 6067020955648.0, 'accelerator_type:GH200': 8.0, 'CPU': 2304.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 120 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 121 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 122 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 123 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 124 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 125 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 126 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 127 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 128 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 129 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 130 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 131 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 132 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 133 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 134 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 135 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 136 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 137 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 138 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 139 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 140 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 141 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 142 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 143 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 144 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 145 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 146 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 147 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 148 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 149 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 150 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 151 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 152 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 153 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 154 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 155 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 156 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 157 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 158 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 159 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 160 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 161 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 162 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 163 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
|
| 164 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 165 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
|
| 166 |
+
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
|
| 167 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
|
| 168 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
|
| 169 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
|
| 170 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
|
| 171 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
|
| 172 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
|
| 173 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
|
| 174 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
|
| 175 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
|
| 176 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
|
| 177 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
|
| 178 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
|
| 179 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
|
| 180 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
|
| 181 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
|
| 182 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
|
| 183 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
|
| 184 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
|
| 185 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
|
| 186 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
|
| 187 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
|
| 188 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
|
| 189 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
|
| 190 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
|
| 191 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
|
| 192 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
|
| 193 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
|
| 194 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
|
| 195 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
|
| 196 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
|
| 197 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
|
| 198 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
|
| 199 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
|
| 200 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
|
| 201 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
|
| 202 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
|
| 203 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
|
| 204 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
|
| 205 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
|
| 206 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
|
| 207 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
|
| 208 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
|
| 209 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
|
| 210 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
|
| 211 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
|
| 212 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
|
| 213 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
|
| 214 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
|
| 215 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
|
| 216 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
|
| 217 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
|
| 218 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
|
| 219 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
|
| 220 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
|
| 221 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
|
| 222 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
|
| 223 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
|
| 224 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
|
| 225 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
|
| 226 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
|
| 227 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
|
| 228 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
|
| 229 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
|
| 230 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
|
| 231 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
|
| 232 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
|
| 233 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
|
| 234 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
|
| 235 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
|
| 236 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
|
| 237 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
|
| 238 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
|
| 239 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
|
| 240 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
|
| 241 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
|
| 242 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
|
| 243 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
|
| 244 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
|
| 245 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
|
| 246 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
|
| 247 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
|
| 248 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
|
| 249 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
|
| 250 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
|
| 251 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
|
| 252 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
|
| 253 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
|
| 254 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
|
| 255 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
|
| 256 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
|
| 257 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
|
| 258 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
|
| 259 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
|
| 260 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
|
| 261 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
|
| 262 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
|
| 263 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
|
| 264 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
|
| 265 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
|
| 266 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
|
| 267 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
|
| 268 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
|
| 269 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
|
| 270 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
|
| 271 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
|
| 272 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
|
| 273 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
|
| 274 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
|
| 275 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
|
| 276 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
|
| 277 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
|
| 278 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
|
| 279 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
|
| 280 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
|
| 281 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
|
| 282 |
+
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
|
| 283 |
+
2026-06-07 14:35:10,656 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.33:6379...
|
| 284 |
+
2026-06-07 14:35:10,666 INFO worker.py:2007 -- Connected to Ray cluster.
|
| 285 |
+
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
| 286 |
+
warnings.warn(
|
| 287 |
+
[33m(raylet, ip=10.128.32.44)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 288 |
+
[33m(raylet, ip=10.128.32.43)[0m [2026-06-07 14:35:49,687 E 2691736 2691736] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 289 |
+
[36m(pid=352159)[0m [2026-06-07 14:35:54,163 E 352159 352769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 290 |
+
[33m(raylet, ip=10.128.32.44)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 291 |
+
[36m(pid=352184)[0m [2026-06-07 14:36:00,280 E 352184 353748] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
| 292 |
+
[33m(raylet, ip=10.128.32.42)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 293 |
+
[2026-06-07 14:36:04,817 E 352114 352155] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 294 |
+
[33m(raylet, ip=10.128.32.44)[0m [2026-06-07 14:36:16,668 E 3005389 3005389] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 295 |
+
[36m(pid=352328)[0m [2026-06-07 14:36:04,815 E 352328 368299] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
| 296 |
+
[33m(raylet, ip=10.128.32.42)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 297 |
+
[33m(raylet, ip=10.128.32.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 298 |
+
[33m(raylet, ip=10.128.32.42)[0m [2026-06-07 14:36:31,723 E 3108436 3108436] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 299 |
+
[33m(raylet, ip=10.128.32.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 300 |
+
[33m(raylet, ip=10.128.32.34)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 301 |
+
[33m(raylet, ip=10.128.32.46)[0m [2026-06-07 14:36:47,990 E 2656361 2656361] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 302 |
+
[33m(raylet, ip=10.128.32.40)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 3x across cluster][0m
|
| 303 |
+
[33m(raylet, ip=10.128.32.34)[0m [2026-06-07 14:37:02,131 E 2733873 2733873] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 304 |
+
[33m(raylet, ip=10.128.32.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 305 |
+
[33m(raylet, ip=10.128.32.40)[0m [2026-06-07 14:37:06,688 E 2571074 2571074] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 306 |
+
[33m(raylet, ip=10.128.32.36)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 307 |
+
[33m(raylet, ip=10.128.32.36)[0m [2026-06-07 14:37:33,965 E 2625779 2625779] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 308 |
+
[33m(raylet, ip=10.128.32.38)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 309 |
+
[33m(raylet, ip=10.128.32.45)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 3x across cluster][0m
|
| 310 |
+
[33m(raylet, ip=10.128.32.38)[0m [2026-06-07 14:38:06,213 E 2699635 2699635] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 311 |
+
[33m(raylet, ip=10.128.32.45)[0m [2026-06-07 14:38:14,810 E 2825651 2825651] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 312 |
+
Traceback (most recent call last):
|
| 313 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
| 314 |
+
main()
|
| 315 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
| 316 |
+
wait_for_cluster(
|
| 317 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
|
| 318 |
+
raise TimeoutError(
|
| 319 |
+
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=10, gpus=40.0)
|
| 320 |
+
srun: error: jpbo-041-33: task 0: Exited with exit code 1
|
| 321 |
+
|
| 322 |
+
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
| 323 |
+
Traceback (most recent call last):
|
| 324 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
|
| 325 |
+
result = subprocess.run(
|
| 326 |
+
^^^^^^^^^^^^^^^
|
| 327 |
+
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
| 328 |
+
raise CalledProcessError(retcode, process.args,
|
| 329 |
+
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
| 330 |
+
|
| 331 |
+
The above exception was the direct cause of the following exception:
|
| 332 |
+
|
| 333 |
+
Traceback (most recent call last):
|
| 334 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1285, in run
|
| 335 |
+
training_exit_code = self._run_with_ray()
|
| 336 |
+
^^^^^^^^^^^^^^^^^^^^
|
| 337 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1532, in _run_with_ray
|
| 338 |
+
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
| 339 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 340 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
|
| 341 |
+
self.start()
|
| 342 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
|
| 343 |
+
self._wait_for_cluster()
|
| 344 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
|
| 345 |
+
raise RuntimeError(
|
| 346 |
+
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
| 347 |
+
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
| 348 |
+
[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
|
| 349 |
+
[RLJobRunner] Launching trace upload (training exit code: 1):
|
| 350 |
+
repo_id: DCAgent/ablation-pymethods2test-seqmean-arm0
|
| 351 |
+
job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0
|
| 352 |
+
episodes: last
|
| 353 |
+
log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/logs/ablation-pymethods2test-seqmean-arm0_trace_upload.log
|
| 354 |
+
[RLJobRunner] Waiting for trace upload to complete...
|
| 355 |
+
[RLJobRunner] Trace upload failed with exit code 1.
|
| 356 |
+
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ray_logs/
|
| 357 |
+
Collecting Ray logs from worker jpbo-041-34...
|
| 358 |
+
Collecting Ray logs from worker jpbo-041-35...
|
| 359 |
+
Collecting Ray logs from worker jpbo-041-36...
|
| 360 |
+
Collecting Ray logs from worker jpbo-041-37...
|
| 361 |
+
Collecting Ray logs from worker jpbo-041-38...
|
| 362 |
+
Collecting Ray logs from worker jpbo-041-39...
|
| 363 |
+
Collecting Ray logs from worker jpbo-041-40...
|
| 364 |
+
Collecting Ray logs from worker jpbo-041-41...
|
| 365 |
+
Collecting Ray logs from worker jpbo-041-42...
|
| 366 |
+
Collecting Ray logs from worker jpbo-041-43...
|
| 367 |
+
Collecting Ray logs from worker jpbo-041-44...
|
| 368 |
+
Collecting Ray logs from worker jpbo-041-45...
|
| 369 |
+
Collecting Ray logs from worker jpbo-041-46...
|
| 370 |
+
Ray log preservation complete
|
training_logs/ablation-pymethods2test-seqmean-arm0_630129.out
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/ablation-pymethods2test-seqmean-arm0_630130.out
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ccadea7fa55f83615cb7841c06afa1a0bae0a0f8f21a8e874cf3b36a35660724
|
| 3 |
+
size 23994623
|
training_logs/ablation-pymethods2test-seqmean-arm0_657667.out
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
The following have been reloaded with a version change:
|
| 3 |
+
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
Lmod is automatically replacing "GCC/14.3.0" with
|
| 7 |
+
"nvidia-compilers/25.9-CUDA-13".
|
| 8 |
+
|
| 9 |
+
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
| 10 |
+
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
| 11 |
+
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
| 12 |
+
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
| 13 |
+
Python executable:
|
| 14 |
+
/var/spool/slurmd/job657667/slurm_script: line 128: python: command not found
|
| 15 |
+
Python path check:
|
| 16 |
+
[ray] RAY_TMPDIR=/tmp/ray/ray_657667
|
| 17 |
+
/var/spool/slurmd/job657667/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
training_logs/ablation-pymethods2test-seqmean-arm0_657668.out
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
The following have been reloaded with a version change:
|
| 3 |
+
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
Lmod is automatically replacing "GCC/14.3.0" with
|
| 7 |
+
"nvidia-compilers/25.9-CUDA-13".
|
| 8 |
+
|
| 9 |
+
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
| 10 |
+
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
| 11 |
+
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
| 12 |
+
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
| 13 |
+
Python executable:
|
| 14 |
+
/var/spool/slurmd/job657668/slurm_script: line 128: python: command not found
|
| 15 |
+
Python path check:
|
| 16 |
+
[ray] RAY_TMPDIR=/tmp/ray/ray_657668
|
| 17 |
+
/var/spool/slurmd/job657668/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
training_logs/ablation-pymethods2test-seqmean-arm0_657669.out
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
The following have been reloaded with a version change:
|
| 3 |
+
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
Lmod is automatically replacing "GCC/14.3.0" with
|
| 7 |
+
"nvidia-compilers/25.9-CUDA-13".
|
| 8 |
+
|
| 9 |
+
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
| 10 |
+
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
| 11 |
+
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
| 12 |
+
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
| 13 |
+
Python executable:
|
| 14 |
+
/var/spool/slurmd/job657669/slurm_script: line 128: python: command not found
|
| 15 |
+
Python path check:
|
| 16 |
+
[ray] RAY_TMPDIR=/tmp/ray/ray_657669
|
| 17 |
+
/var/spool/slurmd/job657669/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
training_logs/ablation-pymethods2test-seqmean-arm0_657670.out
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
The following have been reloaded with a version change:
|
| 3 |
+
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
Lmod is automatically replacing "GCC/14.3.0" with
|
| 7 |
+
"nvidia-compilers/25.9-CUDA-13".
|
| 8 |
+
|
| 9 |
+
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
| 10 |
+
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
| 11 |
+
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
| 12 |
+
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
| 13 |
+
Python executable:
|
| 14 |
+
/var/spool/slurmd/job657670/slurm_script: line 128: python: command not found
|
| 15 |
+
Python path check:
|
| 16 |
+
[ray] RAY_TMPDIR=/tmp/ray/ray_657670
|
| 17 |
+
/var/spool/slurmd/job657670/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
training_logs/ablation-pymethods2test-seqmean-arm0_657671.out
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
The following have been reloaded with a version change:
|
| 3 |
+
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
Lmod is automatically replacing "GCC/14.3.0" with
|
| 7 |
+
"nvidia-compilers/25.9-CUDA-13".
|
| 8 |
+
|
| 9 |
+
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
| 10 |
+
Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
|
| 11 |
+
Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
|
| 12 |
+
/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
|
| 13 |
+
Python executable:
|
| 14 |
+
/var/spool/slurmd/job657671/slurm_script: line 128: python: command not found
|
| 15 |
+
Python path check:
|
| 16 |
+
[ray] RAY_TMPDIR=/tmp/ray/ray_657671
|
| 17 |
+
/var/spool/slurmd/job657671/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory
|
training_logs/ablation-pymethods2test-seqmean-arm0_661733.out
ADDED
|
@@ -0,0 +1,364 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
The following have been reloaded with a version change:
|
| 3 |
+
1) GCCcore/.14.3.0 => GCCcore/14.3.0
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
Lmod is automatically replacing "GCC/14.3.0" with
|
| 7 |
+
"nvidia-compilers/25.9-CUDA-13".
|
| 8 |
+
|
| 9 |
+
Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
|
| 10 |
+
Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
|
| 11 |
+
Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
| 12 |
+
Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
| 13 |
+
[ray] RAY_TMPDIR=/tmp/ray/ray_661733
|
| 14 |
+
[triton_cache] Triton cache: /tmp/triton_cache_feuer1_661733
|
| 15 |
+
[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_661733
|
| 16 |
+
[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
|
| 17 |
+
[proxy] Setting up SSH tunnel to jpbl-s01-01
|
| 18 |
+
[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
|
| 19 |
+
[proxy] Tunnel port: 7003
|
| 20 |
+
[proxy] Node IP: 10.128.16.35 (workers will connect here)
|
| 21 |
+
[proxy] ✓ SSH tunnel started successfully
|
| 22 |
+
[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf
|
| 23 |
+
[proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
|
| 24 |
+
[proxy] - External traffic (internet) → PROXY via tunnel
|
| 25 |
+
[proxy] ✓ Daytona timeout settings configured
|
| 26 |
+
[proxy] Testing proxy connectivity...
|
| 27 |
+
[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf
|
| 28 |
+
[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
|
| 29 |
+
[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 30 |
+
[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
|
| 31 |
+
[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
|
| 32 |
+
[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
|
| 33 |
+
[container_runtime] Using cloud backend: daytona (no local container setup)
|
| 34 |
+
=== Universal RL Training Runner ===
|
| 35 |
+
Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
|
| 36 |
+
Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
|
| 37 |
+
Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
|
| 38 |
+
Python version: Python 3.12.12
|
| 39 |
+
UV_USE_IO_URING: 0
|
| 40 |
+
Proxy: DISABLED (direct internet or not configured)
|
| 41 |
+
========================================
|
| 42 |
+
=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
|
| 43 |
+
[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
|
| 44 |
+
[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
|
| 45 |
+
HF_TOKEN=****pDbg
|
| 46 |
+
HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
|
| 47 |
+
SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
|
| 48 |
+
Environment configured:
|
| 49 |
+
TENSOR_PARALLEL_SIZE=1
|
| 50 |
+
NUM_INFERENCE_ENGINES=56
|
| 51 |
+
POLICY_NUM_NODES=14
|
| 52 |
+
WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
|
| 53 |
+
Starting Ray cluster with 14 nodes, 4 GPUs/node
|
| 54 |
+
Cleaning up existing Ray instances...
|
| 55 |
+
=== Starting Ray Cluster ===
|
| 56 |
+
Nodes: 14
|
| 57 |
+
GPUs per node: 4
|
| 58 |
+
CPUs per node: 288
|
| 59 |
+
Head node: jpbo-001-35 (10.128.16.35)
|
| 60 |
+
Ray port: 6379
|
| 61 |
+
============================
|
| 62 |
+
Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
|
| 63 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 64 |
+
Started Ray head on jpbo-001-35
|
| 65 |
+
Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
|
| 66 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 67 |
+
Started Ray worker 1 on jpbo-001-36
|
| 68 |
+
Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
|
| 69 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 70 |
+
Started Ray worker 2 on jpbo-001-37
|
| 71 |
+
Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
|
| 72 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 73 |
+
Started Ray worker 3 on jpbo-001-38
|
| 74 |
+
Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
|
| 75 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 76 |
+
Started Ray worker 4 on jpbo-001-41
|
| 77 |
+
Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
|
| 78 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 79 |
+
Started Ray worker 5 on jpbo-001-42
|
| 80 |
+
Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
|
| 81 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 82 |
+
Started Ray worker 6 on jpbo-001-43
|
| 83 |
+
Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
|
| 84 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 85 |
+
Started Ray worker 7 on jpbo-001-44
|
| 86 |
+
Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
|
| 87 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 88 |
+
Started Ray worker 8 on jpbo-001-45
|
| 89 |
+
Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
|
| 90 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 91 |
+
Started Ray worker 9 on jpbo-001-46
|
| 92 |
+
Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
|
| 93 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 94 |
+
Started Ray worker 10 on jpbo-001-47
|
| 95 |
+
Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
|
| 96 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 97 |
+
Started Ray worker 11 on jpbo-001-48
|
| 98 |
+
Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
|
| 99 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 100 |
+
Started Ray worker 12 on jpbo-003-37
|
| 101 |
+
Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
|
| 102 |
+
Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
|
| 103 |
+
Started Ray worker 13 on jpbo-003-39
|
| 104 |
+
Waiting for cluster (56 GPUs, 14 nodes)...
|
| 105 |
+
Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
|
| 106 |
+
Ray connection established, polling for resources...
|
| 107 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 108 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 109 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 110 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 111 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 112 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 113 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 114 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 115 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 116 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 117 |
+
[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 118 |
+
[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 40.0, 'memory': 7408839753728.0, 'accelerator_type:GH200': 10.0, 'object_store_memory': 429496729600.0, 'CPU': 2880.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 119 |
+
[Ray wait] nodes=11/14 GPUs=44.0/56.0 resources={'GPU': 44.0, 'memory': 8175128150016.0, 'accelerator_type:GH200': 11.0, 'object_store_memory': 472446402560.0, 'node:10.128.16.135': 1.0, 'CPU': 3168.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
|
| 120 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 121 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 122 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 123 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 124 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 125 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 126 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 127 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 128 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 129 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 130 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 131 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 132 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 133 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 134 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 135 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 136 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 137 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 138 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 139 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 140 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 141 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 142 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 143 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 144 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 145 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 146 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 147 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 148 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 149 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 150 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 151 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 152 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 153 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 154 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 155 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 156 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 157 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 158 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 159 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 160 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 161 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 162 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 163 |
+
[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
|
| 164 |
+
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0}
|
| 165 |
+
[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0}
|
| 166 |
+
srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
|
| 167 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
|
| 168 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
|
| 169 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
|
| 170 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
|
| 171 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
|
| 172 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
|
| 173 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
|
| 174 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
|
| 175 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
|
| 176 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
|
| 177 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
|
| 178 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
|
| 179 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
|
| 180 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
|
| 181 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
|
| 182 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
|
| 183 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
|
| 184 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
|
| 185 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
|
| 186 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
|
| 187 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
|
| 188 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
|
| 189 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
|
| 190 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
|
| 191 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
|
| 192 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
|
| 193 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
|
| 194 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
|
| 195 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
|
| 196 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
|
| 197 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
|
| 198 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
|
| 199 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
|
| 200 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
|
| 201 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
|
| 202 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
|
| 203 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
|
| 204 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
|
| 205 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
|
| 206 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
|
| 207 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
|
| 208 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
|
| 209 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
|
| 210 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
|
| 211 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
|
| 212 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
|
| 213 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
|
| 214 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
|
| 215 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
|
| 216 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
|
| 217 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
|
| 218 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
|
| 219 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
|
| 220 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
|
| 221 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
|
| 222 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
|
| 223 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
|
| 224 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
|
| 225 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
|
| 226 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
|
| 227 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
|
| 228 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
|
| 229 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
|
| 230 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
|
| 231 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
|
| 232 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
|
| 233 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
|
| 234 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
|
| 235 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
|
| 236 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
|
| 237 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
|
| 238 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
|
| 239 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
|
| 240 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
|
| 241 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
|
| 242 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
|
| 243 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
|
| 244 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
|
| 245 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
|
| 246 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
|
| 247 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
|
| 248 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
|
| 249 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
|
| 250 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
|
| 251 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
|
| 252 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
|
| 253 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
|
| 254 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
|
| 255 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
|
| 256 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
|
| 257 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
|
| 258 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
|
| 259 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
|
| 260 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
|
| 261 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
|
| 262 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
|
| 263 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
|
| 264 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
|
| 265 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
|
| 266 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
|
| 267 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
|
| 268 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
|
| 269 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
|
| 270 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
|
| 271 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
|
| 272 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
|
| 273 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
|
| 274 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
|
| 275 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
|
| 276 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
|
| 277 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
|
| 278 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
|
| 279 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
|
| 280 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
|
| 281 |
+
srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
|
| 282 |
+
srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
|
| 283 |
+
2026-06-08 13:34:49,641 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
|
| 284 |
+
2026-06-08 13:34:49,647 WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
|
| 285 |
+
2026-06-08 13:34:49,649 INFO worker.py:2007 -- Connected to Ray cluster.
|
| 286 |
+
/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
| 287 |
+
warnings.warn(
|
| 288 |
+
[36m(pid=1699456)[0m [2026-06-08 13:35:34,739 E 1699456 1700253] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 289 |
+
[36m(pid=1699477)[0m [2026-06-08 13:35:42,369 E 1699477 1701156] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)[0m
|
| 290 |
+
[2026-06-08 13:35:45,201 E 1699412 1699453] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 291 |
+
[33m(raylet, ip=10.128.16.47)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 292 |
+
[36m(pid=1699674)[0m [2026-06-08 13:35:45,144 E 1699674 1715593] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14[32m [repeated 269x across cluster][0m
|
| 293 |
+
[33m(raylet, ip=10.128.16.133)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e[32m [repeated 5x across cluster][0m
|
| 294 |
+
[33m(raylet, ip=10.128.16.47)[0m [2026-06-08 13:37:33,645 E 1598284 1598284] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 295 |
+
[33m(raylet, ip=10.128.16.135)[0m [2026-06-08 13:37:44,343 E 3199851 3199851] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 296 |
+
[33m(raylet, ip=10.128.16.133)[0m [2026-06-08 13:37:47,415 E 2150584 2150584] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
|
| 297 |
+
[33m(raylet, ip=10.128.16.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 298 |
+
[33m(raylet, ip=10.128.16.46)[0m [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
|
| 299 |
+
Traceback (most recent call last):
|
| 300 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
|
| 301 |
+
main()
|
| 302 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
|
| 303 |
+
wait_for_cluster(
|
| 304 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
|
| 305 |
+
raise TimeoutError(
|
| 306 |
+
TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=13, gpus=52.0)
|
| 307 |
+
srun: error: jpbo-001-35: task 0: Exited with exit code 1
|
| 308 |
+
|
| 309 |
+
RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
| 310 |
+
Traceback (most recent call last):
|
| 311 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
|
| 312 |
+
result = subprocess.run(
|
| 313 |
+
^^^^^^^^^^^^^^^
|
| 314 |
+
File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
|
| 315 |
+
raise CalledProcessError(retcode, process.args,
|
| 316 |
+
subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
|
| 317 |
+
|
| 318 |
+
The above exception was the direct cause of the following exception:
|
| 319 |
+
|
| 320 |
+
Traceback (most recent call last):
|
| 321 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1291, in run
|
| 322 |
+
training_exit_code = self._run_with_ray()
|
| 323 |
+
^^^^^^^^^^^^^^^^^^^^
|
| 324 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1538, in _run_with_ray
|
| 325 |
+
with RayCluster.from_slurm(ray_cfg) as ray_cluster:
|
| 326 |
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 327 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
|
| 328 |
+
self.start()
|
| 329 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
|
| 330 |
+
self._wait_for_cluster()
|
| 331 |
+
File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
|
| 332 |
+
raise RuntimeError(
|
| 333 |
+
RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
|
| 334 |
+
[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
|
| 335 |
+
Collecting Ray logs from worker jpbo-001-36...
|
| 336 |
+
Collecting Ray logs from worker jpbo-001-37...
|
| 337 |
+
Collecting Ray logs from worker jpbo-001-38...
|
| 338 |
+
Collecting Ray logs from worker jpbo-001-41...
|
| 339 |
+
Collecting Ray logs from worker jpbo-001-42...
|
| 340 |
+
Collecting Ray logs from worker jpbo-001-43...
|
| 341 |
+
Collecting Ray logs from worker jpbo-001-44...
|
| 342 |
+
Collecting Ray logs from worker jpbo-001-45...
|
| 343 |
+
Collecting Ray logs from worker jpbo-001-46...
|
| 344 |
+
Collecting Ray logs from worker jpbo-001-47...
|
| 345 |
+
Collecting Ray logs from worker jpbo-001-48...
|
| 346 |
+
Collecting Ray logs from worker jpbo-003-37...
|
| 347 |
+
Collecting Ray logs from worker jpbo-003-39...
|
| 348 |
+
[RLJobRunner] Crash-time Ray log preservation complete.
|
| 349 |
+
[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/trace_jobs, skipping upload.
|
| 350 |
+
Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ray_logs/
|
| 351 |
+
Collecting Ray logs from worker jpbo-001-36...
|
| 352 |
+
Collecting Ray logs from worker jpbo-001-37...
|
| 353 |
+
Collecting Ray logs from worker jpbo-001-38...
|
| 354 |
+
Collecting Ray logs from worker jpbo-001-41...
|
| 355 |
+
Collecting Ray logs from worker jpbo-001-42...
|
| 356 |
+
Collecting Ray logs from worker jpbo-001-43...
|
| 357 |
+
Collecting Ray logs from worker jpbo-001-44...
|
| 358 |
+
Collecting Ray logs from worker jpbo-001-45...
|
| 359 |
+
Collecting Ray logs from worker jpbo-001-46...
|
| 360 |
+
Collecting Ray logs from worker jpbo-001-47...
|
| 361 |
+
Collecting Ray logs from worker jpbo-001-48...
|
| 362 |
+
Collecting Ray logs from worker jpbo-003-37...
|
| 363 |
+
Collecting Ray logs from worker jpbo-003-39...
|
| 364 |
+
Ray log preservation complete
|
training_logs/ablation-pymethods2test-seqmean-arm0_661734.out
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
training_logs/ablation-pymethods2test-seqmean-arm0_661735.out
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ffc1dd737ba8b66df4987f88b45897fb347a2b159e9d578fd3f62ac19cd5a2f6
|
| 3 |
+
size 14896573
|