| === P21H V3 dispatch start 2026-05-22T19:05:07Z === |
| variant=P21H_e3 init=qwen seed=1337 |
| base=Qwen/Qwen2.5-1.5B |
| steps=5000 bsz=2 block=512 lr=5e-5 warmup=100 |
| noise_sigma=0.1 lambda_mitosis=0.0 |
| wiki_frac=1.0 corpus_mb=72 |
| langs=en,ko,zh,ru,ja wiki_target_mb_per_lang=10 |
| cost_cap=$25 total (3 variants × ~$8) |
| [create] trying GPU: NVIDIA A100-SXM4-80GB |
| [create] cloud=SECURE gpu=NVIDIA A100-SXM4-80GB resp: {"data":{"podFindAndDeployOnDemand":{"id":"xhjxwzrpadm89y","machineId":"6vjrhbzbke27"}}} |
| [create] pod xhjxwzrpadm89y on NVIDIA A100-SXM4-80GB |
| [ssh-ready] iter 3: 195.26.233.30:46882 - port mapped AND handshake OK |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| [corpus-anima] building corpus_s101 (seed 1337 n=777000) |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| "http://": 0 |
| }, |
| "corpus_total": 0, |
| "PASS": true, |
| "honest_caveat": "external-grep is a proxy over 7 surface tokens; exhaustive provenance audit requires source-attribution tracking, NOT in $0 build scope. §16 generator is anima-OWN substrate, S2 uses anima Ψ-fields only, S5 uses anima-OWN .kosmos files. NO external API calls in build_corpus_s101.py (audit-grep this file)." |
| } |
| } |
| } |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| [corpus-anima] sha=be969af481947da4693618be33a9cc67f2057a53547b6ad21abda06e7f39018b size=603318628 bytes |
| [corpus-multi-wiki] building 5-lang wiki corpus (en,ko,zh,ru,ja, 10 MB/lang) |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
|
|
| [notice] A new release of pip is available: 24.2 -> 26.1.1 |
| [notice] To update, run: python -m pip install --upgrade pip |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| [p21m-build] langs=['en', 'ko', 'zh', 'ru', 'ja'] target_mb/lang=10 |
| [p21m-build] loaded Salesforce/wikitext/wikitext-2-raw-v1 streaming OK |
| [p21m-build] DONE lang=en source=Salesforce/wikitext/wikitext-2-raw-v1 records=17,078 bytes=10,486,184 (10.0 MB) filtered=57 |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [p21m-build] loaded wikimedia/wikipedia/20231101.ko streaming OK |
| [p21m-build] DONE lang=ko source=wikimedia/wikipedia/20231101.ko records=1,004 bytes=10,569,479 (10.1 MB) filtered=2 |
| [p21m-build] loaded wikimedia/wikipedia/20231101.zh streaming OK |
| [p21m-build] DONE lang=zh source=wikimedia/wikipedia/20231101.zh records=521 bytes=10,511,272 (10.0 MB) filtered=3 |
| [p21m-build] loaded wikimedia/wikipedia/20231101.ru streaming OK |
| [p21m-build] DONE lang=ru source=wikimedia/wikipedia/20231101.ru records=264 bytes=10,579,139 (10.1 MB) filtered=0 |
| [p21m-build] loaded wikimedia/wikipedia/20231101.ja streaming OK |
| [p21m-build] DONE lang=ja source=wikimedia/wikipedia/20231101.ja records=470 bytes=10,498,969 (10.0 MB) filtered=0 |
| [p21m-build] CONCAT DONE: 19,337 records, 51.1 MB → /workspace/p21hr/multi_wiki_corpus.jsonl |
| terminate called without an active exception |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| [corpus-multi-wiki] sha=d378923c6039b6bcb571c7c85bf9cd1439d544ece0551a4f9d0564a688aa2241 size=53568645 bytes |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| NVIDIA A100-SXM4-80GB, 81920 MiB |
| [train] P21H V3 launch (qwen) |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| TRAIN_PID 448 |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| [env-verify] [launch] PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True |
| [train] done iter 1 |
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| Warning: Permanently added '[195.26.233.30]:46882' (ED25519) to the list of known hosts.
|
| [pull] success try 1 |
| [pull] result.json present |
| VERDICT= FAIL |
| init= qwen |
| n_strong= 0 /5 n_partial= 1 n_weak= 4 n_pure_memorize= 0 |
| per_lang: |
| en: WEAK score=0/20 gen=20 coh=0 |
| ko: PARTIAL score=15/20 gen=20 coh=15 |
| zh: WEAK score=2/20 gen=20 coh=2 |
| ru: WEAK score=5/20 gen=20 coh=5 |
| ja: WEAK score=6/20 gen=20 coh=6 |
| anima_register_hits= 0 /20 |
| register_regress= True |
| init_CE= 14.792716979980469 final_CE= 6.5491414070129395 |
| n_kosmos_anchors= 7 |
| train_wall_s= 641.2 |
| n_total_params= 2999735296 |
| === P21H V3 pipeline complete 2026-05-22T19:28:55Z === |
| dispatch_p21h_v3_runpod.sh: line 304: 33977 Terminated: 15 ( sleep "$WATCHDOG_SEC"; if [ ! -s "$VDIR/result.json" ]; then |
| echo "WATCHDOG FIRED $(date -u +%FT%TZ): no result.json after ${WATCHDOG_SEC}s - terminating pod ${POD_ID:-?}" > "$FAIL_MARKER"; [ -n "$POD_ID" ] && gql "{\"query\":\"mutation { podTerminate(input:{podId:\\\"$POD_ID\\\"}) }\"}" > /dev/null; pkill -P $$ 2> /dev/null; |
| fi ) |
| [teardown] SAVE_POD=1 - pod xhjxwzrpadm89y RETAINED for manual inspect |
| === P21H V3 dispatch end 2026-05-22T19:28:58Z rc=0 === |
|
|
