Buckets:
| QWEN RGB: CONTINUE CHECKPOINT 3000 ON ANOTHER GPU | |
| Updated 2026-10-15. Bucket Disya/LLM-TO-DIT. | |
| Contains the real full training checkpoint at update3000, not inference-only weights. | |
| Model, AdamW8bit optimizer (429 states), CPU/CUDA/Python RNG, epoch/next_group, | |
| caption counters and76 online vision mean sums/counts are inside full_latest.pt. | |
| Data-order seed20261014; generation comparison seed1234. | |
| Training consumed96000 image presentations; mean counts102816 including6816 seed variants. | |
| Dataset340803 records; both copies retained. Next optimizer group3000 (zero-based). | |
| DOWNLOAD (does not start training): | |
| python -m pip install huggingface-hub==1.24.0 | |
| python -c "from huggingface_hub import HfApi; HfApi().download_bucket_files('Disya/LLM-TO-DIT', [('bootstrap_resume.py','bootstrap_resume.py')])" | |
| python bootstrap_resume.py --dest /workspace/llm-to-dit | |
| The helper fetches the original complete dataset/model archives already in this | |
| bucket, then overlays the UPDATED code and downloads checkpoint3000. | |
| Original manifest.json is unchanged and describes the dataset bundle. | |
| resume/step003000/resume_manifest.json describes this continuation package. | |
| An existing matching fully extracted dataset (root manifest.json) is reused. | |
| Archive copies are removed after extraction. Allow around100GB free disk for | |
| extracted dataset/model/checkpoint plus temporary download/checkpoint writes. | |
| Existing HF images are lossless resized256/384 bucket versions; loader reads them | |
| without second resize. Original-PC loader still supports streaming source resize. | |
| No local-PC datasets or feature caches are needed on the new GPU. | |
| ENVIRONMENT: | |
| cd /workspace/llm-to-dit/qwen_rgb_3060 | |
| Use Python3.12 and install a CUDA-enabled torch2.10.0 build for the GPU/driver. | |
| python -m pip install -r requirements-resume.txt | |
| Linux Triton comes from the matching PyTorch dependency; do not install | |
| triton-windows or force the old triton3.5.1 pin. Vendored FLA0.5.2 is in the base | |
| trainer archive. Actual Linux/GPU execution still needs a smoke on your rental: | |
| python train_large_mixed.py --smoke --workers 2 --out checks/new_gpu_smoke | |
| This smoke does forward/backward only; it does not modify the3000 checkpoint. | |
| CONTINUE ONLY WHEN YOU CHOOSE A DURATION, for example500 MORE updates: | |
| python resume_train.py --additional-updates 500 --render-after | |
| or an absolute final step: | |
| python resume_train.py --until 4000 --render-after | |
| or: ADDITIONAL_UPDATES=500 WORKERS=4 bash run_cloud.sh | |
| None of these commands has been run on a remote GPU by this upload. | |
| Resume restores optimizer/RNG/statistics and starts from step3000. | |
| DO NOT run --prepare-means: all accumulated means already live in the checkpoint. | |
| Updated trainer accepts the portable SQLite layout after logical-content validation; | |
| no raw index-byte-size bypass. IDs, sources, geometry, all captions and remaining | |
| optimizer-group ordering have been verified identical to the original dataset. | |
| Plan microbatch8 is retained on resume independently from runtime limits. | |
| Default physical maximum8, adaptive token budget4096, effective batch32. | |
| You may adjust --token-budget or --physical-batch in resume_train.py; the stored | |
| plan still has packets up to8, so increasing physical maximum does not merge them. | |
| No artificial80% VRAM limit. FP32 trainable weights/grads, BF16 autocast, | |
| AdamW8bit, coreLR5e-5, headsLR5e-4, CFG dropout10%, same flow+0.5feature loss. | |
| All24 independent repr heads stay enabled. Captions cycle without truncation. | |
| Resolution mix75/25 for first2epochs, then1percentage-point shift every2epochs. | |
| Outputs: runs/large_mixed_streaming500_3060/ | |
| full_latest.pt -- complete model/optimizer/RNG/statistics, every25updates and final. | |
| train.jsonl -- total loss, flow_loss, feature_loss, weighted_feature_loss, time, VRAM. | |
| index.html -- CFG1/3 generation panels if --render-after is requested. | |
| A STOP file in that run folder requests a graceful checkpoint at update boundary. | |
| A different GPU/kernel stack can change numerical trajectories; exact pixel/bit | |
| identity across hardware is not promised. Do not reset optimizer or recreate means. | |
Xet Storage Details
- Size:
- 4.14 kB
- Xet hash:
- c8b5bb314f1db426c8d2a91cb25c12e64f4b3306c99a44757b0c720d11ed34fe
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.