Broyojo/world-pretraining-runs

Does Video Pretraining Help a Transformer Learn Language? Runs and code

This repository holds the code, training metrics, and final weights for the experiments in the paper Does Video Pretraining Help a Transformer Learn Language? (David Andrews and Chao Zhang, Georgia Institute of Technology). We take iVideoGPT models trained only on robot video (thuml/ivideogpt-oxe-64-act-free-medium and thuml/ivideogpt-oxe-64-act-free), replace their input and output layers, and train them on FineWeb-Edu with the GPT-2 tokenizer.

Layout

• code/wp/train.py trains one condition. probe.py, rank.py, tables.py, act_table.py, extra_tables.py, and report_figs.py produce the probes, measurements, tables, and figures in the paper.
• code/slurm/ holds the job queue used on the cluster.
• runs/<run>/ holds one training run: metrics.jsonl (training and validation loss by step), args.json (all settings), hardware.txt, and, for most runs, the final weights as a Hugging Face Llama model with the GPT-2 tokenizer.

Run names

Medium model (400M backbone): [prefix-]<init>-<freeze>-lr<peak LR>-<token budget>[-s<seed>]. Small model (113M backbone): act-<source>-<init>-none-lr<peak LR>-<token budget>[-s<seed>].

• init: scratch, ivgpt (pretrained weights as they are), shuffled, rescaled, sp<λ> (shrink-and-perturb), spshuf<λ> (shrink-and-perturb on shuffled weights), nobody (no backbone).
• freeze: none (full training), body (frozen backbone), lora, top4 (four new layers on a frozen backbone).
• source (small model): sm (scratch), base (pretrained), bfree and bact (BAIR fine-tunes without and with actions).
• A run without a seed suffix uses seed 0.

Some runs have metrics but no weights, because an upload failed after the account's private storage limit was reached. Weights were not saved for those runs.