Hugging Face's logo

ML-Intern-lab
/
nanochat

Model card Files
xet
 Community
nanochat / scripts /chat_sft.py

Commit History

  1. delete spurious 'cute' bloat that doesn't matter to a serious research stack

    Andrej Karpathy
    950a1dc

  2. Improve error handling for batch size alignment

    Andrej
    Unverified signature
    6facdee

  1. delete the whole report thing i think it was a bad idea it just bloats the code. doing this subtracts 540 LOC

    Andrej Karpathy
    f10bd75

  1. Merge branch 'master' into fix-batch-size-assertion

    suraj-self
    9f9ef95

  1. Merge pull request #634 from 2bitbit/fix-docs-and-comments⁠

    Andrej
    Unverified signature
    a4ed966

  1. MMLU main split is named auxiliary_train, not train

    Mathieu Lacage
    a641b6c

  1. fix: correct minor typos in help text, README, and comments

    2bitbit
    2bb93b2

  1. Extend informative assertion message to chat_sft.py for consistency

    suraj-self
    f2899a1

  1. delete autocast, an unnecessary thorn in my side, manage dtypes directly

    Andrej Karpathy
    1076f97

  2. Ensure that inputs and targets are contiguous (#569)⁠

    Sofie Van Landeghem
    Unverified signature
    752abc8

  1. Restore completion-only loss masking in SFT dataloader (#582)⁠

    Anish
    Unverified signature
    83dccc2

  1. tune the data mixture a bit, load optimizer by default when SFT. These were confirmed to be best settings from sweeps of sft

    Andrej Karpathy
    8180e1d

  2. a number of upgrades to SFT script to bring it up to date w.r.t. pretraining and tuning some of its kwargs based on sweeps

    Andrej Karpathy
    788dade

  1. fix bug in chat_sft, the attention window must be preserved sigh

    Andrej Karpathy
    8b4849d

  2. fix path which i think was modified during the refactor and this is a bug introduced by claude i believe

    Andrej Karpathy
    eaf49a3

  1. nuke midtraining from orbit, it's not as needed now that we have a BOS-aligned dataloader. Also change the README a lot. midtrianing is not yet fully properly erased across the board, but good enough for step 1

    Andrej Karpathy
    1ddaad1

  1. Combine AdamW and Muon into single MuonAdamW optimizer, cleaner, ty @chrisjmccormick for idea/help

    Andrej Karpathy
    41bb2ea

  1. fix buggy midtrain and update all kwargs to be idiomatic. that is, argparse uses dashes variables use underscores. the underscores are just a remnant of the previous Configurator object. This is the right way

    Andrej Karpathy
    7312ec9

  1. delete the configurator in favor of argparse and clean up a lot of kwarg details to make them more consistent across all scripts

    Andrej Karpathy
    eb7bbc1

  1. clean up model_tag handling across scripts a bit more.

    Andrej
    Unverified signature
    088726a

  2. update to new os env var to get rid of deprecation warning

    Andrej Karpathy
    2874eda

  1. bugfix save and load ckpt from model_tag dir

    Kurt232
    92c6654

  1. remove unneeded iter()

    ericsilberstein
    f37d45c

  1. fix typo

    svlandeg
    7031985

  1. add the SpellingBee task so that nanochat can count r in strawberry etc. along the way we had to add a bunch of new functionality, e.g. extend the calculator to support the count function of python. possibly the current TaskMixture uses way too many synthetic examples of SpellingBee because the eval gives us exactly 100% performance on spelling. We can tune this later to reclaim some wall clock time here I think

    Andrej Karpathy
    8892470

  1. merge and resolve conflict

    Andrej Karpathy
    5bdc99a

  2. add personality to nanochat. breaks previous code on git pull and requires download of a new file from s3, but there is a helpful error message so hopefully its ok

    Andrej Karpathy
    fe5aed9

  1. upgrading all other files to be able to use cpu/mps as well as cuda. various minor other changes ,e.g. changing max_iterations to num_iterations in sft script for consistency in naming

    karpathy
    2e9669e

  1. dont evaluate the sampling evals during SFT they are too slow. keep the multiple choice evals. delete unused imports

    Andrej Karpathy
    190d951

  1. initial commit

    karpathy
    3a5e0bc