Hugging Face's logo

ML-Intern-lab
/
nanochat

Model card Files
xet
 Community
nanochat

Commit History

18 days ago

  1. Fix results-dir default to ~/.cache/nanochat

    abidlabs HF Staff
    Verified signature
    c609d6d

  2. Add scaling-sweep aggregator/plotter

    abidlabs HF Staff
    Verified signature
    f89dce7

  3. Add per-point scaling-sweep runner

    abidlabs HF Staff
    Verified signature
    59db131

  4. Smoke SFT: grad-accum=1 batch + batch-scaled LRs (verified)

    abidlabs HF Staff
    Verified signature
    dfd0c6b

  5. Add hf-jobs README: usage, stages, verified findings, costs

    abidlabs HF Staff
    Verified signature
    5ea70c5

  6. Fix: strip -- separator on plain python smoke invocations

    abidlabs HF Staff
    Verified signature
    f3f214e

  7. Pin trackio 0.42.0 install in launcher

    abidlabs HF Staff
    Verified signature
    a1dbaf5

  8. Add trackio bridge for DummyWandb logging

    abidlabs HF Staff
    Verified signature
    0201bc3

  9. Add hf-jobs launcher for Hugging Face Jobs

    abidlabs HF Staff
    Verified signature
    a93e5a1

  1. clean up fragile code

    Andrej Karpathy
    92d63d4

  2. add inference benchmarking

    Andrej Karpathy
    eb16d01

  3. add a few tests

    Andrej Karpathy
    f8a85a5

  4. unify the two optimizer implementations into one and prevent bugs like the one that just happened where i updated one and didnt update the other

    Andrej Karpathy
    a9d0a86

  5. harden and shorten the execution sandbox

    Andrej Karpathy
    f4f69f9

  6. fix deprecated commit on torchao

    Andrej Karpathy
    f527f76

  7. delete datasets dependency bye

    Andrej Karpathy
    a5a3a33

  8. nuke huggingface and its giant dependency footprint from orbit. datasets still todo

    Andrej Karpathy
    4186540

  9. delete spurious 'cute' bloat that doesn't matter to a serious research stack

    Andrej Karpathy
    950a1dc

  10. remove a ton of UI bloat, we need to make nanochat small and sexy and minimal and forkable. code is ~free now, if you want UI get your favorite LLM to add it. nukes ~1000 LOC from orbit nice

    Andrej Karpathy
    da32e1d

  11. Validate checkpoint file format in find_last_step

    Andrej
    Unverified signature
    21a7774

  12. Improve error handling for batch size alignment

    Andrej
    Unverified signature
    6facdee

  13. Remove inherited parameters from scripts

    Andrej
    Unverified signature
    79aaf1d

  14. Merge pull request #540 from svlandeg/fix/kernel⁠

    Andrej
    Unverified signature
    2fc63c9

  15. Small doc fixes for consistency

    Andrej
    Unverified signature
    dbe6e2a

  16. add a comment clarifying that RoPE here rotates by -theta (transpose of the textbook convention), which is functionally equivalent. Prompted by #489, thank you @qdrk⁠

    Andrej Karpathy +1: Claude Fable 5
    3616a02

  17. fix token_bytes calculation to use raw token bytes instead of round-tripping through str, which corrupts tokens that are not valid standalone UTF-8 (e.g. bytes >= 0x80 became 3-byte U+FFFD). 193/32768 tokens were affected, so bpb of new runs will read ~0.05% higher than old runs. Thank you @ellenjxu #693⁠

    Andrej Karpathy +1: Claude Fable 5
    2ce972a

  18. docs: note that MPS handles bf16 fine on recent macOS (opt in via NANOCHAT_DTYPE=bfloat16, ~25% memory savings). Thank you @eaglstun #793 and @ifsheldon #685⁠

    Andrej Karpathy +1: Claude Fable 5
    5e43f1e

  19. allocate the inference KV cache in COMPUTE_DTYPE instead of hardcoding cuda->bf16, fixing bf16 inference (e.g. NANOCHAT_DTYPE=bfloat16 on MPS) and deleting an old hack. Thank you @mparrett #795⁠

    Andrej Karpathy +1: Claude Fable 5
    ca366eb

  20. fix mixed-dtype ops in the optimizer so that bf16 params (wte, value_embeds) work on MPS, which unlike CUDA refuses to promote dtypes. AdamW math now runs explicitly in fp32; verified bit-identical on CUDA, where inductor was already upcasting in registers. Thank you @mparrett for the report and investigation in #741⁠

    Andrej Karpathy +1: Claude Fable 5
    e02b339

  21. adding MuonEq and Muon+, both seem to be very slight improvements at both d12 and d24. contributed by @jn2clark

    Andrej Karpathy
    4e014d3

  1. fix link

    svlandeg
    9ec05d9

  2. rename fwe to climbmix

    svlandeg
    fbb50ad

  3. Merge branch 'master' into feat/small_fixes

    svlandeg
    07e2103

  4. delete the whole report thing i think it was a bad idea it just bloats the code. doing this subtracts 540 LOC

    Andrej Karpathy
    f10bd75

  1. tried and failed at DyT

    Andrej Karpathy
    dc54a1a

  1. Merge branch 'master' into fix-batch-size-assertion

    suraj-self
    0f83057

  1. Merge pull request #706 from svlandeg/fix/cpu⁠

    Andrej
    Unverified signature
    0aaca56

  1. Merge pull request #686 from marcinbogdanski/fix/init-smear-backout-lambdas⁠

    Andrej
    Unverified signature
    b9b6ce1

  2. add required -i to example chat_eval scripts

    svlandeg
    bc78a71

  3. add comment

    Sofie Van Landeghem
    Unverified signature
    a3ca42a

  4. use nn.init and initialize smear gate's weight as well

    Sofie Van Landeghem
    Unverified signature
    9822cc7

  5. update uv lock

    svlandeg
    12839c1

  6. add setuptools for CPU run

    svlandeg
    8ef90bc

  1. fix: initialize smear and backout lambdas in init_weights

    MarcinGUmeR
    94b73ad

  1. fix layer comment

    svlandeg
    b4e6763

  2. print CORE score in chat_eval

    svlandeg
    c2082b3

  1. update numbers from 3h to 1.5h

    svlandeg
    da13719

  1. Merge branch 'master' into fix-batch-size-assertion

    suraj-self
    d8bbddb

  1. create a group for dev dependencies, there is no need to install all this other stuff just for speedrun and it's exposing people to dependency chain attacks. we need to delete more dependencies. dependencies bad bad bad

    Andrej Karpathy
    a445144

  2. delete non-essential deps from legacy use

    Andrej Karpathy
    03be953