Kernels:

phanerozoic
/

model-as-a-kernel

Kernel card Files Files and versions

phanerozoic commited on Jul 14

Commit

af194d8

·

verified ·

1 Parent(s): e33875a

batched decode

Files changed (1) hide show

README.md +3 -3

README.md CHANGED Viewed

@@ -180,7 +180,7 @@ to 1673 at batch 16 (6.5x aggregate).
 One CUDA device, greedy sampling in-kernel (route the fp32 logits to an
 external sampler for anything else). Llama-family architectures without
 rope attention scaling or attention biases. Weights are dense bf16 or
-bitsandbytes nf4 (4-bit checkpoints load with their weights kept packed
-and dequantized in the kernel). Prompts prefill in chunks of at most eight
-tokens. The program bakes device pointers, so the `MegaModel` owns its
 weights and buffers for its lifetime.

 One CUDA device, greedy sampling in-kernel (route the fp32 logits to an
 external sampler for anything else). Llama-family architectures without
 rope attention scaling or attention biases. Weights are dense bf16 or
+bitsandbytes nf4 (nf4 weights are stored packed and dequantized in the
+kernel). Prompts prefill in chunks of at most eight tokens. The program
+bakes device pointers, so the `MegaModel` owns its
 weights and buffers for its lifetime.