| | --- |
| | license: apache-2.0 |
| | --- |
| | |
| | A toy Llama adapted from [JackFram/llama-160m](https://huggingface.co/JackFram/llama-160m) with special tokens added. |
| |
|
| | This checkpoint can be loaded into MASE's `LlamaQuantized` |
| |
|
| | ```python |
| | from transformers.models.llama import LlamaTokenizer |
| | from chop.models.manual.llama_quantized import ( |
| | LlamaQuantizedConfig, |
| | LlamaQuantizedForCausalLM, |
| | ) |
| | |
| | name="Cheng98/llama-160m" |
| | tokenizer = LlamaTokenizer.from_pretrained(name) |
| | |
| | # override the quant_config to quantized the model |
| | # default does not quantize llama |
| | config = LlamaQuantizedConfig.from_pretrained( |
| | name, |
| | # quant_config="./quant_config_na.toml" |
| | |
| | ) |
| | |
| | llama = LlamaQuantizedForCausalLM.from_pretrained( |
| | name, |
| | config=config, |
| | ) |
| | ``` |
