| --- |
| base_model: SanjiWatsuki/Kunoichi-DPO-v2-7B |
| language: |
| - en |
| license: cc-by-nc-4.0 |
| tags: |
| - text-generation-inference |
| - transformers |
| - quantized |
| - 4-bit |
| - AWQ |
| - text-generation |
| - autotrain_compatible |
| - endpoints_compatible |
| - chatml |
| - mistral |
| model_creator: SanjiWatsuki |
| model_name: Kunoichi-DPO-v2-7B |
| model_type: mistral |
| pipeline_tag: text-generation |
| inference: false |
| --- |
| # SanjiWatsuki/Kunoichi-DPO-v2-7B AWQ |
|
|
| - Model creator: [SanjiWatsuki](https://huggingface.co/SanjiWatsuki) |
| - Original model: [Kunoichi-DPO-v2-7B](https://huggingface.co/SanjiWatsuki/Kunoichi-DPO-v2-7B) |
|
|
| ## Model Summary |
|
|
| | Model | Average | AGIEval | GPT4All | TruthfulQA | Bigbench | |
| |---|---:|---:|---:|---:|---:| |
| | **Kunoichi-DPO-7B**|**58.4**| 45.08 | 74| 66.99| 47.52| |
| | **Kunoichi-DPO-v2-7B**|**58.31**| 44.85| 75.05| 65.69| 47.65| |
| | [Kunoichi-7B](https://huggingface.co/SanjiWatsuki/Kunoichi-7B)|57.54| 44.99| 74.86| 63.72| 46.58| |
| | [OpenPipe/mistral-ft-optimized-1218](https://huggingface.co/OpenPipe/mistral-ft-optimized-1218)| 56.85 | 44.74 | 75.6 | 59.89 | 47.17 | |
| | [Silicon-Maid-7B](https://huggingface.co/SanjiWatsuki/Silicon-Maid-7B) | 56.45| 44.74| 74.26| 61.5| 45.32| |
| | [mlabonne/NeuralHermes-2.5-Mistral-7B](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B) | 53.51 | 43.67 | 73.24 | 55.37 | 41.76 | |
| | [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) | 52.42 | 42.75 | 72.99 | 52.99 | 40.94 | |
| | [openchat/openchat_3.5](https://huggingface.co/openchat/openchat_3.5) | 51.34 | 42.67 | 72.92 | 47.27 | 42.51 | |
| | [berkeley-nest/Starling-LM-7B-alpha](https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha) | 51.16 | 42.06 | 72.72 | 47.33 | 42.53 | |
| | [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) | 50.99 | 37.33 | 71.83 | 55.1 | 39.7 | |
|
|
| ## How to use |
|
|
| ### Install the necessary packages |
|
|
| ```bash |
| pip install --upgrade autoawq autoawq-kernels |
| ``` |
|
|
| ### Example Python code |
|
|
| ```python |
| from awq import AutoAWQForCausalLM |
| from transformers import AutoTokenizer, TextStreamer |
| |
| model_path = "solidrust/Kunoichi-DPO-v2-7B-AWQ" |
| system_message = "You are Kunoichi, incarnated as a powerful AI." |
| |
| # Load model |
| model = AutoAWQForCausalLM.from_quantized(model_path, |
| fuse_layers=True) |
| tokenizer = AutoTokenizer.from_pretrained(model_path, |
| trust_remote_code=True) |
| streamer = TextStreamer(tokenizer, |
| skip_prompt=True, |
| skip_special_tokens=True) |
| |
| # Convert prompt to tokens |
| prompt_template = """\ |
| <|im_start|>system |
| {system_message}<|im_end|> |
| <|im_start|>user |
| {prompt}<|im_end|> |
| <|im_start|>assistant""" |
| |
| prompt = "You're standing on the surface of the Earth. "\ |
| "You walk one mile south, one mile west and one mile north. "\ |
| "You end up exactly where you started. Where are you?" |
| |
| tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt), |
| return_tensors='pt').input_ids.cuda() |
| |
| # Generate output |
| generation_output = model.generate(tokens, |
| streamer=streamer, |
| max_new_tokens=512) |
| |
| ``` |
|
|
| ### About AWQ |
|
|
| AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings. |
|
|
| AWQ models are currently supported on Linux and Windows, with NVidia GPUs only. macOS users: please use GGUF models instead. |
|
|
| It is supported by: |
|
|
| - [Text Generation Webui](https://github.com/oobabooga/text-generation-webui) - using Loader: AutoAWQ |
| - [vLLM](https://github.com/vllm-project/vllm) - version 0.2.2 or later for support for all model types. |
| - [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) |
| - [Transformers](https://huggingface.co/docs/transformers) version 4.35.0 and later, from any code or client that supports Transformers |
| - [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) - for use from Python code |
|
|
| ## Prompt template: ChatML |
|
|
| ```plaintext |
| <|im_start|>system |
| {system_message}<|im_end|> |
| <|im_start|>user |
| {prompt}<|im_end|> |
| <|im_start|>assistant |
| ``` |
|
|
|
|
