| | --- |
| | base_model: Locutusque/Hercules-3.1-Mistral-7B |
| | license: apache-2.0 |
| | library_name: transformers |
| | tags: |
| | - chemistry |
| | - biology |
| | - code |
| | - medical |
| | - quantized |
| | - 4-bit |
| | - AWQ |
| | - text-generation |
| | - autotrain_compatible |
| | - endpoints_compatible |
| | - chatml |
| | datasets: |
| | - Locutusque/Hercules-v3.0 |
| | model-index: |
| | - name: Hercules-3.1-Mistral-7B |
| | results: |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: AI2 Reasoning Challenge (25-Shot) |
| | type: ai2_arc |
| | config: ARC-Challenge |
| | split: test |
| | args: |
| | num_few_shot: 25 |
| | metrics: |
| | - type: acc_norm |
| | value: 61.18 |
| | name: normalized accuracy |
| | source: |
| | url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/Hercules-3.1-Mistral-7B |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: HellaSwag (10-Shot) |
| | type: hellaswag |
| | split: validation |
| | args: |
| | num_few_shot: 10 |
| | metrics: |
| | - type: acc_norm |
| | value: 83.55 |
| | name: normalized accuracy |
| | source: |
| | url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/Hercules-3.1-Mistral-7B |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: MMLU (5-Shot) |
| | type: cais/mmlu |
| | config: all |
| | split: test |
| | args: |
| | num_few_shot: 5 |
| | metrics: |
| | - type: acc |
| | value: 63.65 |
| | name: accuracy |
| | source: |
| | url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/Hercules-3.1-Mistral-7B |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: TruthfulQA (0-shot) |
| | type: truthful_qa |
| | config: multiple_choice |
| | split: validation |
| | args: |
| | num_few_shot: 0 |
| | metrics: |
| | - type: mc2 |
| | value: 42.83 |
| | source: |
| | url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/Hercules-3.1-Mistral-7B |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: Winogrande (5-shot) |
| | type: winogrande |
| | config: winogrande_xl |
| | split: validation |
| | args: |
| | num_few_shot: 5 |
| | metrics: |
| | - type: acc |
| | value: 79.01 |
| | name: accuracy |
| | source: |
| | url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/Hercules-3.1-Mistral-7B |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: GSM8k (5-shot) |
| | type: gsm8k |
| | config: main |
| | split: test |
| | args: |
| | num_few_shot: 5 |
| | metrics: |
| | - type: acc |
| | value: 42.3 |
| | name: accuracy |
| | source: |
| | url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/Hercules-3.1-Mistral-7B |
| | name: Open LLM Leaderboard |
| | language: |
| | - en |
| | model_creator: Locutusque |
| | model_name: Hercules-3.1-Mistral-7B |
| | model_type: mistral |
| | pipeline_tag: text-generation |
| | inference: false |
| | prompt_template: '<|im_start|>system |
| | |
| | {system_message}<|im_end|> |
| | |
| | <|im_start|>user |
| | |
| | {prompt}<|im_end|> |
| | |
| | <|im_start|>assistant |
| | |
| | ' |
| | quantized_by: Suparious |
| | --- |
| | # Model Card: Hercules-3.1-Mistral-7B |
| |
|
| | - Model creator: [Locutusque](https://huggingface.co/Locutusque) |
| | - Original model: [Hercules-3.1-Mistral-7B](https://huggingface.co/Locutusque/Hercules-3.1-Mistral-7B) |
| |
|
| |  |
| |
|
| | ## Model Description |
| |
|
| | Hercules-3.1-Mistral-7B is a fine-tuned language model derived from Mistralai/Mistral-7B-v0.1. It is specifically designed to excel in instruction following, function calls, and conversational interactions across various scientific and technical domains. The dataset used for fine-tuning, also named Hercules-v3.0, expands upon the diverse capabilities of OpenHermes-2.5 with contributions from numerous curated datasets. This fine-tuning has hercules-v3.0 with enhanced abilities in: |
| |
|
| | - Complex Instruction Following: Understanding and accurately executing multi-step instructions, even those involving specialized terminology. |
| | - Function Calling: Seamlessly interpreting and executing function calls, providing appropriate input and output values. |
| | - Domain-Specific Knowledge: Engaging in informative and educational conversations about Biology, Chemistry, Physics, Mathematics, Medicine, Computer Science, and more. |
| |
|
| | ## How to use |
| |
|
| | ### Install the necessary packages |
| |
|
| | ```bash |
| | pip install --upgrade autoawq autoawq-kernels |
| | ``` |
| |
|
| | ### Example Python code |
| |
|
| | ```python |
| | from awq import AutoAWQForCausalLM |
| | from transformers import AutoTokenizer, TextStreamer |
| | |
| | model_path = "solidrust/Hercules-3.1-Mistral-7B-AWQ" |
| | system_message = "You are Senzu, incarnated as a powerful AI." |
| | |
| | # Load model |
| | model = AutoAWQForCausalLM.from_quantized(model_path, |
| | fuse_layers=True) |
| | tokenizer = AutoTokenizer.from_pretrained(model_path, |
| | trust_remote_code=True) |
| | streamer = TextStreamer(tokenizer, |
| | skip_prompt=True, |
| | skip_special_tokens=True) |
| | |
| | # Convert prompt to tokens |
| | prompt_template = """\ |
| | <|im_start|>system |
| | {system_message}<|im_end|> |
| | <|im_start|>user |
| | {prompt}<|im_end|> |
| | <|im_start|>assistant""" |
| | |
| | prompt = "You're standing on the surface of the Earth. "\ |
| | "You walk one mile south, one mile west and one mile north. "\ |
| | "You end up exactly where you started. Where are you?" |
| | |
| | tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt), |
| | return_tensors='pt').input_ids.cuda() |
| | |
| | # Generate output |
| | generation_output = model.generate(tokens, |
| | streamer=streamer, |
| | max_new_tokens=512) |
| | |
| | ``` |
| |
|
| | ### About AWQ |
| |
|
| | AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings. |
| |
|
| | AWQ models are currently supported on Linux and Windows, with NVidia GPUs only. macOS users: please use GGUF models instead. |
| |
|
| | It is supported by: |
| |
|
| | - [Text Generation Webui](https://github.com/oobabooga/text-generation-webui) - using Loader: AutoAWQ |
| | - [vLLM](https://github.com/vllm-project/vllm) - version 0.2.2 or later for support for all model types. |
| | - [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) |
| | - [Transformers](https://huggingface.co/docs/transformers) version 4.35.0 and later, from any code or client that supports Transformers |
| | - [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) - for use from Python code |
| |
|
| | ## Prompt template: ChatML |
| |
|
| | ```plaintext |
| | <|im_start|>system |
| | {system_message}<|im_end|> |
| | <|im_start|>user |
| | {prompt}<|im_end|> |
| | <|im_start|>assistant |
| | ``` |
| |
|
