| | --- |
| | license: gemma |
| | language: |
| | - en |
| | base_model: |
| | - google/gemma-2-2b-it |
| | pipeline_tag: text-generation |
| | library_name: transformers |
| | tags: |
| | - gemma |
| | - 2b |
| | - CoT |
| | - text-generation-inference |
| | - gwq2b |
| | - gemma-with-question |
| | - safetensors |
| | model-index: |
| | - name: GWQ2b |
| | results: |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: IFEval (0-Shot) |
| | type: wis-k/instruction-following-eval |
| | split: train |
| | args: |
| | num_few_shot: 0 |
| | metrics: |
| | - type: inst_level_strict_acc and prompt_level_strict_acc |
| | value: 41.15 |
| | name: averaged accuracy |
| | source: |
| | url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FGWQ2b |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: BBH (3-Shot) |
| | type: SaylorTwift/bbh |
| | split: test |
| | args: |
| | num_few_shot: 3 |
| | metrics: |
| | - type: acc_norm |
| | value: 17.68 |
| | name: normalized accuracy |
| | source: |
| | url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FGWQ2b |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: MATH Lvl 5 (4-Shot) |
| | type: lighteval/MATH-Hard |
| | split: test |
| | args: |
| | num_few_shot: 4 |
| | metrics: |
| | - type: exact_match |
| | value: 6.12 |
| | name: exact match |
| | source: |
| | url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FGWQ2b |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: GPQA (0-shot) |
| | type: Idavidrein/gpqa |
| | split: train |
| | args: |
| | num_few_shot: 0 |
| | metrics: |
| | - type: acc_norm |
| | value: 4.36 |
| | name: acc_norm |
| | source: |
| | url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FGWQ2b |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: MuSR (0-shot) |
| | type: TAUR-Lab/MuSR |
| | args: |
| | num_few_shot: 0 |
| | metrics: |
| | - type: acc_norm |
| | value: 12.76 |
| | name: acc_norm |
| | source: |
| | url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FGWQ2b |
| | name: Open LLM Leaderboard |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | name: MMLU-PRO (5-shot) |
| | type: TIGER-Lab/MMLU-Pro |
| | config: main |
| | split: test |
| | args: |
| | num_few_shot: 5 |
| | metrics: |
| | - type: acc |
| | value: 16.36 |
| | name: accuracy |
| | source: |
| | url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=prithivMLmods%2FGWQ2b |
| | name: Open LLM Leaderboard |
| | --- |
| |  |
| | <a target="_blank" href="https://huggingface.co/spaces/prithivMLmods/GWQ-2B"> |
| | <img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="gwq2b.hf.space"/> |
| | </a> |
| | # **GWQ2b - Gemma with Questions2b** |
| |
|
| | GWQ2b is a family of lightweight, state-of-the-art open models from Google, built using the same research and technology employed to create the Gemini models. These models are text-to-text, decoder-only large language models, available in English, with open weights for both pre-trained and instruction-tuned variants. GWQ2b models are well-suited for a variety of text generation tasks, including question answering, summarization, and reasoning. GWQ2b is fine-tuned on the Chain of Continuous Thought Synthetic Dataset, built upon the Gemma2forCasualLM architecture. |
| |
|
| | # **Running GWQ2b Demo** |
| |
|
| | ```python |
| | # pip install accelerate |
| | from transformers import AutoTokenizer, AutoModelForCausalLM |
| | import torch |
| | |
| | tokenizer = AutoTokenizer.from_pretrained("prithivMLmods/GWQ2b") |
| | model = AutoModelForCausalLM.from_pretrained( |
| | "prithivMLmods/GWQ2b", |
| | device_map="auto", |
| | torch_dtype=torch.bfloat16, |
| | ) |
| | |
| | input_text = "Write me a poem about Machine Learning." |
| | input_ids = tokenizer(input_text, return_tensors="pt").to("cuda") |
| | |
| | outputs = model.generate(**input_ids, max_new_tokens=32) |
| | print(tokenizer.decode(outputs[0])) |
| | ``` |
| |
|
| | You can ensure the correct chat template is applied by using `tokenizer.apply_chat_template` as follows: |
| | ```python |
| | messages = [ |
| | {"role": "user", "content": "Write me a poem about Machine Learning."}, |
| | ] |
| | input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda") |
| | |
| | outputs = model.generate(**input_ids, max_new_tokens=256) |
| | print(tokenizer.decode(outputs[0])) |
| | ``` |
| | # **Key Architecture** |
| |
|
| | 1. **Transformer-Based Design**: |
| | GWQ2b leverages the transformer architecture, utilizing self-attention mechanisms to process input text and capture contextual relationships effectively. |
| |
|
| | 2. **Lightweight and Efficient**: |
| | It is designed to be computationally efficient, with fewer parameters compared to larger models, making it ideal for deployment on resource-constrained devices or environments. |
| |
|
| | 3. **Modular Layers**: |
| | The architecture consists of modular encoder and decoder layers, allowing flexibility in adapting the model for specific tasks like text generation, summarization, or classification. |
| |
|
| | 4. **Attention Mechanisms**: |
| | GWQ2b employs multi-head self-attention to focus on relevant parts of the input text, improving its ability to handle long-range dependencies and complex language structures. |
| |
|
| | 5. **Pre-training and Fine-Tuning**: |
| | The model is pre-trained on large text corpora and can be fine-tuned for specific tasks, such as markdown processing in ReadM.Md, to enhance its performance on domain-specific data. |
| |
|
| | 6. **Scalability**: |
| | The architecture supports scaling up or down based on the application's requirements, balancing performance and resource usage. |
| |
|
| | 7. **Open-Source and Customizable**: |
| | Being open-source, GWQ2b allows developers to modify and extend its architecture to suit specific use cases, such as integrating it into tools like ReadM.Md for markdown-related tasks. |
| |
|
| | # **Intended Use of GWQ2b (Gemma with Questions2b)** |
| |
|
| | 1. **Question Answering:** |
| | The model excels in generating concise and relevant answers to user-provided queries across various domains. |
| |
|
| | 2. **Summarization:** |
| | It can be used to summarize large bodies of text, making it suitable for news aggregation, academic research, and report generation. |
| |
|
| | 3. **Reasoning Tasks:** |
| | GWQ2b is fine-tuned on the Chain of Continuous Thought Synthetic Dataset, which enhances its ability to perform reasoning, multi-step problem solving, and logical inferences. |
| |
|
| | 4. **Text Generation:** |
| | The model is ideal for creative writing tasks such as generating poems, stories, and essays. It can also be used for generating code comments, documentation, and markdown files. |
| |
|
| | 5. **Instruction Following:** |
| | GWQ2b’s instruction-tuned variant is suitable for generating responses based on user instructions, making it useful for virtual assistants, tutoring systems, and automated customer support. |
| |
|
| | 6. **Domain-Specific Applications:** |
| | Thanks to its modular design and open-source nature, the model can be fine-tuned for specific tasks like legal document summarization, medical record analysis, or financial report generation. |
| |
|
| | # **Limitations of GWQ2b** |
| |
|
| | 1. **Resource Requirements:** |
| | Although lightweight compared to larger models, the 9B parameter size still requires significant computational resources, including GPUs with large memory for inference. |
| |
|
| | 2. **Knowledge Cutoff:** |
| | The model’s pre-training data may not include recent information, making it less effective for answering queries on current events or newly developed topics. |
| |
|
| | 3. **Bias in Outputs:** |
| | Since the model is trained on publicly available datasets, it may inherit biases present in those datasets, leading to potentially biased or harmful outputs in sensitive contexts. |
| |
|
| | 4. **Hallucinations:** |
| | Like other large language models, GWQ2b can occasionally generate incorrect or nonsensical information, especially when asked for facts or reasoning outside its training scope. |
| |
|
| | 5. **Lack of Common-Sense Reasoning:** |
| | While GWQ2b is fine-tuned for reasoning, it may still struggle with tasks requiring deep common-sense knowledge or nuanced understanding of human behavior and emotions. |
| |
|
| | 6. **Dependency on Fine-Tuning:** |
| | For optimal performance on domain-specific tasks, fine-tuning on relevant datasets is required, which demands additional computational resources and expertise. |
| | |
| | 7. **Context Length Limitation:** |
| | The model’s ability to process long documents is limited by its maximum context window size. If the input exceeds this limit, truncation may lead to loss of important information. |
| | # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
| | Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/prithivMLmods__GWQ2b-details)! |
| | Summarized results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/contents/viewer/default/train?q=prithivMLmods%2FGWQ2b&sort[column]=Average%20%E2%AC%86%EF%B8%8F&sort[direction]=desc)! |
| |
|
| | | Metric |Value (%)| |
| | |-------------------|--------:| |
| | |**Average** | 16.40| |
| | |IFEval (0-Shot) | 41.15| |
| | |BBH (3-Shot) | 17.68| |
| | |MATH Lvl 5 (4-Shot)| 6.12| |
| | |GPQA (0-shot) | 4.36| |
| | |MuSR (0-shot) | 12.76| |
| | |MMLU-PRO (5-shot) | 16.36| |
| |
|
| |
|
