|
|
--- |
|
|
library_name: transformers |
|
|
tags: [] |
|
|
--- |
|
|
|
|
|
<p align="center"> |
|
|
<img src="figs/bonsai.png" width="200" alt="Bonsai Logo"> |
|
|
|
|
|
<h3 align="center" style="font-size: 30px">Bonsai: A Small Ternary-Weight Language Model</h3> |
|
|
</p> |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
|
|
Bonsai is a small 500 million parameter ternary weight language model trained by deepgrove. Bonsai adopts the Llama architecture and Mistral tokenizer following [Danube 3](https://arxiv.org/pdf/2407.09276v1), with modified linear layers to support ternary weights. The model has been trained primarily using DCLM-Pro and Fineweb-Edu. Bonsai marks a new paradigm of efficiency, being trained in less than 5 billion tokens. |
|
|
|
|
|
- **Developed by:** deepgrove |
|
|
- **Language(s) (NLP):** English |
|
|
- **License:** Apache-2 |
|
|
- **Repository:** https://github.com/deepgrove-ai/Bonsai |
|
|
- **Paper:** https://github.com/deepgrove-ai/Bonsai/tree/main/paper/Bonsai.pdf |
|
|
|
|
|
## Usage |
|
|
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
|
|
Bonsai can be easily used through the Huggingface Transformers library. However, we note that all operations are currently performed in 16 bit precision; we're currently working towards integrating our model design with custom mixed precision kernels. A quick example follows: |
|
|
|
|
|
```{python} |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("deepgrove/Bonsai", trust_remote_code=True) |
|
|
model = AutoModelForCausalLM.from_pretrained("deepgrove/Bonsai", trust_remote_code=True) |
|
|
text = "What is the capital of France?" |
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_length=100) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
We note that Bonsai is not instruction tuned; we highly recommend finetuning the model before usage in a downstream task. |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
|
|
Bonsai achieves competitive performance among its peers, being one of the first ternary models to do so. Evalution results are below; for more detailed results and comparisons to other ternary models, please see the accompanying paper linked above. We use lm-eval for all benchmarks outside of MMLU and lighteval's cloze formulation for MMLU. |
|
|
|
|
|
<div align="center"> |
|
|
|
|
|
| Model | ARC-c | ARC-e | HS. | OBQA | PiQA | Wino. | MMLU | Avg | |
|
|
|-------|--------|--------|------|-------|-------|--------|-------|-----| |
|
|
| MobiLlama 0.5B | 26.62 | 46.68 | 51.66 | 30.00 | 71.65 | 54.50 | 28.61 | 44.25 | |
|
|
| Qwen 2 0.5B | 28.84 | 50.29 | 49.12 | 33.00 | 69.26 | 56.99 | 31.78 | 45.61 | |
|
|
| MobileLLM 600M | 29.01 | 56.65 | 55.35 | 34.00 | 71.65 | 59.75 | 31.40 | 48.13 | |
|
|
| Qwen 2.5 0.5B | 32.25 | 58.29 | 52.18 | 35.40 | 69.91 | 56.12 | 33.40 | 48.22 | |
|
|
| **Bonsai** | 33.36 | 57.95 | 48.04 | 34.00 | 70.24 | 54.85 | 30.28 | 46.96 | |
