| --- |
| library_name: transformers |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - panolm |
| - linear-attention |
| --- |
| |
| # PanoLM-380M |
|
|
| This repo contains the base 380M PanoLM model, which is a linear-attention causal language. |
|
|
| ## Training Data |
|
|
| Pretrained on a weighted mixture of three web-scale English corpora: |
|
|
| | Weight | Dataset | |
| |-------:|------------------------| |
| | 0.45 | FineWeb-Edu (100B-token subset) | |
| | 0.30 | DCLM (100B-token subset) | |
| | 0.25 | FinePDFs-Edu (100B-token subset) | |
|
|
| ## Requirements |
|
|
| ```text |
| torch==2.12.0 |
| transformers==5.8.1 |
| flash-linear-attention==0.5.0 |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "PanocularAI/PanoLM-380M", |
| trust_remote_code=True, |
| ).cuda() # fla's RMSNorm uses Triton kernels that only run on CUDA tensors. |
| print(model) |
| tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True) |
| |
| prompt = "I am PanoLM, an edge device friendly language model." |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| # use_cache=False: HF's generate() would pass a DynamicCache that fla's KDA |
| # layer indexes as a list, which the new transformers API no longer supports. |
| outputs = model.generate( |
| **inputs, |
| max_length=512, |
| top_k=10, |
| use_cache=False, |
| do_sample=True, |
| trust_remote_code=True, |
| ) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Evaluation |
|
|
| All scores are 0-shot, evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). |
| For multi-choice tasks we report length-normalized accuracy (`acc_norm`); for |
| Based-suite recall tasks we report `contains` (soft answer match), which is the |
| metric the suite was designed around. |
|
|
| ### Commonsense reasoning & general QA |
|
|
| | Task | Metric | Value | Stderr | |
| |----------------|----------|-------:|----------| |
| | arc_challenge | acc_norm | 0.2910 | ± 0.0133 | |
| | arc_easy | acc_norm | 0.5349 | ± 0.0102 | |
| | commonsense_qa | acc | 0.1892 | ± 0.0112 | |
| | hellaswag | acc_norm | 0.4137 | ± 0.0049 | |
| | piqa | acc_norm | 0.6741 | ± 0.0109 | |
| | winogrande | acc | 0.5304 | ± 0.0140 | |
| |
| ### Associative recall (Based suite) |
| |
| | Task | Metric | Value | |
| |----------------|----------|-------:| |
| | drop | contains | 0.2286 | |
| | fda | contains | 0.0499 | |
| | nq_2048 | contains | 0.0687 | |
| | squadv2 | contains | 0.3542 | |
| | swde | contains | 0.2439 | |
| | triviaqa | contains | 0.4680 | |
|
|
