---
library_name: transformers
license: apache-2.0
language:
  - en
pipeline_tag: text-generation
tags:
  - panolm
  - linear-attention
---

# PanoLM-380M

This repo contains the base 380M PanoLM model, which is a linear-attention causal language.

## Training Data

Pretrained on a weighted mixture of three web-scale English corpora:

| Weight | Dataset                |
|-------:|------------------------|
|   0.45 | FineWeb-Edu (100B-token subset)  |
|   0.30 | DCLM (100B-token subset)         |
|   0.25 | FinePDFs-Edu (100B-token subset) |

## Requirements

```text
torch==2.12.0
transformers==5.8.1
flash-linear-attention==0.5.0
```

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "PanocularAI/PanoLM-380M",
    trust_remote_code=True,
).cuda()  # fla's RMSNorm uses Triton kernels that only run on CUDA tensors.
print(model)
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)

prompt = "I am PanoLM, an edge device friendly language model."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
# use_cache=False: HF's generate() would pass a DynamicCache that fla's KDA
# layer indexes as a list, which the new transformers API no longer supports.
outputs = model.generate(
    **inputs,
    max_length=512,
    top_k=10,
    use_cache=False,
    do_sample=True,
    trust_remote_code=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Evaluation

All scores are 0-shot, evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).
For multi-choice tasks we report length-normalized accuracy (`acc_norm`); for
Based-suite recall tasks we report `contains` (soft answer match), which is the
metric the suite was designed around.

### Commonsense reasoning & general QA

| Task           | Metric   |  Value | Stderr   |
|----------------|----------|-------:|----------|
| arc_challenge  | acc_norm | 0.2910 | ± 0.0133 |
| arc_easy       | acc_norm | 0.5349 | ± 0.0102 |
| commonsense_qa | acc      | 0.1892 | ± 0.0112 |
| hellaswag      | acc_norm | 0.4137 | ± 0.0049 |
| piqa           | acc_norm | 0.6741 | ± 0.0109 |
| winogrande     | acc      | 0.5304 | ± 0.0140 |

### Associative recall (Based suite)

| Task           | Metric   |  Value |
|----------------|----------|-------:|
| drop     | contains | 0.2286 |
| fda      | contains | 0.0499 |
| nq_2048  | contains | 0.0687 |
| squadv2  | contains | 0.3542 |
| swde     | contains | 0.2439 |
| triviaqa | contains | 0.4680 |