scKevType / README.md
pwwang's picture
Licence: MIT
aa1e368 verified
|
Raw History Blame Contribute Delete
6.16 kB
---
license: mit
library_name: peft
base_model:
- Qwen/Qwen2.5-0.5B
- Qwen/Qwen3-0.6B-Base
- Qwen/Qwen3.5-0.8B-Base
- Qwen/Qwen3.5-4B-Base
- Qwen/Qwen3.5-9B-Base
- jaredpalmer/kev-27b
tags:
- single-cell
- scrna-seq
- cell-type-annotation
- decision-model
- lora
- biology
---
# scKevType
Cell-type annotation as a **decision**, not a generated sentence.
scKevType is a family of fine-tuned decision models for single-cell RNA-seq cell-type annotation.
Given a rendered state built from a cluster's marker evidence, a model answers one typed question
over a presented set of candidate labels in a single forward pass and returns a **scored option**:
one of the options shown, with a probability attached. Declining is an option like any other.
There is no prose to parse. The answer is a label and a number, so it can be calibrated,
thresholded, compared across models, and audited.
## Why this exists
The annotation field has moved to language models, and those systems generate text. Generated text
has no probability vector, so there is nothing to threshold or calibrate; it also cannot decline, so
a model asked about a cluster it cannot place still answers fluently. Exact-match scoring on
generated text needs a parsing rule, and the rule becomes a scoring decision the model never made.
scKevType removes that layer. The comparison it makes possible is between models answering identical
questions under one scoring rule, rather than between parsers.
## The six arms
Each size is a separate LoRA adapter on a frozen base, trained with one shared recipe. Pick by
accuracy against cost: the 4B and 9B lead in distribution, and the 27B is the best arm on the
held-out laboratory.
| dir | base model | in-distribution | held-out | adapter + head |
|---|---|---:|---:|---:|
| `0.5B/` | `Qwen/Qwen2.5-0.5B` | 0.7561 | 0.8115 | 37 MB |
| `0.6B/` | `Qwen/Qwen3-0.6B-Base` | 0.7586 | 0.7281 | 43 MB |
| `0.8B/` | `Qwen/Qwen3.5-0.8B-Base` | 0.7898 | 0.7950 | 45 MB |
| `4B/` | `Qwen/Qwen3.5-4B-Base` | 0.8191 | 0.8492 | 135 MB |
| `9B/` | `Qwen/Qwen3.5-9B-Base` | 0.8191 | 0.8988 | 182 MB |
| `27B/` | `jaredpalmer/kev-27b` | 0.7928 | 0.9100 | 478 MB |
*In-distribution*: served accuracy on 4,035 `panel_choice` questions from the training corpus
(exact match over the option set, abstain available on a subset).
*Held-out*: served accuracy on 2,600 questions from Baron pancreas data, a laboratory absent from
training. The two suites have different label counts and majority-class floors (0.1836 against
0.3865), so **raw levels are not comparable across them**; read each column on its own.
## Repository layout
```
0.5B/ 0.6B/ 0.8B/ 4B/ 9B/ 27B/
adapter_model.safetensors the LoRA adapter, rank 16 on every linear projection
adapter_config.json with the base model and its pinned revision
head.pt the pointer head
tokenizer.json, tokenizer_config.json, chat_template.jinja
provenance.json the run's own record: data read, epochs, seconds, loss
training_config.json hyperparameters and placement
manifest.json per-arm base, revision, checksums, reported scores
```
The base weights are **not** included. Each arm is an adapter over a public base.
## Loading
This is not a `from_pretrained` drop-in, and the card will not pretend otherwise. The pointer head
carries semantics that the standard PEFT loader does not know: it scores the presented options, and
the reported probability is the option logits read at a fixed temperature. A documented loader is
required. Treat this repository as the weights, not as a `transformers` pipeline.
```python
# shape only; see the project code for the loader
adapter = PeftModel.from_pretrained(base, "pwwang/scKevType", subfolder="4B")
head = torch.load(hf_hub_download("pwwang/scKevType", "4B/head.pt"), weights_only=True)
```
## Training
One recipe across all six sizes: LoRA rank 16 on every linear projection of a frozen base,
`lora_alpha` 32, learning rate 2e-4, one record per step accumulated over eight steps, bf16, a
single epoch. The pointer head is trained from its warm start rather than reinitialised. No seed
replicates were run, so no seed spread is reported.
Data: 4,356,215 cells read from six public atlases (pancreas, immune, brain, and multi-organ), labels
mapped into the Cell Ontology register, with unresolvable labels excluded rather than remapped.
Splits are per dataset with donor-level holdout, so every atlas contributes to both training and
test; Baron is the one suite held out entirely.
## Evaluation, honestly
Reported numbers are served accuracy under one scoring rule, on fixed question sets, beside arms
that answered the identical questions. The relevant baselines in the paper are a commercial decision
API and general-purpose language models of the same sizes.
Two things worth knowing before quoting a number:
- **Accuracy alone is not the claim.** On binary marker questions, fine-tuning is what creates an
abstention policy at all; scale does not. A model that never declines can look strong on accuracy
while being worse than a constant rule on a skewed family.
- **Cross-suite comparison is invalid.** A suite with 10 labels and a 0.3865 floor is not the same
task as one with 65 labels and a 0.1836 floor.
## Limitations
- Trained on human atlases in the Cell Ontology register. Labels outside that register are out of
scope, and the model will still answer if asked.
- Seven cell types or sixty-five, the option set is whatever is presented; the model does not know
the true label set of an unseen tissue.
- One epoch, one seed, no replicates: differences smaller than a few points between adjacent sizes
should not be read as real.
- The abstention behaviour is a learned policy, not a calibrated guarantee.
## Licence
Adapter weights and this card: MIT. Each base model keeps its own licence, which governs the
combined work.
## Citation
Citation details will follow the paper.