scKevType / README.md
pwwang's picture
Licence: MIT
aa1e368 verified
|
Raw History Blame Contribute Delete
6.16 kB
metadata
license: mit
library_name: peft
base_model:
  - Qwen/Qwen2.5-0.5B
  - Qwen/Qwen3-0.6B-Base
  - Qwen/Qwen3.5-0.8B-Base
  - Qwen/Qwen3.5-4B-Base
  - Qwen/Qwen3.5-9B-Base
  - jaredpalmer/kev-27b
tags:
  - single-cell
  - scrna-seq
  - cell-type-annotation
  - decision-model
  - lora
  - biology

scKevType

Cell-type annotation as a decision, not a generated sentence.

scKevType is a family of fine-tuned decision models for single-cell RNA-seq cell-type annotation. Given a rendered state built from a cluster's marker evidence, a model answers one typed question over a presented set of candidate labels in a single forward pass and returns a scored option: one of the options shown, with a probability attached. Declining is an option like any other.

There is no prose to parse. The answer is a label and a number, so it can be calibrated, thresholded, compared across models, and audited.

Why this exists

The annotation field has moved to language models, and those systems generate text. Generated text has no probability vector, so there is nothing to threshold or calibrate; it also cannot decline, so a model asked about a cluster it cannot place still answers fluently. Exact-match scoring on generated text needs a parsing rule, and the rule becomes a scoring decision the model never made.

scKevType removes that layer. The comparison it makes possible is between models answering identical questions under one scoring rule, rather than between parsers.

The six arms

Each size is a separate LoRA adapter on a frozen base, trained with one shared recipe. Pick by accuracy against cost: the 4B and 9B lead in distribution, and the 27B is the best arm on the held-out laboratory.

dir base model in-distribution held-out adapter + head
0.5B/ Qwen/Qwen2.5-0.5B 0.7561 0.8115 37 MB
0.6B/ Qwen/Qwen3-0.6B-Base 0.7586 0.7281 43 MB
0.8B/ Qwen/Qwen3.5-0.8B-Base 0.7898 0.7950 45 MB
4B/ Qwen/Qwen3.5-4B-Base 0.8191 0.8492 135 MB
9B/ Qwen/Qwen3.5-9B-Base 0.8191 0.8988 182 MB
27B/ jaredpalmer/kev-27b 0.7928 0.9100 478 MB

In-distribution: served accuracy on 4,035 panel_choice questions from the training corpus (exact match over the option set, abstain available on a subset). Held-out: served accuracy on 2,600 questions from Baron pancreas data, a laboratory absent from training. The two suites have different label counts and majority-class floors (0.1836 against 0.3865), so raw levels are not comparable across them; read each column on its own.

Repository layout

0.5B/ 0.6B/ 0.8B/ 4B/ 9B/ 27B/
    adapter_model.safetensors   the LoRA adapter, rank 16 on every linear projection
    adapter_config.json         with the base model and its pinned revision
    head.pt                     the pointer head
    tokenizer.json, tokenizer_config.json, chat_template.jinja
    provenance.json             the run's own record: data read, epochs, seconds, loss
    training_config.json        hyperparameters and placement
manifest.json                   per-arm base, revision, checksums, reported scores

The base weights are not included. Each arm is an adapter over a public base.

Loading

This is not a from_pretrained drop-in, and the card will not pretend otherwise. The pointer head carries semantics that the standard PEFT loader does not know: it scores the presented options, and the reported probability is the option logits read at a fixed temperature. A documented loader is required. Treat this repository as the weights, not as a transformers pipeline.

# shape only; see the project code for the loader
adapter = PeftModel.from_pretrained(base, "pwwang/scKevType", subfolder="4B")
head    = torch.load(hf_hub_download("pwwang/scKevType", "4B/head.pt"), weights_only=True)

Training

One recipe across all six sizes: LoRA rank 16 on every linear projection of a frozen base, lora_alpha 32, learning rate 2e-4, one record per step accumulated over eight steps, bf16, a single epoch. The pointer head is trained from its warm start rather than reinitialised. No seed replicates were run, so no seed spread is reported.

Data: 4,356,215 cells read from six public atlases (pancreas, immune, brain, and multi-organ), labels mapped into the Cell Ontology register, with unresolvable labels excluded rather than remapped. Splits are per dataset with donor-level holdout, so every atlas contributes to both training and test; Baron is the one suite held out entirely.

Evaluation, honestly

Reported numbers are served accuracy under one scoring rule, on fixed question sets, beside arms that answered the identical questions. The relevant baselines in the paper are a commercial decision API and general-purpose language models of the same sizes.

Two things worth knowing before quoting a number:

  • Accuracy alone is not the claim. On binary marker questions, fine-tuning is what creates an abstention policy at all; scale does not. A model that never declines can look strong on accuracy while being worse than a constant rule on a skewed family.
  • Cross-suite comparison is invalid. A suite with 10 labels and a 0.3865 floor is not the same task as one with 65 labels and a 0.1836 floor.

Limitations

  • Trained on human atlases in the Cell Ontology register. Labels outside that register are out of scope, and the model will still answer if asked.
  • Seven cell types or sixty-five, the option set is whatever is presented; the model does not know the true label set of an unseen tissue.
  • One epoch, one seed, no replicates: differences smaller than a few points between adjacent sizes should not be read as real.
  • The abstention behaviour is a learned policy, not a calibrated guarantee.

Licence

Adapter weights and this card: MIT. Each base model keeps its own licence, which governs the combined work.

Citation

Citation details will follow the paper.