Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,63 @@
+---
+license: other
+tags:
+- computer-use
+- video
+- action-decoder
+- onevision
+- qwen
+---
+
+# OneVision Task-Aware Action Decoder — checkpoints
+
+Lightweight task-aware action decoder for computer-use / instructional-video understanding.
+Built on a frozen **OneVision vision encoder** (`lmms-lab-encoder/onevision-encoder-large-lang`)
+and **Qwen/Qwen3-4B** as the text encoder/decoder, **LoRA**-tuned (r=8, alpha=16, dropout=0.05,
+applied to both vision and text). Trained with `train_onevision_lw_decoder.py` in
+[proteusagi/onevision-compuse](https://github.com/proteusagi/onevision-compuse) under
+`onevision-lightweight-decoder/`.
+
+Given video visible-token embeddings plus a task prompt, the model predicts, per frame:
+- discrete **action / key / modifier** classes,
+- a **click heatmap** (actor query attending over visual tokens),
+- (new_arch only) an **autoregressive per-frame transcript**.
+
+## Checkpoints (~8.7 GB each — LoRA adapters + task heads)
+
+### `new_arch/best.pt` — "parallel heads" architecture (2026-03-16) — RECOMMENDED
+- Adds an **autoregressive transcript decoder** (per-frame transcript via the Qwen LM head:
+  visual features + transcript history of frames 0..t-1 predict frame t) on top of the
+  action/key/modifier classification heads and the click-heatmap head.
+- Modularized `encode`/`decode`, **parallel prediction heads**, and an **efficient-inference**
+  path (git commits: "parallel heads", "efficient inference", "further speedup").
+- Training performance: **train action accuracy ~64-66%**, train loss ~0.9 (see `accuracy.png`
+  and `loss.png` in the repo). The epoch ~85-117 plateau is a curriculum/data change; metrics
+  recover afterward.
+- Choose this by default: most capable (also generates transcripts) and the actively developed arch.
+
+### `onevision_task_action_decoder/best.pt` — original architecture (2026-03-10)
+- The original `OneVisionTaskAwareActionDecoder`: action/key/modifier classification heads plus a
+  click-heatmap head over OneVision visual tokens, with frame self-attention and prompt
+  cross-attention. **Predates the transcript decoder** (which was added 2026-03-16), so it does
+  action/key/modifier + click only.
+- Choose this if you specifically want the lighter, action-only baseline or to reproduce the
+  March-10 results.
+
+> Also in this repo: `onevision_task_action_decoder/epoch_*.pt` (earlier epoch snapshots of the
+> original run) and `onevision_task_action_decoder_8F/best.pt` (an 8-frame-clip variant).
+
+## Which to choose
+- **`new_arch/best.pt`** — full capability (action + transcript), latest architecture. Default pick.
+- **`onevision_task_action_decoder/best.pt`** — original action-only model / March-10 baseline.
+
+## Loading
+```python
+import torch
+ckpt = torch.load("new_arch/best.pt", map_location="cpu")
+# State for OneVisionTaskAwareActionDecoder:
+#   vision_encoder_name = "lmms-lab-encoder/onevision-encoder-large-lang"
+#   text_encoder_name   = "Qwen/Qwen3-4B"
+#   LoRA r=8, alpha=16 on both encoders
+```
+See `onevision-lightweight-decoder/src/model.py` in proteusagi/onevision-compuse for the module
+definition and `train_onevision_lw_decoder.py` for the training/eval pipeline.