xabackus/egoblind-ra-adapters

EgoBlind-RA — LoRA Adapters (Approach 1: Bifurcated)

LoRA adapters for urgency-conditioned egocentric visual question answering, fine-tuned on Kimi-VL-A3B-Instruct.

Component of the EgoBlind-RA project.

Adapters

Adapter	Description	Urgent Sim	Non-urg Sim	Overall Sim
sft_urgent_vision/	Best urgent — SFT, vision, rank 8	0.505	—	—
sft_nonurgent_vision_v2/	Best standalone non-urgent — SFT, vision, rank 8	—	0.268	—
dpo_nonurgent_sft/	DPO on SFT non-urgent (stacked)	—	0.298	—
dpo_urgent_vision/	DPO on SFT urgent (degraded)	0.459	—	—
dpo_urgent_base/	DPO on base, urgent	0.277	—	—
dpo_nonurgent_base/	DPO on base, non-urgent	—	0.270	—

Best overall configuration (oracle routing): sft_urgent_vision + dpo_nonurgent_sft → 0.407 overall similarity.

Best deployed pipeline (CLIP classifier routing): 0.393 overall similarity, closing 88% of the gap to oracle.

Usage

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "moonshotai/Kimi-VL-A3B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Load the urgent adapter
model = PeftModel.from_pretrained(base, "xabackus/egoblind-ra-adapters/sft_urgent_vision")

# For the stacked non-urgent adapter (SFT + DPO):
base = AutoModelForCausalLM.from_pretrained("moonshotai/Kimi-VL-A3B-Instruct", ...)
base = PeftModel.from_pretrained(base, "xabackus/egoblind-ra-adapters/sft_nonurgent_vision_v2")
base = base.merge_and_unload()
model = PeftModel.from_pretrained(base, "xabackus/egoblind-ra-adapters/dpo_nonurgent_sft")

Training

All adapters: LoRA rank 8, α=16, targeting q_proj and v_proj, 1 video frame per example, kimi_vl_nothink template. Trained on MIT Engaging cluster (NVIDIA L40S GPUs).

See the paper for full details.

Citation

@misc{kim2026egoblindra,
  title  = {EgoBlind-RA: Towards Safer Egocentric Assistive AI for Blind Users via Risk-Adaptive Routing},
  author = {Kim, Julia and Backus, Xander},
  year   = {2026},
  url    = {https://github.com/juliavekim/EgoBlind-RA},
}