README.md

---
license: cc-by-sa-4.0
library_name: pytorch
pipeline_tag: image-classification
base_model: facebook/dinov3-vits16-pretrain-lvd1689m
tags:
  - image-classification
  - computer-vision
  - dinov3
  - pytorch
  - safetensors
  - prototype-learning
  - hard-example-mining
  - feedback-routing
  - experimental
metrics:
  - accuracy
  - f1
  - precision
  - recall
---

# ProtoMorph-DINO

**Feedback-Gated Prototype Morphing for Hard-Case Image Classification**

ProtoMorph-DINO is an experimental image classification head designed to run on top of a frozen DINOv3 vision backbone.

This model card is for the Hugging Face repository:

```text
shiowo/DINO-Protomorph
```

This repository currently contains an initial research scaffold and custom ProtoMorph head checkpoint. Evaluation results are **pending** because the repository is being created before full training and benchmarking.

This project is independent and is not affiliated with Meta AI, Hugging Face, or the official DINOv3 project.

---

## Architecture

```text
Image
↓
Frozen DINOv3
↓
Patch map z0
↓
ProtoMorph block 1
↓
Layer Memory Attention
↓
ProtoMorph block 2
↓
Layer Memory Attention
↓
Main logits
↓
Hard-case gate
    ├── easy: return main logits
    └── hard:
          feedback from top-2 probabilities
          modulate DINO patch map
          run Delta-RBF hard expert
          fuse logits
```

---

## Model Summary

ProtoMorph-DINO explores whether a frozen foundation vision backbone can be improved with a custom hard-case refinement head.

For easy images, the model returns the main classifier output directly. For difficult or ambiguous images, the model activates a feedback branch. The feedback branch uses the top-2 predicted probabilities to modulate the DINO patch map, sends the modified representation through a Delta-RBF hard expert, and fuses the refined logits with the main logits.

The main research question is whether feedback-guided hard-case refinement can improve classification performance over simpler frozen-backbone heads such as a linear probe or MLP classifier.

---

## Current Status

**Status: research scaffold / pre-training setup**

The current checkpoint may be randomly initialized or only intended for smoke testing unless a later release says otherwise.

Predictions are **not meaningful** until the ProtoMorph head is trained on a real dataset.

---

## Results

**Evaluation results: Pending**

No benchmark results are reported yet because the repository is being prepared before training and evaluation.

| Metric | Value |
|---|---:|
| Accuracy | Pending |
| F1 | Pending |
| Precision | Pending |
| Recall | Pending |
| Confusion-pair improvement | Pending |
| Hard-case routing benefit | Pending |

Recommended future baselines:

| Baseline | Purpose |
|---|---|
| DINOv3 + Linear Probe | Minimal frozen-backbone baseline |
| DINOv3 + MLP Head | Strong simple head baseline |
| CLIP + Linear Probe | Popular vision-language comparison |
| ConvNeXt | Strong CNN-style baseline |
| ViT | Standard transformer baseline |

---

## Intended Use

This model is intended for:

- image classification research
- hard-example routing experiments
- prototype learning experiments
- frozen-backbone classifier research
- fine-grained classification experiments
- educational computer vision experiments

This model is **not** intended for safety-critical use.

Do not use this model for medical, legal, financial, biometric, security-critical, or production decisions without independent validation.

---

## Model Files

Recommended repository layout:

```text
.
├── README.md
├── LICENSE-WEIGHTS.md
├── config.json
├── labels.txt
├── checkpoints/
│   ├── config.json
│   ├── labels.txt
│   └── protomorph_head.safetensors
├── infer.py
├── scripts/
│   └── upload_to_hf.py
└── src/
    └── protomorph/
```

The main weight file is:

```text
checkpoints/protomorph_head.safetensors
```

This file contains only the custom ProtoMorph classification head.

DINOv3 backbone weights are **not** included in this repository.

---

## Backbone

Default backbone:

```text
facebook/dinov3-vits16-pretrain-lvd1689m
```

The backbone is used as a frozen visual feature extractor.

For RTX 3090-class GPUs, ViT-S/16 is a practical starting point because it keeps VRAM usage manageable while still producing useful patch embeddings.

---

## Installation

Recommended environment:

```text
Python 3.11
PyTorch 2.4.0
CUDA 12.4 PyTorch wheel
```

Install PyTorch:

```bash
pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu124
```

Install dependencies:

```bash
pip install -r requirements-core.txt
```

---

## RunPod Environment Variables

This project supports the RunPod environment variable names shown below:

```text
hf_key=hf_your_huggingface_write_token_here
hf_repo=shiowo/DINO-Protomorph
```

Standard Hugging Face names are also supported:

```text
HF_TOKEN=hf_your_huggingface_write_token_here
HF_REPO_ID=shiowo/DINO-Protomorph
```

Never commit your real Hugging Face token to the repository.

---

## Inference

Run inference from the command line:

```bash
python infer.py \
  --image examples/sample_image.jpg \
  --config checkpoints/config.json \
  --checkpoint checkpoints/protomorph_head.safetensors \
  --labels checkpoints/labels.txt \
  --topk 5
```

For smoke testing only:

```bash
python infer.py --image examples/sample_image.jpg --allow-random-head
```

If the head is untrained, the output is only useful for checking that the pipeline runs.

---

## Upload to Hugging Face from RunPod

After setting `hf_key` and `hf_repo` in RunPod, run:

```bash
cd /workspace/protomorph_dinov3_runpod
source .venv/bin/activate
python scripts/upload_to_hf.py
```

Or use the helper script:

```bash
bash runpod/upload_to_hf.sh
```

Dry run before upload:

```bash
python scripts/upload_to_hf.py --dry-run
```

---

## Config Example

```json
{
  "dino_model_name": "facebook/dinov3-vits16-pretrain-lvd1689m",
  "num_classes": 10,
  "embed_dim": 384,
  "patch_size": 16,
  "proto_count": 64,
  "memory_tokens": 16,
  "rbf_count": 128,
  "num_heads": 8,
  "dropout": 0.0,
  "hard_pmax_threshold": 0.65,
  "hard_margin_threshold": 0.15,
  "hard_entropy_threshold": 1.35,
  "image_size": 512,
  "use_bf16_autocast": true,
  "normalize_patch_tokens": true
}
```

---

## Limitations

Known limitations:

- The architecture is experimental.
- Evaluation results are pending.
- The hard-case gate requires threshold tuning.
- The Delta-RBF hard expert may overfit small datasets.
- Inference may be slower for hard samples.
- The model should be compared against simple baselines before claiming improvement.
- This repository does not include DINOv3 weights.
- The custom head may not generalize outside the dataset it was trained on.

---

## License

The ProtoMorph head weights in this repository are released under:

```text
Creative Commons Attribution-ShareAlike 4.0 International
CC BY-SA 4.0
```

You may use, share, and adapt these weights, including commercially, provided that you give appropriate credit and distribute adapted versions under CC BY-SA 4.0 or a compatible license.

This license applies only to the ProtoMorph head weights and related files released in this repository.

It does not apply to:

- DINOv3
- PyTorch
- Hugging Face Transformers
- third-party datasets
- third-party model weights
- upstream dependencies

DINOv3 is not redistributed in this repository. Users are responsible for obtaining DINOv3 separately and complying with its license.

---

## Attribution

If you use this model or build on it, please credit:

```text
ProtoMorph-DINO: Feedback-Gated Prototype Morphing for Hard-Case Image Classification
Author: shiowo
Repository: https://huggingface.co/shiowo/DINO-Protomorph
```

BibTeX:

```bibtex
@software{protomorph_dino_2026,
  title = {ProtoMorph-DINO: Feedback-Gated Prototype Morphing for Hard-Case Image Classification},
  author = {shiowo},
  year = {2026},
  url = {https://huggingface.co/shiowo/DINO-Protomorph}
}
```

---

## Disclaimer

This is a research prototype.

The model is provided for experimentation and educational use. It should not be used in production or high-stakes environments without independent validation, dataset auditing, robustness testing, and bias evaluation.