aryaaan12
/

T-REN

@@ -1,199 +1,123 @@
----
-library_name: transformers
-tags: []
----
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

+# T-REN: Text-Aligned Region Encoder Network
+**Authors**: [Savya Khosla](https://savya08.github.io/), [Sethuraman TV](https://github.com/sethuramanio), [Aryan Chadha](https://www.linkedin.com/in/aryan-chadha/), [Alex Schwing](https://www.alexander-schwing.de/), [Derek Hoiem](https://dhoiem.cs.illinois.edu/)
+[![GitHub](https://img.shields.io/badge/GitHub-Code-black.svg)](https://github.com/savya08/T-REN)
+T-REN (**T**ext-aligned **R**egion **E**ncoder **N**etwork) is an image encoder that produces region-level tokens aligned with text, built on top of [DINOv3](https://github.com/facebookresearch/dinov3) ViT-L/16. Compared to its patch-based backbone, T-REN delivers:
+- **+5.9 mIoU** on ADE20K open-vocabulary segmentation
+- **+18.4% recall** on COCO object-level text-image retrieval
+- **+15.6% recall** on Ego4D video object localization (VQ2D)
+- **+17.6% mIoU** on VSPW video scene parsing
+- **24× fewer tokens** per image, **187× fewer** per video
+---
+## What's in this repo
+This HuggingFace repo contains:
+- `model.safetensors` — the trained `RegionEncoder` head weights (~1.2 GB)
+- `configuration_tren.py`, `modeling_tren.py`, `model.py`, `task_utils.py` — source code for `trust_remote_code`
+**The DINOv3 ViT-L/16 backbone is NOT included here** — it belongs to Facebook Research and must be obtained separately (see below).
+---
+## Quickstart
+### Step 1 — Install dependencies
+```bash
+pip install transformers torch torchvision kornia
+```
+### Step 2 — Get the DINOv3 weights
+T-REN's backbone is DINOv3 ViT-L/16 with a DINOtxt text-alignment head. You need two weight files from the [DINOv3 release](https://github.com/facebookresearch/dinov3):
+| File | Description |
+|------|-------------|
+| `dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth` | DINOv3 ViT-L/16 backbone |
+| `dinov3_vitl16_dinotxt_vision_head_and_text_encoder-a442d8f5.pth` | DINOtxt vision head + text encoder |
+Place both files in the same directory, e.g. `/path/to/dinov3_weights/`.
+### Step 3 — Load and run T-REN
+```python
+import torch
+import torchvision.transforms as T
+from PIL import Image
+from transformers import AutoModel
+# Load model (downloads T-REN weights from this repo automatically)
+model = AutoModel.from_pretrained("aryaaan12/T-REN", trust_remote_code=True)
+# Load the DINOv3 backbone from your local directory
+model.load_backbone("/path/to/dinov3_weights/")
+model.eval()
+# Prepare an image — resize to 512x512, values in [0, 1]
+transform = T.Compose([
+    T.Resize((512, 512)),
+    T.ToTensor(),
+])
+image = transform(Image.open("your_image.jpg").convert("RGB"))
+image = image.unsqueeze(0)  # (1, 3, 512, 512)
+# Run T-REN
+with torch.no_grad():
+    outputs = model(image)
+# Outputs
+region_tokens = outputs["text_aligned_tokens"]  # list of (N, 1024) per image
+region_masks  = outputs["region_masks"]          # list of (N, 32, 32) per image
+class_token   = outputs["class_tokens"]          # (1, 1024) image-level token
+print(f"Number of region tokens: {len(region_tokens[0])}")
+```
+### Text-guided region matching
+```python
+import torch.nn.functional as F
+texts = ["sky", "car", "building", "tree", "road"]
+with torch.no_grad():
+    outputs = model(image, texts=texts)
+region_tokens = outputs["text_aligned_tokens"][0]   # (N, 1024)
+text_tokens   = outputs["text_encodings"]           # (5, 1024)
+# Cosine similarity: which text label fits each region best?
+sim = F.normalize(region_tokens, dim=-1) @ F.normalize(text_tokens, dim=-1).T
+best_labels = sim.argmax(dim=-1)
+print([texts[i] for i in best_labels])
+```
+---
+## Model details
+| | |
+|---|---|
+| Architecture | RegionEncoder (cross-attention decoder) over DINOv3 ViT-L/16 features |
+| Trainable parameters | 31.5M (RegionEncoder head only; backbone is frozen) |
+| Input resolution | 512 × 512 |
+| Output token dim | 1024 |
+| Multiscale regions | 3 scales per prompt point |
+| Text embedding space | DINOtxt (aligned with DINOv3 text encoder) |
+---
+## Citation
+```bibtex
+@misc{khosla2026tren,
+      title={T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability},
+      author={Savya Khosla and Sethuraman T V and Aryan Chadha and Alexander Schwing and Derek Hoiem},
+      year={2026},
+}
+```