COTTON-DETR: detection and per-boll semantic inference

Complete inference source and task-specific weights for DINOv3-Driven Label-Efficient In-Field Cotton Boll State Detection: From Cold-Start Selection to Semantic Interpretation.

Documentation checked against the manuscript dated October 10, 2026. The manuscript PDF is not distributed here.

Interactive examples · Manuscript: forthcoming · Dataset: forthcoming · GitHub: forthcoming

The interactive Space displays precomputed examples. This repository contains the runnable detector and semantic pipeline for inference on new images, on your own CPU or GPU. It does not require Gemini or another online VLM.

Included weights

File	Purpose
weights/detector.safetensors	SSOD-608 EMA teacher: complete detector, including the trained distilled ViT-Tiny backbone, encoder and decoder
weights/semantic_head.safetensors	Complete balanced semantic MLP checkpoint
weights/damage_prototypes.safetensors	Normalized damage-type and severity centroids derived from train608 features

All exported detector and semantic-head tensors were checked for exact equality against the deployed checkpoints. Optimizer states and unused student weights are excluded. Checksums are recorded in manifest.json.

External dependency: original Meta DINOv3 ViT-S/16 weights are not redistributed here. Obtain them from the official model page under its access and license terms. They encode ROI crops for semantic prediction. No additional ViT-Tiny initialization weights are needed because the trained backbone is included in the detector checkpoint.

Quick start

Python 3.11 is recommended. Tested with PyTorch 2.5.1, torchvision 0.20.1 and transformers 4.57.6.

pip install "huggingface-hub>=0.34,<1"
hf download hwzhanng/COTTON-DETR --local-dir cotton-detr
cd cotton-detr
pip install -r requirements.txt

For CUDA, install the appropriate PyTorch/torchvision binaries for your platform from pytorch.org. For CPU-only usage, the CPU wheels are sufficient.

Accept the upstream DINOv3 access terms on its model page and authenticate with an authorized Hugging Face account:

hf auth login
python download_encoder.py
python predict.py examples/cotton.jpg --output result.json --annotated result.png

If you already have the encoder locally, skip the download and set its path:

# Linux / macOS
export COTTON_DINOV3_PATH=/path/to/dinov3_vits16
# Windows PowerShell
$env:COTTON_DINOV3_PATH='D:/models/dinov3_vits16'

The directory must contain the original Hugging Face config.json and model.safetensors. The expected default location is external/dinov3_vits16/.

python predict.py your-image.jpg --device cpu --threshold 0.35 --output result.json

--device auto (default) selects CUDA when available. A threshold between 0.05 and 1 filters the final output; semantic inference is performed on detector predictions with score ≥ 0.05.

Python and HTTP interfaces

Run Python from the repository directory:

from PIL import Image
from cotton import Detector

model = Detector()
model.load()
with Image.open('examples/cotton.jpg') as image:
    result = model.predict(image.convert('RGB'), 'cotton.jpg')
print(result['detections'][0]['semantics'])

Optional HTTP service:

uvicorn serve:app --host 0.0.0.0 --port 8000
curl -F "file=@examples/cotton.jpg" http://localhost:8000/api/detect

API documentation is available at /docs. The service serializes inference requests and runs the same model path as the CLI.

Output and method

Each detection has an id, class_id, score, pixel-space xyxy box and semantics object. Detection classes are 0 Defected, 1 Flower, 2 Fully opened, 3 Partially opened. Flower denotes the floral organ.

1. Resize the RGB image to 640 × 640 with bilinear interpolation and ImageNet normalization for detection.
2. Expand each eligible detected ROI by 15%, resize to 224 × 224 using bicubic interpolation, and encode using frozen DINOv3 ViT-S/16.
3. The balanced semantic head predicts opening state (flower, closed, partial, full), opening confidence and exposed-lint percentage. The percentage is constrained to the predicted state's range, exactly as in the deployed sidecar.
4. Damage presence follows the detector class. For defective bolls, normalized ROI features are matched to positive training centroids for damage type and severity. Other classes receive none damage fields.
5. Human-readable descriptions summarize those structured predictions deterministically.

The full trained head is included, including auxiliary outputs. The deployed hybrid decision rule uses prototypes for damage subtype/severity; it does not substitute the auxiliary parametric damage outputs.

Checkpoint architecture detail

The released checkpoint has 384-dimensional input features, two 384-unit hidden trunk layers and separate prediction heads; its auxiliary semantic projection has 256 dimensions. These dimensions are taken from the trained checkpoint and cotton/semantic_core.py. The manuscript's §2.5 describes the MLP schematically as 384 → 256 → output; the released implementation does not have that exact trunk layout. We preserve the verified deployed weights rather than silently changing their architecture to match the schematic.

Results reported in the latest manuscript

Table 5 uses all 1,026 ground-truth ROIs from 523 test images. Ground-truth classes determine class mapping and damage presence, so these are attribute-agreement results given known locations and classes, not end-to-end inference accuracy. Reference labels are from Gemini.

Method	Opening-state macro-F1	Opening-% MAE (percentage points)	Severity macro-F1 (4 classes)
Deterministic class map	0.715	8.80	0.449
Original multi-head MLP	0.794	4.96	0.448
Fixed opening + damage majority	0.794	4.65	0.449
Semantic component (§2.5)	0.794	4.65	0.610

The four severity classes include none. Table 5 includes 935 undamaged objects; Figure 13 reports three-class severity macro-F1 of 0.481 on 91 defective ROIs only. These are different evaluation scopes.

Across three seeds, the manuscript reports opening-state macro-F1 of 0.793 ± 0.009 and four-class severity macro-F1 of 0.610 ± 0.000. Prototype matching is deterministic for the fixed training set. A separate conditional evaluation at confidence 0.65 uses 891 one-to-one detections with correct classes and IoU ≥ 0.5, reporting opening / severity macro-F1 of 0.848 / 0.699. It excludes 450 false positives and 135 false negatives.

These are manuscript-reported research results, not new measurements performed by this repository release. The distributed detector is one checkpoint, rather than a three-seed ensemble. The static Space uses predicted boxes/classes for its examples; its default display threshold of 0.35 is not the conditional-evaluation threshold of 0.65.

Validation and limits

• Exported weights retain the original tensor values without quantization.
• On the included defective-boll example, all detections at threshold 0.35, coordinates, scores and full semantic objects exactly matched the original local CUDA inference.
• CPU inference is also supported; floating-point outputs may differ slightly across devices.
• The 608-image training set provides 1,203 labeled ROIs for the semantic pipeline; the learned damage prototypes use positive attribute classes. The raw training feature cache and annotations are not included.
• Semantic supervision/evaluation uses offline Gemini attribute labels. Independent expert annotation remains future work; cosine similarity is not a probability.
• Opening percentage estimates exposed lint for a single ROI, not whole-field opening rate. The demonstration examples are curated, not a full-test-set performance claim.

Source and licensing

The cotton/ pipeline, command-line interface, HTTP interface, configurations, trained task-specific weights and detector implementation are included. Our original inference code and original semantic-head/prototype contributions are available under Apache-2.0. Third-party portions retain their original licenses; DINOv3-derived code and detector weights remain subject to the DINOv3 license, rather than being relicensed as unrestricted Apache-2.0 artifacts. See LICENSE.md and NOTICE.md.

本仓库提供完整的检测与单铃语义推理代码、自训练检测器、平衡语义头、损伤原型和配置。DINOv3 原始编码器权重需按官方许可自行下载。无需在线大模型 API；可使用 CPU 或 GPU。论文、数据集及 GitHub 链接暂留空。