FelixKAI
/

UHR-BAT2 / EVALUATION.md
FelixKAI's picture
Duplicate from RL-MIND/UHR-BAT2
94c9578
|
Raw History Blame Contribute Delete
6.31 kB

UHR-BAT2 evaluation

English · 中文 · Model card

This document reports audited results for the qwen3vl-8b-geollava-ft3-selected-10k checkpoint released as UHR-BAT2. Existing prediction artifacts were checked on 2026-10-13; no new inference was run for this audit. Machine-readable counts, categories, protocol identifiers and source hashes are in evaluation_results.json.

Main results

Accuracy is sample-weighted unless stated otherwise. All three datasets use the corrected dense scoring protocol. Scores from different datasets should not be treated as interchangeable.

Dataset Correct / Total Accuracy Scope
RSHR-Bench 980 / 1,932 50.72% All QA
XLRS-Bench-lite 1,610 / 3,080 52.27% Strict option-set accuracy
MME-RealWorld Remote Sensing 2,123 / 3,738 56.80% Full remote-sensing subset

Denominators and answer formats

RSHR single-turn Overall is 726/1,590 = 45.66%. The remaining multi-turn QA records score 254/342 = 74.27%. The combined 50.72% score must not be labeled as the 1,590-record single-turn Overall metric. The multi-turn group is defined by the three original category identifiers multitrun-future_prediction, multiturn_object_state_judgement, and multturn_anomaly_detection.

XLRS contains 2,980 single-choice records: 1,563 correct (52.45%), and 100 multi-select task records: 47 correct (47.00%). Multi-select examples require an exact match of the complete canonical option set. The evaluated stored split is labeled train; this label describes the benchmark file, not evidence that those records were used for model training. MME evaluates all 3,738 Remote Sensing records, rather than a Lite subset.

Evaluation protocol

The corrected dense run uses 20260913_strict_sets_matched_schedule_v1, BF16 model weights, eager attention, min_pixels=784, and max_pixels=1,310,720. Dense means that visual tokens are not pruned after preprocessing. Images are still resized to the pixel budget; these results do not establish native 4K/8K or full-resolution ultra-high-resolution perception.

For single-choice questions, the evaluator selects the highest candidate-option logit using FP32 rows from the language-model output head. For XLRS multi-select questions, a constrained greedy token-trie decoder with FP32 candidate rows produces a canonical option set, which is scored by exact set matching.

The ordinary model.generate() quickstart is an inference demonstration, not a reproduction of these scores. Reproduction requires the same benchmark records, prompts, image processing, candidate-option scoring or constrained multi-select decoder, and answer normalization. Changing any of these may change the measured accuracy.

Category results

Original dataset category identifiers, including spelling differences, are retained for traceability. All category scores below use the protocol selected for that dataset in the main table.

RSHR-Bench

Category Correct / Total Accuracy
anomaly_detection 37 / 50 74.00%
color 135 / 200 67.50%
detection 32 / 100 32.00%
future_prediction_two_image 27 / 50 54.00%
multi_region_join_contrast 13 / 20 65.00%
multi_region_join_contrast_singal 22 / 50 44.00%
multitrun-future_prediction 100 / 150 66.67%
multiturn_object_state_judgement 112 / 132 84.85%
multturn_anomaly_detection 42 / 60 70.00%
object_clasfication 112 / 200 56.00%
object_counting 33 / 200 16.50%
object_grounding 74 / 200 37.00%
object_state_judgement 30 / 50 60.00%
regional_grouding 33 / 70 47.14%
regionla_counting 43 / 100 43.00%
relation 108 / 200 54.00%
shape 27 / 100 27.00%

XLRS-Bench-lite

Category Correct / Total Accuracy
Complex reasoning/Anomaly Detection and Interpretation 73 / 100 73.00%
Complex reasoning/Environmental condition reasoning 80 / 100 80.00%
Complex reasoning/Route planning 50 / 100 50.00%
Counting/Counting with changing detection 30 / 60 50.00%
Counting/Counting with complex reasoning 55 / 100 55.00%
Counting/Overall counting 22 / 60 36.67%
Counting/Regional counting 47 / 100 47.00%
Land use classification/Overall Land use classification 47 / 100 47.00%
Land use classification/Regional Land use classification 169 / 200 84.50%
Object properties/Object classification 422 / 800 52.75%
Object properties/Object color 374 / 800 46.75%
Object properties/Object motion state 39 / 60 65.00%
Object spatial relationship/Object spatial relationship 202 / 500 40.40%

MME-RealWorld Remote Sensing

Category Correct / Total Accuracy
color 855 / 1,255 68.13%
count 337 / 1,226 27.49%
position 931 / 1,257 74.07%

Verification and interpretation

All corrected dense prediction shards were read in full. Each dataset has complete, unique expected row indices; saved correctness agrees with answer/prediction equality, and protocol sidecars and completion markers are consistent. Recorded checkpoint, evaluator-code and protocol identities match across the corrected runs. Source artifact and evaluator hashes are provided in the JSON.

Excluded historical XLRS results: an earlier evaluator requested one option letter and truncated multi-option ground truths. Those results are not valid strict option-set accuracy and are excluded from this release report. No same-protocol accuracy gain over the original base model is claimed here.

The results show stronger performance on MME position and color questions and XLRS regional land-use classification, while counting is a recurring weakness, especially RSHR object counting and MME count. These are within-benchmark observations, not controlled cross-dataset or model comparisons.

Only aggregate results are included here. Benchmark images, questions and raw prediction files are not redistributed in this evaluation artifact. The hashes identify audited source files; the aggregate JSON alone is not a complete public reproduction package.