google

/

vit-base-patch16-224

Like 1.05k

Follow

Google 73.5k

Could the intended evaluation preprocessing for this checkpoint be clarified and made explicit in its configuration/model card?

The packaged processor at revision 3f49326eb077187dfe1c2a2bb15fbd74e6ab91e3 produces substantially different downsampled inputs from a floating-point bilinear resize control based on the Google evaluation code linked by the model card. This reproduces with synthetic RGB pixels, without loading model weights or using any dataset, converter, quantizer, or accelerator.

Reference recipes

• The checkpoint configuration specifies size 224 and mean/std 0.5, but does not pin a resize backend, interpolation, antialiasing, or crop policy. The ViT PIL processor uses bilinear interpolation. I reproduced the input discrepancy with both explicit PIL and the automatically selected torchvision backend.
• The Google evaluation code linked from the model card resizes directly to the target dimensions in floating point and then normalizes to [-1,1]. The TensorFlow resize defaults are bilinear with antialias=False. PIL's resize/downsampling and integer rounding do not reproduce those operations.
• The model card also says the weights were ported through TIMM. In the related Transformers issue #20430, maintainers discussed TIMM's bicubic resize and center-crop recipe, and this comment says ViT interpolation should be bicubic. That issue is closed. It would help to identify which reference is intended for this exact checkpoint.

Normalization itself is algebraically consistent: rescaling by 1/255 followed by mean/std 0.5 produces [-1,1]. The question is the preceding resize/crop behavior and the backend used to implement it.

Minimal reproduction

The image is generated deterministically. Its high-frequency channel deliberately exposes downsampling differences; it is not a benchmark image. Only the processor configuration is downloaded.

"""No-dataset, no-model-weights reproduction of ViT resize differences."""
import importlib.metadata
import json

import numpy as np
from PIL import Image
import torch
from transformers import AutoImageProcessor

REPO = "google/vit-base-patch16-224"
REVISION = "3f49326eb077187dfe1c2a2bb15fbd74e6ab91e3"

# Deterministic synthetic RGB pixels. The first channel emphasizes downsampling.
y, x = np.mgrid[:480, :640]
rgb = np.stack(
    [((x + y) % 2) * 255, (17 * x + 29 * y) % 256, (11 * x + 7 * y) % 256],
    axis=-1,
).astype(np.uint8)
image = Image.fromarray(rgb)
raw = torch.from_numpy(rgb.copy()).permute(2, 0, 1).unsqueeze(0).float()

pil_processor = AutoImageProcessor.from_pretrained(REPO, revision=REVISION, backend="pil")
auto_processor = AutoImageProcessor.from_pretrained(REPO, revision=REVISION)
pil_values = pil_processor(images=image, return_tensors="pt")["pixel_values"]
auto_values = auto_processor(images=image, return_tensors="pt")["pixel_values"]

# Approximate the resize/normalization operations in Google's evaluation code.
# This uses PyTorch, and does NOT claim bitwise TensorFlow implementation parity.
float_values = torch.nn.functional.interpolate(
    raw, size=(224, 224), mode="bilinear", align_corners=False, antialias=False
)
float_values = (float_values - 127.5) / 127.5


def compare(a, b):
    difference = (a - b).abs()
    return {
        "max_abs": float(difference.max()),
        "mean_abs": float(difference.mean()),
        "fraction_abs_gt_0_01": float((difference > 0.01).float().mean()),
    }


print(json.dumps({
    "versions": {name: importlib.metadata.version(name)
                 for name in ("transformers", "torch", "torchvision", "pillow", "numpy")},
    "repo": REPO,
    "revision": REVISION,
    "explicit_pil_class": type(pil_processor).__name__,
    "auto_class": type(auto_processor).__name__,
    "output_shape": list(pil_values.shape),
    "hf_pil_vs_float_resize": compare(pil_values, float_values),
    "hf_auto_vs_float_resize": compare(auto_values, float_values),
    "hf_auto_vs_pil": compare(auto_values, pil_values),
}, indent=2))

Observed output

Environment: Python 3.13, Transformers 5.21.0, PyTorch 2.14.1+cu130, torchvision 0.29.1+cu130, Pillow 13.0.0, NumPy 2.5.4. The reproduction runs on CPU.

The explicit PIL processor is ViTImageProcessorPil; the automatic backend selects ViTImageProcessor. All outputs have shape [1, 3, 224, 224].

The control uses PyTorch floating-point bilinear interpolation to approximate the operations in Google's code. It does not establish bitwise parity with TensorFlow's implementation. It deliberately starts from identical decoded RGB pixels, so JPEG decoding is not a variable in this reproduction. The tensor differences above are synthetic-input diagnostics, not model-accuracy claims.

Requested clarification/fix

1. Identify the canonical validation preprocessing for these exact weights: the Google pipeline, the TIMM recipe used when porting, or the currently packaged HF recipe.
2. Document the backend, interpolation, antialiasing, crop policy and resize dtype needed to reproduce that reference, and align the checkpoint configuration/processor where appropriate.
3. Add a small golden-input preprocessing test. If the current behavior must remain for compatibility, document the discrepancy and provide an explicit supported way to select the reference recipe.

This report does not propose changing PIL globally or treating an antialiasing flag alone as a complete fix; crop behavior, interpolation and integer rounding may also matter.