Changelog
Dec 12, 2025
- Add CSATV2 model (thanks https://github.com/gusdlf93) -- a lightweight but high res model with DCT stem & spatial attention. https://huggingface.co/Hyunil/CSATv2
- Add AdaMuon and NAdaMuon optimizer support to existing
timm Muon impl. Appears more competitive vs AdamW with familiar hparams for image tasks.
- End of year PR cleanup, merge aspects of several long open PR
- Merge differential attention (
DiffAttention), add corresponding DiffParallelScalingBlock (for ViT), train some wee vits
- Add a few pooling modules,
LsePlus and SimPool
- Cleanup, optimize
DropBlock2d (also add support to ByobNet based models)
- Bump unit tests to PyTorch 2.9.1 + Python 3.13 on upper end, lower still PyTorch 1.13 + Python 3.10
Dec 1, 2025
- Add lightweight task abstraction, add logits and feature distillation support to train script via new tasks.
- Remove old APEX AMP support
Nov 4, 2025
Oct 31, 2025 🎃
- Update imagenet & OOD variant result csv files to include a few new models and verify correctness over several torch & timm versions
- EfficientNet-X and EfficientNet-H B5 model weights added as part of a hparam search for AdamW vs Muon (still iterating on Muon runs)
Oct 16-20, 2025
- Add an impl of the Muon optimizer (based on https://github.com/KellerJordan/Muon) with customizations
- extra flexibility and improved handling for conv weights and fallbacks for weight shapes not suited for orthogonalization
- small speedup for NS iterations by reducing allocs and using fused (b)add(b)mm ops
- by default uses AdamW (or NAdamW if
nesterov=True) updates if muon not suitable for parameter shape (or excluded via param group flag)
- like torch impl, select from several LR scale adjustment fns via
adjust_lr_fn
- select from several NS coefficient presets or specify your own via
ns_coefficients
- First 2 steps of 'meta' device model initialization supported
- Fix several ops that were breaking creation under 'meta' device context
- Add device & dtype factory kwarg support to all models and modules (anything inherting from nn.Module) in
timm
- License fields added to pretrained cfgs in code
- Release 1.0.21
Sept 21, 2025
- Remap DINOv3 ViT weight tags from
lvd_1689m -> lvd1689m to match (same for sat_493m -> sat493m)
- Release 1.0.20
Sept 17, 2025
July 23, 2025
- Add
set_input_size() method to EVA models, used by OpenCLIP 3.0.0 to allow resizing for timm based encoder models.
- Release 1.0.18, needed for PE-Core S & T models in OpenCLIP 3.0.0
- Fix small typing issue that broke Python 3.9 compat. 1.0.19 patch release.
July 21, 2025
- ROPE support added to NaFlexViT. All models covered by the EVA base (
eva.py) including EVA, EVA02, Meta PE ViT, timm SBB ViT w/ ROPE, and Naver ROPE-ViT can be now loaded in NaFlexViT when use_naflex=True passed at model creation time
- More Meta PE ViT encoders added, including small/tiny variants, lang variants w/ tiling, and more spatial variants.
- PatchDropout fixed with NaFlexViT and also w/ EVA models (regression after adding Naver ROPE-ViT)
- Fix XY order with grid_indexing='xy', impacted non-square image use in 'xy' mode (only ROPE-ViT and PE impacted).
July 7, 2025
- MobileNet-v5 backbone tweaks for improved Google Gemma 3n behaviour (to pair with updated official weights)
- Add stem bias (zero'd in updated weights, compat break with old weights)
- GELU -> GELU (tanh approx). A minor change to be closer to JAX
- Add two arguments to layer-decay support, a min scale clamp and 'no optimization' scale threshold
- Add 'Fp32' LayerNorm, RMSNorm, SimpleNorm variants that can be enabled to force computation of norm in float32
- Some typing, argument cleanup for norm, norm+act layers done with above
- Support Naver ROPE-ViT (https://github.com/naver-ai/rope-vit) in
eva.py, add RotaryEmbeddingMixed module for mixed mode, weights on HuggingFace Hub
| model |
img_size |
top1 |
top5 |
param_count |
| vit_large_patch16_rope_mixed_ape_224.naver_in1k |
224 |
84.84 |
97.122 |
304.4 |
| vit_large_patch16_rope_mixed_224.naver_in1k |
224 |
84.828 |
97.116 |
304.2 |
| vit_large_patch16_rope_ape_224.naver_in1k |
224 |
84.65 |
97.154 |
304.37 |
| vit_large_patch16_rope_224.naver_in1k |
224 |
84.648 |
97.122 |
304.17 |
| vit_base_patch16_rope_mixed_ape_224.naver_in1k |
224 |
83.894 |
96.754 |
86.59 |
| vit_base_patch16_rope_mixed_224.naver_in1k |
224 |
83.804 |
96.712 |
86.44 |
| vit_base_patch16_rope_ape_224.naver_in1k |
224 |
83.782 |
96.61 |
86.59 |
| vit_base_patch16_rope_224.naver_in1k |
224 |
83.718 |
96.672 |
86.43 |
| vit_small_patch16_rope_224.naver_in1k |
224 |
81.23 |
95.022 |
21.98 |
| vit_small_patch16_rope_mixed_224.naver_in1k |
224 |
81.216 |
95.022 |
21.99 |
| vit_small_patch16_rope_ape_224.naver_in1k |
224 |
81.004 |
95.016 |
22.06 |
| vit_small_patch16_rope_mixed_ape_224.naver_in1k |
224 |
80.986 |
94.976 |
22.06 |
- Some cleanup of ROPE modules, helpers, and FX tracing leaf registration
- Preparing version 1.0.17 release
June 26, 2025
- MobileNetV5 backbone (w/ encoder only variant) for Gemma 3n image encoder
- Version 1.0.16 released
June 23, 2025
- Add F.grid_sample based 2D and factorized pos embed resize to NaFlexViT. Faster when lots of different sizes (based on example by https://github.com/stas-sl).
- Further speed up patch embed resample by replacing vmap with matmul (based on snippet by https://github.com/stas-sl).
- Add 3 initial native aspect NaFlexViT checkpoints created while testing, ImageNet-1k and 3 different pos embed configs w/ same hparams.
- Support gradient checkpointing for
forward_intermediates and fix some checkpointing bugs. Thanks https://github.com/brianhou0208
- Add 'corrected weight decay' (https://arxiv.org/abs/2506.02285) as option to AdamW (legacy), Adopt, Kron, Adafactor (BV), Lamb, LaProp, Lion, NadamW, RmsPropTF, SGDW optimizers
- Switch PE (perception encoder) ViT models to use native timm weights instead of remapping on the fly
- Fix cuda stream bug in prefetch loader
June 5, 2025
- Initial NaFlexVit model code. NaFlexVit is a Vision Transformer with:
- Encapsulated embedding and position encoding in a single module
- Support for nn.Linear patch embedding on pre-patchified (dictionary) inputs
- Support for NaFlex variable aspect, variable resolution (SigLip-2: https://arxiv.org/abs/2502.14786)
- Support for FlexiViT variable patch size (https://arxiv.org/abs/2212.08013)
- Support for NaViT fractional/factorized position embedding (https://arxiv.org/abs/2307.06304)
- Existing vit models in
vision_transformer.py can be loaded into the NaFlexVit model by adding the use_naflex=True flag to create_model
- Some native weights coming soon
- A full NaFlex data pipeline is available that allows training / fine-tuning / evaluating with variable aspect / size images
- To enable in
train.py and validate.py add the --naflex-loader arg, must be used with a NaFlexVit
- To evaluate an existing (classic) ViT loaded in NaFlexVit model w/ NaFlex data pipe:
python validate.py /imagenet --amp -j 8 --model vit_base_patch16_224 --model-kwargs use_naflex=True --naflex-loader --naflex-max-seq-len 256
- The training has some extra args features worth noting
- The
--naflex-train-seq-lens' argument specifies which sequence lengths to randomly pick from per batch during training
- The
--naflex-max-seq-len argument sets the target sequence length for validation
- Adding
--model-kwargs enable_patch_interpolator=True --naflex-patch-sizes 12 16 24 will enable random patch size selection per-batch w/ interpolation
- The
--naflex-loss-scale arg changes loss scaling mode per batch relative to the batch size, timm NaFlex loading changes the batch size for each seq len
May 28, 2025
- Add a number of small/fast models thanks to https://github.com/brianhou0208
- Update EVA ViT (closest match) to support Perception Encoder models (https://arxiv.org/abs/2504.13181) from Meta, loading Hub weights but I still need to push dedicated
timm weights
- Add some flexibility to ROPE impl
- Big increase in number of models supporting
forward_intermediates() and some additional fixes thanks to https://github.com/brianhou0208
- DaViT, EdgeNeXt, EfficientFormerV2, EfficientViT(MIT), EfficientViT(MSRA), FocalNet, GCViT, HGNet /V2, InceptionNeXt, Inception-V4, MambaOut, MetaFormer, NesT, Next-ViT, PiT, PVT V2, RepGhostNet, RepViT, ResNetV2, ReXNet, TinyViT, TResNet, VoV
- TNT model updated w/ new weights
forward_intermediates() thanks to https://github.com/brianhou0208
- Add
local-dir: pretrained schema, can use local-dir:/path/to/model/folder for model name to source model / pretrained cfg & weights Hugging Face Hub models (config.json + weights file) from a local folder.
- Fixes, improvements for onnx export
Feb 21, 2025
- SigLIP 2 ViT image encoders added (https://huggingface.co/collections/timm/siglip-2-67b8e72ba08b09dd97aecaf9)
- Variable resolution / aspect NaFlex versions are a WIP
- Add 'SO150M2' ViT weights trained with SBB recipes, great results, better for ImageNet than previous attempt w/ less training.
vit_so150m2_patch16_reg1_gap_448.sbb_e200_in12k_ft_in1k - 88.1% top-1vit_so150m2_patch16_reg1_gap_384.sbb_e200_in12k_ft_in1k - 87.9% top-1vit_so150m2_patch16_reg1_gap_256.sbb_e200_in12k_ft_in1k - 87.3% top-1vit_so150m2_patch16_reg4_gap_256.sbb_e200_in12k
- Updated InternViT-300M '2.5' weights
- Release 1.0.15
Feb 1, 2025
- FYI PyTorch 2.6 & Python 3.13 are tested and working w/ current main and released version of
timm
Jan 27, 2025
- Add Kron Optimizer (PSGD w/ Kronecker-factored preconditioner)
Jan 19, 2025
- Fix loading of LeViT safetensor weights, remove conversion code which should have been deactivated
- Add 'SO150M' ViT weights trained with SBB recipes, decent results, but not optimal shape for ImageNet-12k/1k pretrain/ft
vit_so150m_patch16_reg4_gap_256.sbb_e250_in12k_ft_in1k - 86.7% top-1vit_so150m_patch16_reg4_gap_384.sbb_e250_in12k_ft_in1k - 87.4% top-1vit_so150m_patch16_reg4_gap_256.sbb_e250_in12k
- Misc typing, typo, etc. cleanup
- 1.0.14 release to get above LeViT fix out
Jan 9, 2025
- Add support to train and validate in pure
bfloat16 or float16
wandb project name arg added by https://github.com/caojiaolong, use arg.experiment for name- Fix old issue w/ checkpoint saving not working on filesystem w/o hard-link support (e.g. FUSE fs mounts)
- 1.0.13 release
Jan 6, 2025
- Add
torch.utils.checkpoint.checkpoint() wrapper in timm.models that defaults use_reentrant=False, unless TIMM_REENTRANT_CKPT=1 is set in env.
Dec 31, 2024
convnext_nano 384x384 ImageNet-12k pretrain & fine-tune. https://huggingface.co/models?search=convnext_nano%20r384- Add AIM-v2 encoders from https://github.com/apple/ml-aim, see on Hub: https://huggingface.co/models?search=timm%20aimv2
- Add PaliGemma2 encoders from https://github.com/google-research/big_vision to existing PaliGemma, see on Hub: https://huggingface.co/models?search=timm%20pali2
- Add missing L/14 DFN2B 39B CLIP ViT,
vit_large_patch14_clip_224.dfn2b_s39b
- Fix existing
RmsNorm layer & fn to match standard formulation, use PT 2.5 impl when possible. Move old impl to SimpleNorm layer, it's LN w/o centering or bias. There were only two timm models using it, and they have been updated.
- Allow override of
cache_dir arg for model creation
- Pass through
trust_remote_code for HF datasets wrapper
inception_next_atto model added by creator- Adan optimizer caution, and Lamb decoupled weighgt decay options
- Some feature_info metadata fixed by https://github.com/brianhou0208
- All OpenCLIP and JAX (CLIP, SigLIP, Pali, etc) model weights that used load time remapping were given their own HF Hub instances so that they work with
hf-hub: based loading, and thus will work with new Transformers TimmWrapperModel
Nov 28, 2024
- More optimizers
- Add MARS optimizer (https://arxiv.org/abs/2411.10438, https://github.com/AGI-Arena/MARS)
- Add LaProp optimizer (https://arxiv.org/abs/2002.04839, https://github.com/Z-T-WANG/LaProp-Optimizer)
- Add masking from 'Cautious Optimizers' (https://arxiv.org/abs/2411.16085, https://github.com/kyleliang919/C-Optim) to Adafactor, Adafactor Big Vision, AdamW (legacy), Adopt, Lamb, LaProp, Lion, NadamW, RMSPropTF, SGDW
- Cleanup some docstrings and type annotations re optimizers and factory
- Add MobileNet-V4 Conv Medium models pretrained on in12k and fine-tuned in1k @ 384x384
- Add small cs3darknet, quite good for the speed
Nov 12, 2024
- Optimizer factory refactor
- New factory works by registering optimizers using an OptimInfo dataclass w/ some key traits
- Add
list_optimizers, get_optimizer_class, get_optimizer_info to reworked create_optimizer_v2 fn to explore optimizers, get info or class
- deprecate
optim.optim_factory, move fns to optim/_optim_factory.py and optim/_param_groups.py and encourage import via timm.optim
- Add Adopt (https://github.com/iShohei220/adopt) optimizer
- Add 'Big Vision' variant of Adafactor (https://github.com/google-research/big_vision/blob/main/big_vision/optax.py) optimizer
- Fix original Adafactor to pick better factorization dims for convolutions
- Tweak LAMB optimizer with some improvements in torch.where functionality since original, refactor clipping a bit
- dynamic img size support in vit, deit, eva improved to support resize from non-square patch grids, thanks https://github.com/wojtke
Oct 31, 2024
Add a set of new very well trained ResNet & ResNet-V2 18/34 (basic block) weights. See https://huggingface.co/blog/rwightman/resnet-trick-or-treat
Oct 19, 2024
- Cleanup torch amp usage to avoid cuda specific calls, merge support for Ascend (NPU) devices from MengqingCao that should work now in PyTorch 2.5 w/ new device extension autoloading feature. Tested Intel Arc (XPU) in Pytorch 2.5 too and it (mostly) worked.
Oct 16, 2024
Oct 14, 2024
- Pre-activation (ResNetV2) version of 18/18d/34/34d ResNet model defs added by request (weights pending)
- Release 1.0.10
Oct 11, 2024
- MambaOut (https://github.com/yuweihao/MambaOut) model & weights added. A cheeky take on SSM vision models w/o the SSM (essentially ConvNeXt w/ gating). A mix of original weights + custom variations & weights.
- SigLIP SO400M ViT fine-tunes on ImageNet-1k @ 378x378, added 378x378 option for existing SigLIP 384x384 models
- SigLIP SO400M ViT encoder from recent multi-lingual (i18n) variant, patch16 @ 256x256 (https://huggingface.co/timm/ViT-SO400M-16-SigLIP-i18n-256). OpenCLIP update pending.
- Add two ConvNeXt 'Zepto' models & weights (one w/ overlapped stem and one w/ patch stem). Uses RMSNorm, smaller than previous 'Atto', 2.2M params.
Sept 2024
Aug 21, 2024
- Updated SBB ViT models trained on ImageNet-12k and fine-tuned on ImageNet-1k, challenging quite a number of much larger, slower models
- MobileNet-V1 1.25, EfficientNet-B1, & ResNet50-D weights w/ MNV4 baseline challenge recipe
- Add SAM2 (HieraDet) backbone arch & weight loading support
- Add Hiera Small weights trained w/ abswin pos embed on in12k & fine-tuned on 1k
| model |
top1 |
top5 |
param_count |
| hiera_small_abswin_256.sbb2_e200_in12k_ft_in1k |
84.912 |
97.260 |
35.01 |
| hiera_small_abswin_256.sbb2_pd_e200_in12k_ft_in1k |
84.560 |
97.106 |
35.01 |
Aug 8, 2024
July 28, 2024
- Add
mobilenet_edgetpu_v2_m weights w/ ra4 mnv4-small based recipe. 80.1% top-1 @ 224 and 80.7 @ 256.
- Release 1.0.8
July 26, 2024
- More MobileNet-v4 weights, ImageNet-12k pretrain w/ fine-tunes, and anti-aliased ConvLarge models
- Prototype of
set_input_size() added to vit and swin v1/v2 models to allow changing image size, patch size, window size after model creation.
- Improved support in swin for different size handling, in addition to
set_input_size, always_partition and strict_img_size args have been added to __init__ to allow more flexible input size constraints
- Fix out of order indices info for intermediate 'Getter' feature wrapper, check out or range indices for same.
- Add several
tiny < .5M param models for testing that are actually trained on ImageNet-1k
| model |
top1 |
top1_err |
top5 |
top5_err |
param_count |
img_size |
crop_pct |
| test_efficientnet.r160_in1k |
47.156 |
52.844 |
71.726 |
28.274 |
0.36 |
192 |
1.0 |
| test_byobnet.r160_in1k |
46.698 |
53.302 |
71.674 |
28.326 |
0.46 |
192 |
1.0 |
| test_efficientnet.r160_in1k |
46.426 |
53.574 |
70.928 |
29.072 |
0.36 |
160 |
0.875 |
| test_byobnet.r160_in1k |
45.378 |
54.622 |
70.572 |
29.428 |
0.46 |
160 |
0.875 |
| test_vit.r160_in1k |
42.0 |
58.0 |
68.664 |
31.336 |
0.37 |
192 |
1.0 |
| test_vit.r160_in1k |
40.822 |
59.178 |
67.212 |
32.788 |
0.37 |
160 |
0.875 |
- Fix vit reg token init, thanks Promisery
- Other misc fixes
June 24, 2024
- 3 more MobileNetV4 hyrid weights with different MQA weight init scheme
- florence2 weight loading in DaViT model
June 12, 2024
- MobileNetV4 models and initial set of
timm trained weights added:
- Apple MobileCLIP (https://arxiv.org/pdf/2311.17049, FastViT and ViT-B) image tower model support & weights added (part of OpenCLIP support).
- ViTamin (https://arxiv.org/abs/2404.02132) CLIP image tower model & weights added (part of OpenCLIP support).
- OpenAI CLIP Modified ResNet image tower modelling & weight support (via ByobNet). Refactor AttentionPool2d.
May 14, 2024
- Support loading PaliGemma jax weights into SigLIP ViT models with average pooling.
- Add Hiera models from Meta (https://github.com/facebookresearch/hiera).
- Add
normalize= flag for transforms, return non-normalized torch.Tensor with original dytpe (for chug)
- Version 1.0.3 release
May 11, 2024
Searching for Better ViT Baselines (For the GPU Poor) weights and vit variants released. Exploring model shapes between Tiny and Base.
- AttentionExtract helper added to extract attention maps from
timm models. See example in https://github.com/huggingface/pytorch-image-models/discussions/1232#discussioncomment-9320949
forward_intermediates() API refined and added to more models including some ConvNets that have other extraction methods.- 1017 of 1047 model architectures support
features_only=True feature extraction. Remaining 34 architectures can be supported but based on priority requests.
- Remove torch.jit.script annotated functions including old JIT activations. Conflict with dynamo and dynamo does a much better job when used.
April 11, 2024
- Prepping for a long overdue 1.0 release, things have been stable for a while now.
- Significant feature that's been missing for a while,
features_only=True support for ViT models with flat hidden states or non-std module layouts (so far covering 'vit_*', 'twins_*', 'deit*', 'beit*', 'mvitv2*', 'eva*', 'samvit_*', 'flexivit*')
- Above feature support achieved through a new
forward_intermediates() API that can be used with a feature wrapping module or directly.
model = timm.create_model('vit_base_patch16_224')
final_feat, intermediates = model.forward_intermediates(input)
output = model.forward_head(final_feat)
print(final_feat.shape)
torch.Size([2, 197, 768])
for f in intermediates:
print(f.shape)
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
torch.Size([2, 768, 14, 14])
print(output.shape)
torch.Size([2, 1000])
model = timm.create_model('eva02_base_patch16_clip_224', pretrained=True, img_size=512, features_only=True, out_indices=(-3, -2,))
output = model(torch.randn(2, 3, 512, 512))
for o in output:
print(o.shape)
torch.Size([2, 768, 32, 32])
torch.Size([2, 768, 32, 32])
- TinyCLIP vision tower weights added, thx Thien Tran
Feb 19, 2024
- Next-ViT models added. Adapted from https://github.com/bytedance/Next-ViT
- HGNet and PP-HGNetV2 models added. Adapted from https://github.com/PaddlePaddle/PaddleClas by SeeFun
- Removed setup.py, moved to pyproject.toml based build supported by PDM
- Add updated model EMA impl using _for_each for less overhead
- Support device args in train script for non GPU devices
- Other misc fixes and small additions
- Min supported Python version increased to 3.8
- Release 0.9.16
Jan 8, 2024
Datasets & transform refactoring
- HuggingFace streaming (iterable) dataset support (
--dataset hfids:org/dataset)
- Webdataset wrapper tweaks for improved split info fetching, can auto fetch splits from supported HF hub webdataset
- Tested HF
datasets and webdataset wrapper streaming from HF hub with recent timm ImageNet uploads to https://huggingface.co/timm
- Make input & target column/field keys consistent across datasets and pass via args
- Full monochrome support when using e:g:
--input-size 1 224 224 or --in-chans 1, sets PIL image conversion appropriately in dataset
- Improved several alternate crop & resize transforms (ResizeKeepRatio, RandomCropOrPad, etc) for use in PixParse document AI project
- Add SimCLR style color jitter prob along with grayscale and gaussian blur options to augmentations and args
- Allow train without validation set (
--val-split '') in train script
- Add
--bce-sum (sum over class dim) and --bce-pos-weight (positive weighting) args for training as they're common BCE loss tweaks I was often hard coding
Nov 23, 2023
- Added EfficientViT-Large models, thanks SeeFun
- Fix Python 3.7 compat, will be dropping support for it soon
- Other misc fixes
- Release 0.9.12
Nov 20, 2023
- Added significant flexibility for Hugging Face Hub based timm models via
model_args config entry. model_args will be passed as kwargs through to models on creation.
- Updated imagenet eval and test set csv files with latest models
vision_transformer.py typing and doc cleanup by Laureηt- 0.9.11 release
Nov 3, 2023
Oct 20, 2023
- SigLIP image tower weights supported in
vision_transformer.py.
- Great potential for fine-tune and downstream feature use.
- Experimental 'register' support in vit models as per Vision Transformers Need Registers
- Updated RepViT with new weight release. Thanks wangao
- Add patch resizing support (on pretrained weight load) to Swin models
- 0.9.8 release pending
Sep 1, 2023
- TinyViT added by SeeFun
- Fix EfficientViT (MIT) to use torch.autocast so it works back to PT 1.10
- 0.9.7 release
Aug 28, 2023
- Add dynamic img size support to models in
vision_transformer.py, vision_transformer_hybrid.py, deit.py, and eva.py w/o breaking backward compat.
- Add
dynamic_img_size=True to args at model creation time to allow changing the grid size (interpolate abs and/or ROPE pos embed each forward pass).
- Add
dynamic_img_pad=True to allow image sizes that aren't divisible by patch size (pad bottom right to patch size each forward pass).
- Enabling either dynamic mode will break FX tracing unless PatchEmbed module added as leaf.
- Existing method of resizing position embedding by passing different
img_size (interpolate pretrained embed weights once) on creation still works.
- Existing method of changing
patch_size (resize pretrained patch_embed weights once) on creation still works.
- Example validation cmd
python validate.py --data-dir /imagenet --model vit_base_patch16_224 --amp --amp-dtype bfloat16 --img-size 255 --crop-pct 1.0 --model-kwargs dynamic_img_size=True dyamic_img_pad=True
Aug 25, 2023
- Many new models since last release
- Add
--reparam arg to benchmark.py, onnx_export.py, and validate.py to trigger layer reparameterization / fusion for models with any one of reparameterize(), switch_to_deploy() or fuse()
- Including FastViT, MobileOne, RepGhostNet, EfficientViT (MSRA), RepViT, RepVGG, and LeViT
- Preparing 0.9.6 'back to school' release
Aug 11, 2023
- Swin, MaxViT, CoAtNet, and BEiT models support resizing of image/window size on creation with adaptation of pretrained weights
- Example validation cmd to test w/ non-square resize
python validate.py --data-dir /imagenet --model swin_base_patch4_window7_224.ms_in22k_ft_in1k --amp --amp-dtype bfloat16 --input-size 3 256 320 --model-kwargs window_size=8,10 img_size=256,320
Aug 3, 2023
- Add GluonCV weights for HRNet w18_small and w18_small_v2. Converted by SeeFun
- Fix
selecsls* model naming regression
- Patch and position embedding for ViT/EVA works for bfloat16/float16 weights on load (or activations for on-the-fly resize)
- v0.9.5 release prep
July 27, 2023
- Added timm trained
seresnextaa201d_32x8d.sw_in12k_ft_in1k_384 weights (and .sw_in12k pretrain) with 87.3% top-1 on ImageNet-1k, best ImageNet ResNet family model I'm aware of.
- RepViT model and weights (https://arxiv.org/abs/2307.09283) added by wangao
- I-JEPA ViT feature weights (no classifier) added by SeeFun
- SAM-ViT (segment anything) feature weights (no classifier) added by SeeFun
- Add support for alternative feat extraction methods and -ve indices to EfficientNet
- Add NAdamW optimizer
- Misc fixes
May 11, 2023
timm 0.9 released, transition from 0.8.xdev releases
May 10, 2023
- Hugging Face Hub downloading is now default, 1132 models on https://huggingface.co/timm, 1163 weights in
timm
- DINOv2 vit feature backbone weights added thanks to Leng Yue
- FB MAE vit feature backbone weights added
- OpenCLIP DataComp-XL L/14 feat backbone weights added
- MetaFormer (poolformer-v2, caformer, convformer, updated poolformer (v1)) w/ weights added by Fredo Guan
- Experimental
get_intermediate_layers function on vit/deit models for grabbing hidden states (inspired by DINO impl). This is WIP and may change significantly... feedback welcome.
- Model creation throws error if
pretrained=True and no weights exist (instead of continuing with random initialization)
- Fix regression with inception / nasnet TF sourced weights with 1001 classes in original classifiers
- bitsandbytes (https://github.com/TimDettmers/bitsandbytes) optimizers added to factory, use
bnb prefix, ie bnbadam8bit
- Misc cleanup and fixes
- Final testing before switching to a 0.9 and bringing
timm out of pre-release state
April 27, 2023
- 97% of
timm models uploaded to HF Hub and almost all updated to support multi-weight pretrained configs
- Minor cleanup and refactoring of another batch of models as multi-weight added. More fused_attn (F.sdpa) and features_only support, and torchscript fixes.
April 21, 2023
- Gradient accumulation support added to train script and tested (
--grad-accum-steps), thanks Taeksang Kim
- More weights on HF Hub (cspnet, cait, volo, xcit, tresnet, hardcorenas, densenet, dpn, vovnet, xception_aligned)
- Added
--head-init-scale and --head-init-bias to train.py to scale classiifer head and set fixed bias for fine-tune
- Remove all InplaceABN (
inplace_abn) use, replaced use in tresnet with standard BatchNorm (modified weights accordingly).
April 12, 2023
- Add ONNX export script, validate script, helpers that I've had kicking around for along time. Tweak 'same' padding for better export w/ recent ONNX + pytorch.
- Refactor dropout args for vit and vit-like models, separate drop_rate into
drop_rate (classifier dropout), proj_drop_rate (block mlp / out projections), pos_drop_rate (position embedding drop), attn_drop_rate (attention dropout). Also add patch dropout (FLIP) to vit and eva models.
- fused F.scaled_dot_product_attention support to more vit models, add env var (TIMM_FUSED_ATTN) to control, and config interface to enable/disable
- Add EVA-CLIP backbones w/ image tower weights, all the way up to 4B param 'enormous' model, and 336x336 OpenAI ViT mode that was missed.
April 5, 2023
- ALL ResNet models pushed to Hugging Face Hub with multi-weight support
- New ImageNet-12k + ImageNet-1k fine-tunes available for a few anti-aliased ResNet models
resnetaa50d.sw_in12k_ft_in1k - 81.7 @ 224, 82.6 @ 288resnetaa101d.sw_in12k_ft_in1k - 83.5 @ 224, 84.1 @ 288seresnextaa101d_32x8d.sw_in12k_ft_in1k - 86.0 @ 224, 86.5 @ 288 seresnextaa101d_32x8d.sw_in12k_ft_in1k_288 - 86.5 @ 288, 86.7 @ 320
March 31, 2023
- Add first ConvNext-XXLarge CLIP -> IN-1k fine-tune and IN-12k intermediate fine-tunes for convnext-base/large CLIP models.
| model |
top1 |
top5 |
img_size |
param_count |
gmacs |
macts |
| convnext_xxlarge.clip_laion2b_soup_ft_in1k |
88.612 |
98.704 |
256 |
846.47 |
198.09 |
124.45 |
| convnext_large_mlp.clip_laion2b_soup_ft_in12k_in1k_384 |
88.312 |
98.578 |
384 |
200.13 |
101.11 |
126.74 |
| convnext_large_mlp.clip_laion2b_soup_ft_in12k_in1k_320 |
87.968 |
98.47 |
320 |
200.13 |
70.21 |
88.02 |
| convnext_base.clip_laion2b_augreg_ft_in12k_in1k_384 |
87.138 |
98.212 |
384 |
88.59 |
45.21 |
84.49 |
| convnext_base.clip_laion2b_augreg_ft_in12k_in1k |
86.344 |
97.97 |
256 |
88.59 |
20.09 |
37.55 |
- Add EVA-02 MIM pretrained and fine-tuned weights, push to HF hub and update model cards for all EVA models. First model over 90% top-1 (99% top-5)! Check out the original code & weights at https://github.com/baaivision/EVA for more details on their work blending MIM, CLIP w/ many model, dataset, and train recipe tweaks.
| model |
top1 |
top5 |
param_count |
img_size |
| eva02_large_patch14_448.mim_m38m_ft_in22k_in1k |
90.054 |
99.042 |
305.08 |
448 |
| eva02_large_patch14_448.mim_in22k_ft_in22k_in1k |
89.946 |
99.01 |
305.08 |
448 |
| eva_giant_patch14_560.m30m_ft_in22k_in1k |
89.792 |
98.992 |
1014.45 |
560 |
| eva02_large_patch14_448.mim_in22k_ft_in1k |
89.626 |
98.954 |
305.08 |
448 |
| eva02_large_patch14_448.mim_m38m_ft_in1k |
89.57 |
98.918 |
305.08 |
448 |
| eva_giant_patch14_336.m30m_ft_in22k_in1k |
89.56 |
98.956 |
1013.01 |
336 |
| eva_giant_patch14_336.clip_ft_in1k |
89.466 |
98.82 |
1013.01 |
336 |
| eva_large_patch14_336.in22k_ft_in22k_in1k |
89.214 |
98.854 |
304.53 |
336 |
| eva_giant_patch14_224.clip_ft_in1k |
88.882 |
98.678 |
1012.56 |
224 |
| eva02_base_patch14_448.mim_in22k_ft_in22k_in1k |
88.692 |
98.722 |
87.12 |
448 |
| eva_large_patch14_336.in22k_ft_in1k |
88.652 |
98.722 |
304.53 |
336 |
| eva_large_patch14_196.in22k_ft_in22k_in1k |
88.592 |
98.656 |
304.14 |
196 |
| eva02_base_patch14_448.mim_in22k_ft_in1k |
88.23 |
98.564 |
87.12 |
448 |
| eva_large_patch14_196.in22k_ft_in1k |
87.934 |
98.504 |
304.14 |
196 |
| eva02_small_patch14_336.mim_in22k_ft_in1k |
85.74 |
97.614 |
22.13 |
336 |
| eva02_tiny_patch14_336.mim_in22k_ft_in1k |
80.658 |
95.524 |
5.76 |
336 |
- Multi-weight and HF hub for DeiT and MLP-Mixer based models
March 22, 2023
- More weights pushed to HF hub along with multi-weight support, including:
regnet.py, rexnet.py, byobnet.py, resnetv2.py, swin_transformer.py, swin_transformer_v2.py, swin_transformer_v2_cr.py
- Swin Transformer models support feature extraction (NCHW feat maps for
swinv2_cr_*, and NHWC for all others) and spatial embedding outputs.
- FocalNet (from https://github.com/microsoft/FocalNet) models and weights added with significant refactoring, feature extraction, no fixed resolution / sizing constraint
- RegNet weights increased with HF hub push, SWAG, SEER, and torchvision v2 weights. SEER is pretty poor wrt to performance for model size, but possibly useful.
- More ImageNet-12k pretrained and 1k fine-tuned
timm weights:
rexnetr_200.sw_in12k_ft_in1k - 82.6 @ 224, 83.2 @ 288rexnetr_300.sw_in12k_ft_in1k - 84.0 @ 224, 84.5 @ 288regnety_120.sw_in12k_ft_in1k - 85.0 @ 224, 85.4 @ 288regnety_160.lion_in12k_ft_in1k - 85.6 @ 224, 86.0 @ 288regnety_160.sw_in12k_ft_in1k - 85.6 @ 224, 86.0 @ 288 (compare to SWAG PT + 1k FT this is same BUT much lower res, blows SEER FT away)
- Model name deprecation + remapping functionality added (a milestone for bringing 0.8.x out of pre-release). Mappings being added...
- Minor bug fixes and improvements.
Feb 26, 2023
- Add ConvNeXt-XXLarge CLIP pretrained image tower weights for fine-tune & features (fine-tuning TBD) -- see model card
- Update
convnext_xxlarge default LayerNorm eps to 1e-5 (for CLIP weights, improved stability)
- 0.8.15dev0
Feb 20, 2023
- Add 320x320
convnext_large_mlp.clip_laion2b_ft_320 and convnext_large_mlp.clip_laion2b_ft_soup_320 CLIP image tower weights for features & fine-tune
- 0.8.13dev0 pypi release for latest changes w/ move to huggingface org
Feb 16, 2023
safetensor checkpoint support added- Add ideas from 'Scaling Vision Transformers to 22 B. Params' (https://arxiv.org/abs/2302.05442) -- qk norm, RmsNorm, parallel block
- Add F.scaled_dot_product_attention support (PyTorch 2.0 only) to
vit_*, vit_relpos*, coatnet / maxxvit (to start)
- Lion optimizer (w/ multi-tensor option) added (https://arxiv.org/abs/2302.06675)
- gradient checkpointing works with
features_only=True
Feb 7, 2023
- New inference benchmark numbers added in results folder.
- Add convnext LAION CLIP trained weights and initial set of in1k fine-tunes
convnext_base.clip_laion2b_augreg_ft_in1k - 86.2% @ 256x256convnext_base.clip_laiona_augreg_ft_in1k_384 - 86.5% @ 384x384convnext_large_mlp.clip_laion2b_augreg_ft_in1k - 87.3% @ 256x256convnext_large_mlp.clip_laion2b_augreg_ft_in1k_384 - 87.9% @ 384x384
- Add DaViT models. Supports
features_only=True. Adapted from https://github.com/dingmyu/davit by Fredo.
- Use a common NormMlpClassifierHead across MaxViT, ConvNeXt, DaViT
- Add EfficientFormer-V2 model, update EfficientFormer, and refactor LeViT (closely related architectures). Weights on HF hub.
- New EfficientFormer-V2 arch, significant refactor from original at (https://github.com/snap-research/EfficientFormer). Supports
features_only=True.
- Minor updates to EfficientFormer.
- Refactor LeViT models to stages, add
features_only=True support to new conv variants, weight remap required.
- Move ImageNet meta-data (synsets, indices) from
/results to timm/data/_info.
- Add ImageNetInfo / DatasetInfo classes to provide labelling for various ImageNet classifier layouts in
timm
- Update
inference.py to use, try: python inference.py --data-dir /folder/to/images --model convnext_small.in12k --label-type detail --topk 5
- Ready for 0.8.10 pypi pre-release (final testing).
Jan 20, 2023
Add two convnext 12k -> 1k fine-tunes at 384x384
convnext_tiny.in12k_ft_in1k_384 - 85.1 @ 384convnext_small.in12k_ft_in1k_384 - 86.2 @ 384
Push all MaxxViT weights to HF hub, and add new ImageNet-12k -> 1k fine-tunes for rw base MaxViT and CoAtNet 1/2 models
Jan 11, 2023
- Update ConvNeXt ImageNet-12k pretrain series w/ two new fine-tuned weights (and pre FT
.in12k tags)
convnext_nano.in12k_ft_in1k - 82.3 @ 224, 82.9 @ 288 (previously released)convnext_tiny.in12k_ft_in1k - 84.2 @ 224, 84.5 @ 288convnext_small.in12k_ft_in1k - 85.2 @ 224, 85.3 @ 288
Jan 6, 2023
- Finally got around to adding
--model-kwargs and --opt-kwargs to scripts to pass through rare args directly to model classes from cmd line
train.py --data-dir /imagenet --model resnet50 --amp --model-kwargs output_stride=16 act_layer=silutrain.py --data-dir /imagenet --model vit_base_patch16_clip_224 --img-size 240 --amp --model-kwargs img_size=240 patch_size=12
- Cleanup some popular models to better support arg passthrough / merge with model configs, more to go.
Jan 5, 2023
- ConvNeXt-V2 models and weights added to existing
convnext.py
Dec 23, 2022 🎄☃
- Add FlexiViT models and weights from https://github.com/google-research/big_vision (check out paper at https://arxiv.org/abs/2212.08013)
- NOTE currently resizing is static on model creation, on-the-fly dynamic / train patch size sampling is a WIP
- Many more models updated to multi-weight and downloadable via HF hub now (convnext, efficientnet, mobilenet, vision_transformer*, beit)
- More model pretrained tag and adjustments, some model names changed (working on deprecation translations, consider main branch DEV branch right now, use 0.6.x for stable use)
- More ImageNet-12k (subset of 22k) pretrain models popping up:
efficientnet_b5.in12k_ft_in1k - 85.9 @ 448x448vit_medium_patch16_gap_384.in12k_ft_in1k - 85.5 @ 384x384vit_medium_patch16_gap_256.in12k_ft_in1k - 84.5 @ 256x256convnext_nano.in12k_ft_in1k - 82.9 @ 288x288
Dec 8, 2022
- Add 'EVA l' to
vision_transformer.py, MAE style ViT-L/14 MIM pretrain w/ EVA-CLIP targets, FT on ImageNet-1k (w/ ImageNet-22k intermediate for some)
| model |
top1 |
param_count |
gmac |
macts |
hub |
| eva_large_patch14_336.in22k_ft_in22k_in1k |
89.2 |
304.5 |
191.1 |
270.2 |
link |
| eva_large_patch14_336.in22k_ft_in1k |
88.7 |
304.5 |
191.1 |
270.2 |
link |
| eva_large_patch14_196.in22k_ft_in22k_in1k |
88.6 |
304.1 |
61.6 |
63.5 |
link |
| eva_large_patch14_196.in22k_ft_in1k |
87.9 |
304.1 |
61.6 |
63.5 |
link |
Dec 6, 2022
- Add 'EVA g', BEiT style ViT-g/14 model weights w/ both MIM pretrain and CLIP pretrain to
beit.py.
| model |
top1 |
param_count |
gmac |
macts |
hub |
| eva_giant_patch14_560.m30m_ft_in22k_in1k |
89.8 |
1014.4 |
1906.8 |
2577.2 |
link |
| eva_giant_patch14_336.m30m_ft_in22k_in1k |
89.6 |
1013 |
620.6 |
550.7 |
link |
| eva_giant_patch14_336.clip_ft_in1k |
89.4 |
1013 |
620.6 |
550.7 |
link |
| eva_giant_patch14_224.clip_ft_in1k |
89.1 |
1012.6 |
267.2 |
192.6 |
link |
Dec 5, 2022
- Pre-release (
0.8.0dev0) of multi-weight support (model_arch.pretrained_tag). Install with pip install --pre timm
- vision_transformer, maxvit, convnext are the first three model impl w/ support
- model names are changing with this (previous _21k, etc. fn will merge), still sorting out deprecation handling
- bugs are likely, but I need feedback so please try it out
- if stability is needed, please use 0.6.x pypi releases or clone from 0.6.x branch
- Support for PyTorch 2.0 compile is added in train/validate/inference/benchmark, use
--torchcompile argument
- Inference script allows more control over output, select k for top-class index + prob json, csv or parquet output
- Add a full set of fine-tuned CLIP image tower weights from both LAION-2B and original OpenAI CLIP models
| model |
top1 |
param_count |
gmac |
macts |
hub |
| vit_huge_patch14_clip_336.laion2b_ft_in12k_in1k |
88.6 |
632.5 |
391 |
407.5 |
link |
| vit_large_patch14_clip_336.openai_ft_in12k_in1k |
88.3 |
304.5 |
191.1 |
270.2 |
link |
| vit_huge_patch14_clip_224.laion2b_ft_in12k_in1k |
88.2 |
632 |
167.4 |
139.4 |
link |
| vit_large_patch14_clip_336.laion2b_ft_in12k_in1k |
88.2 |
304.5 |
191.1 |
270.2 |
link |
| vit_large_patch14_clip_224.openai_ft_in12k_in1k |
88.2 |
304.2 |
81.1 |
88.8 |
link |
| vit_large_patch14_clip_224.laion2b_ft_in12k_in1k |
87.9 |
304.2 |
81.1 |
88.8 |
link |
| vit_large_patch14_clip_224.openai_ft_in1k |
87.9 |
304.2 |
81.1 |
88.8 |
link |
| vit_large_patch14_clip_336.laion2b_ft_in1k |
87.9 |
304.5 |
191.1 |
270.2 |
link |
| vit_huge_patch14_clip_224.laion2b_ft_in1k |
87.6 |
632 |
167.4 |
139.4 |
link |
| vit_large_patch14_clip_224.laion2b_ft_in1k |
87.3 |
304.2 |
81.1 |
88.8 |
link |
| vit_base_patch16_clip_384.laion2b_ft_in12k_in1k |
87.2 |
86.9 |
55.5 |
101.6 |
link |
| vit_base_patch16_clip_384.openai_ft_in12k_in1k |
87 |
86.9 |
55.5 |
101.6 |
link |
| vit_base_patch16_clip_384.laion2b_ft_in1k |
86.6 |
86.9 |
55.5 |
101.6 |
link |
| vit_base_patch16_clip_384.openai_ft_in1k |
86.2 |
86.9 |
55.5 |
101.6 |
link |
| vit_base_patch16_clip_224.laion2b_ft_in12k_in1k |
86.2 |
86.6 |
17.6 |
23.9 |
link |
| vit_base_patch16_clip_224.openai_ft_in12k_in1k |
85.9 |
86.6 |
17.6 |
23.9 |
link |
| vit_base_patch32_clip_448.laion2b_ft_in12k_in1k |
85.8 |
88.3 |
17.9 |
23.9 |
link |
| vit_base_patch16_clip_224.laion2b_ft_in1k |
85.5 |
86.6 |
17.6 |
23.9 |
link |
| vit_base_patch32_clip_384.laion2b_ft_in12k_in1k |
85.4 |
88.3 |
13.1 |
16.5 |
link |
| vit_base_patch16_clip_224.openai_ft_in1k |
85.3 |
86.6 |
17.6 |
23.9 |
link |
| vit_base_patch32_clip_384.openai_ft_in12k_in1k |
85.2 |
88.3 |
13.1 |
16.5 |
link |
| vit_base_patch32_clip_224.laion2b_ft_in12k_in1k |
83.3 |
88.2 |
4.4 |
5 |
link |
| vit_base_patch32_clip_224.laion2b_ft_in1k |
82.6 |
88.2 |
4.4 |
5 |
link |
| vit_base_patch32_clip_224.openai_ft_in1k |
81.9 |
88.2 |
4.4 |
5 |
link |
- Port of MaxViT Tensorflow Weights from official impl at https://github.com/google-research/maxvit
- There was larger than expected drops for the upscaled 384/512 in21k fine-tune weights, possible detail missing, but the 21k FT did seem sensitive to small preprocessing
| model |
top1 |
param_count |
gmac |
macts |
hub |
| maxvit_xlarge_tf_512.in21k_ft_in1k |
88.5 |
475.8 |
534.1 |
1413.2 |
link |
| maxvit_xlarge_tf_384.in21k_ft_in1k |
88.3 |
475.3 |
292.8 |
668.8 |
link |
| maxvit_base_tf_512.in21k_ft_in1k |
88.2 |
119.9 |
138 |
704 |
link |
| maxvit_large_tf_512.in21k_ft_in1k |
88 |
212.3 |
244.8 |
942.2 |
link |
| maxvit_large_tf_384.in21k_ft_in1k |
88 |
212 |
132.6 |
445.8 |
link |
| maxvit_base_tf_384.in21k_ft_in1k |
87.9 |
119.6 |
73.8 |
332.9 |
link |
| maxvit_base_tf_512.in1k |
86.6 |
119.9 |
138 |
704 |
link |
| maxvit_large_tf_512.in1k |
86.5 |
212.3 |
244.8 |
942.2 |
link |
| maxvit_base_tf_384.in1k |
86.3 |
119.6 |
73.8 |
332.9 |
link |
| maxvit_large_tf_384.in1k |
86.2 |
212 |
132.6 |
445.8 |
link |
| maxvit_small_tf_512.in1k |
86.1 |
69.1 |
67.3 |
383.8 |
link |
| maxvit_tiny_tf_512.in1k |
85.7 |
31 |
33.5 |
257.6 |
link |
| maxvit_small_tf_384.in1k |
85.5 |
69 |
35.9 |
183.6 |
link |
| maxvit_tiny_tf_384.in1k |
85.1 |
31 |
17.5 |
123.4 |
link |
| maxvit_large_tf_224.in1k |
84.9 |
211.8 |
43.7 |
127.4 |
link |
| maxvit_base_tf_224.in1k |
84.9 |
119.5 |
24 |
95 |
link |
| maxvit_small_tf_224.in1k |
84.4 |
68.9 |
11.7 |
53.2 |
link |
| maxvit_tiny_tf_224.in1k |
83.4 |
30.9 |
5.6 |
35.8 |
link |
Oct 15, 2022
- Train and validation script enhancements
- Non-GPU (ie CPU) device support
- SLURM compatibility for train script
- HF datasets support (via ReaderHfds)
- TFDS/WDS dataloading improvements (sample padding/wrap for distributed use fixed wrt sample count estimate)
- in_chans !=3 support for scripts / loader
- Adan optimizer
- Can enable per-step LR scheduling via args
- Dataset 'parsers' renamed to 'readers', more descriptive of purpose
- AMP args changed, APEX via
--amp-impl apex, bfloat16 supportedf via --amp-dtype bfloat16
- main branch switched to 0.7.x version, 0.6x forked for stable release of weight only adds
- master -> main branch rename
Oct 10, 2022
- More weights in
maxxvit series, incl first ConvNeXt block based coatnext and maxxvit experiments:
coatnext_nano_rw_224 - 82.0 @ 224 (G) -- (uses ConvNeXt conv block, no BatchNorm)maxxvit_rmlp_nano_rw_256 - 83.0 @ 256, 83.7 @ 320 (G) (uses ConvNeXt conv block, no BN)maxvit_rmlp_small_rw_224 - 84.5 @ 224, 85.1 @ 320 (G)maxxvit_rmlp_small_rw_256 - 84.6 @ 256, 84.9 @ 288 (G) -- could be trained better, hparams need tuning (uses ConvNeXt block, no BN)coatnet_rmlp_2_rw_224 - 84.6 @ 224, 85 @ 320 (T)- NOTE: official MaxVit weights (in1k) have been released at https://github.com/google-research/maxvit -- some extra work is needed to port and adapt since my impl was created independently of theirs and has a few small differences + the whole TF same padding fun.
Sept 23, 2022
- LAION-2B CLIP image towers supported as pretrained backbones for fine-tune or features (no classifier)
- vit_base_patch32_224_clip_laion2b
- vit_large_patch14_224_clip_laion2b
- vit_huge_patch14_224_clip_laion2b
- vit_giant_patch14_224_clip_laion2b
Sept 7, 2022
- Hugging Face
timm docs home now exists, look for more here in the future
- Add BEiT-v2 weights for base and large 224x224 models from https://github.com/microsoft/unilm/tree/master/beit2
- Add more weights in
maxxvit series incl a pico (7.5M params, 1.9 GMACs), two tiny variants:
maxvit_rmlp_pico_rw_256 - 80.5 @ 256, 81.3 @ 320 (T)maxvit_tiny_rw_224 - 83.5 @ 224 (G)maxvit_rmlp_tiny_rw_256 - 84.2 @ 256, 84.8 @ 320 (T)
Aug 29, 2022
- MaxVit window size scales with img_size by default. Add new RelPosMlp MaxViT weight that leverages this:
maxvit_rmlp_nano_rw_256 - 83.0 @ 256, 83.6 @ 320 (T)
Aug 26, 2022
Aug 15, 2022
- ConvNeXt atto weights added
convnext_atto - 75.7 @ 224, 77.0 @ 288convnext_atto_ols - 75.9 @ 224, 77.2 @ 288
Aug 5, 2022
- More custom ConvNeXt smaller model defs with weights
convnext_femto - 77.5 @ 224, 78.7 @ 288convnext_femto_ols - 77.9 @ 224, 78.9 @ 288convnext_pico - 79.5 @ 224, 80.4 @ 288convnext_pico_ols - 79.5 @ 224, 80.5 @ 288convnext_nano_ols - 80.9 @ 224, 81.6 @ 288
- Updated EdgeNeXt to improve ONNX export, add new base variant and weights from original (https://github.com/mmaaz60/EdgeNeXt)
July 28, 2022
- Add freshly minted DeiT-III Medium (width=512, depth=12, num_heads=8) model weights. Thanks Hugo Touvron!
July 27, 2022
- All runtime benchmark and validation result csv files are finally up-to-date!
- A few more weights & model defs added:
darknetaa53 - 79.8 @ 256, 80.5 @ 288convnext_nano - 80.8 @ 224, 81.5 @ 288cs3sedarknet_l - 81.2 @ 256, 81.8 @ 288cs3darknet_x - 81.8 @ 256, 82.2 @ 288cs3sedarknet_x - 82.2 @ 256, 82.7 @ 288cs3edgenet_x - 82.2 @ 256, 82.7 @ 288cs3se_edgenet_x - 82.8 @ 256, 83.5 @ 320
cs3* weights above all trained on TPU w/ bits_and_tpu branch. Thanks to TRC program!- Add output_stride=8 and 16 support to ConvNeXt (dilation)
- deit3 models not being able to resize pos_emb fixed
- Version 0.6.7 PyPi release (/w above bug fixes and new weighs since 0.6.5)
July 8, 2022
More models, more fixes
- Official research models (w/ weights) added:
- My own models:
- Small
ResNet defs added by request with 1 block repeats for both basic and bottleneck (resnet10 and resnet14)
CspNet refactored with dataclass config, simplified CrossStage3 (cs3) option. These are closer to YOLO-v5+ backbone defs.- More relative position vit fiddling. Two
srelpos (shared relative position) models trained, and a medium w/ class token.
- Add an alternate downsample mode to EdgeNeXt and train a
small model. Better than original small, but not their new USI trained weights.
- My own model weight results (all ImageNet-1k training)
resnet10t - 66.5 @ 176, 68.3 @ 224resnet14t - 71.3 @ 176, 72.3 @ 224resnetaa50 - 80.6 @ 224 , 81.6 @ 288darknet53 - 80.0 @ 256, 80.5 @ 288cs3darknet_m - 77.0 @ 256, 77.6 @ 288cs3darknet_focus_m - 76.7 @ 256, 77.3 @ 288cs3darknet_l - 80.4 @ 256, 80.9 @ 288cs3darknet_focus_l - 80.3 @ 256, 80.9 @ 288vit_srelpos_small_patch16_224 - 81.1 @ 224, 82.1 @ 320vit_srelpos_medium_patch16_224 - 82.3 @ 224, 83.1 @ 320vit_relpos_small_patch16_cls_224 - 82.6 @ 224, 83.6 @ 320edgnext_small_rw - 79.6 @ 224, 80.4 @ 320
cs3, darknet, and vit_*relpos weights above all trained on TPU thanks to TRC program! Rest trained on overheating GPUs.- Hugging Face Hub support fixes verified, demo notebook TBA
- Pretrained weights / configs can be loaded externally (ie from local disk) w/ support for head adaptation.
- Add support to change image extensions scanned by
timm datasets/readers. See (https://github.com/rwightman/pytorch-image-models/pull/1274#issuecomment-1178303103)
- Default ConvNeXt LayerNorm impl to use
F.layer_norm(x.permute(0, 2, 3, 1), ...).permute(0, 3, 1, 2) via LayerNorm2d in all cases.
- a bit slower than previous custom impl on some hardware (ie Ampere w/ CL), but overall fewer regressions across wider HW / PyTorch version ranges.
- previous impl exists as
LayerNormExp2d in models/layers/norm.py
- Numerous bug fixes
- Currently testing for imminent PyPi 0.6.x release
- LeViT pretraining of larger models still a WIP, they don't train well / easily without distillation. Time to add distill support (finally)?
- ImageNet-22k weight training + finetune ongoing, work on multi-weight support (slowly) chugging along (there are a LOT of weights, sigh) ...
May 13, 2022
- Official Swin-V2 models and weights added from (https://github.com/microsoft/Swin-Transformer). Cleaned up to support torchscript.
- Some refactoring for existing
timm Swin-V2-CR impl, will likely do a bit more to bring parts closer to official and decide whether to merge some aspects.
- More Vision Transformer relative position / residual post-norm experiments (all trained on TPU thanks to TRC program)
vit_relpos_small_patch16_224 - 81.5 @ 224, 82.5 @ 320 -- rel pos, layer scale, no class token, avg poolvit_relpos_medium_patch16_rpn_224 - 82.3 @ 224, 83.1 @ 320 -- rel pos + res-post-norm, no class token, avg poolvit_relpos_medium_patch16_224 - 82.5 @ 224, 83.3 @ 320 -- rel pos, layer scale, no class token, avg poolvit_relpos_base_patch16_gapcls_224 - 82.8 @ 224, 83.9 @ 320 -- rel pos, layer scale, class token, avg pool (by mistake)
- Bring 512 dim, 8-head 'medium' ViT model variant back to life (after using in a pre DeiT 'small' model for first ViT impl back in 2020)
- Add ViT relative position support for switching btw existing impl and some additions in official Swin-V2 impl for future trials
- Sequencer2D impl (https://arxiv.org/abs/2205.01972), added via PR from author (https://github.com/okojoalg)
May 2, 2022
- Vision Transformer experiments adding Relative Position (Swin-V2 log-coord) (
vision_transformer_relpos.py) and Residual Post-Norm branches (from Swin-V2) (vision_transformer*.py)
vit_relpos_base_patch32_plus_rpn_256 - 79.5 @ 256, 80.6 @ 320 -- rel pos + extended width + res-post-norm, no class token, avg poolvit_relpos_base_patch16_224 - 82.5 @ 224, 83.6 @ 320 -- rel pos, layer scale, no class token, avg poolvit_base_patch16_rpn_224 - 82.3 @ 224 -- rel pos + res-post-norm, no class token, avg pool
- Vision Transformer refactor to remove representation layer that was only used in initial vit and rarely used since with newer pretrain (ie
How to Train Your ViT)
vit_* models support removal of class token, use of global average pool, use of fc_norm (ala beit, mae).
April 22, 2022
timm models are now officially supported in fast.ai! Just in time for the new Practical Deep Learning course. timmdocs documentation link updated to timm.fast.ai.- Two more model weights added in the TPU trained series. Some In22k pretrain still in progress.
seresnext101d_32x8d - 83.69 @ 224, 84.35 @ 288seresnextaa101d_32x8d (anti-aliased w/ AvgPool2d) - 83.85 @ 224, 84.57 @ 288
March 23, 2022
- Add
ParallelBlock and LayerScale option to base vit models to support model configs in Three things everyone should know about ViT
convnext_tiny_hnf (head norm first) weights trained with (close to) A2 recipe, 82.2% top-1, could do better with more epochs.
March 21, 2022
- Merge
norm_norm_norm. IMPORTANT this update for a coming 0.6.x release will likely de-stabilize the master branch for a while. Branch 0.5.x or a previous 0.5.x release can be used if stability is required.
- Significant weights update (all TPU trained) as described in this release
regnety_040 - 82.3 @ 224, 82.96 @ 288regnety_064 - 83.0 @ 224, 83.65 @ 288regnety_080 - 83.17 @ 224, 83.86 @ 288regnetv_040 - 82.44 @ 224, 83.18 @ 288 (timm pre-act)regnetv_064 - 83.1 @ 224, 83.71 @ 288 (timm pre-act)regnetz_040 - 83.67 @ 256, 84.25 @ 320regnetz_040h - 83.77 @ 256, 84.5 @ 320 (w/ extra fc in head)resnetv2_50d_gn - 80.8 @ 224, 81.96 @ 288 (pre-act GroupNorm)resnetv2_50d_evos 80.77 @ 224, 82.04 @ 288 (pre-act EvoNormS)regnetz_c16_evos - 81.9 @ 256, 82.64 @ 320 (EvoNormS)regnetz_d8_evos - 83.42 @ 256, 84.04 @ 320 (EvoNormS)xception41p - 82 @ 299 (timm pre-act)xception65 - 83.17 @ 299xception65p - 83.14 @ 299 (timm pre-act)resnext101_64x4d - 82.46 @ 224, 83.16 @ 288seresnext101_32x8d - 83.57 @ 224, 84.270 @ 288resnetrs200 - 83.85 @ 256, 84.44 @ 320
- HuggingFace hub support fixed w/ initial groundwork for allowing alternative 'config sources' for pretrained model definitions and weights (generic local file / remote url support soon)
- SwinTransformer-V2 implementation added. Submitted by Christoph Reich. Training experiments and model changes by myself are ongoing so expect compat breaks.
- Swin-S3 (AutoFormerV2) models / weights added from https://github.com/microsoft/Cream/tree/main/AutoFormerV2
- MobileViT models w/ weights adapted from https://github.com/apple/ml-cvnets
- PoolFormer models w/ weights adapted from https://github.com/sail-sg/poolformer
- VOLO models w/ weights adapted from https://github.com/sail-sg/volo
- Significant work experimenting with non-BatchNorm norm layers such as EvoNorm, FilterResponseNorm, GroupNorm, etc
- Enhance support for alternate norm + act ('NormAct') layers added to a number of models, esp EfficientNet/MobileNetV3, RegNet, and aligned Xception
- Grouped conv support added to EfficientNet family
- Add 'group matching' API to all models to allow grouping model parameters for application of 'layer-wise' LR decay, lr scale added to LR scheduler
- Gradient checkpointing support added to many models
forward_head(x, pre_logits=False) fn added to all models to allow separate calls of forward_features + forward_head- All vision transformer and vision MLP models update to return non-pooled / non-token selected features from
forward_features, for consistency with CNN models, token selection or pooling now applied in forward_head
Feb 2, 2022
- Chris Hughes posted an exhaustive run through of
timm on his blog yesterday. Well worth a read. Getting Started with PyTorch Image Models (timm): A Practitioner’s Guide
- I'm currently prepping to merge the
norm_norm_norm branch back to master (ver 0.6.x) in next week or so.
- The changes are more extensive than usual and may destabilize and break some model API use (aiming for full backwards compat). So, beware
pip install git+https://github.com/rwightman/pytorch-image-models installs!
0.5.x releases and a 0.5.x branch will remain stable with a cherry pick or two until dust clears. Recommend sticking to pypi install for a bit if you want stable.
Jan 14, 2022
- Version 0.5.4 w/ release to be pushed to pypi. It's been a while since last pypi update and riskier changes will be merged to main branch soon....
- Add ConvNeXT models /w weights from official impl (https://github.com/facebookresearch/ConvNeXt), a few perf tweaks, compatible with timm features
- Tried training a few small (~1.8-3M param) / mobile optimized models, a few are good so far, more on the way...
mnasnet_small - 65.6 top-1mobilenetv2_050 - 65.9lcnet_100/075/050 - 72.1 / 68.8 / 63.1semnasnet_075 - 73fbnetv3_b/d/g - 79.1 / 79.7 / 82.0
- TinyNet models added by rsomani95
- LCNet added via MobileNetV3 architecture
Jan 5, 2023
- ConvNeXt-V2 models and weights added to existing
convnext.py
Dec 23, 2022 🎄☃
- Add FlexiViT models and weights from https://github.com/google-research/big_vision (check out paper at https://arxiv.org/abs/2212.08013)
- NOTE currently resizing is static on model creation, on-the-fly dynamic / train patch size sampling is a WIP
- Many more models updated to multi-weight and downloadable via HF hub now (convnext, efficientnet, mobilenet, vision_transformer*, beit)
- More model pretrained tag and adjustments, some model names changed (working on deprecation translations, consider main branch DEV branch right now, use 0.6.x for stable use)
- More ImageNet-12k (subset of 22k) pretrain models popping up:
efficientnet_b5.in12k_ft_in1k - 85.9 @ 448x448vit_medium_patch16_gap_384.in12k_ft_in1k - 85.5 @ 384x384vit_medium_patch16_gap_256.in12k_ft_in1k - 84.5 @ 256x256convnext_nano.in12k_ft_in1k - 82.9 @ 288x288
Dec 8, 2022
- Add 'EVA l' to
vision_transformer.py, MAE style ViT-L/14 MIM pretrain w/ EVA-CLIP targets, FT on ImageNet-1k (w/ ImageNet-22k intermediate for some)
| model |
top1 |
param_count |
gmac |
macts |
hub |
| eva_large_patch14_336.in22k_ft_in22k_in1k |
89.2 |
304.5 |
191.1 |
270.2 |
link |
| eva_large_patch14_336.in22k_ft_in1k |
88.7 |
304.5 |
191.1 |
270.2 |
link |
| eva_large_patch14_196.in22k_ft_in22k_in1k |
88.6 |
304.1 |
61.6 |
63.5 |
link |
| eva_large_patch14_196.in22k_ft_in1k |
87.9 |
304.1 |
61.6 |
63.5 |
link |
Dec 6, 2022
- Add 'EVA g', BEiT style ViT-g/14 model weights w/ both MIM pretrain and CLIP pretrain to
beit.py.
| model |
top1 |
param_count |
gmac |
macts |
hub |
| eva_giant_patch14_560.m30m_ft_in22k_in1k |
89.8 |
1014.4 |
1906.8 |
2577.2 |
link |
| eva_giant_patch14_336.m30m_ft_in22k_in1k |
89.6 |
1013 |
620.6 |
550.7 |
link |
| eva_giant_patch14_336.clip_ft_in1k |
89.4 |
1013 |
620.6 |
550.7 |
link |
| eva_giant_patch14_224.clip_ft_in1k |
89.1 |
1012.6 |
267.2 |
192.6 |
link |
Dec 5, 2022
- Pre-release (
0.8.0dev0) of multi-weight support (model_arch.pretrained_tag). Install with pip install --pre timm
- vision_transformer, maxvit, convnext are the first three model impl w/ support
- model names are changing with this (previous _21k, etc. fn will merge), still sorting out deprecation handling
- bugs are likely, but I need feedback so please try it out
- if stability is needed, please use 0.6.x pypi releases or clone from 0.6.x branch
- Support for PyTorch 2.0 compile is added in train/validate/inference/benchmark, use
--torchcompile argument
- Inference script allows more control over output, select k for top-class index + prob json, csv or parquet output
- Add a full set of fine-tuned CLIP image tower weights from both LAION-2B and original OpenAI CLIP models
| model |
top1 |
param_count |
gmac |
macts |
hub |
| vit_huge_patch14_clip_336.laion2b_ft_in12k_in1k |
88.6 |
632.5 |
391 |
407.5 |
link |
| vit_large_patch14_clip_336.openai_ft_in12k_in1k |
88.3 |
304.5 |
191.1 |
270.2 |
link |
| vit_huge_patch14_clip_224.laion2b_ft_in12k_in1k |
88.2 |
632 |
167.4 |
139.4 |
link |
| vit_large_patch14_clip_336.laion2b_ft_in12k_in1k |
88.2 |
304.5 |
191.1 |
270.2 |
link |
| vit_large_patch14_clip_224.openai_ft_in12k_in1k |
88.2 |
304.2 |
81.1 |
88.8 |
link |
| vit_large_patch14_clip_224.laion2b_ft_in12k_in1k |
87.9 |
304.2 |
81.1 |
88.8 |
link |
| vit_large_patch14_clip_224.openai_ft_in1k |
87.9 |
304.2 |
81.1 |
88.8 |
link |
| vit_large_patch14_clip_336.laion2b_ft_in1k |
87.9 |
304.5 |
191.1 |
270.2 |
link |
| vit_huge_patch14_clip_224.laion2b_ft_in1k |
87.6 |
632 |
167.4 |
139.4 |
link |
| vit_large_patch14_clip_224.laion2b_ft_in1k |
87.3 |
304.2 |
81.1 |
88.8 |
link |
| vit_base_patch16_clip_384.laion2b_ft_in12k_in1k |
87.2 |
86.9 |
55.5 |
101.6 |
link |
| vit_base_patch16_clip_384.openai_ft_in12k_in1k |
87 |
86.9 |
55.5 |
101.6 |
link |
| vit_base_patch16_clip_384.laion2b_ft_in1k |
86.6 |
86.9 |
55.5 |
101.6 |
link |
| vit_base_patch16_clip_384.openai_ft_in1k |
86.2 |
86.9 |
55.5 |
101.6 |
link |
| vit_base_patch16_clip_224.laion2b_ft_in12k_in1k |
86.2 |
86.6 |
17.6 |
23.9 |
link |
| vit_base_patch16_clip_224.openai_ft_in12k_in1k |
85.9 |
86.6 |
17.6 |
23.9 |
link |
| vit_base_patch32_clip_448.laion2b_ft_in12k_in1k |
85.8 |
88.3 |
17.9 |
23.9 |
link |
| vit_base_patch16_clip_224.laion2b_ft_in1k |
85.5 |
86.6 |
17.6 |
23.9 |
link |
| vit_base_patch32_clip_384.laion2b_ft_in12k_in1k |
85.4 |
88.3 |
13.1 |
16.5 |
link |
| vit_base_patch16_clip_224.openai_ft_in1k |
85.3 |
86.6 |
17.6 |
23.9 |
link |
| vit_base_patch32_clip_384.openai_ft_in12k_in1k |
85.2 |
88.3 |
13.1 |
16.5 |
link |
| vit_base_patch32_clip_224.laion2b_ft_in12k_in1k |
83.3 |
88.2 |
4.4 |
5 |
link |
| vit_base_patch32_clip_224.laion2b_ft_in1k |
82.6 |
88.2 |
4.4 |
5 |
link |
| vit_base_patch32_clip_224.openai_ft_in1k |
81.9 |
88.2 |
4.4 |
5 |
link |
- Port of MaxViT Tensorflow Weights from official impl at https://github.com/google-research/maxvit
- There was larger than expected drops for the upscaled 384/512 in21k fine-tune weights, possible detail missing, but the 21k FT did seem sensitive to small preprocessing
| model |
top1 |
param_count |
gmac |
macts |
hub |
| maxvit_xlarge_tf_512.in21k_ft_in1k |
88.5 |
475.8 |
534.1 |
1413.2 |
link |
| maxvit_xlarge_tf_384.in21k_ft_in1k |
88.3 |
475.3 |
292.8 |
668.8 |
link |
| maxvit_base_tf_512.in21k_ft_in1k |
88.2 |
119.9 |
138 |
704 |
link |
| maxvit_large_tf_512.in21k_ft_in1k |
88 |
212.3 |
244.8 |
942.2 |
link |
| maxvit_large_tf_384.in21k_ft_in1k |
88 |
212 |
132.6 |
445.8 |
link |
| maxvit_base_tf_384.in21k_ft_in1k |
87.9 |
119.6 |
73.8 |
332.9 |
link |
| maxvit_base_tf_512.in1k |
86.6 |
119.9 |
138 |
704 |
link |
| maxvit_large_tf_512.in1k |
86.5 |
212.3 |
244.8 |
942.2 |
link |
| maxvit_base_tf_384.in1k |
86.3 |
119.6 |
73.8 |
332.9 |
link |
| maxvit_large_tf_384.in1k |
86.2 |
212 |
132.6 |
445.8 |
link |
| maxvit_small_tf_512.in1k |
86.1 |
69.1 |
67.3 |
383.8 |
link |
| maxvit_tiny_tf_512.in1k |
85.7 |
31 |
33.5 |
257.6 |
link |
| maxvit_small_tf_384.in1k |
85.5 |
69 |
35.9 |
183.6 |
link |
| maxvit_tiny_tf_384.in1k |
85.1 |
31 |
17.5 |
123.4 |
link |
| maxvit_large_tf_224.in1k |
84.9 |
211.8 |
43.7 |
127.4 |
link |
| maxvit_base_tf_224.in1k |
84.9 |
119.5 |
24 |
95 |
link |
| maxvit_small_tf_224.in1k |
84.4 |
68.9 |
11.7 |
53.2 |
link |
| maxvit_tiny_tf_224.in1k |
83.4 |
30.9 |
5.6 |
35.8 |
link |
Oct 15, 2022
- Train and validation script enhancements
- Non-GPU (ie CPU) device support
- SLURM compatibility for train script
- HF datasets support (via ReaderHfds)
- TFDS/WDS dataloading improvements (sample padding/wrap for distributed use fixed wrt sample count estimate)
- in_chans !=3 support for scripts / loader
- Adan optimizer
- Can enable per-step LR scheduling via args
- Dataset 'parsers' renamed to 'readers', more descriptive of purpose
- AMP args changed, APEX via
--amp-impl apex, bfloat16 supportedf via --amp-dtype bfloat16
- main branch switched to 0.7.x version, 0.6x forked for stable release of weight only adds
- master -> main branch rename
Oct 10, 2022
- More weights in
maxxvit series, incl first ConvNeXt block based coatnext and maxxvit experiments:
coatnext_nano_rw_224 - 82.0 @ 224 (G) -- (uses ConvNeXt conv block, no BatchNorm)maxxvit_rmlp_nano_rw_256 - 83.0 @ 256, 83.7 @ 320 (G) (uses ConvNeXt conv block, no BN)maxvit_rmlp_small_rw_224 - 84.5 @ 224, 85.1 @ 320 (G)maxxvit_rmlp_small_rw_256 - 84.6 @ 256, 84.9 @ 288 (G) -- could be trained better, hparams need tuning (uses ConvNeXt block, no BN)coatnet_rmlp_2_rw_224 - 84.6 @ 224, 85 @ 320 (T)- NOTE: official MaxVit weights (in1k) have been released at https://github.com/google-research/maxvit -- some extra work is needed to port and adapt since my impl was created independently of theirs and has a few small differences + the whole TF same padding fun.
Sept 23, 2022
- LAION-2B CLIP image towers supported as pretrained backbones for fine-tune or features (no classifier)
- vit_base_patch32_224_clip_laion2b
- vit_large_patch14_224_clip_laion2b
- vit_huge_patch14_224_clip_laion2b
- vit_giant_patch14_224_clip_laion2b
Sept 7, 2022
- Hugging Face
timm docs home now exists, look for more here in the future
- Add BEiT-v2 weights for base and large 224x224 models from https://github.com/microsoft/unilm/tree/master/beit2
- Add more weights in
maxxvit series incl a pico (7.5M params, 1.9 GMACs), two tiny variants:
maxvit_rmlp_pico_rw_256 - 80.5 @ 256, 81.3 @ 320 (T)maxvit_tiny_rw_224 - 83.5 @ 224 (G)maxvit_rmlp_tiny_rw_256 - 84.2 @ 256, 84.8 @ 320 (T)
Aug 29, 2022
- MaxVit window size scales with img_size by default. Add new RelPosMlp MaxViT weight that leverages this:
maxvit_rmlp_nano_rw_256 - 83.0 @ 256, 83.6 @ 320 (T)
Aug 26, 2022
Aug 15, 2022
- ConvNeXt atto weights added
convnext_atto - 75.7 @ 224, 77.0 @ 288convnext_atto_ols - 75.9 @ 224, 77.2 @ 288
Aug 5, 2022
- More custom ConvNeXt smaller model defs with weights
convnext_femto - 77.5 @ 224, 78.7 @ 288convnext_femto_ols - 77.9 @ 224, 78.9 @ 288convnext_pico - 79.5 @ 224, 80.4 @ 288convnext_pico_ols - 79.5 @ 224, 80.5 @ 288convnext_nano_ols - 80.9 @ 224, 81.6 @ 288
- Updated EdgeNeXt to improve ONNX export, add new base variant and weights from original (https://github.com/mmaaz60/EdgeNeXt)
July 28, 2022
- Add freshly minted DeiT-III Medium (width=512, depth=12, num_heads=8) model weights. Thanks Hugo Touvron!
July 27, 2022
- All runtime benchmark and validation result csv files are up-to-date!
- A few more weights & model defs added:
darknetaa53 - 79.8 @ 256, 80.5 @ 288convnext_nano - 80.8 @ 224, 81.5 @ 288cs3sedarknet_l - 81.2 @ 256, 81.8 @ 288cs3darknet_x - 81.8 @ 256, 82.2 @ 288cs3sedarknet_x - 82.2 @ 256, 82.7 @ 288cs3edgenet_x - 82.2 @ 256, 82.7 @ 288cs3se_edgenet_x - 82.8 @ 256, 83.5 @ 320
cs3* weights above all trained on TPU w/ bits_and_tpu branch. Thanks to TRC program!- Add output_stride=8 and 16 support to ConvNeXt (dilation)
- deit3 models not being able to resize pos_emb fixed
- Version 0.6.7 PyPi release (/w above bug fixes and new weighs since 0.6.5)
July 8, 2022
More models, more fixes
- Official research models (w/ weights) added:
- My own models:
- Small
ResNet defs added by request with 1 block repeats for both basic and bottleneck (resnet10 and resnet14)
CspNet refactored with dataclass config, simplified CrossStage3 (cs3) option. These are closer to YOLO-v5+ backbone defs.- More relative position vit fiddling. Two
srelpos (shared relative position) models trained, and a medium w/ class token.
- Add an alternate downsample mode to EdgeNeXt and train a
small model. Better than original small, but not their new USI trained weights.
- My own model weight results (all ImageNet-1k training)
resnet10t - 66.5 @ 176, 68.3 @ 224resnet14t - 71.3 @ 176, 72.3 @ 224resnetaa50 - 80.6 @ 224 , 81.6 @ 288darknet53 - 80.0 @ 256, 80.5 @ 288cs3darknet_m - 77.0 @ 256, 77.6 @ 288cs3darknet_focus_m - 76.7 @ 256, 77.3 @ 288cs3darknet_l - 80.4 @ 256, 80.9 @ 288cs3darknet_focus_l - 80.3 @ 256, 80.9 @ 288vit_srelpos_small_patch16_224 - 81.1 @ 224, 82.1 @ 320vit_srelpos_medium_patch16_224 - 82.3 @ 224, 83.1 @ 320vit_relpos_small_patch16_cls_224 - 82.6 @ 224, 83.6 @ 320edgnext_small_rw - 79.6 @ 224, 80.4 @ 320
cs3, darknet, and vit_*relpos weights above all trained on TPU thanks to TRC program! Rest trained on overheating GPUs.- Hugging Face Hub support fixes verified, demo notebook TBA
- Pretrained weights / configs can be loaded externally (ie from local disk) w/ support for head adaptation.
- Add support to change image extensions scanned by
timm datasets/parsers. See (https://github.com/rwightman/pytorch-image-models/pull/1274#issuecomment-1178303103)
- Default ConvNeXt LayerNorm impl to use
F.layer_norm(x.permute(0, 2, 3, 1), ...).permute(0, 3, 1, 2) via LayerNorm2d in all cases.
- a bit slower than previous custom impl on some hardware (ie Ampere w/ CL), but overall fewer regressions across wider HW / PyTorch version ranges.
- previous impl exists as
LayerNormExp2d in models/layers/norm.py
- Numerous bug fixes
- Currently testing for imminent PyPi 0.6.x release
- LeViT pretraining of larger models still a WIP, they don't train well / easily without distillation. Time to add distill support (finally)?
- ImageNet-22k weight training + finetune ongoing, work on multi-weight support (slowly) chugging along (there are a LOT of weights, sigh) ...
May 13, 2022
- Official Swin-V2 models and weights added from (https://github.com/microsoft/Swin-Transformer). Cleaned up to support torchscript.
- Some refactoring for existing
timm Swin-V2-CR impl, will likely do a bit more to bring parts closer to official and decide whether to merge some aspects.
- More Vision Transformer relative position / residual post-norm experiments (all trained on TPU thanks to TRC program)
vit_relpos_small_patch16_224 - 81.5 @ 224, 82.5 @ 320 -- rel pos, layer scale, no class token, avg poolvit_relpos_medium_patch16_rpn_224 - 82.3 @ 224, 83.1 @ 320 -- rel pos + res-post-norm, no class token, avg poolvit_relpos_medium_patch16_224 - 82.5 @ 224, 83.3 @ 320 -- rel pos, layer scale, no class token, avg poolvit_relpos_base_patch16_gapcls_224 - 82.8 @ 224, 83.9 @ 320 -- rel pos, layer scale, class token, avg pool (by mistake)
- Bring 512 dim, 8-head 'medium' ViT model variant back to life (after using in a pre DeiT 'small' model for first ViT impl back in 2020)
- Add ViT relative position support for switching btw existing impl and some additions in official Swin-V2 impl for future trials
- Sequencer2D impl (https://arxiv.org/abs/2205.01972), added via PR from author (https://github.com/okojoalg)
May 2, 2022
- Vision Transformer experiments adding Relative Position (Swin-V2 log-coord) (
vision_transformer_relpos.py) and Residual Post-Norm branches (from Swin-V2) (vision_transformer*.py)
vit_relpos_base_patch32_plus_rpn_256 - 79.5 @ 256, 80.6 @ 320 -- rel pos + extended width + res-post-norm, no class token, avg poolvit_relpos_base_patch16_224 - 82.5 @ 224, 83.6 @ 320 -- rel pos, layer scale, no class token, avg poolvit_base_patch16_rpn_224 - 82.3 @ 224 -- rel pos + res-post-norm, no class token, avg pool
- Vision Transformer refactor to remove representation layer that was only used in initial vit and rarely used since with newer pretrain (ie
How to Train Your ViT)
vit_* models support removal of class token, use of global average pool, use of fc_norm (ala beit, mae).
April 22, 2022
timm models are now officially supported in fast.ai! Just in time for the new Practical Deep Learning course. timmdocs documentation link updated to timm.fast.ai.- Two more model weights added in the TPU trained series. Some In22k pretrain still in progress.
seresnext101d_32x8d - 83.69 @ 224, 84.35 @ 288seresnextaa101d_32x8d (anti-aliased w/ AvgPool2d) - 83.85 @ 224, 84.57 @ 288
March 23, 2022
- Add
ParallelBlock and LayerScale option to base vit models to support model configs in Three things everyone should know about ViT
convnext_tiny_hnf (head norm first) weights trained with (close to) A2 recipe, 82.2% top-1, could do better with more epochs.
March 21, 2022
- Merge
norm_norm_norm. IMPORTANT this update for a coming 0.6.x release will likely de-stabilize the master branch for a while. Branch 0.5.x or a previous 0.5.x release can be used if stability is required.
- Significant weights update (all TPU trained) as described in this release
regnety_040 - 82.3 @ 224, 82.96 @ 288regnety_064 - 83.0 @ 224, 83.65 @ 288regnety_080 - 83.17 @ 224, 83.86 @ 288regnetv_040 - 82.44 @ 224, 83.18 @ 288 (timm pre-act)regnetv_064 - 83.1 @ 224, 83.71 @ 288 (timm pre-act)regnetz_040 - 83.67 @ 256, 84.25 @ 320regnetz_040h - 83.77 @ 256, 84.5 @ 320 (w/ extra fc in head)resnetv2_50d_gn - 80.8 @ 224, 81.96 @ 288 (pre-act GroupNorm)resnetv2_50d_evos 80.77 @ 224, 82.04 @ 288 (pre-act EvoNormS)regnetz_c16_evos - 81.9 @ 256, 82.64 @ 320 (EvoNormS)regnetz_d8_evos - 83.42 @ 256, 84.04 @ 320 (EvoNormS)xception41p - 82 @ 299 (timm pre-act)xception65 - 83.17 @ 299xception65p - 83.14 @ 299 (timm pre-act)resnext101_64x4d - 82.46 @ 224, 83.16 @ 288seresnext101_32x8d - 83.57 @ 224, 84.270 @ 288resnetrs200 - 83.85 @ 256, 84.44 @ 320
- HuggingFace hub support fixed w/ initial groundwork for allowing alternative 'config sources' for pretrained model definitions and weights (generic local file / remote url support soon)
- SwinTransformer-V2 implementation added. Submitted by Christoph Reich. Training experiments and model changes by myself are ongoing so expect compat breaks.
- Swin-S3 (AutoFormerV2) models / weights added from https://github.com/microsoft/Cream/tree/main/AutoFormerV2
- MobileViT models w/ weights adapted from https://github.com/apple/ml-cvnets
- PoolFormer models w/ weights adapted from https://github.com/sail-sg/poolformer
- VOLO models w/ weights adapted from https://github.com/sail-sg/volo
- Significant work experimenting with non-BatchNorm norm layers such as EvoNorm, FilterResponseNorm, GroupNorm, etc
- Enhance support for alternate norm + act ('NormAct') layers added to a number of models, esp EfficientNet/MobileNetV3, RegNet, and aligned Xception
- Grouped conv support added to EfficientNet family
- Add 'group matching' API to all models to allow grouping model parameters for application of 'layer-wise' LR decay, lr scale added to LR scheduler
- Gradient checkpointing support added to many models
forward_head(x, pre_logits=False) fn added to all models to allow separate calls of forward_features + forward_head- All vision transformer and vision MLP models update to return non-pooled / non-token selected features from
forward_features, for consistency with CNN models, token selection or pooling now applied in forward_head
Feb 2, 2022
- Chris Hughes posted an exhaustive run through of
timm on his blog yesterday. Well worth a read. Getting Started with PyTorch Image Models (timm): A Practitioner’s Guide
- I'm currently prepping to merge the
norm_norm_norm branch back to master (ver 0.6.x) in next week or so.
- The changes are more extensive than usual and may destabilize and break some model API use (aiming for full backwards compat). So, beware
pip install git+https://github.com/rwightman/pytorch-image-models installs!
0.5.x releases and a 0.5.x branch will remain stable with a cherry pick or two until dust clears. Recommend sticking to pypi install for a bit if you want stable.
Jan 14, 2022
- Version 0.5.4 w/ release to be pushed to pypi. It's been a while since last pypi update and riskier changes will be merged to main branch soon....
- Add ConvNeXT models /w weights from official impl (https://github.com/facebookresearch/ConvNeXt), a few perf tweaks, compatible with timm features
- Tried training a few small (~1.8-3M param) / mobile optimized models, a few are good so far, more on the way...
mnasnet_small - 65.6 top-1mobilenetv2_050 - 65.9lcnet_100/075/050 - 72.1 / 68.8 / 63.1semnasnet_075 - 73fbnetv3_b/d/g - 79.1 / 79.7 / 82.0
- TinyNet models added by rsomani95
- LCNet added via MobileNetV3 architecture
