Release DistillPath-KS16-HOpt0 distilled pathology encoder (50k steps)

LICENSE

@@ -0,0 +1,372 @@
+                  Kaiko Non-Commercial Public License
+                         Version 1, May 2024
+ 
+By exercising the Licensed Rights (defined below), You accept and agree to be
+bound by the terms and conditions of this Public License ("Public License")
+from Kaiko B.V., Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
+(the "Licensor"). To the extent this Public License may be interpreted as a
+contract, You are granted the Licensed Rights in consideration of Your
+acceptance of these terms and conditions, and the Licensor grants You such
+rights in consideration of benefits the Licensor receives from making the
+Licensed Material available under these terms and conditions.
+
+Section 1 -- Definitions.
+
+  a. Adapted Material means material subject to Copyright and Similar
+     Rights that is derived from or based upon the Licensed Material
+     and in which the Licensed Material is translated, altered,
+     arranged, transformed, or otherwise modified in a manner requiring
+     permission under the Copyright and Similar Rights held by the
+     Licensor. For purposes of this Public License, where the Licensed
+     Material is a musical work, performance, or sound recording,
+     Adapted Material is always produced where the Licensed Material is
+     synched in timed relation with a moving image.
+
+  b. Adapter's License means the license You apply to Your Copyright
+     and Similar Rights in Your contributions to Adapted Material in
+     accordance with the terms and conditions of this Public License.
+
+  c. Copyright and Similar Rights means copyright and/or similar rights
+     closely related to copyright including, without limitation,
+     performance, broadcast, sound recording, and Sui Generis Database
+     Rights, without regard to how the rights are labeled or
+     categorized. For purposes of this Public License, the rights
+     specified in Section 2(b)(1)-(2) are not Copyright and Similar
+     Rights.
+
+  d. Effective Technological Measures means those measures that, in the
+     absence of proper authority, may not be circumvented under laws
+     fulfilling obligations under Article 11 of the WIPO Copyright
+     Treaty adopted on December 20, 1996, and/or similar international
+     agreements.
+
+  e. Exceptions and Limitations means fair use, fair dealing, and/or
+     any other exception or limitation to Copyright and Similar Rights
+     that applies to Your use of the Licensed Material.
+
+  f. Licensed Material means the software, algorithm, source code,
+     object code, software, database, trained model, or other material
+     to which the Licensor applied this Public License.
+
+  g. Licensed Rights means the rights granted to You subject to the
+     terms and conditions of this Public License, which are limited to
+     all Copyright and Similar Rights that apply to Your use of the
+     Licensed Material and that the Licensor has authority to license.
+
+  h. Licensor means Kaiko B.V.
+
+  i. Non-Commercial means not intended for any commercial advantage or
+     monetary compensation. For purposes of this Public License,
+     Non-Commercial shall exclude any claims of intellectual property
+     rights such as patent applications.
+
+  j. Share means to provide material to the public by any means or
+     process that requires permission under the Licensed Rights, such
+     as reproduction, public display, public performance, distribution,
+     dissemination, communication, or importation, and to make material
+     available to the public including in ways that members of the
+     public may access the material from a place and at a time
+     individually chosen by them.
+
+  k. Sui Generis Database Rights means rights other than copyright
+     resulting from Directive 96/9/EC of the European Parliament and of
+     the Council of 11 March 1996 on the legal protection of databases,
+     as amended and/or succeeded, as well as other essentially
+     equivalent rights anywhere in the world.
+
+  l. You means the individual or entity exercising the Licensed Rights
+     under this Public License. Your has a corresponding meaning.
+
+Section 2 -- Scope.
+
+  a. License grant.
+
+       1. Subject to the terms and conditions of this Public License,
+          the Licensor hereby grants You a worldwide, royalty-free,
+          non-sublicensable, non-exclusive, irrevocable license to
+          exercise the Licensed Rights in the Licensed Material to:
+
+            a. reproduce and Share the Licensed Material, in whole or
+               in part, for Non-Commercial purposes only; and
+
+            b. produce, reproduce, and Share Adapted Material for
+               Non-Commercial purposes only.
+
+          In any case, and without limiting the foregoing, You may
+          not use the Licensed Material for any commercial purpose,
+          or in connection with any patent application (or similar
+          attempts to claim an intellectual property right).
+          The grant of rights expressly set forth in this Section
+          are the complete grant of rights to you in the Licensed
+          Material, and no other licenses are granted, whether by
+          waiver, estoppel, implication, equity or otherwise.
+          Licensor reserve all rights not expressly granted by this
+          License.
+
+       2. Exceptions and Limitations. For the avoidance of doubt, this
+          Public License is not intended to limit fair use, fair 
+          dealing, and/or any other exception or limitation to
+          Copyright and Similar Rights that applies to Your use of
+          the Licensed Material with applicable laws. 
+
+       3. Term. The term of this Public License is specified in Section
+          6(a).
+
+       4. Media and formats; technical modifications allowed. The
+          Licensor authorizes You to exercise the Licensed Rights in
+          all media and formats whether now known or hereafter created,
+          and to make technical modifications necessary to do so. The
+          Licensor waives and/or agrees not to assert any right or
+          authority to forbid You from making technical modifications
+          necessary to exercise the Licensed Rights, including
+          technical modifications necessary to circumvent Effective
+          Technological Measures. For purposes of this Public License,
+          simply making modifications authorized by this Section 2(a)
+          (4) never produces Adapted Material.
+
+       5. Downstream recipients.
+
+            a. Offer from the Licensor -- Licensed Material. Every
+               recipient of the Licensed Material automatically
+               receives an offer from the Licensor to exercise the
+               Licensed Rights under the terms and conditions of this
+               Public License.
+
+            b. Additional offer from the Licensor -- Adapted Material.
+               Every recipient of Adapted Material from You
+               automatically receives an offer from the Licensor to
+               exercise the Licensed Rights in the Adapted Material
+               under the conditions of the Adapter's License You apply.
+
+            c. No downstream restrictions. You may not offer or impose
+               any additional or different terms or conditions on, or
+               apply any Effective Technological Measures to, the
+               Licensed Material if doing so restricts exercise of the
+               Licensed Rights by any recipient of the Licensed
+               Material.
+
+       6. No endorsement. Nothing in this Public License constitutes or
+          may be construed as permission to assert or imply that You
+          are, or that Your use of the Licensed Material is, connected
+          with, or sponsored, endorsed, or granted official status by,
+          the Licensor or others designated to receive attribution as
+          provided in Section 3(a)(1)(A)(i).
+
+  b. Other rights.
+
+       1. Moral rights, such as the right of integrity, are not
+          licensed under this Public License, nor are publicity,
+          privacy, and/or other similar personality rights; however, to
+          the extent possible, the Licensor waives and/or agrees not to
+          assert any such rights held by the Licensor to the limited
+          extent necessary to allow You to exercise the Licensed
+          Rights, but not otherwise.
+
+       2. Patent and trademark rights are not licensed under this
+          Public License.
+
+       3. To the extent possible, the Licensor waives any right to
+          collect royalties from You for the exercise of the Licensed
+          Rights, whether directly or through a collecting society
+          under any voluntary or waivable statutory or compulsory
+          licensing scheme. In all other cases the Licensor expressly
+          reserves any right to collect such royalties, including when
+          the Licensed Material is used other than for Non-Commercial
+          purposes.
+
+Section 3 -- License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+  a. Attribution.
+
+       1. If You Share the Licensed Material (including in modified
+          form), You must:
+
+            a. retain the following if it is supplied by the Licensor
+               with the Licensed Material:
+
+                 i. identification of the creator(s) of the Licensed
+                    Material and any others designated to receive
+                    attribution, in any reasonable manner requested by
+                    the Licensor (including by pseudonym if
+                    designated);
+
+                ii. a copyright notice;
+
+               iii. a notice that refers to this Public License;
+
+                iv. a notice that refers to the disclaimer of
+                    warranties;
+
+                 v. a URI or hyperlink to the Licensed Material to the
+                    extent reasonably practicable;
+
+            b. indicate if You modified the Licensed Material and
+               retain an indication of any previous modifications; and
+
+            c. indicate the Licensed Material is licensed under this
+               Public License, and include the text of, or the URI or
+               hyperlink to, this Public License.
+
+       2. You may satisfy the conditions in Section 3(a)(1) in any
+          reasonable manner based on the medium, means, and context in
+          which You Share the Licensed Material. For example, it may be
+          reasonable to satisfy the conditions by providing a URI or
+          hyperlink to a resource that includes the required
+          information.
+
+       3. If requested by the Licensor, You must remove any of the
+          information required by Section 3(a)(1)(a) to the extent
+          reasonably practicable.
+
+  b. ShareAlike.
+
+     In addition to the conditions in Section 3(a), if You Share
+     Adapted Material You produce, the following conditions also apply.
+
+       1. The Adapter's License You apply must have the same terms and
+          conditions than the License, this version or later. In
+          particular, recipients of the Licensed Material, shall
+          not use it for any purposes other than Non-Commercial.
+
+       2. You must include the text of, or the URI or hyperlink to, the
+          Adapter's License You apply. You may satisfy this condition
+          in any reasonable manner based on the medium, means, and
+          context in which You Share Adapted Material.
+
+       3. You may not offer or impose any additional or different terms
+          or conditions on, or apply any Effective Technological
+          Measures to, Adapted Material that restrict exercise of the
+          rights granted under the Adapter's License You apply.
+
+Section 4 -- Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that
+apply to Your use of the Licensed Material:
+
+  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
+     to extract, reuse, reproduce, and Share all or a substantial
+     portion of the contents of the database for Non-Commercial purposes
+     only;
+
+  b. if You include all or a substantial portion of the database
+     contents in a database in which You have Sui Generis Database
+     Rights, then the database in which You have Sui Generis Database
+     Rights (but not its individual contents) is Adapted Material,
+     including for purposes of Section 3(b); and
+
+  c. You must comply with the conditions in Section 3(a) if You Share
+     all or a substantial portion of the contents of the database.
+
+For the avoidance of doubt, this Section 4 supplements and does not
+replace Your obligations under this Public License where the Licensed
+Rights include other Copyright and Similar Rights.
+
+Section 5 -- Disclaimer of Warranties and Limitation of Liability.
+ 
+  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
+     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
+     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
+     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
+     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
+     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
+     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
+     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
+     KNOWN OR DISCOVERABLE.
+ 
+  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
+     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
+     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
+     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
+     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
+     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
+     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
+     DAMAGES. 
+  c. The disclaimer of warranties and limitation of liability provided
+     above shall be interpreted in a manner that, to the extent
+     possible, most closely approximates an absolute disclaimer and
+     waiver of all liability.
+
+  d. Indemnification. You agree to indemnify, defend, and hold harmless
+     the Licensor, its affiliates, and their respective officers,
+     directors, employees, and agents from and against any and all
+     claims, liabilities, damages, losses, or expenses, including
+     reasonable attorneys' fees and costs, arising out of or in any way
+     connected with Your access to or use of the Licensed Material. 
+
+Section 6 -- Term and Termination.
+
+  a. This Public License applies for the term of the Copyright and
+     Similar Rights licensed here. However, if You fail to comply with
+     this Public License, then Your rights under this Public License
+     terminate automatically.
+
+  b. Where Your right to use the Licensed Material has terminated under
+     Section 6(a), it reinstates:
+
+       1. automatically as of the date the violation is cured, provided
+          it is cured within 30 days of Your discovery of the
+          violation; or
+
+       2. upon express reinstatement by the Licensor.
+
+     For the avoidance of doubt, this Section 6(b) does not affect any
+     right the Licensor may have to seek remedies for Your violations
+     of this Public License.
+
+  c. For the avoidance of doubt, the Licensor may also offer the
+     Licensed Material under separate terms or conditions or stop
+     distributing the Licensed Material at any time. Licensor may
+     modify the terms contained herein at any time without notice.
+     You are responsible for periodically reviewing this Public
+     License for any updates or changes, and continued use of the
+     Licensed Materials constitutes acceptance of any modifications.
+
+  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+     License.
+
+Section 7 -- Other Terms and Conditions.
+
+  a. The Licensor shall not be bound by any additional or different
+     terms or conditions communicated by You unless expressly agreed.
+
+  b. Any arrangements, understandings, or agreements regarding the
+     Licensed Material not stated herein are separate from and
+     independent of the terms and conditions of this Public License.
+
+Section 8 -- Interpretation.
+
+  a. For the avoidance of doubt, this Public License does not, and
+     shall not be interpreted to, reduce, limit, restrict, or impose
+     conditions on any use of the Licensed Material that could lawfully
+     be made without permission under this Public License.
+
+  b. To the extent possible, if any provision of this Public License is
+     deemed unenforceable, it shall be automatically reformed to the
+     minimum extent necessary to make it enforceable. If the provision
+     cannot be reformed, it shall be severed from this Public License
+     without affecting the enforceability of the remaining terms and
+     conditions.
+ 
+  c. No term or condition of this Public License will be waived and no
+     failure to comply consented to unless expressly agreed to by the
+     Licensor.
+
+  d. Nothing in this Public License constitutes or may be interpreted
+     as a limitation upon, or waiver of, any privileges and immunities
+     that apply to the Licensor or You, including from the legal
+     processes of any jurisdiction or authority.
+
+  e. This Public License shall be governed by and construed in
+     accordance with the laws of The Netherlands, without regard to its
+     conflict of law provisions. The parties consent to the exclusive
+     jurisdiction of the courts of, and venue and situs in
+     The Netherlands.
+
+  f. This Public License constitute the entire agreement between You
+     and the Licensor with respect to the Licensed Material, and 
+     supersede all prior or contemporaneous communications and 
+     proposals, whether oral or written.
+ 

README.md

@@ -0,0 +1,113 @@
+---
+license: other
+license_name: kaiko-non-commercial-public-license
+license_link: LICENSE
+library_name: timm
+pipeline_tag: image-feature-extraction
+tags:
+  - computational-pathology
+  - knowledge-distillation
+  - vision-transformer
+  - medical-imaging
+datasets:
+  - TCGA
+---
+
+# DistillPath-KS16-HOpt0
+
+A 22M ViT-S/16 pathology tile encoder distilled from [H-optimus-0](https://huggingface.co/bioptimus/H-optimus-0) (1.1B ViT-g/14) into the [kaiko ViT-S/16](https://huggingface.co/1aurent/vit_small_patch16_224.kaiko_ai_towards_large_pathology_fms) student using backbone-token distillation on 6,000 public TCGA slides.
+
+**Paper:** [DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance](https://github.com/RamonKaspar/DistillPath)
+Ramon Kaspar, Andrey Ignatov, Valentina Boeva. ETH Zurich.
+Published at the ECCV 2026 Workshop on Medical Foundation Models and Benchmarks (MedFM-Bench).
+
+## Model details
+
+| Property | Value |
+|---|---|
+| Architecture | ViT-S/16 (`vit_small_patch16_224` in timm) |
+| Parameters | 21.7M |
+| Feature dimension | 384 |
+| Input size | 224 x 224 |
+| Normalization | mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5) |
+| Student initialization | kaiko ViT-S/16 (pathology-pretrained) |
+| Teacher | H-optimus-0 (1.1B, ViT-g/14, Apache 2.0) |
+| Training data | 6,000 TCGA H&E whole-slide images, 32 cohorts |
+| Training steps | 50,000 (batch size 256, approx. 24-29 GPU-hours on 1x RTX 4090) |
+
+## Benchmark results
+
+| Benchmark | **DistillPath-KS16-HOpt0** | kaiko baseline | H-optimus-0 teacher |
+|---|---|---|---|
+| EVA mean (7 tasks) | **0.769** | 0.764 | 0.803 |
+| HEST mean (9 tasks) | **0.376** | 0.349 | 0.415 |
+| PLISM score | **0.480** | 0.307 | 0.480 |
+
+See the paper for per-task results and comparisons with all four DistillPath variants.
+
+## Usage
+
+Load directly from the Hub with timm:
+
+```python
+import timm
+
+model = timm.create_model(
+    "hf_hub:RamonK/DistillPath-KS16-HOpt0",
+    pretrained=True,
+    num_classes=0,
+)
+model.eval()
+```
+
+Or load manually:
+
+```python
+import timm
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+
+model = timm.create_model("vit_small_patch16_224", pretrained=False, num_classes=0)
+path = hf_hub_download("RamonK/DistillPath-KS16-HOpt0", "model.safetensors")
+state_dict = load_file(path)
+model.load_state_dict(state_dict, strict=True)
+model.eval()
+```
+
+This model uses normalization mean=(0.5, 0.5, 0.5) and std=(0.5, 0.5, 0.5), inherited from the kaiko student. Apply this normalization to input tiles before inference:
+
+```python
+from torchvision import transforms
+
+transform = transforms.Compose([
+    transforms.Resize(224),
+    transforms.CenterCrop(224),
+    transforms.ToTensor(),
+    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
+])
+```
+
+## Distillation recipe
+
+The recipe reads only the teacher's final class and patch tokens (no teacher pretraining heads required):
+- **Class-token loss:** cosine distance + RKD (relational knowledge distillation)
+- **Patch-token loss:** cosine distance after bicubic grid resizing (teacher 16x16 to student 14x14)
+- **Optimizer:** AdamW, lr=1e-4, weight decay 0.05, cosine decay, 500 warmup steps
+- **Projector:** DINO-style MLP (384 to 2048 to 2048 to 256 to d_t), discarded after training
+
+Full details in the paper and the [DistillPath repository](https://github.com/RamonKaspar/DistillPath).
+
+## License
+
+This model is released under the [Kaiko Non-Commercial Public License](LICENSE), inherited from the kaiko ViT-S/16 student weights. The teacher (H-optimus-0) is Apache 2.0.
+
+## Citation
+
+```bibtex
+@inproceedings{kaspar2026distillpath,
+  title     = {DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance},
+  author    = {Kaspar, Ramon and Ignatov, Andrey and Boeva, Valentina},
+  booktitle = {Medical Foundation Models and Benchmarks (MedFM-Bench), ECCV 2026},
+  year      = {2026}
+}
+```

config.json

@@ -0,0 +1,12 @@
+{
+  "architecture": "vit_small_patch16_224",
+  "num_classes": 0,
+  "num_features": 384,
+  "pretrained_cfg": {
+    "num_classes": 0,
+    "input_size": [3, 224, 224],
+    "mean": [0.5, 0.5, 0.5],
+    "std": [0.5, 0.5, 0.5],
+    "custom_load": false
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24f16b09b73b777e33dd2cb612127c120100631b3802ab2854423781b521aafa
+size 86676336