Push model using huggingface_hub.

README.md

@@ -1,69 +1,10 @@
 ---
-license: apache-2.0
-datasets:
-- andrewdalpino/SwissProt-Gene-Ontology
 tags:
-- esmc
+- model_hub_mixin
+- pytorch_model_hub_mixin
 ---
-# ProtHash
 
-A protein language model that outputs amino acid sequence embeddings for use in clustering, classification, locality-sensitive hashing, and more. Distilled from the [ESMC](https://www.evolutionaryscale.ai/blog/esm-cambrian) family of models with deep comprehension of protein structure, ProtHash produces contextual embeddings that align in vector space according to the sequences' atomic structure. Trained on the [SwissProt](https://huggingface.co/datasets/andrewdalpino/SwissProt-Gene-Ontology) dataset to mimic the activations of its ESMC teacher model, ProtHash embeddings have near perfect similarity to ESMC embeddings but at a greatly reduced computational cost.
-
-## Key Features
-
-- **Structurally-relevant embeddings**: Structurally similar proteins will show up nearby in the embedding space enabling downstream tasks such as clustering, classification, and locality-sensitive hashing based on atomic structure.
-
-- **Blazing fast and efficient**: ProtHash uses only 3% of ESMC's parameters to achieve near perfect cosine similarity between the two embedding spaces.
-
-- **Long context**: With a context window of 2048 amino acid tokens you can embed proteins with long sequences.
-
-## Pretrained Models
-
-| Name | Context Length | Embedding Dimensionality | Attention Heads (Q/KV) | Encoder Layers | Total Params |
-|---|---|---|---|---|---|
-| [andrewdalpino/ProtHash-384-Tiny](https://huggingface.co/andrewdalpino/ProtHash-384-Tiny) | 2048 | 384 | 16/4 | 4 | 7M |
-| [andrewdalpino/ProtHash-512-Tiny](https://huggingface.co/andrewdalpino/ProtHash-512-Tiny) | 2048 | 512 | 16/4 | 4 | 12M |
-
-## Pretrained Example
-
-First, you'll need the `prothash` and `esm` packages installed into your environment. We recommend using a virtual environment such as Python's `venv` module to prevent version conflicts with any system packages.
-
-```sh
-pip install prothash esm
-```
-
-Then, load the weights from HuggingFace Hub, tokenize a protein sequence, and pass it to the model. ProtHash adopts the ESM tokenizer as it's amino acids tokenization scheme. The output will be an embedding vector that can be used in downstream tasks such as comparing to other protein sequence embeddings, clustering, and near-duplicate detection.
-
-```python
-import torch
-
-from esm.tokenization import EsmSequenceTokenizer
-
-from prothash.model import ProtHash
-
-tokenizer = EsmSequenceTokenizer()
-
-model_name = "andrewdalpino/ProtHash-512-Tiny"
-
-model = ProtHash.from_pretrained(model_name)
-
-sequence = input("Enter a sequence: ")
-
-out = tokenizer(sequence, max_length=2048)
-
-tokens = out["input_ids"]
-
-x = torch.tensor(tokens, dtype=torch.int64).unsqueeze(0)
-
-y_embed = model.embed(x)
-
-print(y_embed)
-```
-
-## References
-
->- The UniProt Consortium, UniProt: the Universal Protein Knowledgebase in 2025, Nucleic Acids Research, 2025, 53, D609–D617.
->- T. Hayes, et al. Simulating 500 million years of evolution with a language model, 2024.
->- B. Zhang, et al. Root Mean Square Layer Normalization. 33rd Conference on Neural Information Processing Systems, NeurIPS 2019.
->- J. Ainslie, et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Google Research, 2023.
->- T. Kim, et al. Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation, 2021.
+This model has been pushed to the Hub using the [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) integration:
+- Code: [More Information Needed]
+- Paper: [More Information Needed]
+- Docs: [More Information Needed]

config.json

@@ -2,10 +2,11 @@
   "context_length": 2048,
   "dropout": 0.0,
   "embedding_dimensions": 384,
-  "hidden_ratio": 4,
+  "hidden_ratio": 2,
   "kv_heads": 4,
   "num_encoder_layers": 4,
   "padding_index": 1,
   "q_heads": 16,
+  "teacher_dimensions": 960,
   "vocabulary_size": 33
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a67262df3f4087a74a10ed20199663eb7ea4ccf5a9649ac29b765f09666c9739
-size 27984872
+oid sha256:f57d1cc2aaa32651f2cc8beac91d000851a2e82ddeb24af043f041ed7b4e467e
+size 20022320