Upload folder using huggingface_hub

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 768,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": true,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": false,
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,95 @@
+---
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- mathematics
+- scientific-papers
+- retrieval
+- matryoshka
+base_model: allenai/specter2_base
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+language: en
+license: apache-2.0
+---
+
+# math-embed
+
+A 768-dimensional embedding model fine-tuned for mathematical document retrieval, with a focus on **combinatorics** and related areas (representation theory, symmetric functions, algebraic combinatorics). Built on [SPECTER2](https://huggingface.co/allenai/specter2_base) and trained using knowledge-graph-guided contrastive learning.
+
+## Performance
+
+Benchmarked on mathematical paper retrieval (108 queries, 4,794 paper chunks):
+
+| Model | MRR | NDCG@10 |
+|-------|-----|---------|
+| **math-embed (this model)** | **0.816** | **0.736** |
+| OpenAI text-embedding-3-small | 0.461 | 0.324 |
+| SPECTER2 (proximity adapter) | 0.360 | 0.225 |
+| SciNCL | 0.306 | 0.205 |
+
+## Usage
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("RobBobin/math-embed")
+
+# Embed queries and documents
+queries = ["Kostka polynomials", "representation theory of symmetric groups"]
+docs = ["We study the combinatorial properties of Kostka numbers..."]
+
+query_embs = model.encode(queries)
+doc_embs = model.encode(docs)
+```
+
+### Matryoshka dimensions
+
+Trained with Matryoshka Representation Learning — you can truncate embeddings to smaller dimensions (512, 256, 128) with graceful degradation:
+
+```python
+# Use 256-dim embeddings for faster retrieval
+embs = model.encode(texts)
+embs_256 = embs[:, :256]
+```
+
+## Training
+
+### Method
+- **Loss**: MultipleNegativesRankingLoss + MatryoshkaLoss
+- **Training data**: 22,609 (anchor, positive) pairs generated from a knowledge graph of mathematical concepts
+  - **Direct pairs**: concept name/description → chunks from that concept's source papers
+  - **Edge pairs**: cross-concept pairs from knowledge graph edges (e.g., "generalizes", "extends")
+- **Base model**: `allenai/specter2_base` (SciBERT pre-trained on 6M citation triplets)
+
+### Configuration
+- Epochs: 3
+- Batch size: 8 (effective 32 with gradient accumulation)
+- Learning rate: 2e-5
+- Max sequence length: 256 tokens
+- Matryoshka dims: [768, 512, 256, 128]
+
+### Model lineage
+```
+BERT (Google, 110M params)
+  └─ SciBERT (Allen AI, retrained on scientific papers)
+      └─ SPECTER2 base (Allen AI, + 6M citation triplets)
+          └─ math-embed (this model, + KG-derived concept-chunk pairs)
+```
+
+## Approach
+
+The knowledge graph was constructed by an LLM (GPT-4o-mini) from 75 mathematical research papers, identifying 559 concepts and 486 relationships. This graph provides structured ground truth: each concept maps to specific papers, and those papers' chunks serve as positive training examples.
+
+This is a form of **knowledge distillation** — a large language model's understanding of mathematical relationships is distilled into a small, fast embedding model suitable for retrieval.
+
+## Limitations
+
+- Trained specifically on combinatorics papers (symmetric functions, representation theory, partition identities, algebraic combinatorics)
+- May not generalize well to other areas of mathematics or other scientific domains without additional fine-tuning
+- 256-token context window (standard for BERT-based models)
+
+## Citation
+
+See the accompanying paper: *Knowledge-Graph-Guided Fine-Tuning of Embedding Models for Mathematical Document Retrieval*

config.json

@@ -0,0 +1,30 @@
+{
+  "adapters": {
+    "adapters": {},
+    "config_map": {},
+    "fusion_config_map": {},
+    "fusions": {}
+  },
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 31090
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "SentenceTransformer",
+  "__version__": {
+    "sentence_transformers": "5.2.3",
+    "transformers": "4.51.3",
+    "pytorch": "2.9.1"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f49fba1357a7193677791c2ef52d8b989d5ce0f438dd8b7523b3790da99e6d8
+size 439696224

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 256,
+    "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "104": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}