Upload Standard InfoNCE retrieval model

README.md

@@ -0,0 +1,104 @@
+---
+license: apache-2.0
+base_model: google-bert/bert-base-uncased
+tags:
+- retrieval
+- information-retrieval
+- sentence-transformers
+- bert
+- msmarco
+- squad
+pipeline_tag: feature-extraction
+---
+
+# nickcdryan/bitter-retrieval-standard-infonce-bert
+
+This is a retrieval model fine-tuned using **Standard InfoNCE** on MS MARCO dataset with additional validation on SQuAD.
+
+## Model Details
+
+- **Base Model**: google-bert/bert-base-uncased
+- **Training Method**: Standard InfoNCE
+- **Training Data**: MS MARCO soft-labeled dataset
+- **Validation Data**: SQuAD v2 + MS MARCO
+- **Framework**: PyTorch + Transformers
+
+## Training Details
+
+This model was trained using the bitter-retrieval framework with:
+
+- **Training Method**: `Standard InfoNCE`
+- **Encoder**: BERT-base-uncased
+- **Max Sequence Length**: 512 tokens
+- **Batch Size**: 32
+- **Epochs**: 2
+- **Learning Rate**: 2e-5
+- **Temperature**: 0.02
+
+## Usage
+
+```python
+from transformers import AutoModel, AutoTokenizer
+import torch
+import torch.nn.functional as F
+
+# Load model and tokenizer
+model = AutoModel.from_pretrained("nickcdryan/bitter-retrieval-standard-infonce-bert")
+tokenizer = AutoTokenizer.from_pretrained("nickcdryan/bitter-retrieval-standard-infonce-bert")
+
+def encode_text(text, prefix=""):
+    '''Encode text with optional prefix'''
+    full_text = f"{prefix}{text}" if prefix else text
+    inputs = tokenizer(full_text, return_tensors="pt", padding=True, truncation=True, max_length=512)
+    
+    with torch.no_grad():
+        outputs = model(**inputs)
+        # Mean pooling
+        attention_mask = inputs['attention_mask']
+        token_embeddings = outputs.last_hidden_state
+        masked_embeddings = token_embeddings * attention_mask.unsqueeze(-1)
+        sum_embeddings = masked_embeddings.sum(dim=1)
+        count_tokens = attention_mask.sum(dim=1, keepdim=True)
+        embeddings = sum_embeddings / count_tokens
+        # L2 normalize
+        embeddings = F.normalize(embeddings, dim=-1)
+    
+    return embeddings
+
+# Example usage
+query = "What is machine learning?"
+passage = "Machine learning is a subset of artificial intelligence..."
+
+# Encode with prefixes (recommended)
+query_emb = encode_text(query, "query: ")
+passage_emb = encode_text(passage, "passage: ")
+
+# Compute similarity
+similarity = torch.cosine_similarity(query_emb, passage_emb)
+print(f"Similarity: {similarity.item():.4f}")
+```
+
+## Evaluation Metrics
+
+The model was evaluated on both SQuAD and MS MARCO datasets with the following metrics:
+- **Retrieval Accuracy**: How often the correct passage is retrieved
+- **F1 Score**: Token-level F1 between generated and reference answers
+- **Exact Match**: Exact match between generated and reference answers
+- **LLM Judge**: Semantic similarity judged by Gemini-2.0-flash
+
+## Training Framework
+
+This model was trained using the [bitter-retrieval](https://github.com/yourusername/bitter-retrieval) framework, which implements various contrastive learning methods for retrieval tasks.
+
+## Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{bitter-retrieval-standard infonce,
+  title={Bitter Retrieval: Standard InfoNCE Fine-tuned BERT for Information Retrieval},
+  author={Your Name},
+  year={2024},
+  howpublished={\url{https://huggingface.co/nickcdryan/bitter-retrieval-standard-infonce-bert}}
+}
+```

config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.53.2",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0fc83491dbe1b924e899d6f2d62783ede2a7762cb2a7b479f9b97ec8c9988190
+size 437951328