Upload folder using huggingface_hub

README.md

@@ -0,0 +1,177 @@
+---
+license: mit
+language:
+- en
+library_name: transformers
+tags:
+- sentence-transformers
+- feature-extraction
+- text-embeddings
+- semantic-search
+- onnx
+- transformers.js
+- bert
+- knowledge-distillation
+datasets:
+- custom
+pipeline_tag: feature-extraction
+model-index:
+- name: typelevel-bert
+  results:
+  - task:
+      type: retrieval
+      name: Document Retrieval
+    dataset:
+      type: custom
+      name: FP-Doc Benchmark v1
+    metrics:
+    - type: ndcg_at_10
+      value: 0.853
+      name: NDCG@10
+    - type: mrr
+      value: 0.900
+      name: MRR
+    - type: recall_at_10
+      value: 0.967
+      name: Recall@10
+---
+
+# Typelevel-BERT
+
+A compact, browser-deployable text embedding model specialized for searching Typelevel/FP documentation. Distilled from [BAAI/bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) to achieve fast client-side inference.
+
+## Highlights
+
+- **93.3%** of teacher model quality (NDCG@10)
+- **30x smaller** than teacher (11M vs 335M parameters)
+- **10.7 MB** quantized ONNX model
+- **1.5ms** inference latency (CPU, seq_len=128)
+- Optimized for Cats, Cats Effect, FS2, http4s, Doobie, Circe documentation
+
+## Model Details
+
+| Property | Value |
+|----------|-------|
+| **Model Type** | BERT encoder (text embedding) |
+| **Architecture** | 4-layer transformer |
+| **Hidden Size** | 256 |
+| **Attention Heads** | 4 |
+| **Parameters** | 11.2M |
+| **Embedding Dimension** | 256 |
+| **Max Sequence Length** | 512 |
+| **Vocabulary** | bert-base-uncased (30,522 tokens) |
+| **Pooling** | Mean pooling |
+
+## Usage
+
+### Browser/Node.js (transformers.js)
+
+```javascript
+import { pipeline } from '@huggingface/transformers';
+
+// Load the model (downloads automatically)
+const extractor = await pipeline('feature-extraction', 'djspiewak/typelevel-bert', {
+  quantized: true,  // Use INT8 quantized model (10.7 MB)
+});
+
+// Generate embeddings
+const embedding = await extractor("How to sequence effects in cats-effect", {
+  pooling: 'mean',
+  normalize: true,
+});
+
+console.log(embedding.data);  // Float32Array(256)
+```
+
+### Python (ONNX Runtime)
+
+```python
+import onnxruntime as ort
+import numpy as np
+from transformers import AutoTokenizer
+from huggingface_hub import hf_hub_download
+
+# Download and load quantized model
+model_path = hf_hub_download("djspiewak/typelevel-bert", "onnx/model_quantized.onnx")
+tokenizer = AutoTokenizer.from_pretrained("djspiewak/typelevel-bert")
+session = ort.InferenceSession(model_path)
+
+# Tokenize input
+text = "Resource management and safe cleanup"
+inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True)
+
+# Run inference
+outputs = session.run(None, {
+    "input_ids": inputs["input_ids"].astype(np.int64),
+    "attention_mask": inputs["attention_mask"].astype(np.int64),
+})
+
+# Mean pooling
+hidden_states = outputs[0]  # (1, seq_len, 256)
+attention_mask = inputs["attention_mask"]
+mask_expanded = np.expand_dims(attention_mask, -1)
+sum_embeddings = np.sum(hidden_states * mask_expanded, axis=1)
+sum_mask = np.sum(mask_expanded, axis=1)
+embedding = sum_embeddings / sum_mask  # (1, 256)
+
+# L2 normalize
+embedding = embedding / np.linalg.norm(embedding, axis=1, keepdims=True)
+```
+
+## Performance
+
+| Metric | Typelevel-BERT | Teacher (BGE-large) | % of Teacher |
+|--------|----------------|---------------------|--------------|
+| NDCG@10 | 0.853 | 0.915 | 93.3% |
+| MRR | 0.900 | 0.963 | 93.5% |
+| Recall@10 | 96.7% | 96.7% | 100% |
+| Parameters | 11.2M | 335M | 3.3% |
+| Model Size | 10.7 MB | ~1.2 GB | 0.9% |
+| Latency (CPU) | 1.5ms | ~15ms | 10x faster |
+
+## Training
+
+- **Teacher Model**: BAAI/bge-large-en-v1.5 (335M parameters, 1024-dim embeddings)
+- **Training Data**: 30,598 text chunks from Typelevel ecosystem documentation
+- **Distillation Method**: Knowledge distillation with MSE + cosine similarity loss
+- **Hardware**: Apple M3 Max (MPS)
+
+## Intended Use
+
+This model is designed for:
+- **Semantic search** in functional programming documentation
+- **Document retrieval** for Typelevel ecosystem libraries (Cats, Cats Effect, FS2, http4s, Doobie, Circe)
+- **Browser-based inference** via transformers.js or ONNX Runtime Web
+- **Client-side embeddings** for privacy-preserving search applications
+
+## Limitations
+
+1. **Domain Specialization**: Optimized for FP documentation; may underperform on general text
+2. **English Only**: Trained exclusively on English documentation
+3. **Vocabulary**: Uses bert-base-uncased vocabulary; some FP-specific terms may be suboptimally tokenized
+
+## Files
+
+| File | Size | Description |
+|------|------|-------------|
+| `model.safetensors` | 42.6 MB | PyTorch weights |
+| `onnx/model.onnx` | 42.4 MB | Full precision ONNX |
+| `onnx/model_quantized.onnx` | 10.7 MB | INT8 quantized ONNX |
+| `config.json` | - | Model configuration |
+| `tokenizer.json` | - | Fast tokenizer |
+| `vocab.txt` | - | Vocabulary file |
+
+## Citation
+
+```bibtex
+@misc{typelevel-bert,
+  title={Typelevel-BERT: Distilled Text Embeddings for FP Documentation Search},
+  author={Daniel Spiewak},
+  year={2025},
+  url={https://huggingface.co/djspiewak/typelevel-bert}
+}
+```
+
+## License
+
+MIT

config.json

@@ -0,0 +1,19 @@
+{
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "type_vocab_size": 2,
+  "vocab_size": 30522
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62d83e918217f992d91973030d36865eb216388512066f99d4101edb27d7e7a2
+size 44690024

onnx/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:601798ea0a04bc7b3119e061c1a80e081b7b4347f2f064f1e232bc313c5d19cc
+size 44459423

onnx/model_quantized.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:341648c8f684aaf372d224ba83c34e17b709aa62309f1bfd79400a3b6ad7bc90
+size 11248141

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}