reaperdoesntknow
/

MoA-100M

@@ -1,199 +1,252 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
 ## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+language:
+- en
+license: apache-2.0
 library_name: transformers
+pipeline_tag: text-generation
+tags:
+- mixture-of-attentions
+- distance-attention
+- metric-attention
+- mqa
+- hyperffn
+- router-gating
 ---
+# MoAMetricLM-185M — Mixture of Attentions (MoA)
+**A geometry-aware Transformer with a mixture of attention mechanisms and metric-based routing.**
+**Parameters:** ~185M | **Type:** Causal LM (decoder-only) | **KV cache:** not yet implemented
+---
+## Model Index
+- **Model ID:** `your-hf-username/MoAMetricLM-185M`
+- **Task:** text generation (`text-generation`)
+- **Library:** 🤗 Transformers
+- **License:** Apache-2.0 (change here & add LICENSE file if different)
+- **Datasets (examples used in dev runs):**
+  - `WeMake/Intelligent-Content-Understanding` (+ another 250k-token dataset)
+---
+## Overview
+**MoA** replaces standard dot-product attention with **metric-based attention** and blends multiple attentional biases using a **token-wise router** and **feature/router gates**.
+**Heads per block**
+- **LocalConvHead** — depthwise separable 1D conv (local context).
+- **Metric Multi-Head Attention (MetricMHAttention)** — attention via negative distances in learned head subspaces (L2 / cosine / diagonal-Mahalanobis), with per-head **origin** \(o_h\) and **radius** \(r_h\) enabling **ball pruning**.
+- **Metric MQA** — multi-query attention with shared K/V in the same metric space (efficiency).
+- **ChannelMixHead** — per-token MLP for channel interactions.
+**FFN**
+- **HyperFFN** (multi-branch): SwiGLU MLP path, separable-conv path, and low-rank path, combined via a token-wise branch router and optional feature gates. LayerScale + DropPath for stability.
+**Regularization (optional)**
+- **Triangle-inequality (TI) penalty** on sampled triples to encourage true-metric behavior.
+**Design goals:** geometric consistency, diverse inductive biases, structured efficiency, and full HF compatibility.
+---
+## What’s different from a standard Transformer?
+- **Distance-based attention (softmin over distances)** instead of dot product:
+  \[
+  \text{attn}(i,j)\ \propto\ \exp\!\big(-\alpha_h\ \|q_i-k_j\|^2\big)
+  \]
+  with per-head sharpness \(\alpha_h\). Cosine / diag-Mahalanobis variants supported.
+- **Per-head origins & radii** define balls for principled sparsity (**ball pruning**).
+- **Mixture of attentions** (conv / metric MHA / metric MQA / channel MLP) blended by a **token-wise router**, with **feature gates** (FiLM-like) and **router-bias gates** for up/down control.
+- **Up/Down projections** (SwiGLU-style) inside heads to expand/contract the value stream.
+- **HyperFFN** provides non-lazy capacity with token-wise branch routing.
+---
+## Intended Use & Limitations
+**Intended use:** research on geometry-aware attention, structured sparsity, and mixtures of attentional biases; small-scale experimentation and ablations.
+**Limitations:**
+- Dev runs used small token budgets; this is **not** a general-purpose LM.
+- **No KV cache** yet → generation cost scales with context length.
+- No alignment/safety tuning; outputs may be biased or inaccurate.
+**Out-of-scope:** high-stakes applications (medical/legal/etc.) without further training, evaluation, and safeguards.
+---
 ## Training Details
+**Hardware:** CPU (Intel; no CUDA)
+**Precision:** FP32
+### Latest run (v0.2)
+- **Tokens:** ~500,000 (two datasets, ~250k each)
+- **Wall-time:** ~20 minutes (~**417 toks/s** overall)
+- **Tokenizer:** GPT-2 (`gpt2`)
+- **Learning rate:** **5e-4** (AdamW)
+- **Batch / Seq:** batch_size=4, sequence length ≤512
+- **Final train loss:** **≈ 0.30**
+### Prior run (v0.1)
+- **Tokens:** ~196k
+- **Wall-time:** ~14 minutes
+- **Final avg loss:** ≈ 0.417 (min batch ≈ 0.193)
+**Stability aids:** safe softmax (subtract max), PreNorm, LayerScale (≈1e-4), DropPath (optional), label masking (`-100` on padding).
+---
+## Configuration (example)
+```json
+{
+  "model_type": "moa_metric",
+  "vocab_size": 50257,
+  "dim": 768,
+  "num_layers": 12,
+  "attn_heads": 8,
+  "mqa_q_heads": 8,
+  "mixer_hidden": 3072,
+  "ffn_hidden": 3072,
+  "metric": "l2",                     // "l2" | "cosine" | "maha_diag"
+  "alpha_init": 1.0,
+  "learn_alpha": true,
+  "use_balls": true,
+  "radius_init": 3.0,
+  "learn_radius": true,
+  "origin_init_scale": 0.0,
+  "maha_init": 1.0,
+  "ti_reg_weight": 0.0,
+  "ti_reg_samples": 0,
+  "router_hidden": 128,
+  "router_dropout": 0.1,
+  "router_temperature": 1.0,
+  "attn_drop": 0.1,
+  "proj_drop": 0.1,
+  "drop_path": 0.0,
+  "max_position_embeddings": 2048,
+  "pad_token_id": 50256,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256
+}
+If you use gpt2 tokenizer, set pad_token = eos_token and ensure vocab_size/eos/pad match the tokenizer.
+⸻
+Usage
+Inference
+from transformers import AutoTokenizer, AutoModelForCausalLM
+model_id = "your-hf-username/MoAMetricLM-185M"
+tok = AutoTokenizer.from_pretrained(model_id)
+if tok.pad_token is None:
+    tok.pad_token = tok.eos_token
+model = AutoModelForCausalLM.from_pretrained(model_id)
+prompt = "Explain metric-based attention in simple terms:"
+inputs = tok(prompt, return_tensors="pt")
+gen = model.generate(**inputs, max_new_tokens=128, do_sample=False)
+print(tok.decode(gen[0], skip_special_tokens=True))
+KV cache: not yet implemented; generation recomputes full context at each step.
+Training (custom loop sketch)
+from transformers import AutoTokenizer, DataCollatorForLanguageModeling
+from torch.utils.data import DataLoader
+import torch, torch.nn.functional as F
+tok = AutoTokenizer.from_pretrained("gpt2")
+if tok.pad_token is None:
+    tok.pad_token = tok.eos_token
+def collate(examples):
+    batch = tok([e["text"] for e in examples], padding="max_length",
+                truncation=True, max_length=512, return_tensors="pt")
+    labels = batch["input_ids"].clone()
+    labels[batch["attention_mask"] == 0] = -100
+    batch["labels"] = labels
+    return batch
+# dataset = ... (load HF dataset with a 'text' field)
+# loader = DataLoader(dataset, batch_size=4, shuffle=True, collate_fn=collate)
+# model = AutoModelForCausalLM.from_pretrained(model_id)  # or initialize config & model
+# optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, betas=(0.9,0.95), weight_decay=0.01)
+# for batch in loader:
+#     out = model(**batch)
+#     out.loss.backward()
+#     torch.nn.utils.clip_grad_norm_(model.parameters(), 1.2)
+#     optimizer.step(); optimizer.zero_grad()
+⸻
+Evaluation
+For meaningful comparisons, run:
+	•	Validation perplexity on a held-out split.
+	•	Ablations at fixed token budgets:
+	•	L2 vs cosine vs diag-Mahalanobis
+	•	With vs without ball pruning
+	•	With vs without HyperFFN branch router/gates
+	•	With vs without TI regularizer
+Please share results via Issues/PRs.
+⸻
+Efficiency Notes
+	•	Ball pruning: masks keys outside per-head radius → structured sparsity.
+	•	MQA: shared K/V reduce projection cost while retaining diversity via multi-query heads.
+	•	HyperFFN: token-wise branch router (optional top-k) to avoid paying for all branches equally.
+	•	CPU tips: set OMP_NUM_THREADS/MKL_NUM_THREADS to core count; use pad_token = eos_token.
+Roadmap: metric-aware KV cache for long contexts; kernelized distance approximations (e.g., RFF) for sub-quadratic regimes; quantization & mixed precision exploration.
+⸻
+Safety, Bias & Risks
+	•	May produce biased, offensive, or factually incorrect outputs.
+	•	No safety/alignment training included.
+	•	Do not deploy in high-stakes contexts without additional safeguards.
+⸻
+License
+Apache-2.0 (update if different).
+⸻
+Citation
+@misc{moametriclm185m,
+  title  = {MoAMetricLM-185M: A Geometry-Aware Mixture-of-Attentions Language Model},
+  author = {Colca, Roy Shawn and collaborators},
+  year   = {2025},
+  url    = {https://huggingface.co/your-hf-username/MoAMetricLM-185M}
+}
+⸻
+Changelog
+	•	v0.2 (2025-09-20) — 500k-token CPU run, GPT-2 tokenizer, LR=5e-4, final loss ≈ 0.30.
+	•	v0.1 (2025-09-20) — initial public release: metric heads, MQA, ball pruning, HyperFFN, router & gates; HF-compatible; no KV cache.
+⸻
+Maintainers
+	•	Author: reaper (Convergent Intelligence LLC)
+	•	Contact: add preferred contact
+	•	Issues: HF model hub issues tab