| | --- |
| | license: mit |
| | datasets: |
| | - pfb30/multi_woz_v22 |
| | language: |
| | - en |
| | pipeline_tag: text-generation |
| | library_name: transformers |
| | tags: |
| | - Sam-2 |
| | - text-generation |
| | --- |
| | |
| | # 🧠 Model Card: Sam‑2.0 |
| |
|
| | ## 📌 Model Overview |
| | **Sam‑2.0** is a minimal, modular, decoder‑only Transformer architecture designed for chat‑style reasoning tasks. |
| | It emphasizes reproducibility, ablation‑friendly design, and clean benchmarking across input modalities. |
| |
|
| | - **Architecture**: Decoder‑only Transformer with RMSNorm, SwiGLU feed‑forward, and causal masking |
| | - **Training Objective**: Causal language modeling (CLM) with role‑based label masking |
| | - **Checkpoint**: `sam2-epoch35.safetensors` |
| | - **Final Train Loss**: 1.04 |
| | - **Validation Loss**: Not tracked in this run |
| | - **Training Duration**: ~6272 s over 35 epochs |
| | - **Framework**: PyTorch + Hugging Face Transformers (custom model class) |
| |
|
| | ## 🧱 Model Architecture |
| | | Component | Description | |
| | |-------------------|-----------------------------------------------------------------------------| |
| | | Backbone | Decoder‑only Transformer stack | |
| | | Normalization | RMSNorm | |
| | | Attention | Multi‑head self‑attention (causal) | |
| | | Feed‑Forward | SwiGLU activation with dropout | |
| | | Positional Bias | Learned absolute positions (no RoPE in this minimal variant) | |
| | | Head | Tied‑embedding LM head | |
| | | Checkpoint Format | `safetensors` with metadata for reproducibility | |
| |
|
| | ## 🧪 Training Details |
| | - **Dataset**: [pfb30/multi_woz_v22](https://huggingface.co/datasets/pfb30/multi_woz_v22) |
| | - **Batch Size**: 8 |
| | - **Optimizer**: AdamW |
| | - **Learning Rate**: 2 × 10⁻⁴ (constant in this run) |
| | - **Loss Function**: Cross‑entropy over assistant tokens only |
| | - **Hardware**: Kaggle GPU runtime |
| | - **Logging**: Step‑wise loss tracking, no validation during training |
| |
|
| | ## 📊 Evaluation |
| | | Metric | Value | Notes | |
| | |------------------|-------------|---------------------------------------| |
| | | Final Train Loss | 1.04 | Achieved at Epoch 35/35 | |
| | | Validation Loss | — | Not tracked in this run | |
| | | Inference Speed | Fast | Lightweight architecture | |
| | | Generalisation | TBD | To be compared against Sam‑2.5 | |
| |
|
| | ## 🔧 Intended Use |
| | - **Research**: Benchmarking modular architectures and ablation studies |
| | - **Education**: Reasoning scaffolds and logic quizzes |
| | - **Deployment**: Lightweight agents for chat and dialogue modeling |
| |
|
| | ## 🚫 Limitations |
| | - No validation tracking — generalisation must be inferred via external harnesses |
| | - Trained on MultiWOZ v2.2 only — may not generalize to other domains without fine‑tuning |
| | - Minimal architecture — no RoPE/MQA in this variant |
| |
|
| | ## 📁 Files |
| | - `sam2-epoch35.safetensors` — final checkpoint |
| | - `config.json` — architecture and training config |
| | - `tokenizer.json` — tokenizer with special tokens |
| | - `README.md` — training logs and setup instructions |
| |
|
| | ## 🧩 How to Load |
| | ```python |
| | from transformers import AutoTokenizer |
| | import torch |
| | from sam2 import Sam2, Sam2Config # your custom model class |
| | |
| | tok = AutoTokenizer.from_pretrained("Smilyai-labs/Sam-2.0") |
| | cfg = Sam2Config(**json.load(open("config.json"))) |
| | model = Sam2(cfg) |
| | state = torch.load("sam2-epoch35.safetensors", map_location="cpu") |
| | model.load_state_dict(state) |
| | model.eval() |
| | |
| | prompt = "<|user|> Hello! <|eot|>\n<|assistant|>" |
| | ids = tok.encode(prompt, return_tensors="pt") |
| | with torch.no_grad(): |
| | for _ in range(50): |
| | logits = model(ids) |
| | next_id = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True) |
| | ids = torch.cat([ids, next_id], dim=1) |
| | if next_id.item() == tok.eos_token_id: |
| | break |
| | |
| | print(tok.decode(ids[0])) |
| | |
