sep19_round1: 15,894-example dataset (8 buckets incl. general_capability), early-stopped end of epoch 1 -- 86.2% action-correct / 95% tool+args on held-out eval, negative_statement flagged as 0% (real weakness)
bef9487 verified | language: | |
| - hi | |
| - en | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| tags: | |
| - indic | |
| - small-language-model | |
| - qwen3 | |
| - tool-calling | |
| - function-calling | |
| - banking | |
| # NextToken-model-2 — SAM (Small Action Model) | |
| A 272.7M-parameter Hindi-English model fine-tuned into a **small action | |
| model** for banking/NBFC customer service: given a user utterance and a | |
| system prompt listing the available tools, it decides whether one applies | |
| and extracts the arguments. | |
| This checkpoint (`sep19_round1`, end of epoch 1) replaces `sep18_round2`. | |
| **Updated in place** — this repository tracks the current best checkpoint; | |
| check back for updates. | |
| ## What changed vs. `sep18_round2` | |
| Same 5-tool + no-tool scope, but a substantially larger and more | |
| categorized training set: **15,894 examples** (14,196 train / 1,698 val), | |
| organized into 8 explicit buckets — 5 tools, `negative` (with 4 | |
| sub-categories: chitchat, definitional, info, statement), `no_tool`, and a | |
| **`general_capability` bucket** (non-banking text: continuation, | |
| summarization, lists, creative writing) specifically to keep the model | |
| useful outside the banking domain and reduce over-firing a tool on | |
| unrelated prompts. | |
| **Early-stopped again, same lesson as before**: trained to epoch 1 (this | |
| checkpoint), then epoch 2 showed the same overfitting signature already | |
| seen in this project — val loss rose (1.5426 → 1.6514) while train loss | |
| kept falling — so training was stopped before epoch 3. The published | |
| weights are the end-of-epoch-1 checkpoint, not a later one. | |
| ## Verified accuracy — held-out 80-example eval, by category | |
| This is the training team's own held-out evaluation (`eval_epoch1_results.json`, | |
| never trained on), broken down here rather than reported as one aggregate | |
| number: | |
| | Bucket | n | JSON valid | Action correct | Tool correct | Args exact | | |
| |---|---|---|---|---|---| | |
| | get_loan_balance | 8 | 100% | 100% | 100% | 100% | | |
| | get_next_due_date | 8 | 100% | 100% | 100% | 100% | | |
| | get_payment_status | 8 | 100% | 100% | 75% | 75% | | |
| | get_recent_transactions | 8 | 100% | 100% | 100% | 100% | | |
| | get_transaction_details | 8 | 100% | 100% | 100% | 100% | | |
| | general_capability | 8 | 100% | 100% | — | — | | |
| | negative_definitional | 8 | 100% | 100% | — | — | | |
| | negative_info | 8 | 100% | 100% | — | — | | |
| | **negative_chitchat** | 8 | 100% | **62.5%** | — | — | | |
| | **negative_statement** | 8 | 100% | **0%** | — | — | | |
| | **Overall** | **80** | **100%** | **86.2%** | **95.0%** (of 40 tool-target rows) | **95.0%** | | |
| **`negative_statement` is a real, systematic failure, not noise**: all 8 | |
| examples in this bucket are past-tense/statement framings (e.g. "just | |
| checking in about my home loan"), and the model fired the identical | |
| `get_loan_balance`/`get_payment_status`-style tool call on every single | |
| one instead of giving a natural-language acknowledgment — the same | |
| "inconsistent negative-suppression for past-tense/third-person phrasing" | |
| limitation flagged on the previous checkpoint's card, now precisely | |
| quantified rather than described qualitatively. | |
| Independently re-verified here (not just taking the training team's | |
| numbers on faith): re-ran two real training examples (one `get_loan_balance` | |
| positive, one `negative_info`) through the converted HF checkpoint — | |
| both reproduced their exact target string. | |
| ## Architecture | |
| Same base architecture as `somasekhar-dev/NextToken-model-1`: | |
| | | | | |
| |---|---| | |
| | Parameters | 272.7M | | |
| | Attention | GQA — 16 query heads, 4 KV heads | | |
| | Normalization | RMSNorm + per-head QK-norm | | |
| | FFN | SwiGLU | | |
| | Context length | 2048 | | |
| | Tokenizer | [`sarvamai/sarvam-1`](https://huggingface.co/sarvamai/sarvam-1), vocab 68,096 | | |
| ## Usage | |
| Tokenize each message segment separately and concatenate — do not | |
| re-tokenize a merged prompt string (a real train/inference tokenization | |
| mismatch was found and fixed earlier in this project's history): | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| import torch | |
| model = AutoModelForCausalLM.from_pretrained("somasekhar-dev/NextToken-model-2") | |
| tokenizer = AutoTokenizer.from_pretrained("somasekhar-dev/NextToken-model-2") | |
| system_prompt = ( | |
| "You are a banking assistant for an Indian NBFC. Today's date is 2026-09-19 (Saturday).\n" | |
| "Given the user's message, decide whether one of the tools below applies. If it does, " | |
| "call it: reply with a short natural-language line if appropriate, followed by a JSON " | |
| "object {\"tool\": \"<name>\", \"arguments\": {...}} on its own line. Only use arguments " | |
| "that are stated in the conversation or can be computed from today's date -- never invent " | |
| "a value. If no tool applies, reply in natural language only, or with {\"tool\": null} if " | |
| "a bare JSON response is expected.\n\n" | |
| "Available tools:\n" | |
| '{"name": "get_loan_balance", "description": "Outstanding principal on a loan", ' | |
| '"parameters": {"type": "object", "properties": {"loan_type": {"type": "string", ' | |
| '"enum": ["home", "car", "personal", "gold", "business"]}}, "required": []}}' | |
| ) | |
| user_utterance = "मेरे लोन का कितना बकाया है, कृपया मुझे बताएं।" | |
| segments = [f"System: {system_prompt}\n", f"User: {user_utterance}\n", "Assistant: "] | |
| ids = [] | |
| for seg in segments: | |
| ids.extend(tokenizer.encode(seg, add_special_tokens=False)) | |
| ids_t = torch.tensor([ids]) | |
| out = model.generate(ids_t, max_new_tokens=60, do_sample=False, | |
| eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id) | |
| print(tokenizer.decode(out[0][ids_t.shape[1]:], skip_special_tokens=True)) | |
| # -> {"tool": "get_loan_balance", "arguments": {}} | |
| ``` | |
| ## Training | |
| Full fine-tune (not LoRA), 15,894 examples (`kashyap/sep19/raw/*.jsonl`), | |
| 8 buckets (5 tools, negative x4 sub-types, no_tool, general_capability), | |
| val loss 4.8757 -> 1.5426 by end of epoch 1. Stopped before epoch 3 on | |
| detecting overfitting (see above). | |
| ## Status and limitations | |
| - **`negative_statement` framing is unreliable — 0% on the held-out eval.** | |
| Do not rely on this checkpoint to correctly ignore past-tense/statement | |
| mentions of a loan or account; it is likely to fire a tool call anyway. | |
| - **`negative_chitchat` is inconsistent (62.5%)** — casual conversational | |
| openers sometimes still trigger a tool call. | |
| - Tool scope is deliberately narrow: 5 tools + no-tool. The full 41-tool | |
| schema (`model2_tools.json`) does not fit this model's 2048-token | |
| context at all (3,359 tokens alone) — this is an architectural | |
| constraint, not just a training choice. | |
| - Free-text argument values and multi-turn slot-filling are not covered | |
| by this round's eval — treat as unverified until tested. | |
| - **Any state-changing action should be confirmed before executing**, | |
| not trusted blindly from a single model output. | |
| - Conversion to this `Qwen3ForCausalLM` checkpoint was verified | |
| numerically (100% argmax agreement vs. the original training-format | |
| checkpoint on random inputs, a full save/reload round-trip, and two | |
| real training examples reproduced exactly). | |