j0no12/NeroT-86M-Exp

NeroT-86M

Custom encoder-decoder chat-title model trained from scratch on Apple M5 Max (MLX). Task: first user message → short chat title (TitleBench framing).

Results (TitleBench FULL 21,448 rows, official scorer, greedy min_len=3 no_repeat=1)

metric	value
ROUGE-1	0.152
ROUGE-2	0.025
ROUGE-L	0.151
chrF	0.082
BLEU-4	0.284
EM%
Repet%

Params: ~85.6M (tied embeddings). Architecture: deep bidirectional encoder (20L) + shallow AR decoder (6L), d=512, GQA 8q/2kv, RoPE, SwiGLU, RMSNorm.

Training recipe

1. Interleaved pretrain: MASS-style span corruption (15% mask, single 45% of batches).
2. Titles-only finetune (lr 8e-5 cosine).

Data provenance (deduplicated & filtered, seed 42)

• j0no12/chat-titles-unified (real chat titles): 223853 train pairs
• BananaMind-Chat-Title-200K reconstructed locally from gated lmsys-chat-1m by ordinal id: 127335 train pairs
• Self-distilled titles from BananaMind/BananaMind-Title-1.0 (teacher, ROUGE-1 0.471) on 248132 fresh lmsys first-user-messages
• Span-corruption corpus: lmsys-chat-1m conversations (~1M docs)

Leakage control: all 21,448 TitleBench inputs (incl. gated reconstructions) excluded by normalized-key matching before encoding; exact-pair dedup across sources; input-key priority dedup (real > synthetic).

Usage

modeling_titan_mlx.py is the exact MLX implementation used in training (custom Titan architecture; loads model.safetensors + config.json). Tokenizer: tokenizer.json (BPE, vocab 16385; pad=0, eos=1, bos=3; mask=16384). Decode: greedy, min_len=3, no_repeat_ngram=1, max 32 tokens.

Attribution

Built on data derived from lmsys/lmsys-chat-1m (gated, used locally under its terms; only ID-derived labels and self-generated text are redistributed). Teacher: BananaMind/BananaMind-Title-1.0 (LFM2.5-350M base by LiquidAI, LFM Open License v1.0).