Backend: unembedding cosine emission + Devstral tokenizer fallback

Two changes that accumulated together in the working tree.

1. Unembedding cosine decomposition emitted per top-K alternative

For Step 8's "How this token won the vote" disclosure, the backend
now decomposes each alternative's logit into the geometric factors:
logit_v = cos(W_E[v], h_final) × ||W_E[v]|| × ||h_final||

Cosine similarity and the unembedding row norm (||W_E[v]||) are
appended to every entry in the logits payload — both in the
regular and outlier paths, in both the non-SSE and SSE generation
loops. Without this, the SSE endpoint (which the frontend uses)
returned no cosine data while the non-SSE endpoint did, leading to
a silent "cosine not available" state in the lens.

Logging on cosine prep upgraded to info-level on success and
warning-level on failure so future regressions are visible.

2. Tokenizer fallback for Devstral on tokenizers >= 0.20

Devstral-Small-2507 ships only tekken.json. AutoTokenizer with
newer tokenizers is stricter about HF-format file presence and
fails on the official repo. Fall back to the unsloth/Devstral-
Small-2507 mirror, which carries the same Tekken vocabulary in
standard HF format. Same vocab size, same token IDs, same special
tokens — only the on-disk file format differs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/model_service.py

@@ -677,8 +677,24 @@ class ModelManager:
                 attn_implementation="eager"
             ).to(self.device)
             
-            # Load tokenizer
-            self.tokenizer = AutoTokenizer.from_pretrained(self.model_name)
+            # Load tokenizer. Devstral's official repo ships only tekken.json,
+            # which AutoTokenizer can't consume on tokenizers ≥ 0.20. Fall back
+            # to the Unsloth mirror, which carries the same Tekken vocabulary
+            # in standard HF format. Same vocab size, same token IDs, same
+            # special tokens — only the on-disk file format differs.
+            try:
+                self.tokenizer = AutoTokenizer.from_pretrained(self.model_name)
+            except (OSError, EnvironmentError) as tok_err:
+                if self.model_id == "devstral-small":
+                    logger.warning(
+                        f"AutoTokenizer failed for {self.model_name} ({tok_err}); "
+                        f"falling back to unsloth/Devstral-Small-2507 (same Tekken vocab, HF format)."
+                    )
+                    self.tokenizer = AutoTokenizer.from_pretrained(
+                        "unsloth/Devstral-Small-2507"
+                    )
+                else:
+                    raise
             # Set pad_token if the tokenizer allows it (some like MistralCommonTokenizer don't)
             try:
                 self.tokenizer.pad_token = self.tokenizer.eos_token
@@ -2199,16 +2215,85 @@ async def analyze_research_attention(request: Dict[str, Any], authenticated: boo
                 top_n_display = 10  # Get top 10 alternatives for display
                 top_raw_logits, top_raw_indices = torch.topk(raw_logits, k=min(top_n_display, len(raw_logits)))
 
+                # ── Unembedding-cosine prep ──────────────────────────
+                # Each logit is W_E[v] · h_post_norm where h_post_norm
+                # is the post-final-norm residual at the last position.
+                # cos(W_E[v], h_post_norm) is the *direction-only*
+                # alignment — strips out magnitude so a developer can
+                # see whether a high-logit token won by alignment or
+                # by magnitude. Computed once per step (cheap; one
+                # final-norm call + d_model dot product per token).
+                cos_h_norm = None
+                W_E = None
+                try:
+                    final_hidden_pre_norm = outputs.hidden_states[-1][0, -1, :]
+                    if hasattr(manager.model, 'model') and hasattr(manager.model.model, 'norm'):
+                        h_post_norm = manager.model.model.norm(
+                            final_hidden_pre_norm.unsqueeze(0)
+                        )[0]
+                    elif hasattr(manager.model, 'transformer') and hasattr(manager.model.transformer, 'ln_f'):
+                        h_post_norm = manager.model.transformer.ln_f(
+                            final_hidden_pre_norm.unsqueeze(0)
+                        )[0]
+                    else:
+                        h_post_norm = final_hidden_pre_norm
+                    h_norm_val = float(torch.linalg.vector_norm(h_post_norm).item())
+                    if h_norm_val > 1e-9:
+                        cos_h_norm = h_post_norm / h_norm_val
+                        if hasattr(manager.model, 'lm_head'):
+                            W_E = manager.model.lm_head.weight
+                        else:
+                            logger.warning(
+                                "[cosine prep] manager.model has no lm_head — cosine emission skipped"
+                            )
+                    else:
+                        logger.warning(
+                            f"[cosine prep] h_post_norm magnitude too small: {h_norm_val}"
+                        )
+                    if cos_h_norm is not None and W_E is not None and step == 0:
+                        # One-time confirmation per generation that the
+                        # cosine path is live.
+                        logger.info(
+                            f"[cosine prep] enabled · h_post_norm shape {tuple(h_post_norm.shape)} · "
+                            f"W_E shape {tuple(W_E.shape)} · h_norm {h_norm_val:.3f}"
+                        )
+                except Exception as e:
+                    logger.warning(
+                        f"[cosine prep] failed at step {step}: {type(e).__name__}: {e}"
+                    )
+                    cos_h_norm = None
+                    W_E = None
+
+                def _cos_for(token_idx: int):
+                    """cosine(W_E[token_idx], h_post_norm) and ‖W_E row‖."""
+                    if cos_h_norm is None or W_E is None:
+                        return None, None
+                    try:
+                        row = W_E[token_idx]
+                        row_norm = float(torch.linalg.vector_norm(row).item())
+                        if row_norm < 1e-9:
+                            return None, row_norm
+                        cos = float(((row / row_norm) @ cos_h_norm).item())
+                        return cos, row_norm
+                    except Exception:
+                        return None, None
+
                 # Build raw logits entries (before temperature)
                 logits_entries = []
                 for rank, (logit_val, idx) in enumerate(zip(top_raw_logits.tolist(), top_raw_indices.tolist())):
                     token_text = manager.tokenizer.decode([idx], skip_special_tokens=False)
-                    logits_entries.append({
+                    cos, row_norm = _cos_for(idx)
+                    entry = {
                         "token": token_text,
                         "token_id": idx,
                         "logit": logit_val,
-                        "rank": rank + 1
-                    })
+                        "rank": rank + 1,
+                    }
+                    if cos is not None:
+                        entry["cosine_sim"] = round(cos, 4)
+                    if row_norm is not None:
+                        entry["unemb_row_norm"] = round(row_norm, 4)
+                    logits_entries.append(entry)
 
                 # Greedy token (argmax of raw logits, before any sampling)
                 greedy_token_id = torch.argmax(raw_logits).item()
@@ -2332,13 +2417,21 @@ async def analyze_research_attention(request: Dict[str, Any], authenticated: boo
                     })
                     # Also add to logits if not present
                     if next_token_id not in [e["token_id"] for e in logits_entries]:
-                        logits_entries.append({
+                        cos_outlier, row_norm_outlier = _cos_for(next_token_id)
+                        outlier_entry = {
                             "token": next_token_text,
                             "token_id": next_token_id,
                             "logit": selected_logit,
                             "rank": selected_rank,
-                            "is_selected_outlier": True
-                        })
+                            "is_selected_outlier": True,
+                        }
+                        if cos_outlier is not None:
+                            outlier_entry["cosine_sim"] = round(cos_outlier, 4)
+                        if row_norm_outlier is not None:
+                            outlier_entry["unemb_row_norm"] = round(
+                                row_norm_outlier, 4
+                            )
+                        logits_entries.append(outlier_entry)
 
                 # Build sampling metadata. `is_filtered` and `eligible_count`
                 # let the frontend decide whether to show one or two
@@ -3241,15 +3334,81 @@ async def analyze_research_attention_stream(request: Dict[str, Any], authenticat
                         else:
                             return manager.tokenizer.decode([tid], skip_special_tokens=False)
 
+                    # ── Unembedding-cosine prep (SSE path) ──────────
+                    # Mirrors the non-SSE path. Each logit is
+                    # W_E[v] · h_post_norm; cos(W_E[v], h_post_norm)
+                    # strips magnitude so a developer can read whether
+                    # a high-logit token won by direction or by
+                    # row magnitude. Cheap (one matmul per token per
+                    # step) and computed once per step.
+                    cos_h_norm = None
+                    W_E = None
+                    try:
+                        final_hidden_pre_norm = outputs.hidden_states[-1][0, -1, :]
+                        if hasattr(manager.model, 'model') and hasattr(manager.model.model, 'norm'):
+                            h_post_norm = manager.model.model.norm(
+                                final_hidden_pre_norm.unsqueeze(0)
+                            )[0]
+                        elif hasattr(manager.model, 'transformer') and hasattr(manager.model.transformer, 'ln_f'):
+                            h_post_norm = manager.model.transformer.ln_f(
+                                final_hidden_pre_norm.unsqueeze(0)
+                            )[0]
+                        else:
+                            h_post_norm = final_hidden_pre_norm
+                        h_norm_val = float(torch.linalg.vector_norm(h_post_norm).item())
+                        if h_norm_val > 1e-9:
+                            cos_h_norm = h_post_norm / h_norm_val
+                            if hasattr(manager.model, 'lm_head'):
+                                W_E = manager.model.lm_head.weight
+                            else:
+                                logger.warning(
+                                    "[SSE cosine prep] manager.model has no lm_head — cosine emission skipped"
+                                )
+                        else:
+                            logger.warning(
+                                f"[SSE cosine prep] h_post_norm magnitude too small: {h_norm_val}"
+                            )
+                        if cos_h_norm is not None and W_E is not None and step == 0:
+                            logger.info(
+                                f"[SSE cosine prep] enabled · h_post_norm shape {tuple(h_post_norm.shape)} · "
+                                f"W_E shape {tuple(W_E.shape)} · h_norm {h_norm_val:.3f}"
+                            )
+                    except Exception as e:
+                        logger.warning(
+                            f"[SSE cosine prep] failed at step {step}: {type(e).__name__}: {e}"
+                        )
+                        cos_h_norm = None
+                        W_E = None
+
+                    def _cos_for(token_idx: int):
+                        """cosine(W_E[token_idx], h_post_norm) and ‖W_E row‖."""
+                        if cos_h_norm is None or W_E is None:
+                            return None, None
+                        try:
+                            row = W_E[token_idx]
+                            row_norm = float(torch.linalg.vector_norm(row).item())
+                            if row_norm < 1e-9:
+                                return None, row_norm
+                            cos = float(((row / row_norm) @ cos_h_norm).item())
+                            return cos, row_norm
+                        except Exception:
+                            return None, None
+
                     logits_entries = []
                     for rank, (logit_val, idx) in enumerate(zip(top_raw_logits.tolist(), top_raw_indices.tolist())):
                         token_text = decode_token(idx)
-                        logits_entries.append({
+                        cos, row_norm = _cos_for(idx)
+                        entry = {
                             "token": token_text,
                             "token_id": idx,
                             "logit": logit_val,
-                            "rank": rank + 1
-                        })
+                            "rank": rank + 1,
+                        }
+                        if cos is not None:
+                            entry["cosine_sim"] = round(cos, 4)
+                        if row_norm is not None:
+                            entry["unemb_row_norm"] = round(row_norm, 4)
+                        logits_entries.append(entry)
 
                     # Greedy token (argmax of raw logits, before any sampling)
                     greedy_token_id = torch.argmax(raw_logits).item()
@@ -3360,13 +3519,21 @@ async def analyze_research_attention_stream(request: Dict[str, Any], authenticat
                         })
                         # Also add to logits if not present
                         if next_token_id not in [e["token_id"] for e in logits_entries]:
-                            logits_entries.append({
+                            cos_outlier, row_norm_outlier = _cos_for(next_token_id)
+                            outlier_entry = {
                                 "token": next_token_text,
                                 "token_id": next_token_id,
                                 "logit": selected_logit,
                                 "rank": selected_rank,
-                                "is_selected_outlier": True
-                            })
+                                "is_selected_outlier": True,
+                            }
+                            if cos_outlier is not None:
+                                outlier_entry["cosine_sim"] = round(cos_outlier, 4)
+                            if row_norm_outlier is not None:
+                                outlier_entry["unemb_row_norm"] = round(
+                                    row_norm_outlier, 4
+                                )
+                            logits_entries.append(outlier_entry)
 
                     # Build sampling metadata
                     eligible_count = int((probs_filtered > 0).sum().item())
@@ -3780,8 +3947,18 @@ async def analyze_research_attention_stream(request: Dict[str, Any], authenticat
                             # endpoint or a 14,336-d cache entry.
                             layer_entry["ffn_top_neurons"] = ffn_top_neurons[layer_idx]
 
-                        # Phase 5: Logit lens at sampled layers (every 8th layer)
-                        logit_lens_stride = max(1, n_layers // 5)
+                        # Phase 5: Logit lens at every layer.
+                        # The lens projection is a single matmul
+                        # [1, hidden] × [hidden, vocab] ≈ 0.5-1ms on
+                        # Apple Silicon MPS — 40 layers × N_steps adds
+                        # only ~5-10s to a typical analyse run. Worth
+                        # the cost: every-layer sampling eliminates
+                        # the (L17, L24]-style uncertainty windows the
+                        # frontend's crystallisation narrative would
+                        # otherwise have to qualify, giving developers
+                        # a layer-precise commit point for trust
+                        # decisions about the model's prediction.
+                        logit_lens_stride = 1
                         if layer_idx % logit_lens_stride == 0 or layer_idx == n_layers - 1:
                             try:
                                 hidden_for_lens = current_hidden[-1].unsqueeze(0)  # [1, hidden_dim]