echoforest/yt-chat-gemma4-12b-screen-heads-instruct

YouTube live-chat market classifier: Gemma 4 12B, screening arm heads-instruct

Screening checkpoint, not a production model. This is one of four arms trained on the same 15,000-turn sample for one epoch to choose the configuration for the full training run. Scores are on a fixed 4,000-turn validation sample, before calibration. The model classifies one author's turn in Korean bitcoin-futures YouTube live chat into eight labels.

• Base model: google/gemma-4-12B-it (instruct checkpoint), text decoder only.
• Arm: LoRA adapter on the text decoder plus eight linear heads (heads.safetensors) on the final-norm hidden state of the last token. One forward pass per turn.
• Labels: stored reference labels from an LLM labeller. Every score measures agreement with those labels, not ground truth.

This arm did not learn. Training loss fell to the label-prior level (about 8) by step 20 and stayed there (heads-base reached 2.6). Validation agreement is 68.9%, roughly what always predicting the most common option gives, and no direction view reaches 80% precision. The final-token features of the instruct checkpoint are larger in scale than the base checkpoint's (mean norm 278 against 197), and with the shared recipe (heads LR 1e-3, gradient clipping 1.0 shared with LoRA) the heads stayed at the prior. It is published for completeness of the screen. Do not use it.

Screening results

Correct direction views at the lowest gate τ with ≥80% val precision, and mean field agreement, on the same 4,000 val turns. Both backbones were screened with the same recipe and data; the production model is the Qwen3.5-9B heads-instruct arm retrained in full (echoforest/yt-chat-classifier-qwen3.5-9b).

Arm	Gemma 4 12B (this screen): correct views / agreement	Qwen3.5-9B: correct views / agreement
heads-base	936 / 88.3%	707 / 86.4%
heads-instruct (this model)	0 / 68.9%	717 / 86.2%
gen-base	903 / 88.6%	705 / 86.6%
gen-instruct	876 / 88.5%	624 / 86.1%

This arm: mean field agreement 68.9%; all eight fields identical on 3.8% of turns; direction views: no τ reaches 80% precision.

Per-field val agreement: relevance 80.0 · direction 58.5 · direction_basis 62.1 · target 27.7 · own_position_or_trade 90.8 · pnl_state 95.5 · expressed_emotion 52.6 · sarcasm 84.1.

Fields

Field	Options (in order)
relevance	market, off_topic, unclear
direction	bullish, bearish, rangebound, mixed, none, unclear
direction_basis	explicit_forecast, own_position, recommendation_to_others, wish_or_hope, reaction_to_move, not_applicable
target	bitcoin, ethereum, named_altcoin, altcoins_general, crypto_market, traditional, multiple, none
own_position_or_trade	long, short, took_profit, stopped_or_liquidated, hedged, flat_waiting, none
pnl_state	profit, loss, liquidated_or_large_loss, none
expressed_emotion	fear, fomo, greed_euphoria, enthusiasm, frustration, regret, mockery, uncertainty, calm, mixed, none
sarcasm	likely, not_evident, unclear

Input format

Transcripts must be rendered exactly as in training (the project's N1 template): a stream preamble with a slang glossary, recent earlier chat, and the target author's lines fenced as <<<...>>>. Gemma needs exactly one <bos> at the start.

Usage

import json, torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from safetensors.torch import load_file
from transformers import AutoConfig, AutoTokenizer, Gemma4UnifiedForCausalLM

repo = snapshot_download("echoforest/yt-chat-gemma4-12b-screen-heads-instruct")
base = "google/gemma-4-12B-it"
lm = Gemma4UnifiedForCausalLM.from_pretrained(
    base, config=AutoConfig.from_pretrained(base).text_config, dtype=torch.bfloat16, device_map="cuda",
    # Text decoder only. The checkpoint stores it as model.language_model.*, so rename the keys,
    # otherwise every weight is silently re-initialised.
    key_mapping={r"^model\.language_model\.": "model."})
text = PeftModel.from_pretrained(lm.model, repo).eval()
labels = json.load(open(f"{repo}/labels.json"))
heads = {k: v.cuda().float() for k, v in load_file(f"{repo}/heads.safetensors").items()}
tok = AutoTokenizer.from_pretrained(base)
piece = lambda s: tok(s, add_special_tokens=False)["input_ids"]

def classify(transcript):
    ids = torch.tensor([[tok.bos_token_id] + piece(transcript + "\n### LABELS")], device="cuda")  # exactly one <bos>
    with torch.no_grad():
        h = text(input_ids=ids).last_hidden_state[0, -1].float()
    return {f: dict(zip(labels["options"][f],
                     torch.softmax(h @ heads[f + ".weight"].T + heads[f + ".bias"], -1).tolist()))
            for f in labels["fields"]}

Needs transformers ≥ 5.20 (gemma4_unified) and peft ≥ 0.21. No custom kernels are needed. Gemma caps its output logits at 30·tanh(x/30); the generative snippet applies the same cap. Probabilities are uncalibrated (screening runs fit no temperatures).

Training

• Data: export v20261016-1835 (splits by whole stream). A stream-proportional 15,000-turn train sample (seed 13), 1 epoch, max length 1024 tokens, nothing truncated. Turns that needed stream audio trained at half weight.
• Method: LoRA r 32 / α 32 / dropout 0.05 on attention q/k/v/o and MLP gate/up/down; heads trained in full with dropout 0.1, cross-entropy summed over fields, direction ×2. LoRA LR 0.0002, heads LR 0.001, cosine schedule, 3% warmup, AdamW, weight decay 0.01, 32 sequences per step, bf16, gradient checkpointing.
• Compute: 1× H100 80GB, about 3,100 tokens/s.
• Versions: torch 2.8.0+cu128, transformers 5.20.1, peft 0.21.2.

Limitations

• Screening checkpoint: trained on about a fifth of the data for one epoch, uncalibrated, and evaluated only on a val sample from 2 streams. The test set is reserved for the final run.
• Narrow data: 15 train streams over 6 days (2026-10-10 to 10-15), all Korean crypto-futures chat.
• Chat text that imitates the transcript fences or gives instructions can move labels; the robustness suite is run on the final model only.