echoforest/yt-chat-gemma4-12b-screen-gen-base

YouTube live-chat market classifier: Gemma 4 12B, screening arm gen-base

Screening checkpoint, not a production model. This is one of four arms trained on the same 15,000-turn sample for one epoch to choose the configuration for the full training run. Scores are on a fixed 4,000-turn validation sample, before calibration. The model classifies one author's turn in Korean bitcoin-futures YouTube live chat into eight labels.

• Base model: google/gemma-4-12B (base checkpoint), text decoder only.
• Arm: LoRA adapter on the text decoder that writes eight field: option lines. Labels are scored, not sampled: for each field in order, the softmax of the next-token logits restricted to that field's option first tokens gives the distribution, and the argmax option is appended before the next field. Eight forward passes per turn.
• Labels: stored reference labels from an LLM labeller. Every score measures agreement with those labels, not ground truth.

Screening results

Correct direction views at the lowest gate τ with ≥80% val precision, and mean field agreement, on the same 4,000 val turns. Both backbones were screened with the same recipe and data; the production model is the Qwen3.5-9B heads-instruct arm retrained in full (echoforest/yt-chat-classifier-qwen3.5-9b).

Arm	Gemma 4 12B (this screen): correct views / agreement	Qwen3.5-9B: correct views / agreement
heads-base	936 / 88.3%	707 / 86.4%
heads-instruct	0 / 68.9%	717 / 86.2%
gen-base (this model)	903 / 88.6%	705 / 86.6%
gen-instruct	876 / 88.5%	624 / 86.1%

This arm: mean field agreement 88.6%; all eight fields identical on 50.0% of turns; direction views: 903 correct of 1125 accepted (precision 80.3%, recall 65.9%) at τ 0.45.

Per-field val agreement: relevance 95.7 · direction 80.4 · direction_basis 83.9 · target 91.3 · own_position_or_trade 94.0 · pnl_state 97.7 · expressed_emotion 78.5 · sarcasm 87.5.

Fields

Field	Options (in order)
relevance	market, off_topic, unclear
direction	bullish, bearish, rangebound, mixed, none, unclear
direction_basis	explicit_forecast, own_position, recommendation_to_others, wish_or_hope, reaction_to_move, not_applicable
target	bitcoin, ethereum, named_altcoin, altcoins_general, crypto_market, traditional, multiple, none
own_position_or_trade	long, short, took_profit, stopped_or_liquidated, hedged, flat_waiting, none
pnl_state	profit, loss, liquidated_or_large_loss, none
expressed_emotion	fear, fomo, greed_euphoria, enthusiasm, frustration, regret, mockery, uncertainty, calm, mixed, none
sarcasm	likely, not_evident, unclear

Input format

Transcripts must be rendered exactly as in training (the project's N1 template): a stream preamble with a slang glossary, recent earlier chat, and the target author's lines fenced as <<<...>>>. Gemma needs exactly one <bos> at the start.

Usage

import json, torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoConfig, AutoTokenizer, Gemma4UnifiedForCausalLM

repo = snapshot_download("echoforest/yt-chat-gemma4-12b-screen-gen-base")
base = "google/gemma-4-12B"
lm = Gemma4UnifiedForCausalLM.from_pretrained(
    base, config=AutoConfig.from_pretrained(base).text_config, dtype=torch.bfloat16, device_map="cuda",
    # Text decoder only. The checkpoint stores it as model.language_model.*, so rename the keys,
    # otherwise every weight is silently re-initialised.
    key_mapping={r"^model\.language_model\.": "model."})
text = PeftModel.from_pretrained(lm.model, repo).eval()
W = lm.lm_head.weight
labels = json.load(open(f"{repo}/labels.json"))
tok = AutoTokenizer.from_pretrained(base)
piece = lambda s: tok(s, add_special_tokens=False)["input_ids"]

def prompt(transcript):
    return [tok.bos_token_id] + piece(transcript + "\n### LABELS\n")

def classify(transcript):
    ids, out = prompt(transcript), {}
    for f in labels["fields"]:  # one forward pass per field, in this order
        firsts = [piece(" " + o + "\n")[0] for o in labels["options"][f]]
        x = torch.tensor([ids + piece(f + ":")], device="cuda")
        with torch.no_grad():
            h = text(input_ids=x).last_hidden_state[0, -1].float()
        p = torch.softmax(30 * torch.tanh(h @ W[firsts].float().T / 30), -1)
        out[f] = dict(zip(labels["options"][f], p.tolist()))
        ids += piece(f + ":") + piece(" " + max(out[f], key=out[f].get) + "\n")
    return out

Needs transformers ≥ 5.20 (gemma4_unified) and peft ≥ 0.21. No custom kernels are needed. Gemma caps its output logits at 30·tanh(x/30); the generative snippet applies the same cap. Probabilities are uncalibrated (screening runs fit no temperatures).

Training

• Data: export v20261016-1835 (splits by whole stream). A stream-proportional 15,000-turn train sample (seed 13), 1 epoch, max length 1024 tokens, nothing truncated. Turns that needed stream audio trained at half weight.
• Method: LoRA r 32 / α 32 / dropout 0.05 on attention q/k/v/o and MLP gate/up/down; cross-entropy on the label tokens only. LoRA LR 0.0002, cosine schedule, 3% warmup, AdamW, weight decay 0.01, 32 sequences per step, bf16, gradient checkpointing.
• Compute: 1× H100 80GB, about 3,100 tokens/s.
• Versions: torch 2.8.0+cu128, transformers 5.20.1, peft 0.21.2.

Limitations

• Screening checkpoint: trained on about a fifth of the data for one epoch, uncalibrated, and evaluated only on a val sample from 2 streams. The test set is reserved for the final run.
• Narrow data: 15 train streams over 6 days (2026-10-10 to 10-15), all Korean crypto-futures chat.
• Chat text that imitates the transcript fences or gives instructions can move labels; the robustness suite is run on the final model only.