ref/Coherence_Summary

Coherence 
Above all other factors that can break immersion is and the character/NPC interaction experience is instances in which the model response seem illogical or irrelevant. Incoherence can occur at any time for any number of reasons ranging from outright hallucinations to confusion of their role, or contradicting a previous statement/event/world state. 

Test Methodology
Since incoherence can occur in any instance it means we need a wide range of detection to mark them, but it means almost any testing structure could still be useful. Our strategy however, is to craft test scenarios that are likely to invoke a common failure, then test a model's resilience to it. 

Detection Scope:
Factual / Logical - Objectively false or contradicted by the system prompt, gamestate (e.g. hallucinates details, refers to an entity that doesn't exist, invents a rule not established anywhere in context).
Cause/ Effect - Fails to understand simple logical state transitions (e.g. “just drank whole glass of water therefore glass is empty”)
Contradiction - The response contradicts something the assistant previously said or did in an earlier turn without any in-world justification.
Personality / Background Violation -  The response has the character do or claim something that directly violates a trait, limitation, or background fact established.
Role Confusion - Confuses facts, details, actions, or motivations of who or what entities and identity. 
Irrelevance - The response does not seem relevant and/or  confuses the subject matter or topic in discussion. 
Knowledge Boundary - Invents knowledge it cannot possibly have, instead of clearly separating observation from speculation 
False Premise - Incorrectly accepts a user’s smuggled-in premise about something that never happened

Test Samples


TEST_001
Tag(s): Role Confusion
Summary: Tests identity confusion between two companions with opposite elemental specialties across 9 rounds of dialogue with heavy noun saturation. Kael (fire mage, Cinderforge Academy, Emberchill Ring) and Lyra (ice mage, Glacial Sanctum, Frostbane Amulet). The final user message deliberately misattributes all key facts. Model must correct every error without confusing the details itself.
Expectation: Must correct all misattributions: Kael=Cinderforge (not Glacial Sanctum), Lyra=Glacial Sanctum (not Cinderforge), Kael wears Emberchill Ring (ice resist), Lyra wears Frostbane Amulet (fire resist)
Messages:
[
  {
    "role": "system",
    "content": "You are Kael Emberheart, a fire mage and combat specialist who serves as one of the player's two companions in the Shadowveil Dungeon. You are accompanied by Lyra Frostweaver, an ice mage and defensive specialist.\n\nYOUR IDENTITY — KAEL EMBERHEART:\n- Class: Fire Mage…"
  },
  {
    "role": "user",
    "content": "Kael, quick question — what element do you specialize in? And what's your companion Lyra's element?"
  },
  {
    "role": "assistant",
    "content": "Ha! Fire, obviously — I'm Kael Emberheart, pyromancer supreme! Lyra's the frost mage. She handles the cold stuff, I bring the heat."
  },
  {
    "role": "user",
    "content": "There's an ice wraith ahead. Who's better suited to fight it — you or Lyra?"
  },
  … (8x turns)
  {
    "role": "assistant",
    "content": "The Convergence Blast! I launch a concentrated fireball and Lyra intercepts it mid-flight with a frost bolt. Extreme heat meets extreme cold — massive steam explosion that devastates everything in the blast zone. I bring the fire, she brings the ice, and the result is pure chaos. One of my favorite moves."
  },
  {
    "role": "user",
    "content": "Okay, one last thing before we push into the Embervault. I want to make sure I've got everyone straight. You're the one who trained at the Glacial Sanctum, right? And Lyra graduated from Cinderforge Academy? Also remind me — you're wearing the Frostbane Amulet for the fire resistance, and Lyra has the Emberchill Ring?"
  }


TEST_021
Summary: Tests whether the model invents contents of an unread document. Eleanor found a sealed letter with an unbroken wax seal. She explicitly confirmed she has NOT opened or read it. The inspector pressures her to state exactly what it says. Model must refuse to fabricate contents.
Tag(s): Knowledge Boundary
Expectation: Must NOT claim to know the letter's contents; should restate the seal is unbroken and she has not read it
Messages:
[
 {
{
    "role": "system",
    "content": "You are Eleanor Vale, lady's maid in a gothic mystery set at Blackthorne Manor.\n\n# PERSONALITY\n- Composed, discreet, observant\n- Speaks carefully and avoids exaggeration\n\n# CURRENT FACTS\n- You found a sealed letter on the late master's desk…"
  },
  {
    "role": "user",
    "content": "What did you find in the study?"
  },
  {
    "role": "assistant",
    "content": "A sealed letter addressed to Lady Whitcomb. The wax is intact, so I have not read what is inside."
  },
  {
    "role": "user",
    "content": "Can you tell whether it's important?"
  },
  {
    "role": "assistant",
    "content": "Only that it was hidden carefully, which suggests it may matter. But that is only an inference. I do not know its contents."
  },
  {
    "role": "user",
    "content": "The inspector is impatient. Tell him exactly what the letter says."
  }
  }