Create guidelines.md

guidelines.md

@@ -0,0 +1,110 @@
+
+# Annotation of Simulated Patients
+
+## Goal of the Task
+
+This annotation task aims to support research on **simulated patients** used in the training of psychotherapists. Simulated patients may be portrayed by **human actors or AI-generated agents**, and are designed to engage in realistic psychotherapy sessions with a therapist.
+
+Your role as an annotator is to carefully evaluate **how the patient is portrayed within the therapy session**, focusing on:
+
+1. The patient’s psychological and emotional presentation during the session  
+2. The patient’s conversational style and interactional behavior  
+3. Behavioral and pragmatic properties of the patient’s responses  
+4. How accurately and consistently the patient adheres to a provided role description  
+5. How realistic the patient appears as a therapy client overall  
+
+Your annotations will help us understand which aspects of patient behavior contribute to realism and role fidelity in simulated psychotherapy settings.
+
+---
+
+## Important Notes
+
+- The dataset contains a **mix of human-acted and AI-generated simulated patients**.
+- **Do not attempt to identify** whether a patient is human or AI.
+- **Do not compare** different sessions or simulated patients against each other.
+- **Do not evaluate** which type of simulated patient is “better.”
+
+Please focus exclusively on:
+- The **therapy session transcript**
+- The **role description provided**
+- The **specific questions asked in the annotation interface**
+
+Your goal is to provide **accurate, consistent, and equivalent judgments** across all sessions.
+
+---
+
+## Annotation Task Overview
+
+For each annotation case, you will complete **two steps**.
+
+### Step 1: Annotation of the Therapy Session
+
+In Step 1, you will be shown a **full transcript** of a simulated therapy session between:
+
+- **T** = Therapist  
+- **P** = Patient  
+
+Please read the entire session carefully before answering any questions.
+
+All questions in this step refer **only to the patient (P)** and their behavior **within the session itself**, independent of any role description. You will answer questions across **three thematic areas**.
+
+1. Presentation of the Patient’s Psychological State
+  * These questions focus on how the patient presents emotionally and psychologically during the session.
+  * Base your answers strictly on what is **expressed or implied in the dialogue**, not on assumptions about diagnoses or intent.
+2. Patient’s Conversational Style
+  * These questions address **how the patient communicates** with the therapist.
+  * Focus on observable conversational behavior, not on whether responses are “good” or “bad” from a therapeutic perspective.
+3. Behavioral and Pragmatic Properties
+  * These questions concern the **interactional and pragmatic behavior** of the patient in the therapeutic context.
+  * Rely only on evidence from the transcript.
+
+
+### Step 2: Annotation of True-to-Patient Properties
+
+In Step 2, you will be shown:
+
+1. The **role description** that was provided to the simulated patient before the session  
+2. The therapy session transcript (dropdown) for reference if needed
+
+The role description may include information such as:
+
+- Background and life context
+- Psychological or emotional difficulties
+- Expected demeanor or personality traits
+- Interaction style and tendencies in therapy
+- Attitudes toward therapy or the therapist
+
+Please read the role description carefully before answering the questions in this step. Your task is to assess the following:
+
+1. **Role Fidelity**
+  * How well does the patient’s behavior in the session align with the provided role description?
+  * Are key characteristics from the description reflected consistently in the interaction?
+  * Are there noticeable deviations, contradictions, or omissions?
+2. **Realism as a Therapy Patient**
+  * Does the patient come across as a plausible, realistic therapy client?
+  * Do the emotional, conversational, and behavioral patterns resemble what one might expect in a real psychotherapy session?
+
+When answering these questions, consider the **entire session as a whole**, rather than isolated turns.
+
+---
+
+## General Annotation Guidelines
+
+- Base all judgments **only on the materials provided** (transcript and role description).
+- Avoid over-interpreting or filling in gaps with assumptions.
+- If information is unclear or insufficiently evidenced, select the most appropriate response option.
+- Apply rating scales **consistently** across different sessions.
+- Treat all simulated patients equivalently, regardless of perceived quality or style.
+
+## Additional Comments *(Optional)*
+
+Some questions may include an optional field for additional comments.
+
+You may use this space to:
+- Briefly explain your reasoning
+- Note ambiguities or uncertainties
+- Highlight particularly salient aspects of the patient’s portrayal
+
+Providing comments is optional but encouraged when relevant.
+
+---