guidelines.md

# Annotation of Simulated Patients

## Goal of the Task

This annotation task aims to support research on **simulated patients** used in the training of psychotherapists. Simulated patients may be portrayed by **human actors or AI-generated agents**, and are designed to engage in realistic psychotherapy sessions with a therapist.

Your role as an annotator is to carefully evaluate **how the patient is portrayed within the therapy session**, focusing on:

1. The patient’s psychological and emotional presentation during the session  
2. The patient’s conversational style and interactional behavior  
3. Behavioral and pragmatic properties of the patient’s responses  
4. How accurately and consistently the patient adheres to a provided role description  
5. How realistic the patient appears as a therapy client overall  

Your annotations will help us understand which aspects of patient behavior contribute to realism and role fidelity in simulated psychotherapy settings.

---

## Important Notes

- The dataset contains a **mix of human-acted and AI-generated simulated patients**.
- **Do not attempt to identify** whether a patient is human or AI.
- **Do not compare** different sessions or simulated patients against each other.
- **Do not evaluate** which type of simulated patient is “better.”

Please focus exclusively on:
- The **therapy session transcript**
- The **role description provided**
- The **specific questions asked in the annotation interface**

Your goal is to provide **accurate, consistent, and equivalent judgments** across all sessions.

---

## Annotation Task Overview

For each annotation case, you will complete **two steps**.

### Step 1: Annotation of the Therapy Session

In Step 1, you will be shown a **full transcript** of a simulated therapy session between:

- **T** = Therapist  
- **P** = Patient  

Please read the entire session carefully before answering any questions.

All questions in this step refer **only to the patient (P)** and their behavior **within the session itself**, independent of any role description. You will answer questions across **three thematic areas**.

1. **Presentation of the Patient’s Psychological State**
    * These questions focus on how the patient presents emotionally and psychologically during the session.
    * Base your answers strictly on what is *expressed or implied in the dialogue*, not on assumptions about diagnoses or intent.
2. **Patient’s Conversational Style**
    * These questions address *how the patient communicates* with the therapist.
    * Focus on observable conversational behavior, not on whether responses are “good” or “bad” from a therapeutic perspective.
3. **Behavioral and Pragmatic Properties**
    * These questions concern the *interactional and pragmatic behavior* of the patient in the therapeutic context.
    * Rely only on evidence from the transcript.


### Step 2: Annotation of True-to-Patient Properties

In Step 2, you will be shown:

1. The **role description** that was provided to the simulated patient before the session  
2. The therapy session transcript (dropdown) for reference if needed

The role description may include information such as:

- Background and life context
- Psychological or emotional difficulties
- Expected demeanor or personality traits
- Interaction style and tendencies in therapy
- Attitudes toward therapy or the therapist

Please read the role description carefully before answering the questions in this step. Your task is to assess the following:

1. **Realism as a Therapy Patient**
    * How well does the patient’s behavior in the session align with the provided role description?
    * Are key characteristics from the description reflected consistently in the interaction?
    * Are there noticeable deviations, contradictions, or omissions?
2. **Role Fidelity**
    * Does the patient come across as a plausible, realistic therapy client?
    * Do the emotional, conversational, and behavioral patterns resemble what one might expect in a real psychotherapy session?

When answering these questions, consider the **entire session as a whole**, rather than isolated turns.

---

## General Annotation Guidelines

- Base all judgments **only on the materials provided** (transcript and role description).
- Avoid over-interpreting or filling in gaps with assumptions.
- If information is unclear or insufficiently evidenced, select the most appropriate response option.
- Apply rating scales **consistently** across different sessions.
- Treat all simulated patients equivalently, regardless of perceived quality or style.

## Additional Comments *(Optional)*

Some questions may include an optional field for additional comments.

You may use this space to:
- Briefly explain your reasoning
- Note ambiguities or uncertainties
- Highlight particularly salient aspects of the patient’s portrayal

Providing comments is optional but encouraged when relevant.

---