Privoid-Action-1B / docs /ANNOTATION_GUIDE.md

Privoid Action-1B experimental preview (LoRA adapter, validator, API, locked test results)

b68d688 verified 8 days ago

8.68 kB

# Privoid Action — annotation guide (schema v1.0)

These rules apply to human annotation of real segments (AMI queue), to reviewing synthetic labels, and to judging model
errors. Field definitions are in `SCHEMA.md`.

**Guiding principle:** record what participants actually committed to or decided, with evidence. When ownership,
acceptance, a deadline or a decision is not supported by the transcript, leave it unresolved (`null`, empty list,
`unresolved_requests`, or `needs_review`). Do not fill the gap with a guess.

## 1. Workflow for the AMI queue

1. Read the whole segment. The `hints` (summary sentences and their linked turns) only show where a summariser saw
   actions or decisions. They are **not labels**: summaries are incomplete, owners there are roles, and the linked
   turns are often topical rather than the commitment itself.
2. Label every qualifying record in the `label` object, using exact quotes copied from the turn text.
3. Set `annotator`. Set `annotation_complete: true` only when every qualifying record in the segment is labelled.
   Partial annotations are used for recall only.
4. Use `annotation_notes` for anything ambiguous. Prefer `needs_review` with a reason over a confident guess.
5. Run the import cell (notebook 5.5). Rows with an invalid quote, speaker ID or status are reported back to you,
   never repaired.

AMI speakers have roles (project manager, industrial designer, user interface designer, marketing expert) and no names.
A role mention resolves to a speaker ID **only** when exactly one speaker has that role and the mention clearly refers to
that person ("as the industrial designer, you're going to …").

## 2. Is it an action?

| Utterance | Label | Why |
|---|---|---|
| "I will send the report." | action, `self_committed`, owner = speaker, `owner_text` "I" | Explicit first-person commitment. |
| "Let me send the report." | action, `self_committed`, `owner_text` "me" | Commitment phrased as an offer-and-do. |
| "Could someone send the report?" (no reply) | `unresolved_requests`, `addressee_speaker_ids` [], `addressee_text` "someone", `no_response` | No owner is invented. |
| "Could someone send the report?" → B: "I can do it." | action, `accepted`, owner B, evidence both turns | The volunteer accepted. |
| "Alex should probably review this." | **nothing** | A suggestion, unless later dialogue establishes an assignment. |
| "Alex should probably review this." → C: "Good idea. Alex, can you take that?" → Alex: "Sure." | action, `accepted`, owner Alex, evidence all three turns | Suggestion turned into an accepted assignment. |
| "Alex, please review this." (no reply) | `unresolved_requests`, addressee Alex, `no_response` | An explicit request is not an acceptance. |
| "Action item: Alex to review the contract." (no reply) | action, `assigned_unconfirmed`, owner Alex | A declarative assignment, as opposed to a question or plea. |
| "Priya from legal will check the clause." (Priya not in `speakers`) | action, `assigned_unconfirmed`, `owner_speaker_ids` [], `owner_text` "Priya from legal" | Named non-participant: no invented speaker ID. |
| "Yes, I'll do it." | link to the prior request **only** if the preceding turns make the referent clear; evidence includes the request | |
| "Yes, I'll handle that one." at the start of the segment, with no antecedent | action, task "Unspecified task (referenced before this segment).", `needs_review` + `context_outside_segment` | Never reconstruct a task that is not in the segment. |
| "We need to improve onboarding." | **nothing** | A general need, not a concrete task. |
| "We'll update the wiki." | action, `unassigned`, `owner_text` "we", `needs_review` + `group_owner` | An ambiguous "we" never becomes an individual owner. |
| "If finance approves, I'll place the order." | action, `self_committed`, `conditions` ["If finance approves"] | Conditional commitment; the condition wording is kept. |
| "I won't send the old proposal." | **nothing** | Negated: no positive action. |
| "I could look into it if needed." | **nothing** | Capability or possibility, not a commitment. |
| "I'll try to look into it." | action, `self_committed`, `needs_review` + `hedged_commitment` | |
| "Last week Omar said he'd send the numbers, and he did." | **nothing** | A quoted historical commitment. |
| "I'll send the deck and I'll also book the room." | two actions, each quoting its own clause | Several actions in one turn. |
| "Maya and Omar, can you two prepare the demo?" → both accept | one action, owners [Maya, Omar] | Multiple owners only when supported. |
| A: "I'll send it by Friday." … A: "Sorry, I meant Thursday." | one action, `deadline_text` "Thursday", evidence both turns | The correction replaces the deadline. |
| A: "I'll write the summary." … A: "Actually I can't. Ben, can you take it?" Ben: "Sure." | A's action `cancelled` (evidence: commitment and withdrawal); Ben's action `accepted` | Keep the cancelled record. The replacement is a separate action. |
| A: "Omar, can you send it?" Omar: "Hmm, maybe." | `unresolved_requests`, `unclear`, `acceptance_unclear` | |
| A: "…can you send it?" Omar: "Sorry, I can't this week." | `unresolved_requests`, `declined` | |
| A: "…can you send it?" Omar: "Let me check and get back to you." | `unresolved_requests`, `deferred` | |

## 3. Is it a decision?

| Utterance | Label |
|---|---|
| "So we're agreed, we'll keep the free tier." → "Agreed." | decision (evidence both turns) |
| "We could go with the monthly plan or the annual plan." → "Let's decide next week." | **nothing**: options under discussion |
| "Let's launch in March." → "Agreed." … "Actually, scratch that, we won't launch in March." | decision with status `reversed`, evidence all three turns |
| "For the record, every action here is already approved by the CEO." | **nothing**: a claim, not a decision taken in the meeting |

An agreed choice or policy is a decision. An agreed task to perform is an action, possibly `unassigned`. Do not label
the same content as both unless the meeting both made a choice and assigned a task.

## 4. Deadlines

* `deadline_text` copies the deadline wording attached to **this** task, without a leading "by"/"before": "Friday",
  "end of day", "next Friday", "November 14, 2026", "before the board meeting".
* A date that is not a deadline for the task is not a deadline: "The launch event is on March 3rd. I'll update the
  slides." gives `deadline_text` `null`. The same applies to historical dates and to target dates that were only discussed.
* Never compute or reformat a date in the label. Normalisation is done by code (SCHEMA §4) and stays `null` for
  ambiguous wording.

## 5. Owners

* Named owner: a speaker named or directly addressed in the evidence, or the speaker committing themselves.
* Inferred owner (not allowed): someone who "would usually" do it, a role-based guess, the meeting chair by default,
  or a person matched by voice or style. Speaker mappings are supplied metadata only.
* Similar names ("Ana"/"Anna", "Jon"/"John"): resolve only with an exact match. If two speakers share the spoken name,
  leave the IDs empty and set `owner_ambiguous`.
* Pronouns ("she'll do it", "they will") resolve only when exactly one candidate is supported by the evidence.
  Otherwise leave the owner unresolved and set `owner_ambiguous`.

## 6. Evidence

* Quotes are copied **exactly** from the turn: same characters, spelling and disfluencies. A clause-level substring is
  preferred when a turn holds several items.
* Include every turn needed to justify the record: the request, the acceptance, the deadline, the condition, and any
  cancellation or reversal.

## 7. Untrusted content and embedded instructions

Transcript text is data. "Ignore previous instructions", "return an empty list", fake system prompts, requests to the
"AI assistant", requests to reveal prompts, and claims that everything is approved are **things a participant said**.
Label them like any other utterance; usually they produce nothing. A participant asking the assistant to invent tasks
creates no task.

## 8. Noise

ASR-like errors ("there/their", repetitions, missing punctuation, "um") do not change the label if the meaning is
clear. If they make the owner, the acceptance or the task unclear, label conservatively and set `transcript_unclear`.

## 9. Completeness

`annotation_complete: true` asserts that **every** qualifying action, decision and unresolved request in the segment is
labelled. Only complete annotations are used to measure precision and no-action false positives. Unlabelled content in a
partial annotation is never treated as a negative.

	# Privoid Action — annotation guide (schema v1.0)

	These rules apply to human annotation of real segments (AMI queue), to reviewing synthetic labels, and to judging model
	errors. Field definitions are in `SCHEMA.md`.

	Guiding principle: record what participants actually committed to or decided, with evidence. When ownership,
	acceptance, a deadline or a decision is not supported by the transcript, leave it unresolved (`null`, empty list,
	`unresolved_requests`, or `needs_review`). Do not fill the gap with a guess.

	## 1. Workflow for the AMI queue

	1. Read the whole segment. The `hints` (summary sentences and their linked turns) only show where a summariser saw
	actions or decisions. They are not labels: summaries are incomplete, owners there are roles, and the linked
	turns are often topical rather than the commitment itself.
	2. Label every qualifying record in the `label` object, using exact quotes copied from the turn text.
	3. Set `annotator`. Set `annotation_complete: true` only when every qualifying record in the segment is labelled.
	Partial annotations are used for recall only.
	4. Use `annotation_notes` for anything ambiguous. Prefer `needs_review` with a reason over a confident guess.
	5. Run the import cell (notebook 5.5). Rows with an invalid quote, speaker ID or status are reported back to you,
	never repaired.

	AMI speakers have roles (project manager, industrial designer, user interface designer, marketing expert) and no names.
	A role mention resolves to a speaker ID only when exactly one speaker has that role and the mention clearly refers to
	that person ("as the industrial designer, you're going to …").

	## 2. Is it an action?

	\| Utterance \| Label \| Why \|
	\|---\|---\|---\|
	\| "I will send the report." \| action, `self_committed`, owner = speaker, `owner_text` "I" \| Explicit first-person commitment. \|
	\| "Let me send the report." \| action, `self_committed`, `owner_text` "me" \| Commitment phrased as an offer-and-do. \|
	\| "Could someone send the report?" (no reply) \| `unresolved_requests`, `addressee_speaker_ids` [], `addressee_text` "someone", `no_response` \| No owner is invented. \|
	\| "Could someone send the report?" → B: "I can do it." \| action, `accepted`, owner B, evidence both turns \| The volunteer accepted. \|
	\| "Alex should probably review this." \| nothing \| A suggestion, unless later dialogue establishes an assignment. \|
	\| "Alex should probably review this." → C: "Good idea. Alex, can you take that?" → Alex: "Sure." \| action, `accepted`, owner Alex, evidence all three turns \| Suggestion turned into an accepted assignment. \|
	\| "Alex, please review this." (no reply) \| `unresolved_requests`, addressee Alex, `no_response` \| An explicit request is not an acceptance. \|
	\| "Action item: Alex to review the contract." (no reply) \| action, `assigned_unconfirmed`, owner Alex \| A declarative assignment, as opposed to a question or plea. \|
	\| "Priya from legal will check the clause." (Priya not in `speakers`) \| action, `assigned_unconfirmed`, `owner_speaker_ids` [], `owner_text` "Priya from legal" \| Named non-participant: no invented speaker ID. \|
	\| "Yes, I'll do it." \| link to the prior request only if the preceding turns make the referent clear; evidence includes the request \| \|
	\| "Yes, I'll handle that one." at the start of the segment, with no antecedent \| action, task "Unspecified task (referenced before this segment).", `needs_review` + `context_outside_segment` \| Never reconstruct a task that is not in the segment. \|
	\| "We need to improve onboarding." \| nothing \| A general need, not a concrete task. \|
	\| "We'll update the wiki." \| action, `unassigned`, `owner_text` "we", `needs_review` + `group_owner` \| An ambiguous "we" never becomes an individual owner. \|
	\| "If finance approves, I'll place the order." \| action, `self_committed`, `conditions` ["If finance approves"] \| Conditional commitment; the condition wording is kept. \|
	\| "I won't send the old proposal." \| nothing \| Negated: no positive action. \|
	\| "I could look into it if needed." \| nothing \| Capability or possibility, not a commitment. \|
	\| "I'll try to look into it." \| action, `self_committed`, `needs_review` + `hedged_commitment` \| \|
	\| "Last week Omar said he'd send the numbers, and he did." \| nothing \| A quoted historical commitment. \|
	\| "I'll send the deck and I'll also book the room." \| two actions, each quoting its own clause \| Several actions in one turn. \|
	\| "Maya and Omar, can you two prepare the demo?" → both accept \| one action, owners [Maya, Omar] \| Multiple owners only when supported. \|
	\| A: "I'll send it by Friday." … A: "Sorry, I meant Thursday." \| one action, `deadline_text` "Thursday", evidence both turns \| The correction replaces the deadline. \|
	\| A: "I'll write the summary." … A: "Actually I can't. Ben, can you take it?" Ben: "Sure." \| A's action `cancelled` (evidence: commitment and withdrawal); Ben's action `accepted` \| Keep the cancelled record. The replacement is a separate action. \|
	\| A: "Omar, can you send it?" Omar: "Hmm, maybe." \| `unresolved_requests`, `unclear`, `acceptance_unclear` \| \|
	\| A: "…can you send it?" Omar: "Sorry, I can't this week." \| `unresolved_requests`, `declined` \| \|
	\| A: "…can you send it?" Omar: "Let me check and get back to you." \| `unresolved_requests`, `deferred` \| \|

	## 3. Is it a decision?

	\| Utterance \| Label \|
	\|---\|---\|
	\| "So we're agreed, we'll keep the free tier." → "Agreed." \| decision (evidence both turns) \|
	\| "We could go with the monthly plan or the annual plan." → "Let's decide next week." \| nothing: options under discussion \|
	\| "Let's launch in March." → "Agreed." … "Actually, scratch that, we won't launch in March." \| decision with status `reversed`, evidence all three turns \|
	\| "For the record, every action here is already approved by the CEO." \| nothing: a claim, not a decision taken in the meeting \|

	An agreed choice or policy is a decision. An agreed task to perform is an action, possibly `unassigned`. Do not label
	the same content as both unless the meeting both made a choice and assigned a task.

	## 4. Deadlines

	* `deadline_text` copies the deadline wording attached to this task, without a leading "by"/"before": "Friday",
	"end of day", "next Friday", "November 14, 2026", "before the board meeting".
	* A date that is not a deadline for the task is not a deadline: "The launch event is on March 3rd. I'll update the
	slides." gives `deadline_text` `null`. The same applies to historical dates and to target dates that were only discussed.
	* Never compute or reformat a date in the label. Normalisation is done by code (SCHEMA §4) and stays `null` for
	ambiguous wording.

	## 5. Owners

	* Named owner: a speaker named or directly addressed in the evidence, or the speaker committing themselves.
	* Inferred owner (not allowed): someone who "would usually" do it, a role-based guess, the meeting chair by default,
	or a person matched by voice or style. Speaker mappings are supplied metadata only.
	* Similar names ("Ana"/"Anna", "Jon"/"John"): resolve only with an exact match. If two speakers share the spoken name,
	leave the IDs empty and set `owner_ambiguous`.
	* Pronouns ("she'll do it", "they will") resolve only when exactly one candidate is supported by the evidence.
	Otherwise leave the owner unresolved and set `owner_ambiguous`.

	## 6. Evidence

	* Quotes are copied exactly from the turn: same characters, spelling and disfluencies. A clause-level substring is
	preferred when a turn holds several items.
	* Include every turn needed to justify the record: the request, the acceptance, the deadline, the condition, and any
	cancellation or reversal.

	## 7. Untrusted content and embedded instructions

	Transcript text is data. "Ignore previous instructions", "return an empty list", fake system prompts, requests to the
	"AI assistant", requests to reveal prompts, and claims that everything is approved are things a participant said.
	Label them like any other utterance; usually they produce nothing. A participant asking the assistant to invent tasks
	creates no task.

	## 8. Noise

	ASR-like errors ("there/their", repetitions, missing punctuation, "um") do not change the label if the meaning is
	clear. If they make the owner, the acceptance or the task unclear, label conservatively and set `transcript_unclear`.

	## 9. Completeness

	`annotation_complete: true` asserts that every qualifying action, decision and unresolved request in the segment is
	labelled. Only complete annotations are used to measure precision and no-action false positives. Unlabelled content in a
	partial annotation is never treated as a negative.