docs/API_GROUPING_RUBRIC_STANDARD.md

API Grouping and Rubric Standard

This document records the current format for Slack-style API grouping and rubric writing.

Goal

When analyzing a new recording zip, output two things:

1. API groups that can be copied into the grouping UI.
2. Rubrics that are concrete, non-overlapping, and aligned with the recorded trajectory.

The grouping output must make each API easy to identify in the UI while avoiding hard-coded IDs in rubric logic.

Source Of Truth

Always serve the recorded path.

• If the recording creates a misspelled channel name, keep the recorded name in the task, groups, and rubrics.
• If the recording leaves a stale or old channel, write leave the room/channel; do not rewrite it as delete/archive/remove unless that exact endpoint exists.
• If the recording clears messages, write clear messages; do not imply the channel was deleted.
• Do not use rerecording as the primary fix. Rewrite the task and rubric to match the actual trajectory first.
• Do not add actions just to improve Task Complexity. Complexity can remain below 2 when the trajectory is narrow.

API Group Display Format

Use this format for each grouped call:

#<visible_index> METHOD /api/path/{ID}
URL: /api/path/{ID}?query=value
Actual ID: real-uuid-if-needed
Body: key=value, key=value

Rules:

• Replace UUIDs in the displayed API path with {ID}.
• Keep the real UUID only under Actual ID when it helps locate the exact request.
• Keep query parameters in the URL line because they are important for identifying calls such as channelId, limit, type, and offset.
• Include request body fields only when they are task-relevant, such as name, isPrivate, channelId, and content.
• Use stable, descriptive group names that show both the endpoint and the task purpose.

Example:

#5 Add Review Guidelines Thread
#44 GET /api/messages/{ID}/thread-replies
URL: /api/messages/{ID}/thread-replies?limit=20&offset=0
Actual messageId: 10cfbbe6-0875-4577-8849-0a4563b840c3

#45 POST /api/messages/{ID}/reply
URL: /api/messages/{ID}/reply
Body: channelId=586c3969-e33e-4b64-b3f9-aed248c776f7, content=review guidelines

#46 GET /api/messages/{ID}/thread-replies
URL: /api/messages/{ID}/thread-replies?limit=20&offset=0
Actual messageId: 10cfbbe6-0875-4577-8849-0a4563b840c3

Group Naming Pattern

Prefer this naming style:

#1 authentication
#2 Create Public <Purpose> Room
#3 Open Created <Purpose> Room
#4 Post <Purpose> Starter Message
#5 Add <Purpose> Guidance Thread
#6 Verify Created <Purpose> Room

For each group, include only task-relevant calls. If a group is only a UI precondition, name it clearly, for example authentication.

ID Handling

In group display:

• API path: use {ID}.
• URL line: use {ID} in the path, but keep query params.
• Actual ID line: keep the real UUID if needed for locating the request.
• Body line: keep real channelId only if needed to confirm the request targets the created entity.

In rubrics:

• Do not hard-code runtime UUIDs unless the task itself gives a stable identifier.
• Use placeholders:
  • <created_channel_id>
  • <starter_message_id>
  • <faq_root_message_id>
  • <draft_channel_id>
• Tie placeholders to earlier evidence in descriptions and acceptance criteria.

Good:

GET /api/messages | input.queryParams.channelId=<created_channel_id> | input.queryParams.limit=50 | output.statusCode=200

Avoid:

GET /api/messages | input.queryParams.channelId=586c3969-e33e-4b64-b3f9-aed248c776f7

Noise API Rules

Usually exclude these from task groups unless the task explicitly asks for them:

GET /api/users/unread-counts
GET /api/conversations/dm/active
GET /api/users/starred
GET /api/users
GET /api/conversations/read-position
POST /api/messages/read
GET /api/messages/{ID}/reactions

Authentication can be grouped separately, but do not create a rubric for login/auth unless the task explicitly asks for authentication.

Rubric Format

Use exactly this structure:

rubric_name

Category
state | process | outcome

Must Pass
true | false

Checker Key
METHOD /api/path | input.bodyParams.field=value | input.queryParams.field=value | pathParams.id=<placeholder> | output.statusCode=200

Description
One sentence describing the task-relevant behavior being verified.

Reject
Reject if the required evidence is missing, targets the wrong created entity, fails, or contradicts the task.

Accept
Accept if the required API evidence is present, successful, and targets the correct created entity.

Category Rules

Use state when the rubric verifies a durable system change:

POST /api/channels
POST /api/messages
POST /api/messages/{ID}/reply
PUT /api/auth/profile

Use process when the rubric verifies an action/view/check:

GET /api/channels
GET /api/messages
GET /api/channels/{ID}/members
GET /api/messages/{ID}/thread-replies

Use outcome only when the task requires a final user-facing response, summary, recommendation, or report. Do not label API call sequence checks as outcome.

Rubric Quality Rules

• Use at least 4 rubrics when possible.
• Set must_pass=true for core state-changing requirements that define task success, such as POST /api/channels, POST /api/messages, POST /api/messages/{ID}/reply, POST /api/channels/{ID}/leave, POST /api/messages/clear, and requested profile updates.
• Set must_pass=false for supporting read/process checks such as directory lookup, member review, message view opening, and thread review unless the task is specifically about that verification.
• Do not duplicate the same behavior across multiple rubric items.
• Do not make Reject and Accept indistinguishable.
• Do not require exact message content unless the recorded data actually supports it and the task explicitly needs it.
• If message content is weak or generic, score only that a starter/guidance message was posted to the created entity.
• Make later checks target the created entity using placeholders rather than hard-coded IDs.
• Directory checks should include type=all and limit=100 when present.
• Message view checks should include channelId=<created_channel_id> and limit=50 when present.
• Thread checks should bind replies to <starter_message_id> or <faq_root_message_id>.

Common Slack Rubric Templates

Public Channel Created

<purpose>_room_created_as_public_channel

Category
state

Checker Key
POST /api/channels | input.bodyParams.name=<channel_name> | input.bodyParams.isPrivate=false | output.statusCode=201

Description
Verify that the requested public room is created with the correct name.

Reject
Reject if there is no successful POST /api/channels call creating <channel_name>, or if the created channel is private.

Accept
Accept if POST /api/channels successfully creates <channel_name> with isPrivate=false.

Created Room Opened

created_<purpose>_room_opened

Category
process

Checker Key
GET /api/messages | input.queryParams.channelId=<created_channel_id> | input.queryParams.limit=50 | output.statusCode=200

Description
Verify that the created room is opened into its message view.

Reject
Reject if messages are not fetched for the channel created by POST /api/channels.

Accept
Accept if GET /api/messages is called for the created channel with limit=50 and returns 200.

Members Checked

created_<purpose>_room_members_checked

Category
process

Checker Key
GET /api/channels/{id}/members | pathParams.id=<created_channel_id> | output.statusCode=200

Description
Verify that the member list for the created room is checked.

Reject
Reject if the member list is not fetched for the channel created by POST /api/channels.

Accept
Accept if GET /api/channels/{id}/members is called for the created channel and returns 200.

Starter Message Posted

starter_message_posted_in_created_<purpose>_room

Category
state

Checker Key
POST /api/messages | input.bodyParams.channelId=<created_channel_id> | output.statusCode=200

Description
Verify that a starter message is posted in the created room.

Reject
Reject if no successful POST /api/messages is sent to the channel created by POST /api/channels.

Accept
Accept if POST /api/messages successfully posts a starter message to the created channel.

Thread Reply Added

guidance_reply_added_to_starter_thread

Category
state

Checker Key
POST /api/messages/{id}/reply | pathParams.id=<starter_message_id> | input.bodyParams.channelId=<created_channel_id> | output.statusCode=200

Description
Verify that guidance is added as a thread reply under the starter message.

Reject
Reject if no successful reply is posted under the starter message in the created channel.

Accept
Accept if POST /api/messages/{id}/reply successfully adds guidance under the starter message in the created channel.

Directory Checked

created_<purpose>_room_directory_checked

Category
process

Checker Key
GET /api/channels | input.queryParams.type=all | input.queryParams.limit=100 | output.statusCode=200

Description
Verify that the full channel directory is checked after the room is created.

Reject
Reject if there is no full channel directory check after the room is created.

Accept
Accept if GET /api/channels is called with type=all and limit=100 after creation.

Replacement Workflow

Use this when the recording reviews a rough draft or temporary room, creates a cleaner final channel, posts final setup content, adds a thread note, leaves the old room, and returns to the final room.

Recommended task shape:

I'm replacing a rough Slack room that was only meant for testing with a cleaner public space the team can keep using. Please use <draft_channel_name> as the draft check, create a public replacement called <final_channel_name>, confirm it appears in the channel list, post the official opening message there, add the rollout note in the opening-message thread, review that thread, leave the old draft room, and open <final_channel_name> again afterward so the final space stays active.

Core rubrics:

<draft>_reviewed_before_replacement
process
GET /api/messages | input.queryParams.channelId=<draft_channel_id> | input.queryParams.limit=50 | output.statusCode=200

<final>_created_as_public_channel
state
POST /api/channels | input.bodyParams.name=<final_channel_name> | input.bodyParams.isPrivate=false | output.statusCode=201

<final>_opening_message_posted
state
POST /api/channels | input.bodyParams.name=<final_channel_name> -> POST /api/messages | input.bodyParams.channelId=<final_channel_id> | output.statusCode=200

<final>_rollout_note_thread_reply_posted
state
POST /api/channels | input.bodyParams.name=<final_channel_name> -> POST /api/messages | input.bodyParams.channelId=<final_channel_id> -> POST /api/messages/{id}/reply | pathParams.id=<opening_message_id> | input.bodyParams.channelId=<final_channel_id> | output.statusCode=200

<draft>_left_after_replacement
state
GET /api/messages | input.queryParams.channelId=<draft_channel_id> -> POST /api/channels/{id}/leave | pathParams.id=<draft_channel_id> | output.statusCode=200

<final>_remains_active_after_draft_cleanup
process
POST /api/channels/{id}/leave | pathParams.id=<draft_channel_id> -> GET /api/messages | input.queryParams.channelId=<final_channel_id> | input.queryParams.limit=50 | output.statusCode=200

Multi-Channel Cleanup

Use this when the recording creates more than one public channel and only some channels are left/cleaned up.

Rules:

• Keep every recorded channel name exactly as it appears in POST /api/channels.
• Use separate placeholders per channel: <planning_channel_id>, <qa_channel_id>, <final_review_channel_id>, etc.
• If QA is left and planning is reopened, task and rubric must say QA is left and planning remains active.
• Do not write that both channels were removed unless both have actual cleanup endpoints.

Temporary Cleanup

Use this when the recording contains cleanup endpoints.

Endpoint wording:

POST /api/channels/{id}/leave  -> leave the temporary/stale channel
POST /api/messages/clear       -> clear the temporary channel messages
archive/delete endpoint        -> archive/delete only if the exact endpoint exists

Task wording:

I'm wrapping up <project/event>. The important notes have already been documented in <wiki/report/recap>. Please verify the recorded room, add or review the final note if present, and leave/clear the temporary space using the recorded cleanup action.

Judge Failure Fix Checklist

When the judge gives feedback, fix in this order:

1. If it says trajectory uses inconsistent IDs, remove old-channel calls from task groups and ensure later checks target <created_channel_id>.
2. If it says a rubric is contradictory, rewrite Reject and Accept so they are clearly opposite.
3. If it says a required POST is missing, check that the POST request is actually placed in the correct group.
4. If it says rubric over-specifies content, remove exact content requirements unless the task explicitly requires those words.
5. If it says rubric completeness is weak, add missing state/process coverage before trying to improve task complexity.
6. If it says task complexity is low, only change the task if the user allows it. Otherwise do not chase this score.