# API Grouping and Rubric Standard This document records the current format for Slack-style API grouping and rubric writing. ## Goal When analyzing a new recording zip, output two things: 1. API groups that can be copied into the grouping UI. 2. Rubrics that are concrete, non-overlapping, and aligned with the recorded trajectory. The grouping output must make each API easy to identify in the UI while avoiding hard-coded IDs in rubric logic. ## Source Of Truth Always serve the recorded path. - If the recording creates a misspelled channel name, keep the recorded name in the task, groups, and rubrics. - If the recording leaves a stale or old channel, write `leave the room/channel`; do not rewrite it as delete/archive/remove unless that exact endpoint exists. - If the recording clears messages, write `clear messages`; do not imply the channel was deleted. - Do not use rerecording as the primary fix. Rewrite the task and rubric to match the actual trajectory first. - Do not add actions just to improve Task Complexity. Complexity can remain below 2 when the trajectory is narrow. ## API Group Display Format Use this format for each grouped call: ```text # METHOD /api/path/{ID} URL: /api/path/{ID}?query=value Actual ID: real-uuid-if-needed Body: key=value, key=value ``` Rules: - Replace UUIDs in the displayed API path with `{ID}`. - Keep the real UUID only under `Actual ID` when it helps locate the exact request. - Keep query parameters in the `URL` line because they are important for identifying calls such as `channelId`, `limit`, `type`, and `offset`. - Include request body fields only when they are task-relevant, such as `name`, `isPrivate`, `channelId`, and `content`. - Use stable, descriptive group names that show both the endpoint and the task purpose. Example: ```text #5 Add Review Guidelines Thread #44 GET /api/messages/{ID}/thread-replies URL: /api/messages/{ID}/thread-replies?limit=20&offset=0 Actual messageId: 10cfbbe6-0875-4577-8849-0a4563b840c3 #45 POST /api/messages/{ID}/reply URL: /api/messages/{ID}/reply Body: channelId=586c3969-e33e-4b64-b3f9-aed248c776f7, content=review guidelines #46 GET /api/messages/{ID}/thread-replies URL: /api/messages/{ID}/thread-replies?limit=20&offset=0 Actual messageId: 10cfbbe6-0875-4577-8849-0a4563b840c3 ``` ## Group Naming Pattern Prefer this naming style: ```text #1 authentication #2 Create Public Room #3 Open Created Room #4 Post Starter Message #5 Add Guidance Thread #6 Verify Created Room ``` For each group, include only task-relevant calls. If a group is only a UI precondition, name it clearly, for example `authentication`. ## ID Handling In group display: - API path: use `{ID}`. - URL line: use `{ID}` in the path, but keep query params. - Actual ID line: keep the real UUID if needed for locating the request. - Body line: keep real `channelId` only if needed to confirm the request targets the created entity. In rubrics: - Do not hard-code runtime UUIDs unless the task itself gives a stable identifier. - Use placeholders: - `` - `` - `` - `` - Tie placeholders to earlier evidence in descriptions and acceptance criteria. Good: ```text GET /api/messages | input.queryParams.channelId= | input.queryParams.limit=50 | output.statusCode=200 ``` Avoid: ```text GET /api/messages | input.queryParams.channelId=586c3969-e33e-4b64-b3f9-aed248c776f7 ``` ## Noise API Rules Usually exclude these from task groups unless the task explicitly asks for them: ```text GET /api/users/unread-counts GET /api/conversations/dm/active GET /api/users/starred GET /api/users GET /api/conversations/read-position POST /api/messages/read GET /api/messages/{ID}/reactions ``` Authentication can be grouped separately, but do not create a rubric for login/auth unless the task explicitly asks for authentication. ## Rubric Format Use exactly this structure: ```text rubric_name Category state | process | outcome Must Pass true | false Checker Key METHOD /api/path | input.bodyParams.field=value | input.queryParams.field=value | pathParams.id= | output.statusCode=200 Description One sentence describing the task-relevant behavior being verified. Reject Reject if the required evidence is missing, targets the wrong created entity, fails, or contradicts the task. Accept Accept if the required API evidence is present, successful, and targets the correct created entity. ``` ## Category Rules Use `state` when the rubric verifies a durable system change: ```text POST /api/channels POST /api/messages POST /api/messages/{ID}/reply PUT /api/auth/profile ``` Use `process` when the rubric verifies an action/view/check: ```text GET /api/channels GET /api/messages GET /api/channels/{ID}/members GET /api/messages/{ID}/thread-replies ``` Use `outcome` only when the task requires a final user-facing response, summary, recommendation, or report. Do not label API call sequence checks as `outcome`. ## Rubric Quality Rules - Use at least 4 rubrics when possible. - Set `must_pass=true` for core state-changing requirements that define task success, such as `POST /api/channels`, `POST /api/messages`, `POST /api/messages/{ID}/reply`, `POST /api/channels/{ID}/leave`, `POST /api/messages/clear`, and requested profile updates. - Set `must_pass=false` for supporting read/process checks such as directory lookup, member review, message view opening, and thread review unless the task is specifically about that verification. - Do not duplicate the same behavior across multiple rubric items. - Do not make `Reject` and `Accept` indistinguishable. - Do not require exact message content unless the recorded data actually supports it and the task explicitly needs it. - If message content is weak or generic, score only that a starter/guidance message was posted to the created entity. - Make later checks target the created entity using placeholders rather than hard-coded IDs. - Directory checks should include `type=all` and `limit=100` when present. - Message view checks should include `channelId=` and `limit=50` when present. - Thread checks should bind replies to `` or ``. ## Common Slack Rubric Templates ### Public Channel Created ```text _room_created_as_public_channel Category state Checker Key POST /api/channels | input.bodyParams.name= | input.bodyParams.isPrivate=false | output.statusCode=201 Description Verify that the requested public room is created with the correct name. Reject Reject if there is no successful POST /api/channels call creating , or if the created channel is private. Accept Accept if POST /api/channels successfully creates with isPrivate=false. ``` ### Created Room Opened ```text created__room_opened Category process Checker Key GET /api/messages | input.queryParams.channelId= | input.queryParams.limit=50 | output.statusCode=200 Description Verify that the created room is opened into its message view. Reject Reject if messages are not fetched for the channel created by POST /api/channels. Accept Accept if GET /api/messages is called for the created channel with limit=50 and returns 200. ``` ### Members Checked ```text created__room_members_checked Category process Checker Key GET /api/channels/{id}/members | pathParams.id= | output.statusCode=200 Description Verify that the member list for the created room is checked. Reject Reject if the member list is not fetched for the channel created by POST /api/channels. Accept Accept if GET /api/channels/{id}/members is called for the created channel and returns 200. ``` ### Starter Message Posted ```text starter_message_posted_in_created__room Category state Checker Key POST /api/messages | input.bodyParams.channelId= | output.statusCode=200 Description Verify that a starter message is posted in the created room. Reject Reject if no successful POST /api/messages is sent to the channel created by POST /api/channels. Accept Accept if POST /api/messages successfully posts a starter message to the created channel. ``` ### Thread Reply Added ```text guidance_reply_added_to_starter_thread Category state Checker Key POST /api/messages/{id}/reply | pathParams.id= | input.bodyParams.channelId= | output.statusCode=200 Description Verify that guidance is added as a thread reply under the starter message. Reject Reject if no successful reply is posted under the starter message in the created channel. Accept Accept if POST /api/messages/{id}/reply successfully adds guidance under the starter message in the created channel. ``` ### Directory Checked ```text created__room_directory_checked Category process Checker Key GET /api/channels | input.queryParams.type=all | input.queryParams.limit=100 | output.statusCode=200 Description Verify that the full channel directory is checked after the room is created. Reject Reject if there is no full channel directory check after the room is created. Accept Accept if GET /api/channels is called with type=all and limit=100 after creation. ``` ### Replacement Workflow Use this when the recording reviews a rough draft or temporary room, creates a cleaner final channel, posts final setup content, adds a thread note, leaves the old room, and returns to the final room. Recommended task shape: ```text I'm replacing a rough Slack room that was only meant for testing with a cleaner public space the team can keep using. Please use as the draft check, create a public replacement called , confirm it appears in the channel list, post the official opening message there, add the rollout note in the opening-message thread, review that thread, leave the old draft room, and open again afterward so the final space stays active. ``` Core rubrics: ```text _reviewed_before_replacement process GET /api/messages | input.queryParams.channelId= | input.queryParams.limit=50 | output.statusCode=200 ``` ```text _created_as_public_channel state POST /api/channels | input.bodyParams.name= | input.bodyParams.isPrivate=false | output.statusCode=201 ``` ```text _opening_message_posted state POST /api/channels | input.bodyParams.name= -> POST /api/messages | input.bodyParams.channelId= | output.statusCode=200 ``` ```text _rollout_note_thread_reply_posted state POST /api/channels | input.bodyParams.name= -> POST /api/messages | input.bodyParams.channelId= -> POST /api/messages/{id}/reply | pathParams.id= | input.bodyParams.channelId= | output.statusCode=200 ``` ```text _left_after_replacement state GET /api/messages | input.queryParams.channelId= -> POST /api/channels/{id}/leave | pathParams.id= | output.statusCode=200 ``` ```text _remains_active_after_draft_cleanup process POST /api/channels/{id}/leave | pathParams.id= -> GET /api/messages | input.queryParams.channelId= | input.queryParams.limit=50 | output.statusCode=200 ``` ### Multi-Channel Cleanup Use this when the recording creates more than one public channel and only some channels are left/cleaned up. Rules: - Keep every recorded channel name exactly as it appears in `POST /api/channels`. - Use separate placeholders per channel: ``, ``, ``, etc. - If QA is left and planning is reopened, task and rubric must say QA is left and planning remains active. - Do not write that both channels were removed unless both have actual cleanup endpoints. ### Temporary Cleanup Use this when the recording contains cleanup endpoints. Endpoint wording: ```text POST /api/channels/{id}/leave -> leave the temporary/stale channel POST /api/messages/clear -> clear the temporary channel messages archive/delete endpoint -> archive/delete only if the exact endpoint exists ``` Task wording: ```text I'm wrapping up . The important notes have already been documented in . Please verify the recorded room, add or review the final note if present, and leave/clear the temporary space using the recorded cleanup action. ``` ## Judge Failure Fix Checklist When the judge gives feedback, fix in this order: 1. If it says trajectory uses inconsistent IDs, remove old-channel calls from task groups and ensure later checks target ``. 2. If it says a rubric is contradictory, rewrite `Reject` and `Accept` so they are clearly opposite. 3. If it says a required POST is missing, check that the POST request is actually placed in the correct group. 4. If it says rubric over-specifies content, remove exact content requirements unless the task explicitly requires those words. 5. If it says rubric completeness is weak, add missing state/process coverage before trying to improve task complexity. 6. If it says task complexity is low, only change the task if the user allows it. Otherwise do not chase this score.