USER_GUIDE.md

AI Grading Assistant — TA User Guide

Course: CAI 3801 / ISM 6145 — AI for Analytics
Model: Claude Sonnet 4.6 (frontier) or Qwen2.5 (local, free)

---

Before You Start (One-time Setup)

1. Install dependencies

pip install -r requirements.txt

2. Add your API key
Create a file named .env in the project folder and add:

ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxxxxxx

Get your key at: console.anthropic.com

3. Verify it works

python calibrated_grader.py --help

---

Grading a New Assignment

Step 1 — Prepare your files

Collect all student submissions into one folder:

submissions/
  John_Smith.pdf
  Jane_Doe.docx
  Alex_Brown.pdf
  ...

Submissions can be PDF or Word documents. File names become the student identifiers.

Step 2 — Run the grader

Option A — Simple interactive mode (recommended for first-time users)

python calibrated_grader.py

It will ask you for each file path one at a time. You can drag-and-drop files into the terminal to fill the path automatically.

Option B — Command line (faster once you know the paths)

python calibrated_grader.py \
  --instructions  "path/to/assignment_instructions.pdf" \
  --rubric        "path/to/rubric.pdf" \
  --submissions   "path/to/submissions/" \
  --output        "path/to/output/" \
  --assignment    "CAI 3801 Lab 02"

Step 3 — Review results

Open the visual dashboard:

python dashboard.py --input path/to/output/all_results.json

This shows each student's score, criterion breakdown, and feedback. Spot-check 3–5 students to confirm the grading looks reasonable before releasing grades.

---

Output Files

After grading, the output folder contains:

File	What it is
all_results.json	All scores and feedback in structured format
Student_001.json	Individual result per student
gold_standard_template.csv	Fill this in with your human scores for accuracy evaluation

---

Evaluating Accuracy (Optional but Recommended)

To measure how closely the AI matches human grading:

1. Open gold_standard_template.csv and fill in your scores for each student.

2. Run the evaluator:

python evaluator.py \
  --human path/to/output/gold_standard_template.csv \
  --ai    path/to/output/all_results.json

3. You will see:

• MAE — average point difference between AI and human
• Agreement % — how often they match within ±2 points
• QWK — Quadratic Weighted Kappa (published benchmark: 0.68)

---

Adjusting for Your Assignment

Calibration offset

The grader applies a +3.5 point offset by default to match observed human TA generosity. Adjust with:

python calibrated_grader.py --offset 2.0 ...

• Increase if AI grades feel too low
• Decrease if AI grades feel too high
• Set to 0 to use raw AI scores

Using a pre-defined criteria file

If you already have the rubric criteria saved (e.g. from a previous run), skip the parsing step:

python calibrated_grader.py --criteria lab01_data/source_files/lab01_criteria.json ...

---

Using the Free Local Model (No API Cost)

If you don't have API credits, you can grade for free using a local model.

1. Install Ollama: ollama.com

2. Pull a lightweight model:

ollama pull qwen2.5:3b

3. Start Ollama (keep this terminal open):

ollama serve

4. Run the grader with --provider ollama:

python calibrated_grader.py \
  --provider  ollama \
  --model     qwen2.5:3b \
  --criteria  path/to/criteria.json \
  --submissions path/to/submissions/ \
  --output    path/to/output/

Note: Local models are less consistent than Claude. Always spot-check results before releasing grades.

---

Comparing Two Models

To generate a side-by-side comparison report (useful for professor demos):

python model_comparison.py \
  --frontier  path/to/frontier_results/all_results.json \
  --local     path/to/local_results/all_results.json \
  --output    path/to/comparison/

Opens as an HTML file — share it with your professor directly.

---

Privacy and FERPA

• Student names and IDs are stripped locally before any text is sent to Claude
• The API only ever receives anonymized content
• With --provider ollama, nothing leaves your machine at all

---

Quick Reference

Task	Command
Grade an assignment	python calibrated_grader.py
View dashboard	python dashboard.py --input output/all_results.json
Evaluate accuracy	python evaluator.py --human scores.csv --ai output/all_results.json
Compare two models	python model_comparison.py --frontier f.json --local l.json
Grade for free (local)	Add --provider ollama --model qwen2.5:3b

---

Estimated Cost (Claude Sonnet 4.6)

Class size	Cost per assignment
15 students	~$0.45
30 students	~$0.90
100 students	~$3.00 (50% discount via Batch API)

---

Getting Help

Contact: swethasingireddy8@gmail.com
Project repo: /Users/rahulnaidu/Desktop/ta_grader/