Add RL design doc with beginner-friendly explanations

Document covers Double DQN, Actor-Critic, network architecture,
training loop, API endpoints, and hyperparameters. Each section
includes accessible explanations for newcomers to RL concepts.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

App_RL_Design_Doc.md

@@ -0,0 +1,178 @@
+# RL Text Classification Agent - Design Document
+
+## Overview
+
+This application implements a **Double DQN with Actor-Critic** approach for text classification. It classifies user messages into three actions: `TRIP`, `GITHUB`, or `MAIL`.
+
+> **For Beginners:** Think of this as a smart assistant that reads your message and decides what you want to do - book a trip, do something on GitHub, or send an email. Instead of using fixed rules, it learns from examples like a human would.
+
+---
+
+## Architecture
+
+```
+User Message → DistilBERT Encoder → State (768-dim) → RL Agent → Action + Confidence
+```
+
+> **For Beginners:** The flow is simple: your text message gets converted into numbers (768 of them) by a pre-trained language model. These numbers capture the meaning of your message. Then our RL agent looks at these numbers and picks the best action.
+
+---
+
+## Libraries
+
+| Library | Purpose |
+|---------|---------|
+| `torch` | Neural network framework |
+| `transformers` | Text encoding (DistilBERT) |
+| `FastAPI` | REST API server |
+| `pydantic` | Request/response validation |
+
+> **For Beginners:** These are the main tools we use. PyTorch builds the brain (neural networks), Transformers helps understand text, FastAPI creates a web server so other apps can talk to ours, and Pydantic makes sure data is in the right format.
+
+---
+
+## RL Concepts Used
+
+> **For Beginners:** Reinforcement Learning (RL) is like training a dog - the agent tries actions, gets rewards for good ones, and learns to make better choices over time. Below are the specific techniques we use.
+
+### Double DQN
+Separates action selection from evaluation to reduce overestimation:
+```python
+# Select action with online network
+best_actions = self.q_net(next_states).argmax(dim=1)
+# Evaluate with target network
+next_q_values = self.target_q_net(next_states).gather(1, best_actions)
+```
+
+> **For Beginners:** Regular DQN tends to be overconfident about how good actions are. Double DQN uses two networks - one picks the action, another judges it. It's like having a friend double-check your decisions to avoid being too optimistic.
+
+### Actor-Critic
+- **Critic (Q-Network)**: Estimates action values
+- **Actor (Policy Network)**: Outputs action probabilities
+
+> **For Beginners:** Imagine a movie set - the Actor performs actions, while the Critic scores how good they were. The Actor learns to do better based on the Critic's feedback. Together, they improve faster than either would alone.
+
+### Soft Target Update
+Gradual target network updates for stability:
+```python
+target_param = tau * param + (1 - tau) * target_param  # tau=0.005
+```
+
+> **For Beginners:** Instead of suddenly copying all knowledge to the target network (which can cause instability), we blend in just 0.5% of the new knowledge each time. It's like slowly adjusting to new information rather than making sudden changes.
+
+### Entropy Regularization
+Encourages exploration by penalizing confident policies:
+```python
+entropy = -(probs * log_probs).sum(dim=-1).mean()
+policy_loss = -advantage_weighted_loss - 0.05 * entropy
+```
+
+> **For Beginners:** We don't want the agent to become too stubborn and always pick the same action. Entropy measures "randomness" - by rewarding some entropy, we encourage the agent to stay open-minded and keep exploring different options.
+
+### Epsilon-Greedy Exploration
+During training: random actions with probability ε (decays from 1.0 → 0.05)
+
+> **For Beginners:** At the start, the agent picks random actions 100% of the time (exploring). As training progresses, it gradually shifts to using what it learned, eventually only being random 5% of the time. It's like a new employee who tries everything at first, then settles into what works.
+
+### Confidence Scoring
+Combines entropy and probability for uncertainty estimation:
+```python
+confidence = (1 - entropy/max_entropy) * raw_probability
+```
+
+> **For Beginners:** The agent tells us how sure it is about its decision. If it's torn between options (high entropy) or the chosen action has low probability, confidence drops. This helps us know when to trust the agent vs. when to ask a human.
+
+### Outlier Detection
+Uses cosine similarity to class centroids to reject out-of-distribution inputs.
+
+> **For Beginners:** If someone asks "What's the weather?" (not TRIP, GITHUB, or MAIL), the agent shouldn't guess. We measure how similar the input is to known categories - if it's too different from anything we trained on, we return "NONE" instead of a wrong guess.
+
+---
+
+## Network Architecture
+
+> **For Beginners:** Neural networks are layers of math operations stacked together. Data flows through each layer, getting transformed until we get our final answer. Below are our two networks - one decides what to do (Policy), the other evaluates how good actions are (Q-Network).
+
+**Policy Network:**
+```
+Linear(768→128) → LayerNorm → ReLU → Dropout(0.1) →
+Linear(128→128) → LayerNorm → ReLU → Dropout(0.1) →
+Linear(128→3) → Softmax
+```
+
+> **For Beginners:** Takes the 768 numbers from DistilBERT, shrinks them to 128, processes them, then outputs 3 probabilities (one for each action). Dropout randomly ignores some neurons during training to prevent overfitting. Softmax ensures probabilities sum to 1.
+
+**Q-Network:**
+```
+Linear(768→128) → LayerNorm → ReLU →
+Linear(128→128) → LayerNorm → ReLU →
+Linear(128→3)
+```
+
+> **For Beginners:** Similar structure but outputs raw scores (Q-values) for each action instead of probabilities. No Dropout here because we want stable value estimates.
+
+---
+
+## Training Loop
+
+> **For Beginners:** Training is how the agent learns. We show it examples, tell it what's right/wrong, and it adjusts its internal numbers (weights) to do better next time.
+
+1. Encode texts with DistilBERT (frozen)
+2. For each batch:
+   - Create positive examples (correct action → reward +1)
+   - Create negative examples (wrong action → reward -1)
+   - Update Q-network via TD learning
+   - Update policy via advantage-weighted loss
+   - Soft update target network
+3. Decay epsilon
+
+> **For Beginners:** We freeze DistilBERT (don't change it) because it's already great at understanding text. We only train our smaller RL networks. "Frozen" means we use it as a fixed tool, like using a calculator without modifying it.
+
+---
+
+## API Endpoints
+
+> **For Beginners:** APIs let other programs talk to ours over the internet. Think of endpoints as different "phone numbers" - you call the right one depending on what you need.
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | Check model status |
+| `/action` | POST | Classify message → action + score |
+
+**Request:**
+```json
+{"message": "Book a flight to Paris"}
+```
+
+**Response:**
+```json
+{"action": "TRIP", "score": 0.87}
+```
+
+> **For Beginners:** Send a message, get back an action and confidence score. The score (0.87 = 87%) tells you how confident the agent is. If confidence is too low, you'll get "NONE" instead.
+
+---
+
+## Key Hyperparameters
+
+> **For Beginners:** Hyperparameters are settings we choose before training - they control how the learning happens. Think of them as recipe ingredients that affect the final result.
+
+| Parameter | Value | What it means |
+|-----------|-------|---------------|
+| Learning rate | 1e-3 | How big each learning step is (0.001) |
+| Gamma (discount) | 0.95 | How much future rewards matter vs immediate |
+| Batch size | 64 | Examples processed together per step |
+| Epochs | 50 | Times we go through the entire dataset |
+| Confidence threshold | 0.6 | Below this, return "NONE" |
+| Distance threshold | 0.93 | Similarity needed to not be an outlier |
+
+---
+
+## In General
+
+Our OS assistant app communicates with the RL agent. App informs agent what it's on the screen at the moment and RL knows what action should be performed. For the demo we introduced three types of actions:
+- TRIP
+- GITHUB
+- MAIL
+
+> **For Beginners:** This is the big picture - imagine an assistant watching your screen. When you type something, it figures out your intent and triggers the right action automatically. It's like having a smart helper that knows whether you want to travel, code, or email just from what you say.