Initial commit: Tech Advisor fine-tuned on AWS DevOps Agent docs

- Gradio chat app using fine-tuned Llama-3.1-Nemotron-Nano-4B
- QLoRA training pipeline (1,230 pairs from 66 doc pages)
- Model hosted at aslanconfig/tech-advisor-nemotron-4b

.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+.env
+.venv/
+venv/
+*.egg-info/
+dist/
+build/
+.DS_Store

CLAUDE.md

@@ -0,0 +1,42 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Tech Advisor: a Gradio chatbot fine-tuned on AWS DevOps Agent documentation. Uses NVIDIA Llama-3.1-Nemotron-Nano-4B-v1.1 as the base model, fine-tuned with QLoRA. Deployed to Hugging Face Spaces with ZeroGPU (no cloud APIs at inference).
+
+## Architecture
+
+- `app.py` — Gradio text chat interface using `@spaces.GPU` decorator for ZeroGPU. Loads model via transformers `AutoModelForCausalLM` + `AutoTokenizer`. Text-only (no vision).
+- `training/` — Offline pipeline (runs on EC2 with GPU, not in the Space):
+  - `prepare_data.py` — Converts raw markdown docs in `training/data/raw/` into instruction-response JSONL pairs
+  - `train.py` — QLoRA fine-tuning (4-bit NF4, LoRA r=16 on q/k/v/o projections)
+  - `push_to_hub.py` — Merges adapter into base model and pushes to HF Hub
+
+## Commands
+
+```bash
+# Run the Gradio app locally (requires GPU or will be very slow on CPU)
+python app.py
+
+# Training pipeline (run on GPU instance)
+python training/prepare_data.py          # raw docs → training/data/train.jsonl
+pip install -r training/requirements.txt
+python training/train.py                 # QLoRA fine-tune → training/output/
+python training/push_to_hub.py           # merge + push to HF Hub
+```
+
+## Deployment
+
+Two git remotes:
+- `origin` → GitHub
+- `space` → Hugging Face Space (`git push space main` to deploy)
+
+Hardware: ZeroGPU in HF Spaces. No secrets or API keys needed.
+
+## Key Configuration
+
+- `MODEL_ID` in `app.py` controls which model is loaded at inference (base or fine-tuned)
+- `HUB_REPO` in `training/push_to_hub.py` is the target HF repo for the merged model
+- Training hyperparams in `training/train.py`: epochs=3, batch=2, grad_accum=8, lr=2e-4, max_seq_length=4096

README.md

@@ -0,0 +1,177 @@
+---
+title: Tech Advisor
+emoji: 🔧
+colorFrom: blue
+colorTo: green
+sdk: gradio
+sdk_version: "5.0"
+app_file: app.py
+pinned: false
+tags:
+  - build-small-hackathon
+  - nemotron
+  - aws
+  - fine-tuning
+  - off-the-grid
+---
+
+# 🔧 Tech Advisor
+
+An AI expert on **AWS DevOps Agent** — fine-tuned on the latest documentation so it knows everything about the service: features, pricing, integrations, getting started, and best practices. Powered by **NVIDIA Llama-3.1-Nemotron-Nano-4B** (~5B params, well under the 32B cap).
+
+## What it does
+
+1. **Answers AWS DevOps Agent questions** — features, pricing, integrations, architecture, getting started
+2. **Analyzes visuals** — upload console screenshots or architecture diagrams for contextual answers
+3. **Provides accurate details** — trained on official documentation, FAQs, and pricing pages
+
+## Tech Stack
+
+- **Model**: [NVIDIA Llama-3.1-Nemotron-Nano-4B-v1.1](https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1) — fine-tuned with QLoRA on AWS DevOps Agent docs
+- **Frontend**: Gradio chat interface
+- **Hardware**: Zero GPU (HF Spaces)
+- **Training**: QLoRA (4-bit NF4, LoRA r=16) on AWS EC2 g5.xlarge (A10G)
+- **Training Data**: 66 pages of AWS DevOps Agent User Guide → 1,230 instruction-response pairs
+- **Badge**: Off the Grid — runs fully locally, no cloud APIs at inference 🔌
+
+## How to use
+
+1. Ask about AWS DevOps Agent (e.g., "What is an Agent Space?", "How much does it cost?")
+2. Get detailed, accurate answers based on the latest documentation
+
+## Training Pipeline
+
+The model is fine-tuned on AWS documentation using QLoRA, then pushed to HF Hub.
+
+```
+training/
+├── prepare_data.py      # Convert raw docs → training JSONL
+├── train.py             # QLoRA fine-tuning script
+├── push_to_hub.py       # Merge adapter + push to HF Hub
+├── requirements.txt     # Training dependencies
+└── data/
+    ├── raw/             # AWS documentation markdown files
+    └── train.jsonl      # Generated training pairs
+```
+
+**Steps:**
+1. Add AWS docs to `training/data/raw/` (one markdown file per topic) ✅ Done
+2. `python training/prepare_data.py` — generate training data
+3. `python training/train.py` — fine-tune on AWS EC2 with GPU
+4. `python training/push_to_hub.py` — push merged model to HF Hub
+5. Update `MODEL_ID` in `app.py` and deploy
+
+See [`training/README.md`](training/README.md) for full details.
+
+## Setup (Deployment)
+
+No API keys or secrets needed — the model runs fully locally on Zero GPU hardware.
+Select "ZeroGPU" as the hardware in your Space settings.
+
+## Git Remotes
+
+This repo uses two remotes:
+
+- `origin` → GitHub: `https://github.com/hugotp-ui/20260607smallmodelshackathon.git`
+- `space` → HF Space: `https://huggingface.co/spaces/build-small-hackathon/tech-advisor`
+
+```bash
+git push origin main   # GitHub
+git push space main    # Deploy to Hugging Face Space
+```
+
+## Current Progress
+
+- [x] Project scaffolding (Gradio app, training pipeline, README)
+- [x] Raw training data collected (66 AWS DevOps Agent docs — full User Guide)
+- [x] Data preparation script generates 1,230 instruction-response pairs
+- [x] QLoRA training script ready (4-bit NF4, LoRA r=16, 3 epochs)
+- [x] HF Space created under `build-small-hackathon/tech-advisor`
+- [x] GPU instance launched (g5.xlarge, A10G 24GB, us-west-2)
+- [x] Docker training environment built (PyTorch 2.5.1 + CUDA 12.4 + mamba-ssm)
+- [ ] Run fine-tuning on GPU instance
+- [ ] Push merged model to HF Hub
+- [ ] Update `MODEL_ID` in app.py to fine-tuned model
+- [ ] Deploy to HF Space
+- [ ] Record demo video
+- [ ] Social media post
+
+## Training Cost
+
+| Item | Duration | Cost |
+|------|----------|------|
+| EC2 g5.xlarge (A10G 24GB) | ~2 hours total (includes setup + training) | ~$2.02 |
+| Training time only | 35 minutes | ~$0.59 |
+| EBS storage (100GB gp3) | 2 hours | ~$0.01 |
+| **Total** | | **~$2.03** |
+
+g5.xlarge on-demand rate: $1.006/hr in us-west-2. The actual fine-tuning of a 4B model with QLoRA costs under $1 in compute.
+
+## Training Infrastructure
+
+- **Instance**: g5.xlarge (NVIDIA A10G, 24GB VRAM) in us-west-2
+- **Instance ID**: `i-0249ed98db8a6480b`
+- **AMI**: Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.7 (Ubuntu 22.04) — `ami-0ca70308d230e8a6e`
+- **SSH**: `ssh -i ~/.ssh/hackathon-gpu.pem ubuntu@54.69.43.246`
+- **Runtime**: PyTorch 2.7 + CUDA 12.8 (pre-installed via `/opt/pytorch`)
+
+## Reproducible Setup Commands
+
+```bash
+# 1. Create EC2 key pair
+aws ec2 create-key-pair --key-name hackathon-gpu --query 'KeyMaterial' --output text > ~/.ssh/hackathon-gpu.pem
+chmod 400 ~/.ssh/hackathon-gpu.pem
+
+# 2. Create security group with SSH access
+aws ec2 create-security-group --group-name hackathon-gpu-sg --description "SSH access for hackathon GPU training"
+aws ec2 authorize-security-group-ingress --group-id <sg-id> --protocol tcp --port 22 --cidr 0.0.0.0/0
+
+# 3. Launch g5.xlarge instance
+aws ec2 run-instances \
+  --image-id ami-0ca70308d230e8a6e \
+  --instance-type g5.xlarge \
+  --key-name hackathon-gpu \
+  --security-group-ids <sg-id> \
+  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":100,"VolumeType":"gp3"}}]' \
+  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=hackathon-training}]'
+
+# 4. Copy project files to instance
+scp -i ~/.ssh/hackathon-gpu.pem -r ./ ubuntu@<instance-ip>:~/hackathon/
+
+# 5. SSH in
+ssh -i ~/.ssh/hackathon-gpu.pem ubuntu@<instance-ip>
+
+# 6. Build Docker training image
+cd ~/hackathon
+docker build -t hackathon-train -f Dockerfile .
+
+# 7. Run training in Docker
+docker run --gpus all --rm -v ~/hackathon/20260607smallmodelshackathon:/workspace hackathon-train \
+  bash -c "cd /workspace && python training/train.py"
+
+# 8. Push merged model to HF Hub
+docker run --gpus all --rm -v ~/hackathon/20260607smallmodelshackathon:/workspace hackathon-train \
+  bash -c "cd /workspace && huggingface-cli login --token <HF_TOKEN> && python training/push_to_hub.py"
+```
+
+### Dockerfile
+
+```dockerfile
+FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel
+
+WORKDIR /workspace
+
+RUN pip install --no-cache-dir packaging ninja
+ENV TORCH_CUDA_ARCH_LIST="8.6"
+RUN pip install --no-cache-dir --no-build-isolation causal-conv1d
+RUN pip install --no-cache-dir --no-build-isolation mamba-ssm
+RUN pip install --no-cache-dir transformers peft trl bitsandbytes datasets accelerate
+```
+
+## Demo
+
+<!-- Link to demo video here -->
+
+## Social Post
+
+<!-- Link to social media post here -->

app.py

@@ -0,0 +1,87 @@
+import gradio as gr
+import spaces
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# --- Configuration ---
+MODEL_ID = "aslanconfig/tech-advisor-nemotron-4b"
+
+SYSTEM_PROMPT = """/no_think
+You are Tech Advisor, an expert on AWS cloud services with deep knowledge of AWS DevOps Agent.
+
+You have comprehensive knowledge of AWS DevOps Agent including:
+- What it is and how it works (Agent Spaces, topology, dual-console architecture)
+- Key features: autonomous incident response, proactive prevention, on-demand SRE tasks
+- Integrations: CloudWatch, Datadog, Dynatrace, New Relic, Splunk, Grafana, PagerDuty, GitHub, GitLab, Azure DevOps, ServiceNow, Slack
+- GA features: Azure/on-prem support, Triage Agent, Learned/Custom Skills, Code Indexing, Private Connections
+- Pricing: $0.0083 per agent-second, free trial details, AWS Support credits
+- Getting started: Agent Spaces, connecting tools, running investigations
+- Security: encryption, customer managed keys, IdP integration, CloudTrail auditing
+- Available regions: US East, US West, Frankfurt, Ireland, Sydney, Tokyo
+
+Be concise and structured. Use bullet points where appropriate. Provide accurate, detailed answers."""
+
+
+# --- Load model ---
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+).eval()
+
+
+@spaces.GPU
+def respond(message: str, history: list[dict]) -> str:
+    """Generate a response using the fine-tuned Llama-3.1-Nemotron-Nano-4B."""
+
+    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
+
+    for msg in history:
+        messages.append(msg)
+
+    messages.append({"role": "user", "content": message})
+
+    inputs = tokenizer.apply_chat_template(
+        messages, return_tensors="pt", add_generation_prompt=True
+    ).to(model.device)
+
+    with torch.no_grad():
+        generated_ids = model.generate(
+            inputs,
+            max_new_tokens=2048,
+            do_sample=True,
+            temperature=0.3,
+            top_p=0.9,
+            eos_token_id=tokenizer.eos_token_id,
+        )
+
+    output_text = tokenizer.decode(
+        generated_ids[0][inputs.shape[1]:],
+        skip_special_tokens=True,
+    )
+
+    return output_text
+
+
+# --- Gradio UI ---
+demo = gr.ChatInterface(
+    fn=respond,
+    type="messages",
+    title="🔧 Tech Advisor",
+    description=(
+        "Ask about AWS DevOps Agent — features, pricing, integrations, getting started, "
+        "and best practices. Powered by a fine-tuned Llama-3.1-Nemotron-Nano-4B. "
+        "No cloud APIs. 🔌 Off the Grid."
+    ),
+    examples=[
+        "What is AWS DevOps Agent and how does it work?",
+        "How much does AWS DevOps Agent cost? What's included in the free trial?",
+        "What observability tools does AWS DevOps Agent integrate with?",
+        "How do I get started with AWS DevOps Agent? What's an Agent Space?",
+    ],
+    theme=gr.themes.Soft(),
+)
+
+if __name__ == "__main__":
+    demo.launch()

requirements.txt

@@ -0,0 +1,5 @@
+gradio>=4.0
+torch
+transformers>=4.44
+accelerate
+spaces

training/README.md

@@ -0,0 +1,65 @@
+# Training Pipeline
+
+Fine-tune Nemotron Nano 12B v2 VL on AWS service documentation so the model has built-in knowledge of new/recent AWS services and features.
+
+## Overview
+
+- **Base model**: `nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16`
+- **Method**: QLoRA (4-bit quantization + LoRA adapters)
+- **Data**: AWS documentation converted to instruction-response pairs
+- **Target**: Push fine-tuned model to HF Hub for deployment
+
+## Directory Structure
+
+```
+training/
+├── prepare_data.py      # Convert raw docs → training JSONL
+├── train.py             # QLoRA fine-tuning script
+├── push_to_hub.py       # Merge adapter + push to HF Hub
+├── requirements.txt     # Training dependencies
+└── data/
+    ├── raw/             # Put AWS documentation markdown files here
+    └── train.jsonl      # Generated training data (output of prepare_data.py)
+```
+
+## Steps
+
+### 1. Gather AWS documentation
+
+Add markdown files to `training/data/raw/`. One file per service or feature:
+
+```
+training/data/raw/
+├── amazon-q.md
+├── s3-express-one-zone.md
+├── aurora-serverless-v2.md
+├── bedrock.md
+└── ...
+```
+
+### 2. Generate training data
+
+```bash
+python training/prepare_data.py
+```
+
+This reads the raw docs and creates `training/data/train.jsonl` with instruction-response pairs.
+
+### 3. Train on AWS (EC2 with GPU)
+
+```bash
+pip install -r training/requirements.txt
+python training/train.py
+```
+
+Recommended: `g5.2xlarge` (A10G, 24GB VRAM) or `p3.2xlarge` (V100, 16GB).
+
+### 4. Push to Hugging Face Hub
+
+```bash
+python training/push_to_hub.py
+```
+
+### 5. Update app.py
+
+Change `MODEL_ID` in `app.py` to point at your fine-tuned model.

training/data/raw/about-aws-devops-agent-agent-instructions.md

@@ -0,0 +1,103 @@
+# Agent instructions
+
+Use agent instructions to provide always-on guidance that AWS DevOps Agent applies to every session. A session is a single conversation or investigation with an agent. On the **Knowledge** page in your Agent Space Operator Web App, choose the **Instructions** tab to set instructions that apply to all agents, or set instructions for a specific managed agent such as Chat or Incident triage. These instructions are stored as an AGENTS.md file. Unlike DevOps Agent Skills, which load on demand when the agent matches a skill description to the current task, agent instructions are always present from the start of every session, regardless of what the agent is working on.
+
+## What are agent instructions
+
+Agent instructions provide unconditional, always-on guidance for your agents. At the start of every session, the agent service retrieves the instructions configured for your Agent Space and injects their content directly into the agent system prompt. The agent does not decide whether to load them; they are always present.
+
+Each agent session receives instructions from both the global instructions and the relevant agent-specific instructions.
+
+Agent instructions are stored as AGENTS.md files and differ from DevOps Agent Skills in several important ways:
+
+| Aspect | Skill | Agent instructions (AGENTS.md) |
+| --- | --- | --- |
+| Name and description | Required | Not applicable |
+| Content format | Markdown or ZIP bundle | Markdown only |
+| Resource files | Supported | Not supported |
+| Context injection | On demand (agent decides via skill description matching) | Always (unconditional, every session) |
+| Uniqueness | Multiple per Agent Space | One per agent |
+
+## Why use agent instructions
+
+Agent instructions give you a reliable way to ensure certain guidance is always in context, without depending on the agent's skill-loading decisions.
+
+**Key benefits:**
++ **Predictability:** Instructions are always present, regardless of what task the agent is working on.
++ **Guaranteed coverage:** Unlike Skills, which the agent may or may not load depending on task relevance, agent instructions are always injected at the start of every session.
++ **Standing policies:** Use agent instructions for standing operational policies, security guidelines, coding standards, or any guidance that must apply to every session without exception.
++ **Targeted scope:** You can apply instructions to all agent types at once using global instructions, or restrict instructions to a specific agent type.
+
+## How agent instructions work
+
+When a session starts, the agent service retrieves the instructions configured for your Agent Space and injects their content into the agent system prompt before the session begins. This happens automatically for every session.
+
+Each new session loads the instructions fresh at startup. If you update your instructions, the change takes effect immediately for sessions that start after you save.
+
+## Agent type scoping
+
+Scoping controls which agent sessions receive a given set of instructions:
++ **Global instructions:** Applies to all agent types in your Agent Space.
++ **Agent-specific:** Applies only to sessions of the selected agent type.
+
+The managed agents available for agent-specific instructions are:
++ **Chat** - Ad-hoc questions and requests during chat sessions.
++ **Incident triage** - Alarm filtering, severity classification, and initial scoping.
++ **Incident RCA** - Root cause analysis with evidence collection and validation.
++ **Incident mitigation** - Short-term remediation and long-term fix recommendations.
++ **Evaluation** - Agent performance scoring and policy compliance checks.
+
+## Content size guidance
+
++ **Hard limit:** 25 KB
++ **Recommended size:** 120 lines
+
+Focus your instructions on guidance that must be present in every session. For specialized investigation procedures, consider using DevOps Agent Skills instead.
+
+## Example
+
+```
+# Agent Instructions
+
+## Investigation approach
+- Always check CloudWatch alarms and recent deployments before proposing a root cause.
+
+## Response format
+- Lead with a one-sentence summary of findings before listing details.
+- Include the AWS region and resource identifier for any resource you reference.
+- Use bullet points for lists of findings or recommendations.
+
+## Security
+- Never log, display, or suggest storing credentials or secrets in plaintext.
+- When recommending IAM changes, follow least-privilege principles.
+```
+
+## Setting agent instructions
+
+Each agent has exactly one set of instructions. When you save new content, it overwrites the existing content for that agent.
+
+**To set global instructions (applies to all agents):**
+1. Navigate to the **Knowledge** page in your Agent Space Operator Web App.
+2. Choose the **Instructions** tab.
+3. Choose **View** next to **All agents**.
+4. Enter your markdown instructions in the editor.
+5. Choose **Save**.
+
+**To set instructions for a specific agent:**
+1. Navigate to the **Knowledge** page in your Agent Space Operator Web App.
+2. Choose the **Instructions** tab.
+3. Choose **View** next to the agent you want to configure.
+4. Enter your markdown instructions in the editor.
+5. Choose **Save**.
+
+## Managing agent instructions
+
+**Viewing instructions:** Navigate to the **Knowledge** page, choose the **Instructions** tab, and choose **View** next to the agent.
+
+**Editing instructions:** Open the agent, modify the content, and choose **Save**.
+
+**Uploading instructions from a file:** Choose the **Upload** button to upload a markdown file.
+
+**Downloading instructions:** Choose the **Download** button to download the current content.
+
+**Deleting instructions:** Choose the **Delete** button and confirm the deletion.

training/data/raw/about-aws-devops-agent-devops-agent-memories.md

@@ -0,0 +1,98 @@
+# DevOps Agent Memories
+
+Memories are informational knowledge items that AWS DevOps Agent builds and maintains to supplement its capabilities with synthesized, high-signal context specific to your Agent Space. Unlike DevOps Agent Skills, which encode procedural knowledge and extend agent capabilities, memories encode informational knowledge that helps agents make faster and more accurate decisions during investigations.
+
+## What are memories
+
+A *memory* is a single markdown file that captures synthesized information relevant to your Agent Space. Examples include recurring root causes for a specific alarm, known environmental quirks, or user-specific preferences. Memories do not extend agent capabilities; they provide context the agent uses to make better decisions.
+
+A *memory store* is a collection of related memory files. Each memory store has a name and description that agents use to decide whether to browse its contents. Memory stores are the organizational containers that group memories by topic.
+
+Memories differ from other knowledge item types in key ways:
+
+| Aspect | Skill | Agent instructions | Memory |
+| --- | --- | --- | --- |
+| Knowledge type | Procedural (instructions) | Procedural (always-on instructions) | Informational (synthesized context) |
+| Content format | Markdown or ZIP bundle | Markdown only | Markdown only |
+| Context injection | On demand (agent decides via description matching) | Always (every session) | On demand (agent decides via description matching) |
+| Created by | User (UI, CLI), DevOps Agent | User (UI, CLI) | User (via Chat), DevOps Agent (Learning agent) |
+
+## Why use memories
+
+Memories give agents access to historical patterns and environmental knowledge that would otherwise be lost between sessions.
+
+**Key benefits:**
++ **Faster investigations** - Agents recall recent root causes for specific monitors, avoiding redundant diagnostic steps when a recurring issue fires again.
++ **Environmental awareness** - Memories capture environment-specific details such as known noisy alarms, infrastructure quirks, or component relationships that are difficult to re-discover each session.
++ **Continuous improvement** - As the DevOps Agent resolves more incidents, it automatically builds a richer knowledge base of patterns and root causes specific to your infrastructure.
++ **User preferences** - Memories record communication preferences and directives so the agent consistently behaves according to your expectations.
+
+## How memories work
+
+When an agent session starts, the agent receives a list of memory stores with their names and descriptions. During an investigation, the agent evaluates whether a memory store is relevant to the current task. If it is, the agent lists the memories within that store and reads the specific memory files it needs. This progressive-disclosure pattern keeps context consumption low while making all relevant knowledge accessible.
+
+Each memory is versioned. Every update creates a new immutable version, enabling audit trails and rollback to previous content.
+
+## Managed memory stores
+
+DevOps Agent creates and maintains the following managed memory stores automatically as it learns from sessions in your Agent Space:
+
+### monitors
+
+Per-monitor recurring root cause history. Each memory file corresponds to a specific monitor (alarm or metric) and lists the cause categories that have produced incidents for that alarm, with per-investigation evidence. Agents read the entry matching the firing alarm before triaging to quickly identify whether the current incident matches a known pattern.
+
+When there are investigations in the past 2 weeks in the Agent Space, a Learning agent runs once per day to analyze recent investigations, then extract and store memories in this store. Memory items in this store are deleted when they have no updates for 2 weeks.
+
+### directives
+
+User-authored directives that steer agent behavior. Use this store to record standing instructions the agent should follow, such as infrastructure conventions or naming preferences.
+
+Examples:
++ "Lambdas are no longer used. The service uses Fargate."
++ "The storage service is called Orders Storage Service."
+
+## Viewing memories
+
+You can view all memory stores and their contents from the **Knowledge** page in your Agent Space Operator Web App.
+
+**To view memory stores:**
+1. Navigate to the **Knowledge** page in your Agent Space Operator Web App.
+2. Choose the **Memories** tab.
+3. Browse the list of managed memory stores.
+
+**To view memories within a store:**
+1. On the **Memories** tab, choose **View** next to the memory store you want to explore.
+2. The store detail page lists all memories with their name, description, version number, and last modified date.
+
+**To view a specific memory:**
+1. From the store detail page, choose **View** next to the memory you want to read.
+2. The memory detail page displays the rendered markdown content along with creation date, last modified date, and a version selector.
+
+## Activating and deactivating memories
+
+You can toggle individual memory stores or individual memories active or inactive without deleting them.
+
+**To toggle a memory store:**
+1. On the **Memories** tab, use the toggle switch next to the memory store.
+2. When inactive, agents do not access any memories within that store.
+
+**To toggle an individual memory:**
+1. Navigate into a memory store to see its list of memories.
+2. Use the toggle switch next to the memory you want to activate or deactivate.
+
+## Editing memories with Chat
+
+You can create, update, or delete memories by chatting with the DevOps Agent in the Chat interface.
+
+**Examples of Chat commands:**
++ "Remember that the storage service is called Orders Storage Service"
++ "Update the memory for the canary alarm to note that it is flaky during deployments"
++ "Delete the memory about the old database connection string"
+
+## Memory limits
+
+| Resource | Limit |
+| --- | --- |
+| Memory stores per Agent Space | 50 |
+| Memories per memory store | 200 |
+| Individual memory content size | 100 KB |

training/data/raw/about-aws-devops-agent-devops-agent-skills.md

@@ -0,0 +1,255 @@
+# DevOps Agent Skills
+
+AWS DevOps Agent Skills are modular instruction sets that extend the agent's capabilities with specialized domain knowledge and investigation methodologies tailored to your infrastructure and operational workflows.
+
+## What are Skills
+
+Skills are self-contained directories containing Markdown instructions that provide specialized capabilities to AWS DevOps Agent. AWS DevOps Agent supports a subset of the Agent Skills specification—an open standard for packaging agent instructions and resources—supporting only non-executable documents: Markdown instructions, PDFs, images, and data files.
+
+Every skill requires a SKILL.md file containing instructions you want to provide for your AWS DevOps Agent. In addition to the required SKILL.md file, skills can include:
++ **Investigation workflows** for specific scenarios or infrastructure types.
++ **Reference materials** including architecture patterns and operational procedures.
++ **Agent type targeting** – Skills can be targeted to specific agent types (Generic, On-demand, Incident Triage, Incident RCA, Incident Mitigation, Evaluation) to reduce context consumption and improve agent focus.
+
+## Why use Skills
+
+Skills transform AWS DevOps Agent from a general-purpose assistant into a specialist for your infrastructure and operational workflows. Unlike one-time instructions provided in a chat message, Skills are reusable capabilities that load automatically when relevant to tasks performed by AWS DevOps Agent.
+
+**Key benefits:**
++ **Specialize your agent** – Tailor AWS DevOps Agent with investigation procedures, best practices, and organizational knowledge specific to your infrastructure and operational patterns.
++ **Reduce repetition** – Create investigation workflows once and AWS DevOps Agent uses them automatically across all relevant investigations, eliminating the need to provide the same guidance repeatedly.
++ **Compose capabilities** – Combine multiple Skills to build end-to-end investigation workflows. AWS DevOps Agent reads multiple skills during execution, such as a skill to retrieve deployments from your custom CI/CD pipeline and a skill to search your code repositories.
++ **Amplify custom tools** – Create skills that guide AWS DevOps Agent in using your custom MCP server tools effectively. Skills can document when to invoke specific tools, what parameters to use for different scenarios, and how to interpret results to accomplish workflows specific to your infrastructure.
+
+## How Skills work
+
+When AWS DevOps Agent encounters a relevant task, it loads the appropriate skills and follows the instructions to guide its investigation. For example, a "Database Performance Investigation" skill might include step-by-step procedures for analyzing RDS throttling issues, enabling the agent to systematically check alarm status, analyze connection metrics, and identify slow queries.
+
+## Skill structure
+
+A skill is organized as a directory containing:
+
+```
+my-skill/
+├── SKILL.md              # Main skill instructions
+├── references/           # Optional: additional reference documentation
+└── assets/               # Optional: images, diagrams, data files
+```
+
+### SKILL.md
+
+The `SKILL.md` is the only mandatory file. It contains the core instructions written in Markdown format. This file should:
++ Describe when and how to use the skill.
++ Provide step-by-step investigation procedures.
++ Include decision trees for different scenarios.
++ Document expected outputs and success criteria.
+
+### Frontmatter
+
+Frontmatter is the metadata block at the top of a `SKILL.md` file, enclosed between `---` delimiters. It contains the `name` and `description` fields that AWS DevOps Agent uses to determine when to activate the Skill during an investigation or task.
+
+```
+---
+name: rds-performance-investigation
+description: Investigation procedures for RDS performance issues including
+  connection exhaustion, slow queries, replication lag, and storage capacity.
+  Use this skill when investigating database latency, connection errors, or
+  read/write performance degradation.
+---
+```
+
+**name** – A unique identifier for the Skill. Use lowercase letters, numbers, and hyphens only (maximum 64 characters). Must not start or end with a hyphen.
+
+**description** – A detailed explanation of when and why AWS DevOps Agent should use this Skill. AWS DevOps Agent evaluates this field to decide whether the Skill is relevant to the current task. A vague or missing description can cause the agent to skip the Skill entirely, even if the instructions are well-written.
+
+**Important** – Write the description from the agent's perspective. Include the specific scenarios, services, error types, or symptoms that should trigger the Skill.
+
+When you create a Skill in the UI, the system generates frontmatter automatically from the name and description you provide. Skills uploaded as zip files must include frontmatter in the `SKILL.md` file.
+
+## Example: Complete skill
+
+The following example shows a complete, well-formed skill for investigating RDS performance issues.
+
+**Directory structure:**
+
+```
+rds-performance-investigation/
+├── SKILL.md
+├── references/
+│   └── rds-metrics-reference.md
+└── assets/
+    └── rds-investigation-flowchart.png
+```
+
+**SKILL.md:**
+
+```
+---
+name: rds-performance-investigation
+description: Investigation procedures for RDS performance issues including
+  connection exhaustion, slow queries, replication lag, and storage capacity.
+  Use this skill when investigating database latency, connection errors, or
+  read/write performance degradation.
+---
+
+# RDS Performance Investigation
+
+Use this skill when customers report database latency, connection errors,
+query timeouts, or read/write performance degradation.
+
+## Step 1: Check alarm status
+
+Query CloudWatch for active alarms on the affected RDS instance. Look for:
+- `DatabaseConnections` exceeding 80% of max_connections
+- `ReadLatency` or `WriteLatency` above 20ms
+- `FreeStorageSpace` below 20% of total storage
+- `ReplicaLag` above 30 seconds (read replicas only)
+
+## Step 2: Analyze connection metrics
+
+Retrieve `DatabaseConnections` over the past hour. If connections are near
+the max_connections limit, check for connection pool misconfiguration or
+long-running idle connections.
+
+## Step 3: Identify slow queries
+
+Use Performance Insights (`pi:GetResourceMetrics`) to retrieve the top SQL
+statements by average active sessions. Focus on queries with high `db.load`
+contribution or frequent I/O waits.
+
+## Step 4: Summarize findings
+
+Provide a summary with:
+1. Current performance status (healthy / degraded / critical)
+2. Root cause hypothesis with supporting metrics
+3. Recommended remediation steps ranked by priority
+```
+
+**references/rds-metrics-reference.md:**
+
+```
+# RDS CloudWatch Metrics Reference
+
+| Metric | Normal Range | Investigation Threshold |
+|---|---|---|
+| DatabaseConnections | < 70% max_connections | > 80% max_connections |
+| ReadLatency | < 5ms | > 20ms |
+| WriteLatency | < 5ms | > 20ms |
+| FreeStorageSpace | > 30% total storage | < 20% total storage |
+| ReplicaLag | < 5 seconds | > 30 seconds |
+| CPUUtilization | < 70% | > 85% |
+```
+
+## Example: Incident filtering skill
+
+Skills targeted to the **Incident Triage** agent type can define criteria for automatically skipping incidents. Use this to filter incidents that don't require investigation.
+
+**SKILL.md:**
+
+```
+---
+name: skip-scheduled-maintenance
+description: Skip low-priority incidents during a scheduled maintenance window.
+  Use this skill to automatically filter MEDIUM and LOW severity alarms that
+  fire during planned maintenance, avoiding unnecessary investigations for
+  expected disruptions.
+---
+
+# Skip Scheduled Maintenance
+
+Skip all incidents that meet BOTH of the following criteria:
+
+1. The incident arrived between **2025-03-15 02:00 UTC** and **2025-03-15 06:00 UTC**
+2. Severity is MEDIUM or LOW
+
+Do NOT skip HIGH or CRITICAL severity incidents, even during the maintenance window.
+```
+
+## Creating Skills
+
+You can create skills in two ways depending on your workflow preferences and skill complexity.
+
+### Creating a skill in the UI
+
+Skills created in the AWS DevOps Agent Operator Web App contain a name, description, and instructions in a single SKILL.md file.
+
+**To create a skill in the UI:**
++ Navigate to the **Knowledge** page in your Agent Space Operator Web App and choose the **Skills** tab.
++ Click "Add skill".
++ Select "Create skill" from the modal.
++ Fill out the skill form:
+  + **Name** – Lowercase letters, numbers, and hyphens only (maximum 64 characters). Must not start or end with a hyphen.
+  + **Description** – Brief explanation of when to use this skill (minimum 100 characters recommended, maximum 1,024 characters).
+  + **Status** – Set to Active (default) or Inactive.
+  + **Agent Type** – Select one or more agent types that can use this skill.
+  + **Instructions** – Step-by-step procedures in Markdown format.
++ Choose "Create" to save the skill.
+
+### Uploading a skill
+
+Skills uploaded as zip files contain a SKILL.md file plus additional resources such as reference materials or assets.
+
+**Skill structure:**
+
+```
+my-skill.zip
+├── SKILL.md              # Required: main skill instructions
+├── references/           # Optional: reference documentation
+│   ├── architecture.md
+│   └── troubleshooting.md
+└── assets/               # Optional: images, diagrams, data files
+    ├── topology.png
+    └── metrics.csv
+```
+
+**To create a skill via zip upload:**
++ Create a directory with your skill files following the structure above.
++ Ensure SKILL.md includes proper frontmatter (name and description).
++ Compress the directory into a .zip file.
++ Navigate to the **Knowledge** page and choose the **Skills** tab.
++ Click "Add skill".
++ Select "Upload skill" from the modal.
++ Drag and drop your .zip file or choose to browse (ZIP files only, maximum 6 MB).
++ Select one or more agent types.
++ Choose "Upload" to add the skill to your Agent Space.
+
+### Importing a skill from a repository
+
+You can import skills directly from a GitHub repository directory. AWS DevOps Agent fetches the skill content from the repository, extracts the name and description from the SKILL.md frontmatter, and creates the skill in your Agent Space.
+
+**Prerequisites:**
++ A GitHub account associated with your Agent Space.
++ A GitHub repository containing a valid skill directory with a SKILL.md file at the root.
+
+**To import a skill from a repository:**
++ Navigate to the **Knowledge** page and choose the **Skills** tab.
++ Click "Add skill".
++ Select "Import from repository" from the modal.
++ Enter the GitHub directory URL that contains your SKILL.md file.
++ Select one or more agent types.
++ Set the lifecycle status (Active or Inactive).
++ Click "Import skill".
+
+## Sample skills
+
+The AWS DevOps Agent Skills samples repository on GitHub contains community-contributed skills you can use as-is or as a starting point for writing your own.
+
+## Managing Skills
+
+AWS DevOps Agent provides comprehensive skill management capabilities through the Operator Web App:
+
+**Listing skills** – View all Skills in your Agent Space on the Knowledge page Skills tab.
+
+**Viewing skills** – Choose any skill to see its detail view.
+
+**Selecting agents for a skill** – Configure which agent types can use each skill.
+
+**Activating and deactivating skills** – Temporarily disable skills without deleting them.
+
+**Updating skills** – Modify existing skills based on how they were created.
+
+**Deleting skills** – Permanently remove skills from your Agent Space.
+
+## Migrating from Runbooks
+
+Existing Runbooks are automatically migrated to Skills with no customer action required. When your Agent Space transitions to the Skills model, all Runbooks are converted to Skills and appear in your Skills UI.

training/data/raw/about-aws-devops-agent-learned-skills.md

@@ -0,0 +1,42 @@
+# Learned Skills
+
+## What Are Learned Skills?
+
+Learned skills are structured knowledge files that the DevOps Agent generates from your Agent Space data. Each learned skill encodes a specific type of knowledge that the AWS DevOps Agent uses as it performs tasks. At launch, two learned skills are available: Agent Space Understanding and Tool Use Best Practices.
+
+### Agent Space Understanding
+
+The Agent Space Understanding skill (`understanding-agent-space`) analyzes your connected cloud accounts, code repositories, and telemetry integrations to build a map of the resources and relationships in an Agent Space.
+
+The skill produces a main `SKILL.md` file and a set of reference files. The main file contains a plain-language system overview with key domain concepts, the deployment environments (AWS account and region pairs, Azure subscriptions and regions, and so on), a container-level architecture diagram showing how logical services connect, the request paths that are central to your application with the components they traverse, and a mapping of code repositories to containers.
+
+Each logical container receives a dedicated reference file describing its internal components (compute, data, messaging, network, and others) with resource types and physical identifiers such as ARNs, table names, and queue URLs. The reference file also captures observability coverage, including the alarms, dashboards, and monitors linked to each component. It also maps each component to its associated code repositories, packages, and infrastructure-as-code definitions, providing a complete traceability chain from source code to deployed resources.
+
+Each critical request path receives a dedicated reference file describing the full end-to-end request flow at component granularity, from the entry point through each intermediate service, data store, and external dependency. The file includes a sequenced flow diagram showing the order of operations and interaction mechanisms between components, along with the responsibility of each participant. It also catalogs the observability signals relevant to the path: log group patterns for each hop, key metrics (latency, error rates, throttling, token quotas) with their alarm names and dimensions, and distributed trace spans that can be correlated across services and accounts.
+
+### Tool Use Best Practices
+
+The Tool Use Best Practices skill analyzes past investigation tool uses to extract effective usage patterns, common failure modes, and parameter guidance. This helps the DevOps Agent avoid known pitfalls and run investigations with fewer wasted steps. The skill produces a main file and a set of per-tool reference files. The main file serves as a routing index that lists each tool with the investigation scenarios it supports and links to the corresponding reference file.
+
+Each per-tool reference file can include up to three sections:
++ **Best Practices** — Investigation-driven techniques extracted from successful tool usage, such as CloudWatch Logs Insights query templates, environment-specific metric namespaces and dimensions, and CloudTrail event source filters.
++ **Common Errors** — Recurring failure modes and their fixes. Each entry describes a specific error condition and provides a corrective action so the agent can avoid or recover from the error without wasting investigation steps.
++ **Output Management** — Guidance for tool calls that tend to return large responses. Each entry describes a parameter change or processing strategy that reduces output size while preserving diagnostic value.
+
+When live infrastructure access is available, the skill validates patterns against your environment before including them. Confirmed patterns are stated with confidence, unconfirmed patterns use cautious language, and disproved patterns are excluded.
+
+## Managing Learned Skills
+
+**Updates** — The DevOps Agent automatically generates and updates learned skills based on activity in your Agent Space.
+
+The DevOps Agent generates an updated **Tool Use Best Practices** skill every 30 investigations.
+
+The **Agent Space Understanding** skill is generated by the learning agent, which runs whenever you add, update, or remove an Agent Space capability or integration. It is also periodically refreshed every 3 days for active Agent Spaces. An Agent Space is considered active if it has had at least one investigation in the last 30 days. If your Agent Space has no investigations for 30 days, skill refresh pauses automatically. It resumes when a new investigation starts.
+
+To regenerate learned skills manually, choose the **Regenerate** button on the Topology page in the operator app, or chat with the agent and ask it to update the learned skills.
+
+**Deactivation** — Learned skills are active by default. When active, the DevOps Agent loads them at the start of each DevOps Agent task. To stop a learned skill from being applied, deactivate it from the **Knowledge** page **Skills** tab in the operator app.
+
+**Topology view** — The Topology page in your Agent Space's web app uses the Agent Space Understanding Skill to visually display your Agent Space environment as logical containers and components.
+
+**Summary report** — The summary report is a versioned, read-only view of what the DevOps Agent knows about your environment, derived from the Agent Space Understanding skill. You can find it in two places: the **Summary report** tab on the Agent Space details page in the AWS DevOps Agent admin console, and the **Artifacts** section of the DevOps Agent web app.

training/data/raw/about-aws-devops-agent-managing-assets.md

@@ -0,0 +1,112 @@
+# Managing Assets
+
+AWS DevOps Agent stores the configuration and reference material for an Agent Space as **assets**, the customer-managed resources that shape what the agent knows and how it behaves. Skills, AGENTS.md files, and attachments are all assets, and you can create, read, update, and delete them programmatically through the Asset API.
+
+## When to use the Asset API
+
+The Operator Web App is the fastest way to author a single skill or upload an AGENTS.md file interactively. The Asset API exposes the same operations programmatically so that scripts and automation can manage assets without going through the Web App. Common reasons to call the Asset API directly include:
++ Authoring or updating an asset from a script, terminal, or notebook instead of the Web App.
++ Bulk-loading a starter set of skills or AGENTS.md files into a new Agent Space.
++ Reading an asset's contents to back it up or compare versions.
+
+Every operation in the Asset API is exposed through the AWS CLI as `aws devops-agent <operation>` and through the AWS SDKs as the `devops-agent` client.
+
+## Asset API operations
+
+| Operation | Description | IAM action | Resource |
+| --- | --- | --- | --- |
+| ListAssetTypes | List the asset types supported by AWS DevOps Agent. | aidevops:ListAssetTypes | * |
+| CreateAsset | Create a new asset in an Agent Space. | aidevops:CreateAsset | Agent Space |
+| GetAsset | Retrieve an asset's metadata and version information. | aidevops:GetAsset | Agent Space |
+| UpdateAsset | Update the metadata or content of an existing asset. | aidevops:UpdateAsset | Agent Space |
+| DeleteAsset | Delete an asset and all of its files from an Agent Space. | aidevops:DeleteAsset | Agent Space |
+| ListAssets | List assets in an Agent Space, with optional filtering. | aidevops:ListAssets | Agent Space |
+| ListAssetVersions | List the historical versions of an asset. | aidevops:ListAssetVersions | Agent Space |
+| GetAssetContent | Download an asset's full content as a zip bundle. | aidevops:GetAssetContent | Agent Space |
+| CreateAssetFile | Add a new file to an existing asset. | aidevops:CreateAssetFile | Agent Space |
+| GetAssetFile | Retrieve a single file from an asset by its path. | aidevops:GetAssetFile | Agent Space |
+| UpdateAssetFile | Replace the content or metadata of an existing file. | aidevops:UpdateAssetFile | Agent Space |
+| DeleteAssetFile | Remove a single file from an asset. | aidevops:DeleteAssetFile | Agent Space |
+| ListAssetFiles | List the files within an asset. | aidevops:ListAssetFiles | Agent Space |
+
+## Asset types
+
+Every asset has an `assetType` string that identifies what kind of resource it is. Three asset types can be created through the Asset API: `skill`, `agents_md`, and `attachment`.
+
+### skill
+
+A `skill` asset packages instructions and reference material that the agent loads when relevant. A simple skill is a single `SKILL.md` file; a complex skill is a zip bundle.
+
+**Required metadata properties:**
++ **name** (string) – A unique identifier for the skill. Lowercase letters, numbers, and hyphens only, 1-64 characters.
++ **description** (string) – A 1-1024 character explanation of when the agent should use the skill.
++ **agent_types** (array of strings) – One or more agent types this skill applies to.
+
+**Optional metadata properties:**
++ **skill_type** (string) – Defaults to `USER`.
++ **status** (string) – `ACTIVE` or `INACTIVE`. Defaults to `ACTIVE`.
++ **enable_tools** (array of strings) – A list of tool identifiers the agent can call when it loads this skill.
+
+### agents_md
+
+An `agents_md` asset is a markdown file containing standing agent instructions for a specific agent type.
+
+**Required metadata properties:**
++ **agent_type** (string) – The agent type the AGENTS.md file applies to.
+
+**Limits:** Each Agent Space can contain at most one AGENTS.md per agent_type. Maximum 25 KB.
+
+### attachment
+
+An `attachment` asset stores a binary or text file that the agent can reference during investigations.
+
+**Required metadata properties:**
++ **filename** (string) – The original file name.
++ **extension** (string) – The file extension without the leading dot.
++ **size** (number) – The size of the file in bytes.
+
+## Managing a skill end-to-end
+
+### Create a skill from a single text file
+
+```
+aws devops-agent create-asset \
+  --agent-space-id 8f6187a7-0388-4926-8217-3a0fe32f757c \
+  --asset-type skill \
+  --metadata '{
+    "name": "rds-performance-investigation",
+    "description": "Investigation procedures for RDS performance issues.",
+    "agent_types": ["GENERIC"]
+  }' \
+  --content '{
+    "file": {
+      "path": "SKILL.md",
+      "body": {
+        "text": "# RDS Performance Investigation\n\nUse this skill when customers report database latency."
+      }
+    }
+  }'
+```
+
+### Create a skill from a zip bundle
+
+```
+base64 -w 0 rds-performance-investigation.zip > skill.zip.b64
+aws devops-agent create-asset --cli-input-json file://create-skill.json
+```
+
+### Get, list, update, and delete
+
+```
+aws devops-agent get-asset --agent-space-id <id> --asset-id <assetId>
+aws devops-agent list-assets --agent-space-id <id> --max-results 50
+aws devops-agent update-asset --agent-space-id <id> --asset-id <assetId> --metadata '{ "agent_types": ["INCIDENT_TRIAGE"] }'
+aws devops-agent delete-asset --agent-space-id <id> --asset-id <assetId>
+```
+
+### Activating and deactivating skills
+
+```
+aws devops-agent update-asset --agent-space-id <id> --asset-id <assetId> --metadata '{ "status": "INACTIVE" }'
+aws devops-agent update-asset --agent-space-id <id> --asset-id <assetId> --metadata '{ "status": "ACTIVE" }'
+```

training/data/raw/about-aws-devops-agent-supported-regions.md

@@ -0,0 +1,49 @@
+# Supported Regions
+
+This topic describes the AWS Regions where you can use AWS DevOps Agent.
+
+## Cross-Region resource monitoring
+
+AWS DevOps Agent can monitor and investigate resources in AWS accounts located in any AWS Region, regardless of which supported Region you create your Agent Space in. When you associate an AWS account with an Agent Space, the agent discovers and maps resources across all Regions within that account. This means you do not need an Agent Space in every Region where your workloads run.
+
+Choose a supported Region based on your preferred data residency, proximity to your operations team, or organizational requirements.
+
+## Supported Regions
+
+AWS DevOps Agent is available in the following AWS Regions.
+
+| Region Name | Region Code | 
+| --- | --- |
+| US East (N. Virginia) | us-east-1 |
+| US West (Oregon) | us-west-2 |
+| Canada (Central) | ca-central-1 |
+| South America (Sao Paulo) | sa-east-1 |
+| Asia Pacific (Mumbai) | ap-south-1 |
+| Asia Pacific (Singapore) | ap-southeast-1 |
+| Asia Pacific (Sydney) | ap-southeast-2 |
+| Asia Pacific (Tokyo) | ap-northeast-1 |
+| Europe (Frankfurt) | eu-central-1 |
+| Europe (Ireland) | eu-west-1 |
+| Europe (London) | eu-west-2 |
+
+## Service endpoints
+
+| Region Name | Region Code | Endpoint | Protocol |
+| --- | --- | --- | --- |
+| US East (N. Virginia) | us-east-1 | aidevops.us-east-1.amazonaws.com | HTTPS |
+| US West (Oregon) | us-west-2 | aidevops.us-west-2.amazonaws.com | HTTPS |
+| Canada (Central) | ca-central-1 | aidevops.ca-central-1.amazonaws.com | HTTPS |
+| South America (Sao Paulo) | sa-east-1 | aidevops.sa-east-1.amazonaws.com | HTTPS |
+| Asia Pacific (Mumbai) | ap-south-1 | aidevops.ap-south-1.amazonaws.com | HTTPS |
+| Asia Pacific (Singapore) | ap-southeast-1 | aidevops.ap-southeast-1.amazonaws.com | HTTPS |
+| Asia Pacific (Sydney) | ap-southeast-2 | aidevops.ap-southeast-2.amazonaws.com | HTTPS |
+| Asia Pacific (Tokyo) | ap-northeast-1 | aidevops.ap-northeast-1.amazonaws.com | HTTPS |
+| Europe (Frankfurt) | eu-central-1 | aidevops.eu-central-1.amazonaws.com | HTTPS |
+| Europe (Ireland) | eu-west-1 | aidevops.eu-west-1.amazonaws.com | HTTPS |
+| Europe (London) | eu-west-2 | aidevops.eu-west-2.amazonaws.com | HTTPS |
+
+## Considerations
+
++ **Agent Space Region selection** — An Agent Space and its data are stored in the Region where you create it. Choose a Region that meets your data residency requirements.
++ **Cross-Region monitoring** — Resources in AWS accounts associated with an Agent Space are monitored regardless of which Region those resources are deployed in.
++ **Third-party integrations** — Connections to CI/CD providers, observability tools, and MCP servers are configured per Agent Space and are not Region-dependent.

training/data/raw/about-aws-devops-agent-what-are-devops-agent-spaces.md

@@ -0,0 +1,29 @@
+# What are DevOps Agent Spaces?
+
+A DevOps Agent Space is a logical container that defines the tools and infrastructure that AWS DevOps Agent has access to. Each Agent Space operates independently with its own AWS account access, third-party integrations, and user permissions.
+
+An Agent Space represents the boundary of what AWS DevOps Agent can access and investigate during incident response. When you create an Agent Space, you define which AWS accounts the agent can access, which external tools it can connect to, and which users in your organization can interact with the agent.
+
+Each Agent Space functions as an independent deployment of AWS DevOps Agent. You configure the Agent Space through the AWS Management Console, while your operations teams use the Agent Space's web app to conduct investigations and review recommendations within that space.
+
+## How Agent Spaces are isolated
+
+Agent Spaces maintain isolation to ensure security and prevent unintended access across different environments or teams:
++ **AWS account isolation** – Each Agent Space uses dedicated IAM roles that grant access only to specific AWS accounts and resources. The agent cannot access AWS resources outside of those explicitly configured for the Agent Space.
++ **User access isolation** – You control which users or groups can access each Agent Space. This allows you to align access permissions with your organizational structure, ensuring teams only interact with their designated Agent Spaces.
++ **Data isolation** – Investigation data, incident history, and recommendations are maintained separately within each Agent Space. Information from one Agent Space is not visible or accessible from another Agent Space.
++ **Chat data isolation** - Chat conversation history is also isolated within each Agent Space. Conversations and queries in one Agent Space are not visible or accessible from another Agent Space.
+
+## Agent Space Web App
+
+Each Agent Space has a dedicated web app that is accessible outside of the AWS Management Console. See What is a DevOps Agent Web App? to learn more about the web app.
+
+## When to use multiple Agent Spaces
+
+Consider creating multiple Agent Spaces to support different organizational needs:
++ **Team separation** – Create dedicated Agent Spaces for different application teams or business units to maintain clear ownership boundaries in the Agent Space.
++ **Environment isolation** – Separate production and non-production environments into different Agent Spaces to prevent accidental cross-environment access.
++ **Service boundaries** – Align Agent Spaces with specific services or application boundaries to keep investigations focused and relevant.
++ **Compliance requirements** – Configure separate Agent Spaces with different access controls or data residency settings to meet regulatory requirements.
+
+**Note:** When creating multiple Agent Spaces, you can use a dedicated AWS account as the primary account for an Agent Space and connect distinct application accounts as secondary accounts. This approach allows you to maintain granular access controls while ensuring that each Agent Space can access only the resources specific to its intended scope, even when using automatic role creation.

training/data/raw/about-aws-devops-agent-what-is-a-devops-agent-topology.md

@@ -0,0 +1,54 @@
+# What is a DevOps Agent Topology?
+
+AWS DevOps Agent's automatically discovers and visualizes the resources and relationships within your applications and uses the resulting topology to understand your infrastructure during incident investigations and when making preventative recommendations.
+
+## How topology graphs are created
+
+AWS DevOps Agent builds topology graphs through several automated processes:
++ **Resource discovery** – The agent automatically scans your AWS accounts to identify resources like compute instances, storage services, networking components, and databases that are part of your applications.
++ **Relationship detection** – The agent analyzes configuration data, CloudFormation stacks, and resource tags to determine how resources relate to one another.
++ **Code and deployment mapping** – When connected to CI/CD pipelines, the agent links infrastructure resources back to their deployment processes and changed application and infrastructure code.
++ **Observability behavior mapping** – Data from observability systems such as Amazon CloudWatch Application Signals and Dynatrace are used to identify observed behaviors that indicate relationships between resources.
+
+## Key capabilities
+
+Resource mapping provides several capabilities that enhance incident investigation and prevention:
++ **Interactive visualization** – Explore your application topology through an interactive graph in the Operator Web App. You can zoom and navigate the topology to understand complex relationships between resources. You can also use Chat to query topology information using natural language, such as 'Show me all Lambda functions connected to this DynamoDB table' or 'What resources are affected by this alarm?'.
++ **Contextual investigation** – During incident investigations, AWS DevOps Agent is assisted by the resource topology to identify affected components, understand blast radius, and trace the impact path through your systems.
++ **Root cause analysis** – The detailed understanding of resource relationships helps pinpoint where issues originate, even in complex distributed systems with many interdependencies.
++ **Impact assessment** – When analyzing incidents, the agent can better determine which downstream services might be affected by identifying dependency chains in the topology.
++ **Preventative recommendations** – The agent uses topology insights to make targeted recommendations for resilience improvements, suggesting changes that will have the most significant impact on system stability.
+
+## Topology views
+
+The topology visualization in the Topology page in the Operator Web App offers multiple levels of detail, organized into two categories:
+
+### Learned
+
+Learned view is generated from the Agent Space understanding and pipeline skill and display structured summaries of your infrastructure. The following learned views are available:
++ **Topology** – The default view, generated from the Agent Space Understanding skill. Displays a structured summary of your infrastructure organized by logical services and request paths.
++ **Pipeline** – Displays your CI/CD pipeline topology, showing deployment stages, actions, and their relationships to infrastructure resources. This view is only available when pipeline topology has been generated for your agent space.
+
+### Others
+
+This view shows your infrastructure at different levels of granularity based on raw resource discovery data:
++ **System** – Shows high-level account and region boundaries.
++ **Container** – Displays deployment stacks like CloudFormation stacks that contain related resources.
++ **Components** – Shows individual components within containers and their relationships.
++ **All Resources** – Shows the complete view with all discovered resources and their relationships.
+
+## Resource discovery
+
+Resources are discovered through two methods:
++ **CloudFormation stacks** – The agent lists all CloudFormation stacks and their resources in the primary AWS account and any connected secondary accounts. This is supported for any infrastructure-as-code tooling that uses CloudFormation for deployment, including AWS Cloud Development Kit (AWS CDK).
++ **Resource Explorer** – For resources not deployed from CloudFormation, tagged resources are discovered from AWS Resource Explorer. The target AWS account must have Resource Explorer enabled. This is useful for identifying application boundaries for resources deployed through the AWS Management Console, the AWS service APIs, or other infrastructure-as-code frameworks.
+
+## Investigation scope beyond topology
+
+While the application topology provides important context during investigations, AWS DevOps Agent is not limited to investigating only the resources shown in the topology. The agent may use additional data sources, such as AWS service APIs or connected observability tools, to investigate resources that are not in the application topology.
+
+To limit the resources the agent has access to, restrict the policy for the role assigned to the agent to access cross-account resources. For more information, see Limiting Agent Access in an AWS Account.
+
+## Topology and the Agent Space Understanding skill
+
+The topology graph feeds into the Agent Space Understanding learned skill, which encodes a structured summary of your infrastructure for use during investigations. When topology discovery completes for a new agent space, the system automatically generates the Agent Space Understanding skill. For more information about learned skills, see Learned Skills.

training/data/raw/about-aws-devops-agent-what-is-a-devops-agent-web-app.md

@@ -0,0 +1,25 @@
+# What is a DevOps Agent Web App?
+
+AWS DevOps Agent uses a dual-console architecture that separates administrative functions from day-to-day operational activities. This design enables administrators to configure the service while operations teams focus on incident response and prevention.
+
+## Consoles
+
+AWS DevOps Agent provides two distinct interfaces:
++ **AWS Management Console** – Administrators use the AWS Management Console to set up and manage AWS DevOps Agent. In this console, you can create an Agent Space, connect AWS services and third-party tools, and manage access permissions for your organization.
++ **DevOps Agent web app** – Operations teams use DevOps Agent Space web apps for daily incident response activities. This standalone application provides an interface where on-call engineers can launch investigations, interact with the agent through natural language chat, view application topologies, and review incident prevention recommendations.
+
+## Web app capabilities
+
+The DevOps Agent web app provides the following primary capabilities:
++ **Incident Response** – The page is where you create and track incident investigations as well as generate mitigation plans to resolve incidents.
++ **Incident Prevention** – Found in the Prevention page, this is where you will find recommendations to improve your observability posture, delivery processes, and infrastructure architecture to prevent future incidents.
++ **Topology** – The Topology page provides an interactive visual representation of the account resources and their relationships across all of the resources in the connected accounts. You can view the topology with different levels of detail using the "Show" dropdown to switch between System, Container, and Resource views.
++ **Skills** – Modular instruction sets that extend AWS DevOps Agent with specialized capabilities. Skills contain domain knowledge, investigation methodologies, and tool configurations tailored to your infrastructure. Each skill enables specific tools and provides progressive disclosure of instructions only when relevant to the investigation.
++ **Natural language Chat interface** – Available throughout the web app, Chat is an AI-powered conversational assistant that enables you to query your infrastructure, analyze system health, and work with investigations using natural language. Chat provides context-aware responses based on the page you're viewing.
+
+## Authentication
+
+AWS DevOps Agent supports flexible authentication methods to accommodate different organizational requirements:
++ **IAM Identity Center integration (User access)** – Organizations can use AWS Identity Center (IAM Identity Center) to centrally manage user access to the DevOps Agent Space web apps. IAM Identity Center can federate with external identity providers through standard OIDC and SAML protocols, including providers like Okta, Ping Identity, and Microsoft Entra ID. This method supports multi-factor authentication from your identity provider.
++ **External identity provider (IdP) authentication** – Organizations can connect an OIDC-compatible identity provider, such as Okta or Microsoft Entra ID, directly to the Agent Space web app without requiring IAM Identity Center. Users sign in with their corporate credentials through the IdP.
++ **IAM authentication link (Admin access)** – An alternative method provides direct access to the web app from the AWS Management Console using your existing console session. This option is useful before implementing full Identity Center integration, but sessions are limited to 10 minutes.

training/data/raw/about-aws-devops-agent.md

@@ -0,0 +1,52 @@
+# About AWS DevOps Agent
+
+AWS DevOps Agent is a frontier agent that resolves and proactively prevents incidents, continuously improving reliability and performance.
+
+AWS DevOps Agent investigates incidents and identifies operational improvements as an experienced DevOps engineer.
+
+The agent works by:
++ Learning your resources and their relationships.
++ Working with your observability tools, skills, code repositories, and CI/CD pipelines.
++ Correlating telemetry, code, and deployment data to understand relationships between your application resources.
++ Supporting applications in multicloud and hybrid environments.
+
+## Key features
+
+AWS DevOps Agent provides comprehensive incident response and prevention capabilities through the following features:
+
+### Always-on, autonomous incident response
+
+AWS DevOps Agent autonomously investigates issues the moment they occur:
++ **Automated incident investigation** – Begins investigating immediately when an alert or support ticket comes in
++ **AWS DevOps Agent Chat** - Query your infrastructure, analyze system health, and guide investigations using natural language throughout the DevOps Agent Space web app. Chat provides context-aware responses based on the page you're viewing, whether asking about resources in Topology, steering an investigation, or filtering recommendations in Prevention.
++ **Detailed mitigation plans** – Provides specific actions to resolve incidents, validate success, and revert changes if needed
++ **Automated incident coordination** – Routes observations, findings, and mitigation steps through your preferred communication channels like Slack and ServiceNow
++ **AWS Support integration** – Create AWS Support cases directly from an investigation with immediate context provided to AWS Support experts
+
+### Prevent future incidents
+
+AWS DevOps Agent analyzes patterns across historical incidents to help you move from reactive firefighting to proactive operational improvement:
++ **Targeted recommendations** – Delivers specific, actionable improvements that strengthen four key areas: observability (monitoring, alerting, logging), infrastructure optimization (autoscaling, capacity tuning), and deployment pipeline enhancement (testing, validation).
++ **Continuous learning** – Refines recommendations based on your team's feedback
+
+### Get more from your DevOps tools
+
+AWS DevOps Agent integrates with your existing tools without changing your workflows:
++ **Application resource mapping** – Builds a topology graph of your application resources and their relationships
++ **Built-in integrations** – Works with popular observability tools (Amazon CloudWatch, Dynatrace, Datadog, New Relic, and Splunk), code repositories, and CI/CD pipelines (GitHub Actions and repositories, GitLab workflows and repositories)
++ **Custom tool integration** – Extend capabilities by connecting to your own Model Context Protocol (MCP) servers for additional tools
++ **Conversational infrastructure queries** – Use natural language to query AWS resources, system metrics and alarm status without navigating multiple consoles. Chat understands context and maintains conversation history for follow-up questions.
+
+## How AWS DevOps Agent works
+
+AWS DevOps Agent operates through a dual-console architecture. Administrators use the AWS Management Console to create and manage Agent Spaces, configure integrations, and set up access controls. Operations teams use the AWS DevOps Agent web app for day-to-day incident response and investigation activities. The web app is where operators can interact with agent investigations, browse cross-account application topology, and learn about preventative improvements to observability, code, pipelines, and infrastructure architectures. To learn more, see Proactive incident prevention.
+
+The service is organized around Agent Spaces, which are logical containers that define what AWS DevOps Agent can access and investigate. Each Agent Space contains your AWS account configurations, third-party tool integrations, and access permissions. To learn more, see What are DevOps Agent Spaces?.
+
+AWS DevOps Agent automatically builds an application topology that maps your resources and their relationships. This topology helps the service understand your application architecture during investigations. To learn more, see What is a DevOps Agent Topology?.
+
+## Benefits
++ **Reduce mean time to resolution (MTTR)** – Autonomous investigation starts immediately, accelerating incident resolution from hours to minutes
++ **Prevent recurring incidents** – Targeted recommendations address root causes and strengthen system resilience
++ **Improve operational efficiency** – Free your team from repetitive investigation tasks to focus on innovation
++ **Work within existing workflows** – Integrates with your existing tools and processes without disruption

training/data/raw/aws-devops-agent-security-compliance-validation.md

@@ -0,0 +1,7 @@
+# Compliance validation for AWS DevOps Agent
+
+AWS DevOps Agent is in scope for the following compliance programs: BIO, C5, CISPE, CPSTIC, ENS High, FINMA, GNS, GSMA, HITRUST, IRAP, ISMAP, ISO (ISO/IEC 27001, 27017, 27018, 27701, 22301, 20000, 9001), CSA STAR, MTCS, OSPAR, PCI, Pinakes, PiTuKri, and SOC.
+
+AWS DevOps Agent is HIPAA eligible.
+
+Download third-party audit reports using AWS Artifact.

training/data/raw/aws-devops-agent-security-devops-agent-iam-permissions.md

@@ -0,0 +1,41 @@
+# DevOps Agent IAM permissions
+
+AWS DevOps Agent uses IAM actions in the `aidevops:` namespace to control access.
+
+## Agent Space management actions
++ aidevops:GetAgentSpace - View Agent Space details
++ aidevops:GetAssociation - View account association details
++ aidevops:ListAssociations - List AWS account associations
++ aidevops:CreateAgentSpace - Create new Agent Spaces
++ aidevops:DeleteAgentSpace - Delete Agent Spaces
++ aidevops:UpdateAgentSpace - Update Agent Space configuration
++ aidevops:ListAgentSpaces - List all Agent Spaces
+
+## Investigation and execution actions
++ aidevops:ListExecutions - View execution metadata
++ aidevops:GetExecution - Get execution details
++ aidevops:CreateTask - Create investigation tasks
++ aidevops:ListTasks - List tasks
++ aidevops:GetTask - Get task details
++ aidevops:UpdateTask - Update task priority/status
+
+## Chat actions
++ aidevops:ListChats - View chat history
++ aidevops:CreateChat - Create new conversations
++ aidevops:SendMessage - Send messages and receive responses
+
+## Knowledge management actions
++ aidevops:CreateAsset - Create skills, AGENTS.md, attachments
++ aidevops:GetAsset - Retrieve asset metadata
++ aidevops:UpdateAsset - Update assets
++ aidevops:DeleteAsset - Delete assets
++ aidevops:ListAssets - List assets
+
+## Service and integration actions
++ aidevops:RegisterService - Register third-party services
++ aidevops:AssociateService - Associate services with Agent Spaces
++ aidevops:EnableOperatorApp - Enable web app access
+
+## AWS managed policies
++ **AIDevOpsAgentAccessPolicy** - Read-only permissions for agent investigations
++ **AIDevOpsOperatorAppAccessPolicy** - Operator web app access permissions

training/data/raw/aws-devops-agent-security-encryption-at-rest-for-devops-agent.md

@@ -0,0 +1,38 @@
+# Encryption at rest for AWS DevOps Agent
+
+AWS DevOps Agent encrypts all customer data at rest. By default, AWS owned keys are used at no additional charge. You can optionally use a customer managed KMS key.
+
+## Customer managed keys
+
+You can specify a customer managed key when creating:
++ **Agent Space** – Encrypts Agent Space details, investigations, skills, and chat
++ **Service** – Encrypts third-party service credentials
+
+### Step 1: Create a customer managed key
+Requirements:
+| Property | Requirement |
+| --- | --- |
+| Key type | Symmetric |
+| Key spec | SYMMETRIC_DEFAULT |
+| Key usage | ENCRYPT_DECRYPT |
+
+### Step 2: Set the key policy
+Required KMS actions: kms:DescribeKey, kms:GenerateDataKey, kms:Decrypt, kms:Encrypt, kms:ReEncrypt
+
+Grant permissions to both caller credentials (synchronous operations) and aidevops.amazonaws.com service principal (asynchronous operations like investigations).
+
+### Step 3: Specify the key when creating a resource
+In console: Advanced Configuration > Encryption key type > Customer managed key
+In API: Include kmsKeyArn parameter
+
+## Encryption context
+```
+{
+  "aws-crypto-ec:aws:aidevops:arn": "arn:aws:aidevops:{region}:{accountId}:{resourceType}/{resourceId}"
+}
+```
+
+## Key management
++ Disabling/deleting KMS key prevents data access
++ Cannot change key after resource creation
++ Monitor key usage via CloudTrail

training/data/raw/aws-devops-agent-security-limiting-agent-access-in-an-aws-account.md

@@ -0,0 +1,59 @@
+# Limiting Agent Access in an AWS Account
+
+Control what AWS service APIs and resources the agent can access through IAM policies.
+
+## Understanding IAM roles
++ **Primary account role** - Access to Agent Space account resources
++ **Secondary account roles** - Access to additional connected accounts
+
+## Understanding permission guardrails
+
+The agent applies a session policy (guardrail) that limits effective permissions to the intersection of:
+1. Your IAM role policies (what you intend)
+2. The permission guardrail (maximum the agent can ever do)
+
+### Default permissions
+AIDevOpsAgentAccessPolicy provides default read-only permissions included in the guardrail.
+
+### Supported additional permissions (opt-in)
+| Service | Actions | Use case |
+| --- | --- | --- |
+| Amazon Athena | athena:GetQuery*, athena:StartQueryExecution | Run Athena queries |
+| Amazon S3 | s3:GetObject, s3:ListBucket | Read logs/data in S3 |
+| AWS Direct Connect | directconnect:Describe* | Network connectivity issues |
+| AWS Glue | glue:GetPartitions | Glue Data Catalog for Athena |
+| AWS KMS | kms:Decrypt | Decrypt encrypted resources |
+
+### Permissions blocked by guardrail
+Write operations (s3:PutObject, ec2:TerminateInstances, etc.) are always blocked.
+
+## Restricting service access
+Grant only read-only permissions for necessary services:
+```json
+{
+  "Effect": "Allow",
+  "Action": ["cloudwatch:GetMetricData", "logs:GetLogEvents", "ec2:DescribeInstances"],
+  "Resource": "*"
+}
+```
+
+## Restricting resource access
+Use ARN patterns:
+```json
+{"Resource": "arn:aws:lambda:*:*:function:production-*"}
+```
+
+Use tag-based restrictions:
+```json
+{"Condition": {"StringEquals": {"aws:ResourceTag/Environment": "production"}}}
+```
+
+## Restricting regional access
+```json
+{"Condition": {"StringEquals": {"aws:RequestedRegion": ["us-east-1", "us-west-2"]}}}
+```
+
+## Best practices
++ Grant only read-only permissions
++ Use resource-level permissions when possible
++ Regularly review and audit permissions

training/data/raw/aws-devops-agent-security-setting-up-external-identity-provider-idp-authentication.md

@@ -0,0 +1,52 @@
+# Setting Up External Identity Provider (IdP) Authentication
+
+External IdP authentication allows using an OIDC-compatible identity provider (Okta or Microsoft Entra ID) to manage user access without requiring IAM Identity Center.
+
+## Prerequisites
++ OIDC-compatible identity provider (Okta or Microsoft Entra ID)
++ Administrator access to your identity provider
++ Administrator permissions for AWS DevOps Agent console
+
+## How it works
++ Users navigate to the Agent Space web app URL
++ Redirected to identity provider login page
++ After authenticating, redirected back to web app
++ Sessions valid for up to 8 hours with automatic refresh
+
+## Configuring external IdP authentication
+
+### Step 1: Register an application in your identity provider
+
+#### Option A: Okta
+1. Create OIDC Web Application
+2. Enable Authorization Code and Refresh Token grant types
+3. Note Client ID, Client secret, and Okta domain (Issuer URL)
+
+#### Option B: Microsoft Entra ID
+1. Create App Registration
+2. Create client secret
+3. Issuer URL: https://login.microsoftonline.com/{tenant-id}/v2.0
+
+### Step 2: Enable the Operator App with IdP authentication
+1. Select Agent Space > Access tab
+2. Choose External identity provider
+3. Configure: Identity Provider, Issuer URL, Client ID, Client Secret
+4. Choose role option
+5. Copy the Callback URL
+6. Choose Connect
+
+### Step 3: Add the callback URL to your identity provider
+Add: https://{agentSpaceId}.aidevops.global.app.aws/authorizer/idp/callback
+
+## Session management
++ Session duration: up to 8 hours
++ Automatic refresh using OIDC refresh tokens
++ MFA supported when configured in IdP
+
+## Updating IdP configuration
++ Rotate client secret without disconnecting via Rotate client secret option
++ Other changes require disconnect and reconfigure
+
+## Disconnecting external IdP
+1. Access tab > Disconnect
+2. Removes IdP configuration and user history

training/data/raw/aws-devops-agent-security-setting-up-iam-identity-center-authentication.md

@@ -0,0 +1,53 @@
+# Setting Up IAM Identity Center Authentication
+
+IAM Identity Center authentication provides a centralized way to manage user access to the AWS DevOps Agent Space web application.
+
+## Prerequisites
++ IAM Identity Center enabled in your organization or account
++ Administrator permissions in AWS DevOps Agent
++ An Agent Space configured
+
+## Authentication options
+
++ **IAM Identity Center authentication** – Recommended for production. Centralized user management, sessions up to 12 hours.
++ **Admin access (IAM authentication)** – Quick access for administrators. Sessions limited to 30 minutes.
+
+## Configuring IAM Identity Center during Agent Space creation
+
+### Step 1: Navigate to the Web app configuration
+1. After configuring Agent Space details, proceed to the Access tab
+
+### Step 2: Configure IAM Identity Center integration
+1. Verify the IAM Identity Center instance
+2. Select role option:
+   + Auto-create a new DevOps Agent role (recommended)
+   + Assign an existing role
+   + Create a new role using a policy template
+
+The system automatically creates the IAM Identity Center application, trust relationships, and OAuth 2.0 flows.
+
+## Adding users and groups
+
+Required permissions: aidevops:EnableOperatorApp, sso:CreateApplicationAssignment, sso:DeleteApplicationAssignment, sso:ListApplicationAssignments, sso:DescribeApplication
+
+1. Select Agent Space > Access tab
+2. Under User Access, choose Manage Users and Groups
+3. Choose Add Users or Groups
+4. Search and select users/groups from IAM Identity Center directory
+
+## How users access the web app
+
+1. Navigate to the Agent Space web app URL
+2. Redirected to IAM Identity Center login page
+3. Enter credentials (and MFA if configured)
+4. Redirected back to web app
+
+## Session management
++ Default session duration: 8 hours (configurable 1-12 hours)
++ HTTP-only cookies for security
++ Multi-factor authentication supported
++ 15-minute SigV4 credentials renewed automatically
+
+## Disconnecting Identity Center
+1. Choose Actions > Disconnect from IAM Identity Center
+2. Confirm in dialog

training/data/raw/aws-devops-agent-security-vpc-endpoints-aws-privatelink.md

@@ -0,0 +1,27 @@
+# VPC Endpoints (AWS PrivateLink)
+
+Use AWS PrivateLink to create a private connection between your VPC and AWS DevOps Agent without using an internet gateway.
+
+## VPC endpoint service names
+
+| Category | Endpoint suffix |
+| --- | --- |
+| Control Plane API Actions | aidevops |
+| Runtime Operations | aidevops-dataplane |
+| Webhook Events | event-ai |
+
+## Create an interface endpoint
+
+Service names:
++ com.amazonaws.{region}.aidevops
++ com.amazonaws.{region}.aidevops-dataplane
++ com.amazonaws.{region}.event-ai
+
+Enable Private DNS Name for default Regional DNS:
++ cp.aidevops.{region}.api.aws
++ dp.aidevops.{region}.api.aws
++ event-ai.{region}.api.aws
+
+## Endpoint policy
+
+Attach custom endpoint policies to control access from your VPC to AWS DevOps Agent.

training/data/raw/aws-devops-agent-security.md

@@ -0,0 +1,46 @@
+# AWS DevOps Agent Security
+
+## Multi-layered security
+AWS DevOps Agent implements security at multiple layers with internal access controls limiting scope of actions.
+
+## Agent Spaces
+Primary security boundary. Each Agent Space operates independently with its own configurations, permissions, and data isolation.
+
+## Regional processing and data flow
++ Data stored in Agent Space region
++ Inference processed within geography (EU in EU, US in US, Australia in Australia, Japan in Japan)
++ Cross-region inference for Singapore, Mumbai, Sao Paulo routes globally
+
+## Identity and access management
++ IAM Identity Center (OAuth 2.0, sessions up to 12 hours)
++ External IdP (Okta, Microsoft Entra ID)
++ IAM authentication link (10-minute sessions)
+
+## Data protection
++ Encryption at rest with AWS-managed or customer-managed KMS keys
++ Encryption in transit for all data
++ PII not filtered - recommended to redact before storing in logs
+
+## Agent journal and audit logging
++ Logs every reasoning step and action
++ Immutable once recorded
++ CloudTrail integration for all API calls
+
+## Prompt injection protection
++ Limited write capabilities (only tickets/support cases)
++ Account boundary enforcement
++ AI safety protections (ASL-3)
++ Immutable audit trail
+
+## Integration security
++ Native bidirectional integrations
++ MCP servers (OAuth 2.0, API keys)
++ Webhook triggers (HMAC)
++ Outbound communication (Slack, ticketing)
+
+## Shared responsibility model
+**AWS:** Security of retrieved data, native tools, infrastructure
+**Customer:** User access management, trusted data sources, MCP server security, IAM role scoping, PII redaction
+
+## Data usage
+AWS does not use agent data to train models or improve the product.

training/data/raw/configuring-capabilities-for-aws-devops-agent-aws-eks-access-setup.md

@@ -0,0 +1,20 @@
+# AWS EKS access setup
+
+You can enable AWS DevOps Agent to investigate issues in your Amazon EKS clusters by running read-only kubectl commands against both public and private clusters.
+
+Once connected, the agent can help diagnose operational issues — describing resources, retrieving pod logs, inspecting cluster events, checking node health, and more. The agent cannot create, modify, or delete any resources in your cluster.
+
+## Prerequisites
+
+Ensure that your EKS cluster's authentication mode includes the EKS API. Check on the Access tab in the Amazon EKS console.
+
+## Setup
+
+1. Go to the Access tab in the EKS console.
+2. Create a new IAM access entry with your primary cloud source IAM role ARN.
+3. Select the AmazonAIOpsAssistantPolicy access policy with Cluster access scope.
+4. Review and create the access entry.
+
+## Troubleshooting
+
+If the agent can't reach your cluster, verify that the access entry is using the correct IAM role ARN and that the AmazonAIOpsAssistantPolicy access policy is attached.

training/data/raw/configuring-capabilities-for-aws-devops-agent-connecting-azure-index.md

@@ -0,0 +1,17 @@
+# Connecting Azure
+
+Azure integration enables AWS DevOps Agent to investigate resources in your Azure environment and correlate Azure DevOps pipeline deployments with operational incidents.
+
+Azure integration consists of two independent capabilities:
++ **Azure Resources** – Enables the agent to discover and investigate Azure cloud resources such as virtual machines, AKS clusters, databases, and networking components.
++ **Azure DevOps** – Enables the agent to access Azure DevOps repositories and pipeline execution history.
+
+## Registration methods
+
++ **Admin Consent** – A consent-based flow where you authorize the AWS DevOps Agent Entra application in your Azure tenant.
++ **App Registration** – A self-managed approach where you create your own Entra application with federated identity credentials.
+
+## Known limitations
++ Admin Consent: one AWS account per Azure tenant
++ App Registration: unique application per registration
++ Azure DevOps: source code access requires separate repository connection

training/data/raw/configuring-capabilities-for-aws-devops-agent-connecting-mcp-servers.md

@@ -0,0 +1,46 @@
+# Connecting MCP Servers
+
+Model Context Protocol (MCP) servers extend AWS DevOps Agent's investigation capabilities by providing access to data from your external observability tools, custom monitoring systems, and operational data sources.
+
+## Requirements
++ Streamable HTTP transport protocol
++ Authentication: OAuth 2.0 (Client Credentials or 3LO), API key/token, or AWS SigV4
+
+## Security considerations
++ Allowlist only specific tools needed per Agent Space
++ Maximum tool name length: 64 characters
++ Only allowlist read-only MCP tools
++ Custom MCP servers can introduce prompt injection risks
+
+## Registering an MCP server (account-level)
+
+### Step 1: MCP server details
++ Name, Endpoint URL, Description
++ Optional: Enable Dynamic Client Registration
++ Optional: Connect using private connection
+
+### Step 2: Authorization flow
+Select: OAuth Client Credentials, OAuth 3LO, API Key, or AWS SigV4
+
+### Step 3: Authorization configuration
+
+**OAuth Client Credentials:** Client ID, Client Secret, Exchange URL, Scopes
+**OAuth 3LO:** Client ID, Client Secret, Exchange URL, Authorization URL, Scopes
+**API Key:** Key name, header name, key value
+**AWS SigV4:** IAM role, Region, Service Name, Custom Headers
+
+### Step 4: Review and submit
+
+## Configuring MCP tools in an Agent Space
+
+1. Select Agent Space > Capabilities tab
+2. MCP Servers section > Add
+3. Select MCP server
+4. Choose Allow all tools or Select specific tools
+5. Choose Add
+
+## Creating an IAM role for SigV4 authentication
+
+Trust policy must allow aidevops.amazonaws.com with confused deputy protection (aws:SourceAccount, aws:SourceArn conditions).
+
+For multi-region signing (SigV4a), enter * as the Region.

training/data/raw/configuring-capabilities-for-aws-devops-agent-connecting-multiple-aws-accounts.md

@@ -0,0 +1,118 @@
+# Connecting Multiple AWS Accounts
+
+You can connect secondary AWS accounts to AWS DevOps Agent for cross-account investigation. This allows the agent to access resources, logs, and metrics in accounts other than the one where it is deployed.
+
+## Overview
+
+When DevOps Agent needs to investigate issues that span multiple AWS accounts, you must configure cross-account IAM roles that grant the agent access to the secondary accounts. The agent assumes these roles to perform read operations in those accounts.
+
+## Steps to Connect a Secondary Account
+
+### Step 1: Start Configuration
+
+In the DevOps Agent console, navigate to **Settings > AWS Accounts** and select **Add account**.
+
+Provide:
+- The AWS account ID of the secondary account
+- A descriptive name for the account connection
+- The IAM role name that DevOps Agent will assume
+
+### Step 2: Specify the Role Name
+
+Choose a role name for the cross-account role. The recommended naming convention is:
+
+```
+AIDevOpsAgentCrossAccountRole
+```
+
+This role will be created in the secondary account and assumed by DevOps Agent from the primary account.
+
+### Step 3: Create the Role in the Secondary Account
+
+In the secondary AWS account, create an IAM role with a trust policy that allows the DevOps Agent service role in your primary account to assume it.
+
+Trust policy:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Principal": {
+        "AWS": "arn:aws:iam::<primary-account-id>:role/AIDevOpsAgentServiceRole"
+      },
+      "Action": "sts:AssumeRole",
+      "Condition": {
+        "StringEquals": {
+          "sts:ExternalId": "<external-id-from-console>"
+        }
+      }
+    }
+  ]
+}
+```
+
+### Step 4: Attach AIDevOpsAgentAccessPolicy
+
+Attach the AWS managed policy `AIDevOpsAgentAccessPolicy` to the role. This policy grants read-only access to the AWS services that DevOps Agent needs to investigate issues:
+
+- CloudWatch Logs and Metrics
+- AWS CloudTrail
+- Amazon ECS and EKS
+- AWS Lambda
+- Amazon EC2
+- AWS Systems Manager
+- Amazon S3 (for log access)
+
+### Step 5: Create an Inline Policy
+
+Create an inline policy for any additional permissions specific to your environment. For example, if you use custom KMS keys for encryption:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "kms:Decrypt",
+        "kms:DescribeKey"
+      ],
+      "Resource": "arn:aws:kms:<region>:<secondary-account-id>:key/<key-id>"
+    }
+  ]
+}
+```
+
+### Step 6: Complete Configuration
+
+Return to the DevOps Agent console and complete the account connection setup. DevOps Agent will verify it can assume the role in the secondary account.
+
+## Understanding Required Policies
+
+### Trust Policy
+
+The trust policy defines who can assume the role. It must specify:
+- The principal (DevOps Agent service role ARN from the primary account)
+- An external ID condition for security
+- The `sts:AssumeRole` action
+
+### Managed Policy (AIDevOpsAgentAccessPolicy)
+
+This AWS-managed policy provides baseline read-only permissions across services commonly used during incident investigation. AWS maintains and updates this policy as DevOps Agent capabilities expand.
+
+### Inline Policy
+
+The inline policy provides account-specific permissions that go beyond the managed policy, such as:
+- Access to customer-managed KMS keys
+- Access to specific S3 buckets with custom resource policies
+- Permissions for custom or third-party AWS services
+
+## Best Practices
+
+- Use the external ID condition in the trust policy to prevent confused deputy attacks.
+- Regularly audit cross-account role usage via CloudTrail.
+- Apply the principle of least privilege when creating inline policies.
+- Tag cross-account roles for easy identification and management.
+- Test the connection after setup to verify DevOps Agent can access resources.

training/data/raw/configuring-capabilities-for-aws-devops-agent-connecting-remote-a2a-agents.md

@@ -0,0 +1,109 @@
+# Connecting Remote A2A Agents
+
+You can extend the capabilities of AWS DevOps Agent by connecting remote agents that support the Agent-to-Agent (A2A) protocol. Remote A2A agents allow DevOps Agent to delegate specialized tasks to external agent systems.
+
+## Requirements
+
+To connect a remote A2A agent, it must meet the following requirements:
+
+- **A2A Protocol Support**: The remote agent must implement the A2A protocol specification, exposing a valid agent card at a discoverable endpoint.
+- **Authentication**: The remote agent must support one of the authentication methods recognized by DevOps Agent.
+- **Network Accessibility**: The agent endpoint must be reachable from AWS DevOps Agent (either publicly or via a private connection).
+
+## Security Considerations
+
+When connecting remote A2A agents, consider the following:
+
+- Remote agents receive context from DevOps Agent investigations, which may include sensitive operational data.
+- Ensure the remote agent endpoint uses TLS encryption.
+- Use the least-privilege authentication method appropriate for your use case.
+- Regularly rotate credentials used for agent authentication.
+- Monitor agent invocations through CloudWatch logs and metrics.
+
+## Registration
+
+### Agent Card Endpoint
+
+The remote agent must expose an agent card at a well-known endpoint. The agent card describes the agent's capabilities, supported tasks, and authentication requirements.
+
+```
+https://your-agent-host/.well-known/agent.json
+```
+
+The agent card must include:
+
+- Agent name and description
+- Supported task types
+- Input/output schemas
+- Authentication requirements
+
+### Authentication Methods
+
+DevOps Agent supports the following authentication methods when connecting to remote A2A agents:
+
+| Method | Description | Use Case |
+|--------|-------------|----------|
+| API Key | Static API key passed in a header | Simple integrations, development |
+| Bearer Token | OAuth 2.0 bearer token | Token-based auth systems |
+| OAuth 2.0 | Full OAuth 2.0 client credentials flow | Enterprise identity providers |
+| SigV4 | AWS Signature Version 4 | AWS-hosted agents, IAM-based auth |
+
+#### API Key Authentication
+
+```json
+{
+  "auth_type": "api_key",
+  "api_key_header": "X-API-Key",
+  "api_key_secret_arn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-agent-key"
+}
+```
+
+#### Bearer Token Authentication
+
+```json
+{
+  "auth_type": "bearer_token",
+  "token_secret_arn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-agent-token"
+}
+```
+
+#### OAuth 2.0 Authentication
+
+```json
+{
+  "auth_type": "oauth2",
+  "token_endpoint": "https://auth.example.com/oauth/token",
+  "client_id": "your-client-id",
+  "client_secret_arn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:oauth-secret",
+  "scopes": ["agent:invoke"]
+}
+```
+
+#### SigV4 Authentication
+
+```json
+{
+  "auth_type": "sigv4",
+  "service_name": "execute-api",
+  "region": "us-east-1"
+}
+```
+
+## Associating with Agent Spaces
+
+After registering a remote A2A agent, you must associate it with one or more Agent Spaces for it to be available during investigations.
+
+1. Navigate to the Agent Space configuration.
+2. Under **Connected Agents**, select **Add remote agent**.
+3. Choose the registered A2A agent from the list.
+4. Configure invocation permissions and any task-specific settings.
+5. Save the Agent Space configuration.
+
+The remote agent will now be available to DevOps Agent when working within that Agent Space. DevOps Agent will automatically determine when to delegate tasks to the remote agent based on the agent card's declared capabilities and the current investigation context.
+
+## Managing Remote Agents
+
+- **List registered agents**: View all registered remote A2A agents in the DevOps Agent console under Settings > Remote Agents.
+- **Update credentials**: Rotate authentication credentials by updating the referenced Secrets Manager secret.
+- **Remove an agent**: Disassociate from all Agent Spaces first, then delete the registration.
+- **Monitor usage**: Check CloudWatch metrics for invocation counts, latency, and error rates per remote agent.

training/data/raw/configuring-capabilities-for-aws-devops-agent-connecting-telemetry-sources-index.md

@@ -0,0 +1,65 @@
+# Connecting Telemetry Sources
+
+AWS DevOps Agent can ingest telemetry data from multiple observability platforms to enrich its investigations. There are three ways to connect telemetry sources.
+
+## Integration Types
+
+### Built-in 2-Way Integrations
+
+Two-way integrations allow DevOps Agent to both receive alerts from and query data in the telemetry platform.
+
+| Platform | Capabilities |
+|----------|-------------|
+| Dynatrace | Receive incidents, query metrics/traces/logs, create mitigations |
+
+With 2-way integrations, DevOps Agent can proactively pull additional context during investigations, such as related metrics, traces, and topology information.
+
+### Built-in 1-Way Integrations
+
+One-way integrations allow DevOps Agent to receive alerts and query data from the telemetry platform via MCP (Model Context Protocol) endpoints.
+
+| Platform | Protocol | Capabilities |
+|----------|----------|-------------|
+| Amazon CloudWatch | Native AWS | Metrics, logs, alarms |
+| Amazon S3 | Native AWS | Log files, data objects |
+| Datadog | MCP over OAuth | Metrics, monitors, events |
+| Grafana | MCP over Service Account | Dashboards, alerts, datasources |
+| New Relic | MCP over API Key | APM, infrastructure, alerts |
+| Splunk | MCP over Bearer Token | Search, alerts, dashboards |
+
+One-way integrations receive webhook notifications for alerts and can query the platform for additional data during investigations.
+
+### Bring-Your-Own via Webhook/MCP
+
+For observability platforms not covered by built-in integrations, you can connect custom sources using:
+
+- **Generic Webhooks**: Send alert data to DevOps Agent using the generic webhook endpoint with a standardized payload format.
+- **Custom MCP Servers**: Connect any MCP-compatible server to provide DevOps Agent with access to custom data sources.
+
+## Choosing an Integration Type
+
+| Consideration | 2-Way | 1-Way | Bring-Your-Own |
+|---------------|-------|-------|----------------|
+| Setup complexity | Medium | Low | Variable |
+| Data richness | Highest | High | Depends on implementation |
+| Bidirectional actions | Yes | No | No |
+| Maintenance | Managed by AWS | Managed by AWS | Customer-managed |
+
+## Getting Started
+
+1. Identify which telemetry platforms your team uses.
+2. Check if a built-in integration exists for your platform.
+3. Follow the platform-specific setup guide to register the integration.
+4. Associate the telemetry source with an Agent Space.
+5. Configure webhooks in your telemetry platform to send alerts to DevOps Agent.
+
+## Common Configuration Steps
+
+Regardless of the integration type, most telemetry connections follow this pattern:
+
+1. **Register** the telemetry source with DevOps Agent (provide credentials and endpoint information).
+2. **Enable** the source in one or more Agent Spaces.
+3. **Configure** the external platform to send webhooks to DevOps Agent.
+4. **Verify** the connection by triggering a test alert.
+
+See the individual platform guides for detailed instructions.

training/data/raw/configuring-capabilities-for-aws-devops-agent-connecting-to-cicd-pipelines-index.md

@@ -0,0 +1,11 @@
+# Connecting to CI/CD pipelines
+
+CI/CD pipeline integration enables AWS DevOps Agent to monitor deployments and correlate code changes with operational incidents during investigations.
+
+Two-step process:
+1. Account-level registration – Register your CI/CD provider once
+2. Agent Space connection – Connect specific projects or repositories
+
+## Supported CI/CD providers
++ **GitHub** – Connect repositories from GitHub.com using the AWS DevOps Agent GitHub app.
++ **GitLab** – Connect projects from GitLab.com, managed GitLab instances, or publicly accessible self-hosted GitLab deployments.

training/data/raw/configuring-capabilities-for-aws-devops-agent-connecting-to-privately-hosted-tools.md

@@ -0,0 +1,48 @@
+# Connecting to privately hosted tools
+
+## Private connections overview
+
+Private connections let you securely connect your Agent Space to services in your VPC without public internet exposure. Uses Amazon VPC Lattice for connectivity.
+
+Works with: MCP servers, self-hosted Grafana/Splunk, GitHub Enterprise Server, GitLab Self-Managed.
+
+## How private connections work
+
+1. You provide VPC, subnets, and security groups
+2. AWS DevOps Agent creates a service-managed resource gateway with ENIs
+3. Traffic routes privately through the resource gateway to your service
+
+## Security
++ No public internet exposure
++ Service-controlled resource gateway (read-only in your account)
++ Your security groups control traffic
++ Service-linked role with least privilege
+
+## Create a private connection
+
+### Prerequisites
++ Active Agent Space
++ Privately reachable target service (HTTPS, TLS 1.2+)
++ 1-20 subnets in your VPC
++ (Optional) Up to 5 security groups
+
+### Using the console
+1. Capability providers > Private connections > Create
+2. Enter Name, VPC, Subnets, IP address type
+3. (Optional) Security groups, Port ranges
+4. Enter Host address (IP or DNS)
+5. (Optional) Certificate public key for private CA
+6. Create connection (up to 10 minutes)
+
+### Using AWS CLI
+```
+aws devops-agent create-private-connection \
+    --name my-connection \
+    --mode '{"serviceManaged": {"hostAddress": "mcp.internal.example.com", "vpcId": "vpc-xxx", "subnetIds": ["subnet-xxx"], "portRanges": ["443"]}}'
+```
+
+## Use with capability providers
+Select "Connect to endpoint using a private connection" during registration of MCP Server, GitHub, GitLab, or Grafana.
+
+## Advanced: Self-managed mode
+Provide your own VPC Lattice resource configuration ARN instead of having AWS DevOps Agent create one.

training/data/raw/configuring-capabilities-for-aws-devops-agent-connecting-to-ticketing-and-chat-index.md

@@ -0,0 +1,8 @@
+# Connecting to ticketing and chat
+
+AWS DevOps Agent participates in your team's existing communication channels. Connect to ticketing and alarming systems like ServiceNow and PagerDuty to automatically launch investigations from incident tickets. Connect to Slack to receive activity summaries in a chat channel.
+
+Supported integrations:
++ **PagerDuty** - Trigger investigations from PagerDuty incidents using OAuth 2.0 Scoped OAuth
++ **ServiceNow** - Trigger investigations from ServiceNow tickets and receive updates in originating tickets
++ **Slack** - Receive investigation findings, root cause analyses, and mitigation plans in Slack channels

training/data/raw/configuring-capabilities-for-aws-devops-agent-integrating-devops-agent-into-event-driven-applications-using-amazon-eventbridge-index.md

@@ -0,0 +1,39 @@
+# Integrating AWS DevOps Agent with Amazon EventBridge
+
+AWS DevOps Agent sends events to Amazon EventBridge when investigation or mitigation states change. Create EventBridge rules to take action based on these events.
+
+## How EventBridge routes events
+1. AWS DevOps Agent sends event to default event bus
+2. EventBridge evaluates against your rules
+3. Matching events sent to specified targets
+
+## Supported investigation events
+| detail-type | Description |
+| --- | --- |
+| Investigation Created | Investigation created |
+| Investigation In Progress | Active analysis started |
+| Investigation Completed | Finished with findings |
+| Investigation Failed | Error encountered |
+| Investigation Timed Out | Exceeded max duration |
+| Investigation Cancelled | Canceled before completion |
+| Investigation Pending Triage | Awaiting triage |
+| Investigation Linked | Linked to related incident |
+| Investigation Skipped | Matched skip criteria |
+
+## Supported mitigation events
+| detail-type | Description |
+| --- | --- |
+| Mitigation In Progress | Started |
+| Mitigation Completed | Finished |
+| Mitigation Failed | Error |
+| Mitigation Timed Out | Exceeded duration |
+| Mitigation Cancelled | Canceled |
+
+## Event patterns
+
+Match all: `{"source": ["aws.aidevops"]}`
+Match investigations only: `{"source": ["aws.aidevops"], "detail-type": [{"prefix": "Investigation"}]}`
+Match specific space: `{"source": ["aws.aidevops"], "detail": {"metadata": {"agent_space_id": ["your-id"]}}}`
+
+## Permissions
+No additional permissions needed for event delivery. Configure target-specific permissions as needed.

training/data/raw/configuring-capabilities-for-aws-devops-agent-invoking-devops-agent-through-webhook.md

@@ -0,0 +1,56 @@
+# Invoking DevOps Agent through Webhook
+
+Webhooks allow external systems to automatically trigger AWS DevOps Agent investigations.
+
+## Webhook types
++ **Integration-specific webhooks** – Auto-generated with third-party integrations
++ **Generic webhooks** – Manually created, use HMAC authentication
++ **Grafana alert webhooks** – Through Grafana contact points
+
+## Authentication methods
+
+**HMAC authentication** – Used by Dynatrace and generic webhooks
+**Bearer token authentication** – Used by Splunk, Datadog, New Relic, ServiceNow, Slack, Grafana
+
+| Consideration | HMAC | Bearer token |
+| --- | --- | --- |
+| Setup complexity | More complex | Simpler |
+| Payload integrity | Verified | Not verified |
+| Replay protection | Built-in | Not built-in |
+| Secret exposure risk | Lower | Higher |
+
+## Webhook request format
+
+### HMAC (Version 1)
+Headers: Content-Type, x-amzn-event-signature, x-amzn-event-timestamp
+
+### Bearer token (Version 2)
+Headers: Content-Type, Authorization: Bearer <token>
+
+### Payload schema
+```json
+{
+    "eventType": "incident",
+    "incidentId": "string",
+    "action": "created | updated | closed | resolved",
+    "priority": "CRITICAL | HIGH | MEDIUM | LOW | MINIMAL",
+    "title": "string",
+    "description": "string",
+    "timestamp": "string",
+    "service": "string",
+    "data": {}
+}
+```
+
+## Example (cURL, HMAC)
+```bash
+TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%S.000Z)
+PAYLOAD='{"eventType":"incident","incidentId":"test-123","action":"created","priority":"HIGH","title":"Test Alert"}'
+SIGNATURE=$(echo -n "${TIMESTAMP}:${PAYLOAD}" | openssl dgst -sha256 -hmac "$SECRET" -binary | base64)
+curl -X POST "$WEBHOOK_URL" -H "Content-Type: application/json" -H "x-amzn-event-timestamp: $TIMESTAMP" -H "x-amzn-event-signature: $SIGNATURE" -d "$PAYLOAD"
+```
+
+## Troubleshooting
++ No 200 response: Check authentication/headers
++ 200 but no investigation: Check payload format, ensure unique incidentId/timestamp
++ Investigation immediately cancelled: May have hit monthly limit

training/data/raw/configuring-capabilities-for-aws-devops-agent-migrating-from-public-preview-to-general-availability.md

@@ -0,0 +1,41 @@
+# Migrating from public preview to general availability
+
+If you used AWS DevOps Agent during public preview, you must update your IAM roles before the GA release.
+
+## What's changing
+
+1. On-demand chat histories during preview are no longer accessible
+2. New managed policies replace policies available during preview
+3. Agent Spaces may have an outdated IAM Identity Center application access scope
+
+## New Managed Policies
+
+| Role type | Remove | Add |
+| --- | --- | --- |
+| Monitoring | AIOpsAssistantPolicy managed policy | AIDevOpsAgentAccessPolicy managed policy |
+| Operator (IAM and IDC) | Inline policy | AIDevOpsOperatorAppAccessPolicy managed policy |
+
+## Step 1: Update monitoring roles
+
+Update the monitoring role in your primary account and each secondary account:
+1. Remove the AIOpsAssistantPolicy AWS managed policy
+2. Attach the AIDevOpsAgentAccessPolicy managed policy
+3. Update inline policy for Resource Explorer service-linked role creation
+
+## Step 2: Update the operator role (IAM)
+
+1. Remove the existing inline policy
+2. Attach the AIDevOpsOperatorAppAccessPolicy managed policy
+3. Update trust policy to include sts:TagSession
+
+## Step 3: Update operator roles (IDC)
+
+If you use IAM Identity Center:
+1. Remove existing inline policy
+2. Attach AIDevOpsOperatorAppAccessPolicy
+3. Update trust policy with TrustedIdentityPropagation statement
+4. Create inline policy for sso:ListInstances, identitystore:DescribeUser
+
+## Reconnect IAM Identity Center (if applicable)
+
+Agent Spaces created during preview may have outdated scope (awsaidevops:read_write instead of aidevops:read_write). Disconnect and reconnect IAM Identity Center to update.

training/data/raw/configuring-capabilities-for-aws-devops-agent-vended-logs-and-metrics.md

@@ -0,0 +1,43 @@
+# Vended Logs and Metrics
+
+Monitor agent spaces using CloudWatch metrics and vended logs.
+
+## Vended CloudWatch metrics
+
+Namespace: `AWS/AIDevOps`
+
+| Metric name | Description | Unit |
+| --- | --- | --- |
+| ConsumedChatRequests | Chat requests consumed | Count |
+| ConsumedInvestigationTime | Time spent on investigations | Seconds |
+| ConsumedEvaluationTime | Time spent on evaluations | Seconds |
+| TopologyCompletionCount | Topology processing completions | Count |
+
+Dimension: AgentSpaceUUID
+
+### Service-Linked Role
+AWS DevOps Agent creates AWSServiceRoleForAIDevOps automatically. Customers before March 13, 2026 must create manually.
+
+## Vended logs
+
+Log type: APPLICATION_LOGS
+
+Destinations:
++ Amazon CloudWatch Logs
++ Amazon S3
++ Amazon Data Firehose
+
+### Log event types
+| Event | Level |
+| --- | --- |
+| Agent inbound event received | INFO |
+| Topology creation queued/started/finished | INFO |
+| Resource discovery failed | ERROR |
+| Service registration failed | ERROR |
+| Webhook validation fails | ERROR |
+| Association validation status updates | ERROR/INFO |
+
+### Configure log delivery
+Use the AWS DevOps Agent console (Settings or Agent Space Configuration tab) or CloudWatch Logs API (PutDeliverySource, PutDeliveryDestination, CreateDelivery).
+
+Required permission: aidevops:AllowVendedLogDeliveryForResource

training/data/raw/configuring-capabilities-for-aws-devops-agent.md

@@ -0,0 +1,16 @@
+# Configuring capabilities for AWS DevOps Agent
+
+AWS DevOps Agent capabilities extend your agent's functionality by connecting it to your existing tools and infrastructure. Configure these capabilities to enable comprehensive incident investigation, automated response workflows, and seamless integration with your DevOps ecosystem.
+
+The following capabilities help you maximize your DevOps Agent's effectiveness:
++ **AWS EKS Access Setup** - Enable introspection of Kubernetes clusters, pod logs, and cluster events for both public and private EKS environments
++ **Azure Integration** - Connect Azure subscriptions and Azure DevOps organizations to investigate Azure resources and correlate Azure DevOps deployments with incidents
++ **CI/CD Pipeline Integration** - Connect GitHub and GitLab pipelines to correlate deployments with incidents and track code changes during investigations
++ **MCP Server Connections** - Extend investigation capabilities by connecting external observability tools and custom monitoring systems through Model Context Protocol
++ **Multi-Account AWS Access** - Configure secondary AWS accounts to investigate resources across your entire organization during incident response
++ **Telemetry Source Integration** - Connect monitoring platforms like Datadog, Dynatrace, Grafana, New Relic, and Splunk for comprehensive observability data access
++ **Ticketing and Chat Integration** - Connect ServiceNow, PagerDuty, and Slack to automate incident response workflows and enable team collaboration
++ **Webhook Configuration** - Allow external systems to automatically trigger DevOps Agent investigations through HTTP requests
++ **Amazon EventBridge Integration** - Incorporate AWS DevOps Agent into event-driven applications by routing investigation and mitigation lifecycle events to Amazon EventBridge targets
+
+You can configure each capability independently based on your team's specific needs and existing tool stack.

training/data/raw/connecting-azure-connecting-azure-devops.md

@@ -0,0 +1,31 @@
+# Connecting Azure DevOps
+
+Azure DevOps integration enables AWS DevOps Agent to access repositories and pipeline execution history in your Azure DevOps organization.
+
+## Registering Azure DevOps via Admin Consent
+
+1. Navigate to Capability Providers > Azure DevOps > Register
+2. Enter Azure DevOps organization name
+3. Complete admin consent flow
+4. Complete user authorization
+5. Grant access in Azure DevOps
+
+## Registering Azure DevOps via App Registration
+
+If already completed for Azure Resources, skip to Granting access.
+
+## Granting access in Azure DevOps
+
+1. In Azure DevOps: Organization Settings > Users > Add Users
+2. Search for the application
+3. Set access level to Basic
+4. Select projects to access
+5. Assign Project Readers group
+
+**Security Requirement:** Assign only the Project Readers group for read-only access.
+
+## Associating a project with an Agent Space
+
+1. Select Agent Space > Capabilities tab
+2. Pipelines section > Add > Azure DevOps
+3. Select the project

training/data/raw/connecting-azure-connecting-azure-resources.md

@@ -0,0 +1,38 @@
+# Connecting Azure Resources
+
+Azure Resources integration enables AWS DevOps Agent to discover and investigate resources in your Azure subscriptions during incident investigations.
+
+## Registering Azure Resources via Admin Consent
+
+1. Navigate to Capability Providers page and Register Azure Cloud
+2. Select Admin Consent method
+3. Complete admin consent on Microsoft Entra page
+4. Complete user authorization
+5. Assign Reader role to the application in Azure Portal
+
+## Registering Azure Resources via App Registration
+
+1. Navigate to Capability Providers and Register
+2. Select App Registration method
+3. Create/configure Entra application with federated identity credentials
+4. Provide Tenant ID, Client ID, Audience
+5. Assign Reader role
+
+## Assigning Azure roles
+
+1. In Azure Portal, navigate to subscription > Access Control (IAM)
+2. Add role assignment > Reader role
+3. Select the application (AWS DevOps Agent or your Entra app)
+
+**Security Requirement:** Only assign the Reader role. The Reader role restricts the agent to read-only operations.
+
+### AKS access setup (optional)
+
+1. Assign Azure Kubernetes Service Cluster User Role
+2. For Kubernetes API access, use Azure RBAC (recommended) or Azure AD + Kubernetes RBAC
+
+## Associating a subscription with an Agent Space
+
+1. Select Agent Space > Capabilities tab
+2. Secondary sources > Add > Azure
+3. Provide Subscription ID

training/data/raw/connecting-telemetry-sources-connecting-datadog.md

@@ -0,0 +1,98 @@
+# Connecting Datadog
+
+AWS DevOps Agent supports a 1-way integration with Datadog. This integration allows DevOps Agent to receive alert notifications from Datadog and query Datadog for metrics, monitors, and events during investigations via an MCP endpoint.
+
+## Prerequisites
+
+- Datadog account with admin permissions
+- An API Key and Application Key with appropriate scopes
+- AWS DevOps Agent deployed and configured
+
+## Step 1: Connect (MCP Endpoint with OAuth)
+
+Register the Datadog integration with DevOps Agent by providing the MCP endpoint and OAuth credentials.
+
+1. In the DevOps Agent console, navigate to **Telemetry Sources > Add > Datadog**.
+2. Select your Datadog site region (determines the API endpoint).
+3. Provide your Datadog API Key and Application Key.
+4. DevOps Agent will configure the MCP endpoint for querying Datadog.
+
+### Datadog Site Endpoints
+
+| Site | Region | API Endpoint |
+|------|--------|-------------|
+| US1 | US (Virginia) | `https://api.datadoghq.com` |
+| US3 | US (Oregon) | `https://api.us3.datadoghq.com` |
+| US5 | US (Ohio) | `https://api.us5.datadoghq.com` |
+| EU1 | EU (Frankfurt) | `https://api.datadoghq.eu` |
+| AP1 | Asia Pacific (Tokyo) | `https://api.ap1.datadoghq.com` |
+| AP2 | Asia Pacific (Sydney) | `https://api.ap2.datadoghq.com` |
+
+Ensure you select the correct site that matches your Datadog organization.
+
+## Step 2: Enable in Agent Space
+
+Associate the Datadog integration with one or more Agent Spaces.
+
+1. Navigate to the Agent Space configuration.
+2. Under **Telemetry Sources**, select **Add source**.
+3. Choose the registered Datadog integration.
+4. Configure which Datadog services or tags to scope the integration (optional).
+5. Save the Agent Space configuration.
+
+## Step 3: Configure Webhooks
+
+Configure Datadog to send alert notifications to DevOps Agent when monitors trigger.
+
+1. In Datadog, navigate to **Integrations > Webhooks**.
+2. Create a new webhook:
+   - **Name**: `aws-devops-agent`
+   - **URL**: The webhook URL provided in the DevOps Agent console
+   - **Payload**: Use the following JSON template:
+
+```json
+{
+  "event_type": "alert",
+  "source": "datadog",
+  "monitor_id": "$ALERT_ID",
+  "monitor_name": "$ALERT_TITLE",
+  "status": "$ALERT_TRANSITION",
+  "priority": "$ALERT_PRIORITY",
+  "tags": "$TAGS",
+  "hostname": "$HOSTNAME",
+  "message": "$TEXT_ONLY_MSG",
+  "snapshot_url": "$SNAPSHOT",
+  "timestamp": "$DATE"
+}
+```
+
+3. **Custom Headers**: Add the authentication header:
+   - For HMAC: `X-Signature: <computed-hmac>`
+   - For Bearer: `Authorization: Bearer <token>`
+
+4. **Assign to monitors**: Add the `@webhook-aws-devops-agent` notification to the monitors you want to route to DevOps Agent.
+
+## Capabilities
+
+With the Datadog integration, DevOps Agent can:
+
+- Receive real-time monitor alert notifications
+- Query metric timeseries data for specific hosts and services
+- List and inspect monitor configurations
+- Retrieve recent events and their context
+- Correlate Datadog alerts with AWS resource state
+
+## Scoping the Integration
+
+You can limit which Datadog data DevOps Agent can access by:
+
+- Restricting the Application Key's scopes
+- Using tag-based filtering when associating with Agent Spaces
+- Configuring monitor-specific webhook notifications
+
+## Troubleshooting
+
+- **Authentication failures**: Verify the API Key and Application Key are valid and have not been revoked.
+- **Wrong endpoint**: Confirm you selected the correct Datadog site region.
+- **Webhooks not firing**: Ensure the `@webhook-aws-devops-agent` mention is included in monitor notification messages.
+- **Missing metrics**: Check that the Application Key has the `metrics_read` scope.

training/data/raw/connecting-telemetry-sources-connecting-dynatrace.md

@@ -0,0 +1,121 @@
+# Connecting Dynatrace
+
+AWS DevOps Agent supports a 2-way integration with Dynatrace SaaS. This integration allows DevOps Agent to receive incident notifications from Dynatrace and actively query the Dynatrace environment for metrics, traces, logs, and topology data during investigations.
+
+## Prerequisites
+
+- Dynatrace SaaS environment (not Dynatrace Managed)
+- Dynatrace account with admin permissions to create OAuth clients
+- AWS DevOps Agent deployed and configured
+
+## Onboarding
+
+### Step 1: Connect (OAuth Client Setup)
+
+Create an OAuth client in Dynatrace to allow DevOps Agent to authenticate.
+
+1. In Dynatrace, navigate to **Account Management > Identity & access management > OAuth clients**.
+2. Create a new OAuth client with the following scopes:
+   - `storage:logs:read`
+   - `storage:metrics:read`
+   - `storage:events:read`
+   - `storage:entities:read`
+   - `storage:bizevents:read`
+   - `environment-api:metrics:read`
+   - `environment-api:entities:read`
+   - `environment-api:events:read`
+   - `environment-api:problems:read`
+   - `environment-api:slo:read`
+3. Note the client ID and client secret.
+4. In the DevOps Agent console, navigate to **Telemetry Sources > Add > Dynatrace**.
+5. Enter the OAuth client ID and client secret.
+
+### Step 2: Enable (Environment and Entity Configuration)
+
+Configure which Dynatrace environment and entities DevOps Agent should monitor.
+
+1. Provide the **Environment ID** of your Dynatrace SaaS environment.
+   - Found in your Dynatrace URL: `https://<environment-id>.live.dynatrace.com`
+2. Specify **Entity IDs** to scope the integration:
+   - Host groups
+   - Management zones
+   - Specific services or applications
+3. Associate the Dynatrace connection with an Agent Space.
+
+### Step 3: Configure Dynatrace Environment
+
+Install and configure the SRE Agents app in your Dynatrace environment.
+
+1. In Dynatrace, go to the **Hub** and install the **SRE Agents** app.
+2. Open the SRE Agents app and configure the webhook destination:
+   - Webhook URL: Provided in the DevOps Agent console after registration
+   - Authentication: HMAC signature verification
+3. Configure which problem types should trigger notifications to DevOps Agent.
+
+## Supported Event Schemas
+
+### Incident Events
+
+Dynatrace sends incident events when problems are detected:
+
+```json
+{
+  "event_type": "incident",
+  "source": "dynatrace",
+  "problem_id": "P-12345",
+  "title": "High CPU on host-001",
+  "severity": "CRITICAL",
+  "status": "OPEN",
+  "impact_level": "SERVICE",
+  "affected_entities": [
+    {
+      "entity_id": "HOST-ABC123",
+      "entity_type": "HOST",
+      "entity_name": "host-001"
+    }
+  ],
+  "root_cause": {
+    "entity_id": "PROCESS_GROUP_INSTANCE-XYZ",
+    "entity_type": "PROCESS_GROUP_INSTANCE"
+  },
+  "start_time": "2024-01-15T10:30:00Z"
+}
+```
+
+### Mitigation Events
+
+DevOps Agent can send mitigation actions back to Dynatrace:
+
+```json
+{
+  "event_type": "mitigation",
+  "source": "aws_devops_agent",
+  "problem_id": "P-12345",
+  "action": "scale_up",
+  "status": "EXECUTED",
+  "details": {
+    "resource": "arn:aws:ecs:us-east-1:123456789012:service/my-service",
+    "previous_count": 2,
+    "new_count": 4
+  },
+  "timestamp": "2024-01-15T10:35:00Z"
+}
+```
+
+## Capabilities
+
+With the 2-way Dynatrace integration, DevOps Agent can:
+
+- Receive real-time problem notifications
+- Query metrics for affected entities
+- Retrieve distributed traces for failing services
+- Access log data related to detected problems
+- Understand service topology and dependencies
+- Correlate Dynatrace problems with AWS resource events
+- Report mitigation actions taken back to Dynatrace
+
+## Troubleshooting
+
+- **OAuth errors**: Verify the client ID and secret, and ensure all required scopes are granted.
+- **No events received**: Check that the SRE Agents app is installed and the webhook URL is correctly configured.
+- **Missing data in queries**: Ensure the entity IDs are correct and the OAuth client has the necessary read scopes.

training/data/raw/connecting-telemetry-sources-connecting-grafana.md

@@ -0,0 +1,115 @@
+# Connecting Grafana
+
+AWS DevOps Agent supports integration with Grafana 9.0 and later. This integration allows DevOps Agent to receive alert notifications from Grafana and query dashboards, datasources, and alert rules during investigations.
+
+## Prerequisites
+
+- Grafana 9.0 or later (self-hosted or Grafana Cloud)
+- Grafana service account with appropriate permissions
+- AWS DevOps Agent deployed and configured
+
+## Step 1: Register with Service Account Token
+
+Create a Grafana service account and generate a token for DevOps Agent.
+
+1. In Grafana, navigate to **Administration > Service Accounts**.
+2. Create a new service account:
+   - **Name**: `aws-devops-agent`
+   - **Role**: Viewer (minimum required)
+3. Generate a token for the service account and note it securely.
+4. In the DevOps Agent console, navigate to **Telemetry Sources > Add > Grafana**.
+5. Provide:
+   - **Grafana URL**: The base URL of your Grafana instance (e.g., `https://grafana.example.com`)
+   - **Service Account Token**: The token generated above
+6. DevOps Agent will validate the connection and register the MCP endpoint.
+
+## Step 2: Add to Agent Space
+
+Associate the Grafana integration with an Agent Space.
+
+1. Navigate to the Agent Space configuration.
+2. Under **Telemetry Sources**, select **Add source**.
+3. Choose the registered Grafana integration.
+4. Optionally scope to specific folders or dashboards.
+5. Save the configuration.
+
+## Step 3: Configure Alert Webhooks
+
+Configure Grafana to send alert notifications to DevOps Agent.
+
+### Create a Contact Point
+
+1. In Grafana, go to **Alerting > Contact points**.
+2. Create a new contact point:
+   - **Name**: `aws-devops-agent`
+   - **Type**: Webhook
+   - **URL**: The webhook URL from the DevOps Agent console
+   - **HTTP Method**: POST
+   - **Authorization Header**: Bearer token or HMAC as configured
+
+### Custom Notification Template
+
+Create a custom notification template for the webhook payload:
+
+1. Go to **Alerting > Notification templates**.
+2. Create a new template:
+
+```
+{{ define "devops_agent_payload" }}
+{
+  "event_type": "alert",
+  "source": "grafana",
+  "alert_name": "{{ .CommonLabels.alertname }}",
+  "status": "{{ .Status }}",
+  "severity": "{{ .CommonLabels.severity }}",
+  "summary": "{{ .CommonAnnotations.summary }}",
+  "description": "{{ .CommonAnnotations.description }}",
+  "dashboard_url": "{{ .CommonAnnotations.dashboard_url }}",
+  "panel_id": "{{ .CommonAnnotations.panel_id }}",
+  "values": {{ .Values | toJSON }},
+  "labels": {{ .CommonLabels | toJSON }},
+  "starts_at": "{{ .StartsAt }}",
+  "ends_at": "{{ .EndsAt }}"
+}
+{{ end }}
+```
+
+3. Assign the template to the `aws-devops-agent` contact point.
+
+### Create a Notification Policy
+
+1. Go to **Alerting > Notification policies**.
+2. Create a policy that routes relevant alerts to the `aws-devops-agent` contact point.
+3. Use label matchers to control which alerts are sent (e.g., `team=platform` or `severity=critical`).
+
+## Capabilities
+
+With the Grafana integration, DevOps Agent can:
+
+- Receive real-time alert notifications
+- Query dashboard panels for metric visualizations
+- List and inspect alert rules and their states
+- Access datasource queries for additional context
+- Retrieve annotation data for correlation
+
+## Limitations
+
+### ClickHouse Not Supported
+
+DevOps Agent cannot directly query ClickHouse datasources through the Grafana integration. If your Grafana dashboards use ClickHouse as a backend, the agent can view dashboard panels but cannot execute arbitrary ClickHouse queries.
+
+### Amazon Managed Grafana Webhook Limitations
+
+When using Amazon Managed Grafana:
+
+- Webhooks must be configured through the AWS-managed alert notification channels.
+- Some self-hosted Grafana webhook features may not be available.
+- Ensure the Managed Grafana workspace has outbound network access to the DevOps Agent webhook endpoint.
+- VPC endpoint configuration may be required for private connectivity.
+
+## Troubleshooting
+
+- **Authentication errors**: Verify the service account token is valid and has Viewer role.
+- **Connection timeout**: Ensure the Grafana URL is reachable from DevOps Agent. Use a private connection if Grafana is not publicly accessible.
+- **Missing dashboards**: Check that the service account has access to the folders containing the dashboards.
+- **Webhooks not received**: Verify the notification policy routes alerts to the correct contact point.

training/data/raw/connecting-telemetry-sources-connecting-new-relic.md

@@ -0,0 +1,120 @@
+# Connecting New Relic
+
+AWS DevOps Agent supports a 1-way integration with New Relic. This integration allows DevOps Agent to receive alert notifications from New Relic and query APM, infrastructure, and alert data during investigations.
+
+## Prerequisites
+
+- New Relic account with a **Full Platform User** license
+- New Relic User API Key (not the license key or ingest key)
+- AWS DevOps Agent deployed and configured
+
+## Step 1: Register
+
+Register the New Relic integration with DevOps Agent.
+
+1. In the DevOps Agent console, navigate to **Telemetry Sources > Add > New Relic**.
+2. Provide the following:
+   - **API Key**: A New Relic User API Key with Full Platform User permissions
+   - **Account ID**: Your New Relic account ID (found in **Administration > Account settings**)
+   - **Region**: Select your New Relic data center region
+
+### New Relic Regions
+
+| Region | API Endpoint |
+|--------|-------------|
+| US | `https://api.newrelic.com` |
+| EU | `https://api.eu.newrelic.com` |
+
+3. DevOps Agent will validate the credentials and configure the MCP endpoint.
+
+## Step 2: Enable in Agent Space
+
+Associate the New Relic integration with an Agent Space.
+
+1. Navigate to the Agent Space configuration.
+2. Under **Telemetry Sources**, select **Add source**.
+3. Choose the registered New Relic integration.
+4. Optionally configure:
+   - Specific application names to monitor
+   - Tag-based filtering for entities
+5. Save the Agent Space configuration.
+
+## Step 3: Configure Webhooks
+
+Configure New Relic to send alert notifications to DevOps Agent.
+
+### Create a Notification Destination
+
+1. In New Relic, navigate to **Alerts & AI > Destinations**.
+2. Create a new destination:
+   - **Type**: Webhook
+   - **Name**: `aws-devops-agent`
+   - **Endpoint URL**: The webhook URL from the DevOps Agent console
+   - **Authentication**: Add the appropriate auth header (Bearer token or HMAC)
+
+### Create a Notification Channel
+
+1. Navigate to **Alerts & AI > Workflows**.
+2. Create a new workflow or edit an existing one.
+3. Add a notification channel:
+   - **Destination**: Select the `aws-devops-agent` webhook destination
+   - **Payload template**:
+
+```json
+{
+  "event_type": "alert",
+  "source": "new_relic",
+  "condition_name": "{{ conditionName }}",
+  "policy_name": "{{ policyName }}",
+  "account_id": "{{ accountId }}",
+  "status": "{{ state }}",
+  "severity": "{{ priority }}",
+  "incident_id": "{{ incidentId }}",
+  "details": "{{ details }}",
+  "entity": {
+    "name": "{{ entitiesData.names }}",
+    "type": "{{ entitiesData.types }}",
+    "id": "{{ entitiesData.ids }}"
+  },
+  "timestamp": "{{ createdAt }}",
+  "violation_url": "{{ violationChartUrl }}"
+}
+```
+
+### Configure Workflow Filters
+
+Use workflow filters to control which alerts are sent to DevOps Agent:
+
+- Filter by policy name
+- Filter by condition priority (Critical, High, Medium, Low)
+- Filter by entity tags
+
+## Capabilities
+
+With the New Relic integration, DevOps Agent can:
+
+- Receive real-time alert and incident notifications
+- Query APM data (transactions, errors, throughput)
+- Access infrastructure metrics (CPU, memory, disk)
+- Retrieve alert condition configurations
+- Query NRQL for custom data exploration
+- Correlate New Relic incidents with AWS events
+
+## API Key Requirements
+
+The API Key must belong to a user with **Full Platform User** access. This ensures DevOps Agent can:
+
+- Read APM application data
+- Query infrastructure metrics
+- Access alert policies and conditions
+- Execute NRQL queries
+- Read entity relationships
+
+Keys from Basic or Core users will not have sufficient permissions.
+
+## Troubleshooting
+
+- **401 Unauthorized**: Verify the API Key is valid and belongs to a Full Platform User.
+- **Wrong region**: Ensure US vs. EU region matches your New Relic account's data center.
+- **Missing data**: Check that the account ID is correct and the API Key has access to the relevant accounts.
+- **Webhook delivery failures**: Verify the endpoint URL is correct and the authentication header matches.

training/data/raw/connecting-telemetry-sources-connecting-splunk.md

@@ -0,0 +1,46 @@
+# Connecting Splunk
+
+## Built-in, 1 way integration
+
+AWS DevOps Agent supports Splunk with built-in 1-way integration enabling:
++ **Automated Investigation triggering** - Splunk events can trigger investigations via webhooks.
++ **Telemetry introspection** - AWS DevOps Agent can introspect Splunk telemetry via the remote MCP server.
+
+## Prerequisites
+
+### Splunk Administrator steps
++ Enable REST API access
++ Enable token authentication
++ Create a new role 'mcp_user'
++ Assign 'mcp_user' role to authorized users
++ Create tokens with audience 'mcp'
+
+### Splunk User steps
++ Get a token with audience 'mcp'
+
+## Onboarding
+
+### Step 1: Connect
+
+1. Go to Capability Providers page
+2. Find Splunk under Telemetry and choose Register
+3. Enter details:
+   + Server Name - Unique identifier
+   + Endpoint URL: `https://<YOUR_SPLUNK_DEPLOYMENT_NAME>.api.scs.splunk.com/<YOUR_SPLUNK_DEPLOYMENT_NAME>/mcp/v1/`
+   + Token Name and Token Value
+
+### Step 2: Enable
+
+1. Select Agent Space > Capabilities tab
+2. Telemetry section > Add > Splunk
+3. Review and Save
+4. Copy Webhook URL and API Key
+
+### Step 3: Configure webhooks
+
+Use bearer token authentication with the API Key from Step 2.
+
+## Removal
+
+1. Remove from agent space (Capabilities > Telemetry > Splunk > Remove)
+2. Deregister from account (Capability Providers > Deregister)

training/data/raw/connecting-to-cicd-pipelines-connecting-github.md

@@ -0,0 +1,37 @@
+# Connecting GitHub
+
+GitHub integration enables AWS DevOps Agent to access code repositories and receive deployment events during incident investigations. Supports both GitHub.com and GitHub Enterprise Server.
+
+## Prerequisites
++ AWS DevOps Agent admin console access
++ GitHub user account or organization with admin permissions
++ Authorization to install GitHub apps
+
+## Registering GitHub (account-level)
+
+1. Navigate to Capabilities tab > Pipeline > Add > GitHub
+2. Choose connection type (User or Organization)
+3. For GitHub Enterprise Server, check "Use GitHub Enterprise Server" and enter HTTPS URL
+4. Submit to begin GitHub App setup
+5. Install the AWS DevOps Agent GitHub app
+6. Select repositories (All or specific)
+7. Choose Install & Authorize
+
+## Connecting repositories to an Agent Space
+
+1. Select Agent Space > Capabilities tab
+2. Pipeline section > Add > GitHub
+3. Select repositories
+4. Choose Add
+
+## Understanding the GitHub app
+
+The app:
++ Requests repository access
++ Receives deployment events
++ Allows correlating code changes with incidents
++ Can be uninstalled through GitHub settings
+
+## GitHub App permission updates
+
+AWS DevOps Agent may request permission updates for new features. Accept requests when notified.

training/data/raw/connecting-to-cicd-pipelines-connecting-gitlab.md

@@ -0,0 +1,29 @@
+# Connecting GitLab
+
+GitLab integration enables AWS DevOps Agent to monitor deployments from GitLab Pipelines to inform causal investigations.
+
+## Registering GitLab (account-level)
+
+1. Navigate to Capability Providers > GitLab > Register
+2. Select connection type (Personal or Group)
+3. Select instance type (GitLab.com or self-hosted)
+4. Provide personal access token with permissions:
+   + read_repository
+   + read_virtual_registry
+   + read_registry
+   + api
+   + self_rotate
+5. Set token expiration (max 365 days)
+6. Submit registration
+
+## Connecting projects to an Agent Space
+
+1. Select Agent Space > Capabilities tab
+2. Pipeline section > Add > GitLab
+3. Select projects
+4. Choose Save
+
+## Managing GitLab connections
++ Update access token at account level if expired
++ View connected projects in Capabilities tab
++ Remove connections from Pipeline section

training/data/raw/connecting-to-ticketing-and-chat-connecting-pagerduty.md

@@ -0,0 +1,29 @@
+# Connecting PagerDuty
+
+PagerDuty integration enables AWS DevOps Agent to access and update incident data, on-call schedules, and service information using OAuth 2.0.
+
+**Important:** Only PagerDuty OAuth 2.0 (Scoped OAuth) is supported. Legacy OAuth with redirect URI is not supported.
+
+## PagerDuty requirements
++ PagerDuty account with OAuth client ID and client secret
++ PagerDuty account subdomain
+
+## Registering PagerDuty
+
+1. Go to Capability Providers > PagerDuty > Register
+2. Select region (US or EU) and enter subdomain
+3. Create app in PagerDuty: Integrations > App Registration > OAuth 2.0 Scoped OAuth
+4. Required scopes: incidents.read, incidents.write, services.read, webhook_subscriptions.read, webhook_subscriptions.write
+5. Enable Events Integration
+6. Enter Client name, Client ID, Client secret
+7. Submit
+
+## Adding PagerDuty to an Agent Space
+
+1. Select Agent Space > Capabilities tab
+2. Communications > Add > PagerDuty
+3. Save
+
+## Webhook support
+
+Only PagerDuty V3 webhooks are supported.

training/data/raw/connecting-to-ticketing-and-chat-connecting-servicenow.md

@@ -0,0 +1,34 @@
+# Connecting ServiceNow
+
+Connect ServiceNow to automatically initiate investigations when tickets are created and post findings into originating tickets.
+
+## Initial Setup
+
+### Create a ServiceNow OAuth application client
+1. Enable client credential system property: glide.oauth.inbound.client.credential.grant_type.enabled = true
+2. Navigate to System OAuth > Application Registry
+3. Create New > OAuth - Client Credentials Grant
+4. Set OAuth application user to "Problem Administrator"
+
+### Connect OAuth client to AWS DevOps Agent
+1. Navigate to Capability Providers > ServiceNow > Register
+2. Enter Client Name, Client ID, Client Secret, Instance URL
+3. Save webhook URL and secret
+
+### Configure ServiceNow Business Rule
+1. Navigate to Activity Subscriptions > Business Rules > New
+2. Set Table to "Incident [incident]"
+3. Check Advanced, set to run after Insert, Update, Delete
+4. Add webhook script with your webhook secret and URL
+
+## ServiceNow ticket updates
+
+During investigations, DevOps Agent posts findings, root cause analyses, and mitigation plans as comments on the originating ticket.
+
+## Ticket routing examples
+
+### Filtering incidents
+Use Business Rule filter conditions to send only specific tickets (e.g., by source, category, urgency).
+
+### Routing across multiple Agent Spaces
+Create separate Business Rules with different conditions routing to different Agent Space webhooks.

training/data/raw/connecting-to-ticketing-and-chat-connecting-slack.md

@@ -0,0 +1,25 @@
+# Connecting Slack
+
+Configure AWS DevOps Agent to update Slack channels with investigation findings, root cause analyses, and mitigation plans.
+
+## Prerequisites
++ Slack workspace with ability to install third-party apps
++ Identified target Slack channels
+
+## Register Slack integration
+
+1. From Capability Providers > Slack > Register
+2. Redirected to Slack for authorization
+3. Install directly to workspaces (not organization level)
+4. Choose workspace, review scopes, choose Allow
+
+## Associate Slack with Agent Space
+
+1. From Capabilities tab > Communications > Slack
+2. Select Add Slack
+3. Enter Channel ID
+4. Choose Create
+
+**Note:** The bot must be added to private channels before it can post messages.
+
+**Important:** Avoid uninstalling the Slack app as it may prevent reinstallation.