plans/ci-cd-devops.md

# CI/CD & DevOps Enhancement Plan

## Current State

**Existing**:
- Basic Go build workflow
- Dockerfile for containerization
- Docker Compose setup
- GoReleaser configuration

**Missing**:
- Comprehensive testing in CI
- Security scanning
- Multi-environment deployment
- Infrastructure as Code
- Automated rollback
- Release automation

## Phase 1: Enhanced CI Pipeline (Week 1)

### 1.1 Main CI Workflow
**File**: `.github/workflows/ci.yml`

```yaml
name: CI

on:
  push:
    branches: [main, develop]
    tags: ['v*']
  pull_request:
    branches: [main, develop]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-go@v5
        with:
          go-version: '1.24'
          cache: true

      - name: golangci-lint
        uses: golangci/golangci-lint-action@v6
        with:
          version: latest
          args: --timeout=5m --out-format=colored-line-number

      - name: Check formatting
        run: |
          fmt_files=$(gofmt -l .)
          if [ -n "$fmt_files" ]; then
            echo "The following files need formatting:"
            echo "$fmt_files"
            exit 1
          fi

  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432

      redis:
        image: redis:7-alpine
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379

      minio:
        image: minio/minio:latest
        env:
          MINIO_ROOT_USER: minioadmin
          MINIO_ROOT_PASSWORD: minioadmin
        ports:
          - 9000:9000
        options: >-
          server /data

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-go@v5
        with:
          go-version: '1.24'
          cache: true

      - name: Download dependencies
        run: go mod download

      - name: Run unit tests
        run: go test -race -coverprofile=coverage.out ./internal/... ./sdk/...
        env:
          CGO_ENABLED: 1

      - name: Run integration tests
        run: go test -v -tags=integration ./test/integration/...
        env:
          TEST_POSTGRES_URL: postgres://postgres:postgres@localhost:5432/test?sslmode=disable
          TEST_REDIS_URL: redis://localhost:6379
          TEST_MINIO_ENDPOINT: localhost:9000
          TEST_MINIO_ACCESS_KEY: minioadmin
          TEST_MINIO_SECRET_KEY: minioadmin

      - name: Generate coverage report
        run: |
          go tool cover -html=coverage.out -o coverage.html
          go tool cover -func=coverage.out

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage.out
          fail_ci_if_error: true
          verbose: true

  build:
    runs-on: ubuntu-latest
    needs: [lint, test]
    strategy:
      matrix:
        include:
          - platform: linux/amd64
            goos: linux
            goarch: amd64
          - platform: linux/arm64
            goos: linux
            goarch: arm64
          - platform: darwin/amd64
            goos: darwin
            goarch: amd64
          - platform: darwin/arm64
            goos: darwin
            goarch: arm64
          - platform: windows/amd64
            goos: windows
            goarch: amd64
            extension: .exe

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-go@v5
        with:
          go-version: '1.24'
          cache: true

      - name: Build binary
        env:
          GOOS: ${{ matrix.goos }}
          GOARCH: ${{ matrix.goarch }}
          CGO_ENABLED: 0
        run: |
          go build -ldflags "-s -w -X main.version=${{ github.ref_name }} -X main.commit=${{ github.sha }} -X main.date=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
            -o dist/cliproxy-${{ matrix.goos }}-${{ matrix.goarch }}${{ matrix.extension }} \
            ./cmd/server

      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: cliproxy-${{ matrix.goos }}-${{ matrix.goarch }}
          path: dist/cliproxy-${{ matrix.goos }}-${{ matrix.goarch }}${{ matrix.extension }}
          retention-days: 7

  docker:
    runs-on: ubuntu-latest
    needs: [lint, test]
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Login to GHCR
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: |
            ${{ secrets.DOCKER_USERNAME }}/cliproxy
            ghcr.io/${{ github.repository }}
          tags: |
            type=ref,event=branch
            type=ref,event=pr
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=sha,prefix=,suffix=,format=short

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          build-args: |
            VERSION=${{ github.ref_name }}
            COMMIT=${{ github.sha }}
            DATE=${{ github.event.head_commit.timestamp }}

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ghcr.io/${{ github.repository }}:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'

      - name: Upload Trivy results
        uses: github/codeql-action/upload-sarif@v2
        if: always()
        with:
          sarif_file: 'trivy-results.sarif'
```

### 1.2 PR Automation
**File**: `.github/workflows/pr-automation.yml`

```yaml
name: PR Automation

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]

jobs:
  assign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Auto-assign PR
        uses: actions/github-script@v7
        with:
          script: |
            const pr = context.payload.pull_request;
            if (pr.user.type === 'User') {
              github.rest.issues.addAssignees({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: pr.number,
                assignees: [pr.user.login]
              });
            }

  size-label:
    runs-on: ubuntu-latest
    steps:
      - name: Label based on size
        uses: codelytv/pr-size-labeler@v1
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          xs_label: 'size/xs'
          xs_max_size: 50
          s_label: 'size/s'
          s_max_size: 200
          m_label: 'size/m'
          m_max_size: 500
          l_label: 'size/l'
          l_max_size: 1000
          xl_label: 'size/xl'
          fail_if_xl: false

  dependency-review:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/dependency-review-action@v3
        with:
          fail-on-severity: high
```

## Phase 2: Release Automation (Week 2)

### 2.1 Release Workflow
**File**: `.github/workflows/release.yml`

```yaml
name: Release

on:
  push:
    tags:
      - 'v*'

permissions:
  contents: write
  packages: write

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-go@v5
        with:
          go-version: '1.24'
          cache: true

      - name: Generate changelog
        id: changelog
        uses: mikepenz/release-changelog-builder-action@v4
        with:
          configuration: .github/changelog-config.json
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Run GoReleaser
        uses: goreleaser/goreleaser-action@v5
        with:
          distribution: goreleaser
          version: latest
          args: release --clean
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}

  deploy-hf:
    runs-on: ubuntu-latest
    needs: release
    steps:
      - uses: actions/checkout@v4

      - name: Trigger Hugging Face Space update
        run: |
          curl -X POST \
            -H "Authorization: Bearer ${{ secrets.HF_TOKEN }}" \
            https://huggingface.co/api/spaces/${{ secrets.HF_SPACE }}/sync

      - name: Wait for deployment
        run: |
          sleep 60
          curl -s https://huggingface.co/api/spaces/${{ secrets.HF_SPACE }} | jq '.runtime.stage'
```

### 2.2 GoReleaser Config
**File**: `.goreleaser.yml`

```yaml
version: 2

before:
  hooks:
    - go mod tidy
    - go generate ./...

builds:
  - id: cliproxy
    main: ./cmd/server
    binary: cliproxy
    env:
      - CGO_ENABLED=0
    goos:
      - linux
      - darwin
      - windows
    goarch:
      - amd64
      - arm64
    goarm:
      - '7'
    ldflags:
      - -s -w
      - -X main.version={{.Version}}
      - -X main.commit={{.Commit}}
      - -X main.date={{.Date}}

archives:
  - id: cliproxy
    name_template: >-
      {{ .ProjectName }}_
      {{- title .Os }}_
      {{- if eq .Arch "amd64" }}x86_64
      {{- else if eq .Arch "386" }}i386
      {{- else }}{{ .Arch }}{{ end }}
    format_overrides:
      - goos: windows
        format: zip
    files:
      - README.md
      - LICENSE
      - config.example.yaml

dockers:
  - image_templates:
      - "ghcr.io/echyai/cliproxy:{{ .Tag }}-amd64"
      - "echyai/cliproxy:{{ .Tag }}-amd64"
    dockerfile: Dockerfile
    use: buildx
    build_flag_templates:
      - --platform=linux/amd64
      - --label=org.opencontainers.image.title={{ .ProjectName }}
      - --label=org.opencontainers.image.description={{ .ProjectName }}
      - --label=org.opencontainers.image.url=https://github.com/echyai/cliproxy
      - --label=org.opencontainers.image.source=https://github.com/echyai/cliproxy
      - --label=org.opencontainers.image.version={{ .Version }}
      - --label=org.opencontainers.image.created={{ .Date }}
      - --label=org.opencontainers.image.revision={{ .FullCommit }}
  - image_templates:
      - "ghcr.io/echyai/cliproxy:{{ .Tag }}-arm64"
      - "echyai/cliproxy:{{ .Tag }}-arm64"
    dockerfile: Dockerfile
    use: buildx
    goarch: arm64
    build_flag_templates:
      - --platform=linux/arm64
      - --label=org.opencontainers.image.title={{ .ProjectName }}
      - --label=org.opencontainers.image.description={{ .ProjectName }}
      - --label=org.opencontainers.image.url=https://github.com/echyai/cliproxy
      - --label=org.opencontainers.image.source=https://github.com/echyai/cliproxy
      - --label=org.opencontainers.image.version={{ .Version }}
      - --label=org.opencontainers.image.created={{ .Date }}
      - --label=org.opencontainers.image.revision={{ .FullCommit }}

docker_manifests:
  - name_template: "echyai/cliproxy:{{ .Tag }}"
    image_templates:
      - "echyai/cliproxy:{{ .Tag }}-amd64"
      - "echyai/cliproxy:{{ .Tag }}-arm64"
  - name_template: "echyai/cliproxy:latest"
    image_templates:
      - "echyai/cliproxy:{{ .Tag }}-amd64"
      - "echyai/cliproxy:{{ .Tag }}-arm64"
  - name_template: "ghcr.io/echyai/cliproxy:{{ .Tag }}"
    image_templates:
      - "ghcr.io/echyai/cliproxy:{{ .Tag }}-amd64"
      - "ghcr.io/echyai/cliproxy:{{ .Tag }}-arm64"
  - name_template: "ghcr.io/echyai/cliproxy:latest"
    image_templates:
      - "ghcr.io/echyai/cliproxy:{{ .Tag }}-amd64"
      - "ghcr.io/echyai/cliproxy:{{ .Tag }}-arm64"

changelog:
  use: github
  sort: asc
  filters:
    exclude:
      - '^docs:'
      - '^test:'
      - '^chore:'
      - '^ci:'
      - Merge pull request
      - Merge branch

release:
  github:
    owner: echyai
    name: cliproxy
  draft: false
  prerelease: auto
  mode: replace
  header: |
    ## CLIProxyAPI {{ .Tag }}

    ### Docker Images
    ```bash
    docker pull echyai/cliproxy:{{ .Tag }}
    docker pull ghcr.io/echyai/cliproxy:{{ .Tag }}
    ```

  footer: |
    ## Quick Start

    ```bash
    # Download and run
    curl -L https://github.com/echyai/cliproxy/releases/download/{{ .Tag }}/cliproxy_Linux_x86_64.tar.gz | tar xz
    ./cliproxy --config config.yaml
    ```
```

## Phase 3: Infrastructure as Code (Week 3)

### 3.1 Terraform for AWS
**File**: `infrastructure/terraform/main.tf`

```hcl
terraform {
  required_version = ">= 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }

  backend "s3" {
    bucket         = "cliproxy-terraform-state"
    key            = "infrastructure/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "cliproxy-terraform-locks"
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Project     = "cliproxy"
      Environment = var.environment
      ManagedBy   = "terraform"
    }
  }
}

# VPC
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.project_name}-${var.environment}"
  cidr = var.vpc_cidr

  azs             = var.availability_zones
  private_subnets = var.private_subnets
  public_subnets  = var.public_subnets

  enable_nat_gateway   = true
  single_nat_gateway   = var.environment != "production"
  enable_dns_hostnames = true
  enable_dns_support   = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }
}

# EKS Cluster
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

  cluster_name    = "${var.project_name}-${var.environment}"
  cluster_version = "1.29"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  control_plane_subnet_ids       = module.vpc.intra_subnets
  cluster_endpoint_public_access = true

  eks_managed_node_groups = {
    general = {
      desired_size = var.node_desired_size
      min_size     = var.node_min_size
      max_size     = var.node_max_size

      instance_types = var.node_instance_types
      capacity_type  = var.environment == "production" ? "ON_DEMAND" : "SPOT"

      labels = {
        workload = "general"
      }

      update_config = {
        max_unavailable_percentage = 25
      }
    }
  }

  # Fargate profiles for serverless workloads
  fargate_profiles = {
    default = {
      name = "default"
      selectors = [
        { namespace = "kube-system" },
        { namespace = "default" }
      ]
    }
  }
}

# RDS PostgreSQL
module "rds" {
  source  = "terraform-aws-modules/rds/aws"
  version = "~> 6.0"

  identifier = "${var.project_name}-${var.environment}"

  engine               = "postgres"
  engine_version       = "16"
  family               = "postgres16"
  major_engine_version = "16"
  instance_class       = var.db_instance_class

  allocated_storage     = var.db_allocated_storage
  max_allocated_storage = var.db_max_allocated_storage

  db_name  = var.db_name
  username = var.db_username
  port     = 5432

  multi_az               = var.environment == "production"
  db_subnet_group_name   = module.vpc.database_subnet_group
  vpc_security_group_ids = [aws_security_group.rds.id]

  backup_retention_period = var.environment == "production" ? 7 : 1
  backup_window          = "03:00-04:00"
  maintenance_window     = "Mon:04:00-Mon:05:00"

  deletion_protection = var.environment == "production"

  enabled_cloudwatch_logs_exports = ["postgresql"]

  performance_insights_enabled = var.environment == "production"
}

# ElastiCache Redis
module "redis" {
  source  = "terraform-aws-modules/elasticache/aws"
  version = "~> 1.0"

  cluster_id = "${var.project_name}-${var.environment}"

  engine          = "redis"
  engine_version  = "7.1"
  node_type       = var.redis_node_type
  num_cache_nodes = 1

  subnet_group_name  = aws_elasticache_subnet_group.redis.name
  security_group_ids = [aws_security_group.redis.id]

  at_rest_encryption_enabled = true
  transit_encryption_enabled = true
}

# Application Load Balancer
module "alb" {
  source  = "terraform-aws-modules/alb/aws"
  version = "~> 9.0"

  name = "${var.project_name}-${var.environment}"

  load_balancer_type = "application"

  vpc_id  = module.vpc.vpc_id
  subnets = module.vpc.public_subnets

  security_groups = [aws_security_group.alb.id]

  listeners = {
    https = {
      port            = 443
      protocol        = "HTTPS"
      certificate_arn = aws_acm_certificate.main.arn

      fixed_response = {
        content_type = "text/plain"
        message_body = "OK"
        status_code  = "200"
      }
    }
  }
}

# Route53
resource "aws_route53_record" "main" {
  zone_id = var.route53_zone_id
  name    = var.domain_name
  type    = "A"

  alias {
    name                   = module.alb.dns_name
    zone_id                = module.alb.zone_id
    evaluate_target_health = true
  }
}
```

### 3.2 Kubernetes Manifests
**File**: `infrastructure/k8s/base/deployment.yaml`

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cliproxy
  labels:
    app: cliproxy
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 10%
  selector:
    matchLabels:
      app: cliproxy
  template:
    metadata:
      labels:
        app: cliproxy
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      serviceAccountName: cliproxy
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: cliproxy
          image: echyai/cliproxy:latest
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "1000m"
          livenessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          env:
            - name: CONFIG_TYPE
              value: "postgres"
            - name: POSTGRES_HOST
              valueFrom:
                secretKeyRef:
                  name: cliproxy-db
                  key: host
            - name: POSTGRES_PORT
              valueFrom:
                secretKeyRef:
                  name: cliproxy-db
                  key: port
            - name: POSTGRES_DATABASE
              valueFrom:
                secretKeyRef:
                  name: cliproxy-db
                  key: database
            - name: POSTGRES_USERNAME
              valueFrom:
                secretKeyRef:
                  name: cliproxy-db
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cliproxy-db
                  key: password
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: cache
              mountPath: /cache
      volumes:
        - name: tmp
          emptyDir: {}
        - name: cache
          emptyDir:
            sizeLimit: 500Mi
---
apiVersion: v1
kind: Service
metadata:
  name: cliproxy
  labels:
    app: cliproxy
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: cliproxy
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cliproxy
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/...
spec:
  rules:
    - host: api.cliproxy.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: cliproxy
                port:
                  number: 80
```

### 3.3 Kustomize Overlays
**File**: `infrastructure/k8s/overlays/production/kustomization.yaml`

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: cliproxy-production

resources:
  - ../../base
  - hpa.yaml
  - pdb.yaml

replicas:
  - name: cliproxy
    count: 5

images:
  - name: echyai/cliproxy
    newTag: v1.2.3

patchesStrategicMerge:
  - deployment-patch.yaml

configMapGenerator:
  - name: cliproxy-config
    literals:
      - LOG_LEVEL=info
      - METRICS_ENABLED=true

secretGenerator:
  - name: cliproxy-db
    envs:
      - .env.db

commonLabels:
  environment: production
```

## Phase 4: Monitoring Stack (Week 4)

### 4.1 Prometheus Rules
**File**: `infrastructure/monitoring/prometheus-rules.yaml`

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cliproxy-alerts
spec:
  groups:
    - name: cliproxy
      rules:
        - alert: CLProxyHighErrorRate
          expr: |
            (
              sum(rate(cliproxy_http_requests_total{status=~"5.."}[5m]))
              /
              sum(rate(cliproxy_http_requests_total[5m]))
            ) > 0.05
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High error rate detected"
            description: "Error rate is above 5% for 5 minutes"

        - alert: CLProxyHighLatency
          expr: |
            histogram_quantile(0.99,
              sum(rate(cliproxy_http_request_duration_seconds_bucket[5m])) by (le)
            ) > 2
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High latency detected"
            description: "P99 latency is above 2 seconds"

        - alert: CLProxyProviderErrors
          expr: |
            sum(rate(cliproxy_provider_requests_total{status="error"}[5m])) by (provider) > 10
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Provider {{ $labels.provider }} has high error rate"

        - alert: CLProxyCircuitBreakerOpen
          expr: cliproxy_circuit_breaker_state == 2
          for: 1m
          labels:
            severity: warning
          annotations:
            summary: "Circuit breaker is open"
            description: "Circuit breaker {{ $labels.name }} has opened"

        - alert: CLProxyRateLimitHits
          expr: |
            sum(rate(cliproxy_rate_limit_hits_total[5m])) > 100
          for: 5m
          labels:
            severity: info
          annotations:
            summary: "High rate limit hits"
```

### 4.2 Grafana Dashboard
**File**: `infrastructure/monitoring/dashboard.json` (simplified)

```json
{
  "dashboard": {
    "title": "CLIProxyAPI",
    "tags": ["cliproxy", "api"],
    "timezone": "UTC",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(cliproxy_http_requests_total[5m])) by (status)"
          }
        ]
      },
      {
        "title": "Latency",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(cliproxy_http_request_duration_seconds_bucket[5m])) by (le))"
          }
        ]
      },
      {
        "title": "Provider Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(cliproxy_provider_requests_total[5m])) by (provider)"
          }
        ]
      },
      {
        "title": "Active Connections",
        "type": "singlestat",
        "targets": [
          {
            "expr": "cliproxy_active_connections"
          }
        ]
      }
    ]
  }
}
```

## Success Metrics

- [ ] Build time < 5 minutes
- [ ] Test coverage > 70%
- [ ] Zero-downtime deployments
- [ ] Automated rollback capability
- [ ] Full observability (logs, metrics, traces)
- [ ] Infrastructure as Code coverage 100%
- [ ] Security scanning in CI
- [ ] Multi-arch container images

## Deployment Flow

```
1. Developer pushes code
   ↓
2. CI: Lint, Test, Build
   ↓
3. Docker image built (multi-arch)
   ↓
4. Security scan (Trivy)
   ↓
5. Push to registry
   ↓
6. Deploy to staging (auto)
   ↓
7. Integration tests
   ↓
8. Deploy to production (manual approval)
   ↓
9. Smoke tests
   ↓
10. Monitor and alert
```