docs/wiki/02-api-specification-and-implementation.md

Wiki: 02. API Specification & Implementation

The Ever-Brain API is the content-access layer built on TypeScript + Hono (Node.js). It serves as a reliable source layer that resolves GitHub repositories and streams their content.

---

1. Dual Ingestion Strategy

To balance fast response times with strict GitHub API rate-limit conservation, the API utilizes a two-tier ingestion mechanism:

               ┌────────────────────────────────────────────────┐
               │                 Ever-Brain API                 │
               └────────┬──────────────────────────────┬────────┘
                        │                              │
             1. Tree Discovery               2. Content Streaming
             (api.github.com)             (raw.githubusercontent.com)
                        │                              │
                        ▼                              ▼
             GitHub Git Trees API               Raw GitHub CDN
         (1 req / 60s cached per repo)           (0 REST API quota cost)

1. Discovery Tier:
   • Discovers project trees via GET https://api.github.com/repos/{owner}/{repo}/git/trees/{branch}?recursive=1.
   • Single HTTP request discovers entire repository hierarchies (up to 100,000 objects in 45–380ms).
   • Rate limit tracked via response headers (x-ratelimit-remaining, x-ratelimit-limit).
2. Streaming Tier:
   • Streams raw file contents directly from https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{path}.
   • Consumes 0 GitHub REST API quota, allowing unlimited content reads for public repositories.

---

2. In-Memory Tree Cache Engine

• Implementation: packages/api/src/services/tree-cache.ts
• TTL: 60,000ms (60 seconds) by default (configurable via CACHE_TTL_MS).
• Key: ${owner.toLowerCase()}/${repo.toLowerCase()}:${branch.toLowerCase()}.
• Cache Invalidation:
  • Automatically expires after TTL.
  • Can be bypassed on any request by passing query parameter ?fresh=true.
  • In-memory only: Server restarts cleanly drop cache with zero data loss or corruption.

---

3. Path Normalization & Binary Filtering

Case-Insensitive Path Normalization

Many autonomous AI agents request uppercase filenames (e.g. README.md, INDEX.md, LICENSE.txt) even when the repository uses lowercase (readme.md).

• The API builds a lowercase lookup index during tree discovery: pathMap: Map<lowercasePath, exactCasedPath>.
• When an endpoint receives /repos/acme/project/file/README.md, the resolver looks up readme.md, discovers the true casing, and streams the correct raw CDN file without raising a 404.

Binary and Noise Filter

The API excludes non-textual files and .git/ repository internals during discovery:

• Git internals: .git/, .git/HEAD, .git/objects/...
• Images: .png, .jpg, .jpeg, .gif, .ico, .webp, .bmp, .tiff, .svg
• Audio/Video: .mp3, .mp4, .wav, .mov, .avi, .flac, .mkv, etc.
• Binaries & Archives: .zip, .tar, .gz, .7z, .rar, .exe, .dll, .so, .wasm, .bin
• Lockfiles & DBs: package-lock.json, pnpm-lock.yaml, yarn.lock, Cargo.lock, .sqlite, .db

(Note: The API does not enforce arbitrary framework ignore lists like node_modules or dist to avoid restricting genuine content repositories).

---

4. Substring Search Engine

• Implementation: packages/api/src/services/search-service.ts
• Design: Exact substring matching across discovered text/Markdown files.
• Concurrency: Streams candidate files concurrently in controlled batches of 10.
• Response: Returns matching file paths, total match count, and 1-indexed line numbers where the query occurs.

---

5. Complete REST API Reference

GET /health

Operational probe returning service status, uptime, and timestamp.

{
  "status": "ok",
  "version": "1.0.0",
  "uptimeSeconds": 142,
  "timestamp": "2026-08-21T14:36:28.656Z"
}

GET /repos/:owner/:repo

Resolves repository metadata and detects the default branch (main, master, develop, etc.).

{
  "owner": "abrez-rizvi",
  "repo": "trial-markdown",
  "defaultBranch": "main"
}

GET /repos/:owner/:repo/files

Returns flat array of discovered project files.

• Query Parameters:
  • prefix (optional): Filter by path prefix (e.g. ?prefix=knowledge/).
  • fresh (optional): Pass true or 1 to bypass the 60s in-memory tree cache.
  • branch (optional): Target a specific branch instead of the default branch.

{
  "repository": "abrez-rizvi/trial-markdown",
  "branch": "main",
  "totalFiles": 1,
  "files": [
    {
      "path": "text.md",
      "type": "file",
      "sizeBytes": 92,
      "mimeType": "text/markdown"
    }
  ]
}

GET /repos/:owner/:repo/file/*path

Streams the exact raw text or Markdown content with appropriate headers.

• Response Headers:
  • Content-Type: MIME type (e.g. text/markdown; charset=utf-8, text/plain; charset=utf-8).
  • X-Ever-Brain-Path: Exact cased path in the repository.
  • X-Ever-Brain-Branch: Target branch resolved.
• Aliases: GET /repos/:owner/:repo/files/*path and GET /projects/:owner/:repo/files/*path.

GET /repos/:owner/:repo/search

Searches text/Markdown documents for a string query.

• Query Parameters:
  • q (required): The search string.
  • prefix (optional): Search only within a specific folder prefix.
  • branch (optional): Target branch.

{
  "query": "architecture",
  "totalMatches": 3,
  "results": [
    {
      "path": "knowledge/architecture.md",
      "matches": 3,
      "lines": [1, 14, 42]
    }
  ]
}