Spaces:
Sleeping
Sleeping
File size: 5,945 Bytes
a5d1c1b 61b1da1 a5d1c1b 61b1da1 a5d1c1b 61b1da1 a5d1c1b 61b1da1 a5d1c1b 61b1da1 a5d1c1b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | # Wiki: 02. API Specification & Implementation
The **Ever-Brain API** is the content-access layer built on **TypeScript + Hono (Node.js)**. It serves as a reliable source layer that resolves GitHub repositories and streams their content.
---
## 1. Dual Ingestion Strategy
To balance fast response times with strict GitHub API rate-limit conservation, the API utilizes a two-tier ingestion mechanism:
```text
┌────────────────────────────────────────────────┐
│ Ever-Brain API │
└────────┬──────────────────────────────┬────────┘
│ │
1. Tree Discovery 2. Content Streaming
(api.github.com) (raw.githubusercontent.com)
│ │
▼ ▼
GitHub Git Trees API Raw GitHub CDN
(1 req / 60s cached per repo) (0 REST API quota cost)
```
1. **Discovery Tier**:
- Discovers project trees via `GET https://api.github.com/repos/{owner}/{repo}/git/trees/{branch}?recursive=1`.
- Single HTTP request discovers entire repository hierarchies (up to 100,000 objects in 45–380ms).
- Rate limit tracked via response headers (`x-ratelimit-remaining`, `x-ratelimit-limit`).
2. **Streaming Tier**:
- Streams raw file contents directly from `https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{path}`.
- **Consumes 0 GitHub REST API quota**, allowing unlimited content reads for public repositories.
---
## 2. In-Memory Tree Cache Engine
- **Implementation**: `packages/api/src/services/tree-cache.ts`
- **TTL**: 60,000ms (60 seconds) by default (configurable via `CACHE_TTL_MS`).
- **Key**: `${owner.toLowerCase()}/${repo.toLowerCase()}:${branch.toLowerCase()}`.
- **Cache Invalidation**:
- Automatically expires after TTL.
- Can be bypassed on any request by passing query parameter `?fresh=true`.
- In-memory only: Server restarts cleanly drop cache with zero data loss or corruption.
---
## 3. Path Normalization & Binary Filtering
### Case-Insensitive Path Normalization
Many autonomous AI agents request uppercase filenames (e.g. `README.md`, `INDEX.md`, `LICENSE.txt`) even when the repository uses lowercase (`readme.md`).
- The API builds a lowercase lookup index during tree discovery:
`pathMap: Map<lowercasePath, exactCasedPath>`.
- When an endpoint receives `/repos/acme/project/file/README.md`, the resolver looks up `readme.md`, discovers the true casing, and streams the correct raw CDN file without raising a 404.
### Binary and Noise Filter
The API excludes non-textual files and `.git/` repository internals during discovery:
- **Git internals**: `.git/`, `.git/HEAD`, `.git/objects/...`
- **Images**: `.png`, `.jpg`, `.jpeg`, `.gif`, `.ico`, `.webp`, `.bmp`, `.tiff`, `.svg`
- **Audio/Video**: `.mp3`, `.mp4`, `.wav`, `.mov`, `.avi`, `.flac`, `.mkv`, etc.
- **Binaries & Archives**: `.zip`, `.tar`, `.gz`, `.7z`, `.rar`, `.exe`, `.dll`, `.so`, `.wasm`, `.bin`
- **Lockfiles & DBs**: `package-lock.json`, `pnpm-lock.yaml`, `yarn.lock`, `Cargo.lock`, `.sqlite`, `.db`
*(Note: The API does not enforce arbitrary framework ignore lists like `node_modules` or `dist` to avoid restricting genuine content repositories).*
---
## 4. Substring Search Engine
- **Implementation**: `packages/api/src/services/search-service.ts`
- **Design**: Exact substring matching across discovered text/Markdown files.
- **Concurrency**: Streams candidate files concurrently in controlled batches of 10.
- **Response**: Returns matching file paths, total match count, and 1-indexed line numbers where the query occurs.
---
## 5. Complete REST API Reference
### `GET /health`
Operational probe returning service status, uptime, and timestamp.
```json
{
"status": "ok",
"version": "1.0.0",
"uptimeSeconds": 142,
"timestamp": "2026-08-21T14:36:28.656Z"
}
```
### `GET /repos/:owner/:repo`
Resolves repository metadata and detects the default branch (`main`, `master`, `develop`, etc.).
```json
{
"owner": "abrez-rizvi",
"repo": "trial-markdown",
"defaultBranch": "main"
}
```
### `GET /repos/:owner/:repo/files`
Returns flat array of discovered project files.
- **Query Parameters**:
- `prefix` *(optional)*: Filter by path prefix (e.g. `?prefix=knowledge/`).
- `fresh` *(optional)*: Pass `true` or `1` to bypass the 60s in-memory tree cache.
- `branch` *(optional)*: Target a specific branch instead of the default branch.
```json
{
"repository": "abrez-rizvi/trial-markdown",
"branch": "main",
"totalFiles": 1,
"files": [
{
"path": "text.md",
"type": "file",
"sizeBytes": 92,
"mimeType": "text/markdown"
}
]
}
```
### `GET /repos/:owner/:repo/file/*path`
Streams the exact raw text or Markdown content with appropriate headers.
- **Response Headers**:
- `Content-Type`: MIME type (e.g. `text/markdown; charset=utf-8`, `text/plain; charset=utf-8`).
- `X-Ever-Brain-Path`: Exact cased path in the repository.
- `X-Ever-Brain-Branch`: Target branch resolved.
- **Aliases**: `GET /repos/:owner/:repo/files/*path` and `GET /projects/:owner/:repo/files/*path`.
### `GET /repos/:owner/:repo/search`
Searches text/Markdown documents for a string query.
- **Query Parameters**:
- `q` *(required)*: The search string.
- `prefix` *(optional)*: Search only within a specific folder prefix.
- `branch` *(optional)*: Target branch.
```json
{
"query": "architecture",
"totalMatches": 3,
"results": [
{
"path": "knowledge/architecture.md",
"matches": 3,
"lines": [1, 14, 42]
}
]
}
```
|