Spaces:
Runtime error
Runtime error
File size: 9,539 Bytes
cd8bd0a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 | ---
title: "Compression Rules Format"
version: 3.8.2
lastUpdated: 2026-05-13
---
# Compression Rules Format
Compression rules are JSON files loaded at runtime. They are intentionally data-only so new
language packs and RTK command filters can be reviewed without changing engine code.
> **Canonical schema (source of truth):** [`open-sse/services/compression/rules/_schema.json`](../../open-sse/services/compression/rules/_schema.json) (JSON Schema draft 2020-12).
> The examples below are illustrative — when in doubt, validate your pack against `_schema.json`.
## Caveman Rule Packs
Caveman rule packs live under:
```txt
open-sse/services/compression/rules/<language>/<pack>.json
```
Each pack contains replacements that apply to normal prose after protected regions are isolated.
```json
{
"language": "en",
"category": "filler",
"rules": [
{
"name": "question_to_directive",
"pattern": "\\b(?:Can you explain why|Could you show me how)\\b\\s*",
"replacement": "Explain why ",
"replacementMap": {
"can you explain why": "Explain why ",
"could you show me how": "Show how "
},
"flags": "gi",
"context": "all",
"category": "context",
"minIntensity": "lite",
"description": "Convert verbose questions into direct requests."
}
]
}
```
### Caveman Fields
| Field | Required | Description |
| ------------------------ | -------- | ---------------------------------------------------------------- |
| `language` | yes | BCP-47-like language key such as `en`, `pt-BR`, `es` |
| `category` | yes | Pack category filename/category, for example `filler` or `dedup` |
| `rules` | yes | Array of regex replacement rules |
| `rules[].name` | yes | Stable rule name |
| `rules[].pattern` | yes | JavaScript regex source |
| `rules[].flags` | no | JavaScript regex flags; default `gi` |
| `rules[].replacement` | no | Replacement string or fallback when `replacementMap` misses |
| `rules[].replacementMap` | no | Match-specific replacements keyed by normalized matched text |
| `rules[].context` | no | `all`, `user`, `assistant`, or `system`; default `all` |
| `rules[].category` | no | `filler`, `context`, `structural`, `dedup`, `terse`, or `ultra` |
| `rules[].minIntensity` | no | `lite`, `full`, or `ultra`; default `lite` |
| `rules[].description` | no | Human-readable rule summary |
Use `flags` when case-sensitive matching matters, for example article removal before lowercase prose
without stripping `the OpenAI API`. Use `replacementMap` when one regex has multiple alternatives
that need different outputs; this keeps JSON rule packs data-only while preserving the behavior of
the richer built-in TypeScript replacement functions.
## RTK Filter Packs
RTK filters live under:
```txt
open-sse/services/compression/engines/rtk/filters/<filter>.json
```
Each filter describes how to recognize and compress a command-output family.
```json
{
"id": "test-vitest",
"label": "Vitest output",
"category": "test",
"priority": 92,
"match": {
"outputTypes": ["test-vitest"],
"commands": ["vitest", "npm test", "npm run test"],
"patterns": ["\\bFAIL\\b", "\\bPASS\\b", "\\bTest Files\\b"]
},
"rules": {
"stripAnsi": true,
"replace": [{ "pattern": "\\s+\\[[0-9]+ms\\]", "replacement": "" }],
"matchOutput": [
{ "pattern": "All tests passed", "message": "vitest: ok", "unless": "FAIL|Error:" }
],
"includePatterns": ["FAIL", "Error:", "Test Files", "Tests"],
"dropPatterns": ["^\\s*$", "Duration\\s+\\d+"],
"collapsePatterns": ["^\\s+at "],
"deduplicate": true,
"truncateLineAt": 240,
"maxLines": 160,
"headLines": 24,
"tailLines": 40,
"onEmpty": "vitest: ok",
"filterStderr": false
},
"preserve": {
"errorPatterns": ["FAIL", "Error:", "AssertionError"],
"summaryPatterns": ["Test Files", "Tests", "Snapshots"]
},
"tests": [
{
"name": "keeps failing tests",
"command": "vitest",
"input": "FAIL test/a.test.ts\\nError: boom\\nTest Files 1 failed",
"expected": "FAIL test/a.test.ts\\nError: boom\\nTest Files 1 failed"
}
]
}
```
### RTK Fields
| Field | Required | Description |
| -------------------------- | -------- | ------------------------------------------------------------------------------ |
| `id` | yes | Stable filter id |
| `label` | yes | Dashboard-readable name |
| `category` | yes | Filter family: git, test, build, shell, docker, package, infra, cloud, generic |
| `priority` | no | Higher priority wins when multiple filters match |
| `match.outputTypes` | no | Detector output ids that select this filter |
| `match.commands` | no | Command tokens that select this filter |
| `match.patterns` | no | Regex patterns that select this filter from output text |
| `rules.stripAnsi` | no | Remove ANSI escape sequences before regex stages |
| `rules.replace` | no | Ordered regex substitutions applied line by line |
| `rules.matchOutput` | no | Short-circuit output rules with optional `unless` guard |
| `rules.includePatterns` | no | Lines to prefer preserving |
| `rules.dropPatterns` | no | Lines to remove as noise |
| `rules.collapsePatterns` | no | Repeated matching lines that can be collapsed |
| `rules.deduplicate` | no | Collapse duplicate normalized lines |
| `rules.truncateLineAt` | no | Unicode-safe per-line character limit |
| `rules.maxLines` | no | Maximum retained lines before tail preservation |
| `rules.headLines` | no | Head lines retained during truncation |
| `rules.tailLines` | no | Tail lines retained for recent context |
| `rules.onEmpty` | no | Fallback message when filtering removes all content |
| `rules.filterStderr` | no | Normalize common stderr prefixes before later filtering stages |
| `preserve.errorPatterns` | no | Error lines that should survive truncation |
| `preserve.summaryPatterns` | no | Summary lines that should survive truncation |
| `tests[]` | no | Inline verification samples used by the RTK verify gate |
RTK applies declarative stages in this order: `stripAnsi`, `filterStderr`, `replace`,
`matchOutput`, `dropPatterns`/`includePatterns`, `truncateLineAt`, `headLines`/`tailLines`,
`maxLines`, and `onEmpty`.
Custom filters can be loaded from:
1. Project `.rtk/filters.json` files only after a matching `.rtk/trust.json` hash is present or
`trustProjectFilters` is enabled.
2. Global `DATA_DIR/rtk/filters.json`.
3. Built-in filters.
Project/global custom files may contain one filter object or an array of filter objects. Invalid
custom filters are skipped with diagnostics; invalid built-in filters fail validation.
Project trust file:
```json
{
"filtersSha256": "0123456789abcdef..."
}
```
The environment override `OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=1` trusts project filters without a
hash and should be limited to controlled local development.
## Safety Rules
- Keep rules idempotent: running the same filter twice should not corrupt output.
- Preserve exact error text, file paths, line numbers, and command summaries where possible.
- Avoid rules that modify code blocks, JSON payloads, URLs, or secrets.
- Add unit coverage for new command families in detector/filter tests.
- Add `tests[]` samples to every built-in filter and to shared custom filters.
## Validation
Rule packs are validated before use. Built-in Caveman packs and built-in RTK filters fail fast
during validation so broken release assets are caught before shipment. Custom RTK filters are
skipped with diagnostics when parsing or trust validation fails.
Focused validation:
```bash
node --import tsx/esm --test tests/unit/compression/rule-loader.test.ts tests/unit/compression/language-packs.test.ts
node --import tsx/esm --test tests/unit/compression/rtk-verify.test.ts tests/unit/compression/rtk-dsl-pipeline.test.ts
```
|