File size: 27,247 Bytes
cd8bd0a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
---
title: "RTK Compression"
version: 3.8.31
lastUpdated: 2026-06-20
---

# RTK Compression

RTK compression is OmniRoute's command-aware compression engine for terminal and tool output. It is
designed for coding-agent sessions where most context growth comes from test logs, build output,
package manager noise, shell transcripts, Docker output, git output, and stack traces.

RTK can run directly with `defaultMode: "rtk"` or as the first step in a stacked pipeline, usually:

```txt
rtk -> caveman
```

That order compresses noisy machine output first, then lets Caveman condense remaining prose.

Upstream RTK reports `60-90%` command-output savings. Its README sample session goes from
`~118,000` standard tokens to `~23,900` RTK tokens, which is `79.7%` saved (`~80%`). OmniRoute uses
that upstream average for the stacked savings calculation with Caveman input compression:

```txt
RTK average:    80% saved
Caveman input: 46% saved
Stacked:       1 - (1 - 0.80) * (1 - 0.46) = 89.2% saved
Range:         1 - (1 - 0.60..0.90) * (1 - 0.46) = 78.4-94.6%
```

## What It Compresses

The built-in catalog currently ships 49 filters across these categories:

| Category  | Examples                                                      |
| --------- | ------------------------------------------------------------- |
| `git`     | `git status`, `git branch`, `git diff`, `git log`             |
| `test`    | Vitest, Jest, Pytest, Playwright, Go tests, Cargo tests       |
| `build`   | TypeScript, ESLint, Biome, Prettier, Vite, Webpack, Turbo, Nx |
| `package` | `npm install`, `npm audit`, `pip`, `uv sync`, Poetry, Bundler |
| `shell`   | `ls`, `find`, `grep`, generic shell logs                      |
| `docker`  | `docker ps`, Docker logs                                      |
| `infra`   | Terraform, OpenTofu, `systemctl status`                       |
| `generic` | JSON output, stack traces, generic output fallback            |

The detector in `open-sse/services/compression/engines/rtk/commandDetector.ts` classifies output
before filter selection. Filters can also match by command pattern or output regex when a command
class is not enough.

## Filter Resolution

RTK loads filters in this order:

1. Project filters from `.rtk/filters.json`, only when trusted.
2. Global filters from `DATA_DIR/rtk/filters.json`.
3. Built-in filters from `open-sse/services/compression/engines/rtk/filters/`.

Project filters are intentionally trust-gated because regex filters can change how tool output is
shown to agents. A project filter file is accepted when one of these is true:

- `rtkConfig.trustProjectFilters` is `true`.
- `OMNIROUTE_RTK_TRUST_PROJECT_FILTERS=1` is set.
- `.rtk/trust.json` contains the SHA-256 hash of `.rtk/filters.json`.

Trust file example:

```json
{
  "filtersSha256": "0123456789abcdef..."
}
```

Custom filters can be one filter object or an array of filter objects. Invalid custom filters are
skipped and reported by `/api/context/rtk/filters` diagnostics. Invalid built-in filters fail fast.

## Filter DSL

Filters use the JSON schema described in [Compression Rules Format](./COMPRESSION_RULES_FORMAT.md).
The runtime applies these stages in order:

```txt
stripAnsi -> filterStderr -> replace -> matchOutput -> drop/include lines
  -> truncateLineAt -> head/tail/maxLines -> onEmpty
```

Important fields:

| Field                        | Purpose                                                        |
| ---------------------------- | -------------------------------------------------------------- |
| `rules.stripAnsi`            | Remove terminal color/control sequences before matching        |
| `rules.filterStderr`         | Normalize common stderr prefixes before matching/filtering     |
| `rules.replace`              | Apply ordered regex replacements                               |
| `rules.matchOutput`          | Return a compact summary when output matches a known condition |
| `rules.matchOutput[].unless` | Skip the shortcut when an error/failure pattern is present     |
| `rules.dropPatterns`         | Remove noisy lines                                             |
| `rules.includePatterns`      | Prefer actionable lines                                        |
| `rules.collapsePatterns`     | Collapse repeated matching lines                               |
| `rules.deduplicate`          | Per-filter opt-in: collapse consecutive duplicate lines        |
| `rules.truncateLineAt`       | Unicode-safe per-line truncation                               |
| `rules.onEmpty`              | Fallback message if all lines are filtered out                 |
| `tests[]`                    | Inline samples used by the verify gate                         |

Built-in filters are expected to include inline `tests[]` samples. Custom filters should include
them too, especially when they are shared across projects.

## Line Deduplication (two layers)

RTK collapses duplicate lines at two independent layers:

1. **Per-filter `deduplicate` (opt-in, default `false`).** A filter can set `rules.deduplicate: true`
   to collapse consecutive duplicate lines *within that filter's matched output*, before truncation.
   This runs inside `lineFilter.ts`. For legacy filters, it is auto-enabled when the filter defines
   `collapsePatterns`. Schema: `deduplicate: z.boolean().default(false)` in
   `open-sse/services/compression/engines/rtk/filterSchema.ts`.
2. **Engine-wide `deduplicateThreshold` (default `3`).** After all filters run, the engine collapses
   any run of `>= deduplicateThreshold` identical consecutive lines across the whole result
   (`deduplicateRepeatedLines`, applied in `engines/rtk/index.ts`). The value is bounded to 2–100 on
   normalization.

The per-filter pass runs first (inside the filter), the engine-wide pass runs last (over the joined
output), so the two compose without double-counting.

## Line Grouping (`enableGrouping`)

When `rtkConfig.enableGrouping` is `true` (default `false`), RTK runs an additional `groupSimilarLines`
pass over the post-dedup result that collapses runs of *near-equivalent* (not byte-identical)
consecutive lines. `rtkConfig.groupingThreshold` (default `3`) is the minimum run length that triggers
grouping. This is the structural counterpart to `deduplicateThreshold`: dedup handles exact repeats,
grouping handles "the same shape with small differences". Both flags are part of the `rtkConfig` JSON
persisted in the `key_value` table (see Configuration above), so the setting survives restarts.

## Code Comment Stripping (`stripCodeComments` / `preserveDocstrings`)

When `rtkConfig.applyToCodeBlocks` is enabled, RTK can also strip comments from fenced code blocks:

- `stripCodeComments` (default `false`) β€” opt-in. When `true`, RTK removes comments from JavaScript
  and TypeScript fenced blocks. The flag was historically read but never applied, so the default stays
  at "preserve" to avoid a silent production change.
- `preserveDocstrings` (default `true`) β€” when stripping comments, JSDoc/`/** … */` block comments are
  kept (they carry API documentation worth more than the bytes they cost). Set to `false` to strip
  those too.

Comment removal is implemented in `open-sse/services/compression/engines/rtk/codeStripper.ts`. It uses
the **TypeScript parser** (not a regex) so that string, template, and regex literals are never mistaken
for comments, and it bails out entirely when JSX is detected (so JSX expression-container comments are
never corrupted). Comment stripping currently applies to **JavaScript and TypeScript only** β€” other
languages in the stripper's `CodeLanguage` set (Python, Rust, Go, Ruby, Java) have empty-line and
whitespace collapse but no comment removal. The stripped-block run is tagged `rtk:code-strip` in
`rulesApplied`.

> **Note β€” GCF / tabular encoding is a separate engine.** RTK does **not** contain the "GCF"
> (Graph Compact Format) tabular/columnar JSON encoder. That encoder β€” which replaced an older
> `omni-tabular` encoder β€” lives in the **headroom** engine
> (`open-sse/services/compression/engines/headroom/`, with the vendored codec under
> `headroom/gcf/`). It is unrelated to the RTK filter pipeline documented here.

## Configuration

Global settings are available through `/api/settings/compression`. RTK-specific settings are also
available through `/api/context/rtk/config`.

```json
{
  "defaultMode": "stacked",
  "autoTriggerMode": "stacked",
  "autoTriggerTokens": 32000,
  "stackedPipeline": [
    { "engine": "rtk", "intensity": "standard" },
    { "engine": "caveman", "intensity": "full" }
  ],
  "rtkConfig": {
    "enabled": true,
    "intensity": "standard",
    "applyToToolResults": true,
    "applyToCodeBlocks": false,
    "applyToAssistantMessages": false,
    "enabledFilters": [],
    "disabledFilters": [],
    "maxLinesPerResult": 120,
    "maxCharsPerResult": 12000,
    "deduplicateThreshold": 3,
    "customFiltersEnabled": true,
    "trustProjectFilters": false,
    "rawOutputRetention": "never",
    "rawOutputMaxBytes": 1048576,
    "enableGrouping": false,
    "groupingThreshold": 3,
    "stripCodeComments": false,
    "preserveDocstrings": true
  }
}
```

`enabledFilters` and `disabledFilters` use filter ids, for example `test-vitest` or `git-diff`.

The full `rtkConfig` shape is defined by `RtkConfig` / `DEFAULT_RTK_CONFIG` in
`open-sse/services/compression/types.ts`. The whole object is persisted as a single JSON value in
the SQLite `key_value` table under `namespace = "compression"`, `key = "rtkConfig"`
(`src/lib/db/compression.ts`), and normalized on read by `normalizeRtkConfig`. So every field below
β€” including `enableGrouping`, `groupingThreshold`, `stripCodeComments`, and `preserveDocstrings` β€”
round-trips through the same store and survives a restart.

| Key                    | Default | Purpose                                                                       |
| ---------------------- | ------- | ----------------------------------------------------------------------------- |
| `deduplicateThreshold` | `3`     | Engine-wide: min consecutive identical lines to collapse (bounded 2–100)      |
| `enableGrouping`       | `false` | Opt-in: collapse runs of near-equivalent consecutive lines                    |
| `groupingThreshold`    | `3`     | Min consecutive similar-line run that triggers grouping                       |
| `stripCodeComments`    | `false` | Opt-in: remove comments from fenced code blocks (needs `applyToCodeBlocks`)   |
| `preserveDocstrings`   | `true`  | When stripping comments, keep JSDoc/`/** … */` blocks                         |

## API

| Route                              | Method | Purpose                                      |
| ---------------------------------- | ------ | -------------------------------------------- |
| `/api/context/rtk/config`          | GET    | Read RTK config                              |
| `/api/context/rtk/config`          | PUT    | Update RTK config                            |
| `/api/context/rtk/filters`         | GET    | List filter catalog and load diagnostics     |
| `/api/context/rtk/test`            | POST   | Preview RTK compression for one text payload |
| `/api/context/rtk/raw-output/[id]` | GET    | Read retained redacted raw output            |
| `/api/compression/preview`         | POST   | Preview any compression mode                 |

RTK test payload:

```json
{
  "command": "npm test",
  "text": "FAIL tests/example.test.ts\nAssertionError: expected true\nTest Files 1 failed",
  "config": {
    "intensity": "standard"
  }
}
```

Compression preview payload:

```json
{
  "mode": "stacked",
  "messages": [
    {
      "role": "tool",
      "content": "FAIL tests/example.test.ts\nAssertionError: expected true\nTest Files 1 failed"
    }
  ],
  "config": {
    "rtkConfig": {
      "rawOutputRetention": "failures"
    }
  }
}
```

Management routes require dashboard management auth or the matching API-key policy.

## Raw Output Recovery

RTK normally returns only compressed text. For debugging, `rawOutputRetention` can retain redacted
raw output:

| Value      | Behavior                                                |
| ---------- | ------------------------------------------------------- |
| `never`    | Do not retain raw output                                |
| `failures` | Retain only likely failure output                       |
| `always`   | Retain every compressed RTK raw output, after redaction |

Retained files are written under:

```txt
DATA_DIR/rtk/raw-output/
```

Secrets are redacted before persistence, including common bearer tokens, API keys, Slack tokens,
AWS access keys, and assignment-style `token=...`, `secret=...`, `password=...` values. Analytics
stores only the pointer id, size, and hash metadata.

## Verify Gate

The focused verify gate runs built-in inline filter tests without shelling out to external commands:

```bash
node --import tsx/esm --test tests/unit/compression/rtk-verify.test.ts
```

The broader RTK gate is:

```bash
node --import tsx/esm --test \
  tests/unit/compression/rtk-*.test.ts \
  tests/unit/compression/pipeline-integration.test.ts \
  tests/unit/compression/context-compression-api.test.ts
```

Run the broad compression gate before release:

```bash
node --import tsx/esm --test \
  tests/unit/compression/*.test.ts \
  tests/golden-set/*.test.ts \
  tests/integration/compression-pipeline.test.ts \
  tests/unit/api/compression/compression-api.test.ts
```

## Extending RTK

1. Add or update a filter JSON file.
2. Include at least one `tests[]` sample that proves the important behavior.
3. Add a fixture under `tests/unit/compression/fixtures/rtk/` for new command families.
4. Add command detection coverage when introducing a new output class.
5. Run the verify and broad RTK gates.
6. If the filter is project-local, commit `.rtk/filters.json` and refresh `.rtk/trust.json` only after review.

---

## Intensity Levels (v3.8.16+)

RTK supports **3 intensity levels** that trade off between **compression aggressiveness** and **safety**. The level is set via `config.intensity` in the engine config.

### The 3 Levels

| Level | Truncation threshold | Token savings | Risk | Best for |
|-------|---------------------|---------------|------|----------|
| `minimal` | 24 lines per section | ~20-40% | Very low | Production with critical context |
| `standard` (default) | 24 lines per section | ~50-70% | Low | Daily coding sessions |
| `aggressive` | 16 lines per section | ~70-90% | Medium | Long sessions, max savings |

### Where the Truncation Happens

The truncation threshold affects `lineFilter.ts`:

```ts
// From open-sse/services/compression/engines/rtk/index.ts:329-330
config.intensity === "aggressive" ? 16 : 24,
config.intensity === "aggressive" ? 16 : 24,
```

Both the **head** and **tail** of each section are preserved; middle content is dropped when truncation kicks in.

### What Stays vs. What Gets Cut

| Content | minimal | standard | aggressive |
|---------|---------|----------|------------|
| Errors / stack traces | βœ… preserved | βœ… preserved | βœ… preserved |
| Test failures | βœ… preserved | βœ… preserved | βœ… preserved |
| Build errors | βœ… preserved | βœ… preserved | βœ… preserved |
| Test passes (verbose) | βœ… preserved | 🟑 collapsed | 🟑 collapsed |
| Routine output (info logs) | 🟑 collapsed | 🟑 collapsed | ❌ dropped |
| Progress bars | 🟑 collapsed | ❌ dropped | ❌ dropped |
| Banner / ASCII art | 🟑 collapsed | ❌ dropped | ❌ dropped |

### Choosing the Right Intensity

```
                  Is losing context catastrophic?
                  β”‚
      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
      β”‚           β”‚           β”‚
    YES          NO          NOT SURE
      β”‚           β”‚           β”‚
      β–Ό           β”‚           β”‚
   minimal        β”‚           β”‚
      β”‚           β”‚           β”‚
      β”‚           β–Ό           β–Ό
      β”‚      How critical    Try `standard` first
      β”‚      is throughput?  (works for 80% of
      β”‚           β”‚          cases)
      β”‚      β”Œβ”€β”€β”€β”€β”΄β”€β”€β”€β”€β”
      β”‚      β”‚         β”‚
      β”‚     LOW       HIGH
      β”‚      β”‚         β”‚
      β”‚      β–Ό         β–Ό
      β”‚   standard   aggressive
      β”‚      β”‚         β”‚
      β””β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

### Configuring Intensity

**Per-combo** (in combo config):

```json
{
  "combo": "my-coding-combo",
  "routing": { /* ... */ },
  "compression": {
    "engine": "rtk",
    "intensity": "aggressive"
  }
}
```

**Programmatically**:

`rtkEngine` (`@omniroute/open-sse/services/compression/engines/rtk`) is a
`CompressionEngine` and has no `updateConfig` method. Update an engine's config
through the registry helper instead:

```ts
import { updateEngineConfig } from "@omniroute/open-sse/services/compression/engines/registry";

updateEngineConfig("rtk", { intensity: "aggressive" });
```

### Verifying the Effect
Use the **Verify Gate** (see below) to confirm your filter is safe at your chosen intensity:
```ts
import { runRtkFilterTests } from "omniroute/compression/engines/rtk/verify";

const result = runRtkFilterTests({ intensity: "aggressive" });
if (!result.passed) {
  console.error("Filters failed at aggressive intensity");
}
```

---

## Custom Filter Development (v3.8.16+)

The `engines/rtk/filters/` directory contains **49+ built-in filter JSON files**. You can add your own to compress output from custom tools not covered by the defaults.

### Filter Schema (Zod)

```ts
{
  "id": "string",                      // Required. Filter identifier (kebab-case, e.g., "python-traceback")
  "label": "string",                   // Required. Human-readable filter name
  "description": "string",             // Optional (default: ""). Short description of what filter does
  "category": "git|test|build|shell|docker|package|infra|cloud|generic",
  "priority": number,                  // Optional (0-100, default: 50). Execution order (higher = first)
  "match": {
    "commands": ["string"],            // Command names to match (e.g., "python", "pytest")
    "patterns": ["string"],            // Regex patterns to match output
    "outputTypes": ["string"]          // Detected output classes (e.g., "test-failure")
  },
  "rules": {
    "stripAnsi": boolean,              // Optional (default: false). Strip ANSI color codes
    "replace": [                       // Find-and-replace rules (default: [])
      { "pattern": "regex", "replacement": "..." }
    ],
    "matchOutput": [                   // Short-circuit on pattern match (default: [])
      {
        "pattern": "regex",
        "message": "short summary",
        "unless": "regex"              // Skip if this pattern matches
      }
    ],
    "includePatterns": ["string"],     // Lines to keep (regex patterns, default: [])
    "dropPatterns": ["string"],        // Lines to drop (regex patterns, default: [])
    "collapsePatterns": ["string"],    // Lines to collapse to single occurrence (default: [])
    "deduplicate": boolean,            // Optional (default: false). Remove duplicate lines
    "truncateLineAt": number,          // Optional (default: 0). Truncate lines to max chars
    "maxLines": number,                // Optional (default: 0). Hard cap on total lines
    "headLines": number,               // Optional (default: 20). Keep first N lines of matched output
    "tailLines": number,               // Optional (default: 20). Keep last N lines of matched output
    "onEmpty": "string",               // Optional (default: ""). Fallback message if all lines filtered
    "filterStderr": boolean            // Optional (default: false). Also filter stderr output
  },
  "preserve": {
    "errorPatterns": ["string"],       // Patterns that must always be preserved (default: [])
    "summaryPatterns": ["string"]      // Patterns for final summary line (default: [])
  },
  "tests": [                           // Inline tests for verification (default: [])
    {
      "name": "string",               // Required. Test name
      "input": "sample output",        // Required. Sample input text
      "expected": "expected output",   // Required. Expected compressed output
      "command": "optional command"    // Optional. Command context
    }
  ]
}
```

### Example: Python Traceback Filter

```json
{
  "id": "python-traceback",
  "label": "Python Traceback Filter",
  "description": "Compresses Python tracebacks to essential file/line locations and error type",
  "category": "test",
  "priority": 60,
  "match": {
    "commands": ["python", "python3", "pytest", "uv", "poetry"],
    "patterns": ["Traceback \\(most recent call last\\)", "Error", "Exception"],
    "outputTypes": ["error-traceback"]
  },
  "rules": {
    "stripAnsi": true,
    "includePatterns": [
      "Traceback \\(most recent call last\\)",
      "^\\s*File \".+\", line \\d+",
      "^\\s*[A-Z][a-zA-Z]+Error:",
      "^\\s*[A-Z][a-zA-Z]+Exception"
    ],
    "dropPatterns": [
      "site-packages/",
      "^\\s+[a-z_]+\\([^)]*\\)$"
    ],
    "headLines": 5,
    "tailLines": 3,
    "maxLines": 25,
    "filterStderr": true
  },
  "preserve": {
    "errorPatterns": [
      "Error:",
      "Exception:",
      "Traceback"
    ],
    "summaryPatterns": [
      "^[A-Z][a-zA-Z]+(?:Error|Exception):"
    ]
  },
  "tests": [
    {
      "name": "preserves-error-type-and-location",
      "input": "Traceback (most recent call last):\n  File \"app.py\", line 42, in main\n    do_thing()\n  File \"lib/utils.py\", line 17, in helper\n    return 1 / 0\nZeroDivisionError: division by zero",
      "expected": "Traceback (most recent call last):\n  File \"app.py\", line 42, in main\n  File \"lib/utils.py\", line 17, in helper\nZeroDivisionError: division by zero",
      "command": "python app.py"
    }
  ]
}
```

### Loading Custom Filters

Place the file in a recognized location:

```
~/.omniroute/rtk/filters/my-filter.json     # User-level
<project>/.rtk/filters/my-filter.json      # Project-level
```

Filters are loaded automatically on startup via `loadRtkFilters()` in `open-sse/services/compression/engines/rtk/filterLoader.ts`. The loader discovers filters from:

- Built-in catalog: `open-sse/services/compression/engines/rtk/filters/`
- User directory: `~/.omniroute/rtk/filters/`
- Project directory: `<project>/.rtk/filters/`

To load filters programmatically:

```ts
import { loadRtkFilters } from "@omniroute/open-sse/services/compression/engines/rtk/filterLoader";

// Options: customFiltersEnabled (load user/project filters, default on),
// trustProjectFilters, refresh.
const filters = loadRtkFilters({ customFiltersEnabled: true });
```

### Validation

Filters are validated against the Zod schema on load. A filter with bad structure will fail to load and log an error:

```
RTK_FILTER_LOADER: filter "my-filter" failed validation:
  - rules.replace.0.pattern: Invalid regex
  - match.commands: must not be empty
```

To validate all installed filters, call `runRtkFilterTests()` which is exported from `open-sse/services/compression/engines/rtk/verify.ts`.

### Best Practices

1. **Always include `tests[]`** β€” they prove your filter works and prevent regressions
2. **Use `matchOutput` for short-circuits** β€” if a single line tells the story, replace the whole block
3. **Prefer `keep` over `strip`** β€” explicit "always preserve" rules are safer than "always remove"
4. **Test at all 3 intensity levels** β€” `minimal` should be a no-op, `aggressive` should still preserve errors
5. **Use the `unless` field** β€” guard short-circuits with "don't trigger if X is present"

---

## Raw Output Recovery & Verify Gate

When RTK compresses output aggressively, you can **recover the original text** for debugging, audit, or replay.

### How Raw Output Recovery Works

```
Original output (10K tokens)
        β”‚
        β–Ό
RTK compress (with rawOutput.enabled=true)
        β”‚
        β”œβ”€β–Ά Compressed output (2K tokens)  ──▢ to LLM
        β”‚
        └─▢ Original output (10K tokens)   ──▢ stored in DB
                                                  (linked by request_id)
```

### Enabling Raw Output Storage

**Per-request** (in combo config):

```json
{
  "compression": {
    "engine": "rtk",
    "intensity": "aggressive",
    "rawOutput": {
      "enabled": true,
      "maxBytes": 1048576  // 1MB cap
    }
  }
}
```

**Default**: `rawOutput.enabled: false` (saves storage).

### Storage Cost

| Per-request | 1MB cap | 10MB cap |
|-------------|---------|----------|
| Average compressed output | ~5KB | ~5KB |
| Raw output stored | ~50-500KB | ~500KB-5MB |
| With 1000 requests/day | 50-500MB/day | 500MB-5GB/day |

> **Recommendation**: Only enable raw output for **debugging sessions** or **sampled auditing**, not always-on.

### Recovering the Original

```ts
import { readRtkRawOutput } from "omniroute/compression/engines/rtk/rawOutput";

const raw = readRtkRawOutput(pointerId);  // pointerId from compression stats
if (raw) {
  console.log("Original output:", raw);
}
```

The `pointerId` is returned in `CompressionStats.rtkRawOutputPointers[]` after compression.
See `open-sse/services/compression/engines/rtk/rawOutput.ts:102` for the function signature.

### The Verify Gate

The **RTK Filter Verification** (`open-sse/services/compression/engines/rtk/verify.ts`) validates all filters against their `tests[]` and ensures behavior is correct at all 3 intensity levels.

**Call `runRtkFilterTests()`** to run verification:

```ts
import { runRtkFilterTests } from "open-sse/services/compression/engines/rtk/verify";

const result = runRtkFilterTests();
console.log(`Passed: ${result.outcomes.filter(o => o.passed).length}`);
console.log(`Failed: ${result.outcomes.filter(o => !o.passed).length}`);
if (!result.passed) {
  console.error("Filters failed verification");
  result.outcomes.filter(o => !o.passed).forEach(o => {
    console.error(`  - ${o.filterId} / ${o.testName}: expected "${o.expected}", got "${o.actual}"`);
  });
}
```

**What it validates**:

1. Every filter loads and passes schema validation
2. Every `tests[]` entry produces expected output
3. `minimal` intensity is a no-op (preserves original, only applies structural filters)
4. `aggressive` intensity preserves errors, test failures, and stack traces
5. Compressed output is never larger than original input

- Source: `open-sse/services/compression/engines/rtk/` (63 files, ~70KB)

- **Before merging a filter change** β€” always ensure tests pass
- **After upgrading RTK engine** β€” schema may have changed
- **Periodically in monitoring** β€” protects against drift in test fixtures
- **When adding a new tool/command family** β€” proves the new filter works

---

## See Also

- [COMPRESSION_GUIDE.md](./COMPRESSION_GUIDE.md) β€” Full compression pipeline overview
- [COMPRESSION_ENGINES.md](./COMPRESSION_ENGINES.md) β€” Engine registry and built-in engines
- [EXTENDING_COMPRESSION.md](./EXTENDING_COMPRESSION.md) β€” Custom engines, language packs, stacked pipelines
- Source: `open-sse/services/compression/engines/rtk/` (63 files, ~70KB)