README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-0.6B-Base
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- pretraining-data
- data-curation
- noise-pruning
- dataorchestra
---

# DataOrchestra — Noise Pruning (NP) Model

This is the **Noise Pruning (NP)** tool model of [DataOrchestra](https://arxiv.org/abs/2607.24717). It is the lightest of the three cleaning stages: given a document chunk, it emits whole-line deletion operations that strip line-level noise (site navigation, ads, share bars, boilerplate, catalog metadata, etc.) without rewriting any surviving text. It is used together with the [orchestrator](https://huggingface.co/DataOrchestra/Orchestrator) and the SR/PA rewriter, but can also be run on its own.

## Model Details

| | |
| --- | --- |
| Base model | [`Qwen/Qwen3-0.6B-Base`](https://huggingface.co/Qwen/Qwen3-0.6B-Base) |
| Role | Noise Pruning (NP) tool model |
| Input | one line-numbered chunk (≤ 1024 Qwen3 tokens) wrapped in `[DOC]` / `[/DOC]` |
| Output | one or more `remove_lines(start, end)` ops, or `skip()` |
| Inference mode | non-thinking, greedy decoding |

The model is trained ProX-style (following [ProX](https://arxiv.org/abs/2409.17115)): unlike ProX/RefineX, it keeps **only** whole-line removal and drops in-line substring edits, which simplifies the duty of this small tool model.

## Usage

The model sees the chunk with a **0-indexed, zero-padded `[NNN]` prefix on every line**, wrapped in `[DOC]` / `[/DOC]`, under a one-line system prompt. It responds with `remove_lines(start, end)` calls (both ends inclusive, line-indices referring to the `[NNN]` prefixes) or the sentinel `skip()` when nothing should be removed. You line-number the chunk, parse the ops, and delete those lines.

```python
import re
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "DataOrchestra/NP-0.6B"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto", device_map="auto")

SYSTEM_PROMPT = "You are an excellent noise pruning model for pretraining data cleaning."
REMOVE_LINES_RE = re.compile(r"remove_lines\s*\(\s*(\d+)\s*,\s*(\d+)\s*\)")


def prune(chunk: str) -> str:
    lines = chunk.split("\n")
    # NP sees a 0-indexed [NNN] prefix on every line (add_line_numbers()).
    numbered = "\n".join(f"[{i:03d}] {line}" for i, line in enumerate(lines))
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"[DOC]\n{numbered}\n[/DOC]"},   # wrap_doc()
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,           # NP runs non-thinking
    )
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    generated = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=False,                 # greedy: temperature 0.0 / top_p 1.0
    )
    response = tokenizer.decode(
        generated[0][inputs.input_ids.shape[1]:], skip_special_tokens=True
    )

    # Parse remove_lines(start, end); no ops (e.g. skip()) -> keep the chunk as-is.
    remove = set()
    for start, end in REMOVE_LINES_RE.findall(response):
        for i in range(int(start), int(end) + 1):
            if 0 <= i < len(lines):
                remove.add(i)
    return "\n".join(line for i, line in enumerate(lines) if i not in remove)


chunk = (
    "Home | About | Contact\n"
    "The French Revolution began in 1789 and reshaped European politics.\n"
    "Share this on Facebook | Twitter\n"
    "It led to the rise of Napoleon Bonaparte."
)
print(prune(chunk))
# -> keeps the two content lines, drops the nav header and the share bar
```

### Serving with vLLM

For high-throughput curation, serve the model with an OpenAI-compatible endpoint. Note the chunk must still be line-numbered by the caller before sending:

```bash
vllm serve DataOrchestra/NP-0.6B --served-model-name DataOrchestra-NP-0.6B --trust-remote-code
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="DataOrchestra-NP-0.6B",
    messages=[
        {"role": "system", "content": "You are an excellent noise pruning model for pretraining data cleaning."},
        {"role": "user", "content": "[DOC]\n[000] Home | About | Contact\n[001] <your content line>\n[/DOC]"},
    ],
    temperature=0.0,
    max_tokens=1024,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)
```

## Output Format

The model emits one operation per line:

```
remove_lines(0, 0)
remove_lines(2, 2)
```

- `remove_lines(start, end)` — delete lines `start` through `end` **inclusive**, where indices refer to the `[NNN]` prefixes of the input. Only whole-line removal is supported.
- `skip()` (or an empty / op-free response) — remove nothing; the chunk is kept unchanged.

Apply the ops by deleting the referenced lines (process them bottom-up, or collect all removed indices first, so earlier deletions do not shift later indices).

## Citation

If you find this work useful, please cite:

```bibtex
@article{dataorchestra2026,
  title   = {DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data},
  author  = {Huang, Zhen and Wang, Yikun and Xia, Shijie and Liu, Pengfei},
  year    = {2026},
  journal = {arXiv preprint arXiv:2607.24717}
}
```