README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.6-27B
base_model_relation: finetune
pipeline_tag: text-generation
library_name: transformers
language:
- en
- vi
tags:
- function-calling
- tool-use
- tool-calling
- agent
- agentic
- workflow-automation
- vllm
- sglang
- qwen3.6
- vietnamese
- enterprise-automation
---

# Ops-I — a 27B agent for business workflow automation

**Ops-I** is a fine-tune of **Qwen3.6-27B** aimed at *workflow* agents: episodes where the
model reads a business request, discovers the right tool out of a large surface, chains
several calls together, and — just as importantly — knows when **not** to act.

On the public 600-task [AutomationBench](https://github.com/zapier/AutomationBench)
(Zapier), Ops-I is the **strongest of four open-weight models** we measured, ahead of the
Qwen3.6-27B base it was tuned from.

**Use it if you need:** an open-weight backend for business automation that holds up across long multi-step tool episodes. Really want to hear about

| | |
|---|---|
| Base | `Qwen/Qwen3.6-27B` (dense, VL architecture) |
| Precision / size | bf16, 27.8B params, 15 shards (~56 GB) |
| Context | base supports 262 144; 16K validated in our runs |
| Reasoning | thinking **on** — how it was evaluated |
| Tool-call format | Qwen3.5-style XML (`<function=...>`), **not** JSON — see gotchas |
| License | Apache-2.0 |

---

## Results — AutomationBench 600


| Model | Pass rate | Passed / 600 | Partial credit | Wall clock |
|---|---|---|---|---|
| **beyoru/Ops-I** | **10.17%** | **61** | **45.54%** | 1.96 h |
| gemma-4-31B-it (thinking on) | 10.17% | 61 | 41.64% | 2.15 h |
| Qwen3.6-27B (base) | 9.67% | 58 | 46.00% | 1.95 h |

For scale, the reference models in the upstream README (run at their highest reasoning
effort) land at Claude Opus 4.8 30.33%, GPT-5.6 Sol 29.17%, GLM 5.2 20.33%,
Gemini 3.5 Flash 14.83%.

### Per domain

Tasks passed out of 100, partial credit in parentheses.

| Model | Finance | HR | Marketing | Operations | Sales | Support |
|---|---|---|---|---|---|---|
| **beyoru/Ops-I** | **20** (57.0) | 3 (10.1) | 5 (51.4) | **18** (60.5) | **10** (38.2) | 5 (55.9) |
| Qwen3.6-27B base | 21 (57.7) | 3 (12.4) | 3 (45.1) | 17 (60.9) | 8 (41.4) | 6 (58.4) |
| gemma-4-31B-it | 18 (48.7) | 1 (5.2) | **14** (56.3) | 15 (56.2) | 9 (34.3) | 4 (49.1) |

Finance and operations carry the score. HR is the floor for every model tested (1-3 / 100).

---

## Known weakness: it acts too much

The dominant failure mode is **not** inability to act — it is failure to *refrain*.

| Assertion type | Ops-I |
|---|---|
| Positive ("do send / do create") | 3028 / 5636 = **53.7%** |
| Negative ("do **not** send / do **not** create") | 65 / 585 = **11.1%** |

Some negative assertion types never pass, aggregated across all four models tested:
`gmail_message_not_sent` 0/203, `slack_message_not_in_channel` 0/166,
`google_sheets_row_not_exists` 0/161.

This is uniform across two model families and four checkpoints, so treat it as a property
of the current open-weight generation rather than of Ops-I specifically. **If you deploy
this in a workflow that can send email, post to chat, or write to a sheet, gate those
actions behind confirmation.**

---

## Quickstart

### Serve with SGLang

```bash
python3 -m sglang.launch_server --model-path beyoru/Ops-I --served-model-name Ops-I \
  --host 0.0.0.0 --port 30014 --tp 4 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --mem-fraction-static 0.85
```

### Serve with vLLM

```bash
vllm serve beyoru/Ops-I --served-model-name Ops-I \
  -tp 4 --max-model-len 16384 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3
```

### Call it

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30014/v1", api_key="EMPTY")

tools = [{
    "type": "function",
    "function": {
        "name": "crm_update_contact",
        "description": "Update a contact's details in the CRM.",
        "parameters": {
            "type": "object",
            "properties": {"contact_id": {"type": "string"}, "phone": {"type": "string"}},
            "required": ["contact_id", "phone"],
        },
    },
}]

r = client.chat.completions.create(
    model="Ops-I",
    messages=[{"role": "user", "content": "Update Le Van A's phone to 0912345678."}],
    tools=tools,
    tool_choice="auto",
)
print(r.choices[0].message.tool_calls)
```

---

## Training


---

## Evaluation notes

- All numbers above come from a single run per model (k=1) on the full public 600-task
  set, temperature left at server default. No task was excluded.
- AutomationBench has **no separate train split** — the 600 public tasks are both the
  dataset and the scoring set. Ops-I was **not** trained on them; if you fine-tune against
  this benchmark, its scores stop being comparable to anything.
- Numbers were produced with SGLang 0.5.12. A vLLM baseline was not measured, so do not
  mix serving stacks when comparing.

# Limitaion

**The model only take a few steps training because of LIMITATION gpu usage in our org**

## Community Feedback

*Ops-I* is an ongoing project, and community feedback is invaluable.

If you discover failure cases, benchmark results, or have ideas for future improvements, feel free to start a Discussion. Every report helps improve future releases.

## Citation

Benchmark tasks, tools and assertions are Zapier's:

```bibtex
@misc{automationbench,
  title  = {AutomationBench},
  author = {Zapier},
  year   = {2026},
  url    = {https://github.com/zapier/AutomationBench}
}
```