Add model card
Browse files- MODEL_CARD.md +30 -0
- README.md +34 -0
MODEL_CARD.md
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SimpleTool-VLM-4B model card
|
| 2 |
+
|
| 3 |
+
## Summary
|
| 4 |
+
|
| 5 |
+
`corrected_mixed_frozen_r4` is a Qwen3-VL-4B multimodal checkpoint fine-tuned for SimpleTool's parallel structured output protocol. The model receives an image plus a policy/task prompt and a function-tool schema. It produces independent textual branches for `function` and the arguments required by the schema.
|
| 6 |
+
|
| 7 |
+
## Input contract
|
| 8 |
+
|
| 9 |
+
A request contains a PNG image, system/user messages, and JSON-schema tools. The image is the source of invoice values in the demo; the client does not insert the printed numbers into the prompt. The model may still fail or return an illegal schema, so callers must inspect `legal` and retain `raw` outputs.
|
| 10 |
+
|
| 11 |
+
## Modes
|
| 12 |
+
|
| 13 |
+
- **Direct**: emit function and active argument heads.
|
| 14 |
+
- **Adaptive**: emit a bounded short `content` field, then the active heads. Empty content is valid and common.
|
| 15 |
+
|
| 16 |
+
Each head terminates on its own protocol stop marker. The public runtime dynamically activates only the argument heads needed by the largest tool schema.
|
| 17 |
+
|
| 18 |
+
## Training and evaluation boundary
|
| 19 |
+
|
| 20 |
+
The checkpoint was trained from a Qwen3-VL-4B base with mixed text/image structured-call data. The internal deployment smoke test passed the prepared invoice contract on the released serving path. Those probes do not establish general OCR accuracy, accounting correctness, or performance on arbitrary images.
|
| 21 |
+
|
| 22 |
+
Strict adapter-to-merged numerical parity was not established. The merged checkpoint was independently loaded and probed. Do not describe this release as a drop-in OCR replacement or as a reasoning trace generator.
|
| 23 |
+
|
| 24 |
+
## Quantization
|
| 25 |
+
|
| 26 |
+
BF16 is the reference release. W4A16 and FP8 files are separate exports with their own backend requirements and should be validated with the provided protocol before use.
|
| 27 |
+
|
| 28 |
+
## Safety
|
| 29 |
+
|
| 30 |
+
Tool calls should be validated against an allow-list and business rules before execution. Do not use the invoice example to approve payments automatically.
|
README.md
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SimpleTool-VLM
|
| 2 |
+
|
| 3 |
+
SimpleTool-VLM is a research release of a Qwen3-VL-4B model adapted for structured tool calls from images and text. It emits a function name and only the argument heads required by the supplied tool schema. The public release contains inference code and reproducibility notes; model weights are hosted separately on Hugging Face.
|
| 4 |
+
|
| 5 |
+
## Highlights
|
| 6 |
+
|
| 7 |
+
- Direct and Adaptive modes.
|
| 8 |
+
- Function plus dynamically sized `arg1...argN` heads (`N` is the maximum property count in the supplied tools).
|
| 9 |
+
- Image input is passed as pixels; invoice values are not copied into hidden prompt fields.
|
| 10 |
+
- Raw branch outputs and legality are preserved for auditing.
|
| 11 |
+
|
| 12 |
+
## Intended use
|
| 13 |
+
|
| 14 |
+
Research on real-time multimodal structured decisions, tool calling, and embodied-agent interfaces. This is not an OCR benchmark or a general invoice accounting system. Validate every action before execution.
|
| 15 |
+
|
| 16 |
+
## Model variants
|
| 17 |
+
|
| 18 |
+
- BF16 merged checkpoint: `Cialtion/SimpleTool-VLM`.
|
| 19 |
+
- W4A16 checkpoint: `Cialtion/SimpleTool-VLM-W4A16` (when the upload is complete).
|
| 20 |
+
- FP8 checkpoint: `Cialtion/SimpleTool-VLM-FP8` (when the upload is complete).
|
| 21 |
+
|
| 22 |
+
The quantized variants use the same task adapter and tokenizer contract; backend support depends on the serving stack.
|
| 23 |
+
|
| 24 |
+
## Quick start
|
| 25 |
+
|
| 26 |
+
The lightweight HTTP implementation is in `inference/`. It accepts one PNG, text messages, and OpenAI-style function tools. See `examples/invoice_request.json`. The service returns `prediction.raw`, `prediction.legal`, `events`, and server timings.
|
| 27 |
+
|
| 28 |
+
## Limitations
|
| 29 |
+
|
| 30 |
+
This release does not claim strict adapter-to-merged numerical parity, arbitrary-image robustness, OCR superiority, or native Qwen chain-of-thought. Adaptive `content` is a short task-controlled field. The included invoice demo is an end-to-end protocol demonstration, not a financial approval system.
|
| 31 |
+
|
| 32 |
+
## License and attribution
|
| 33 |
+
|
| 34 |
+
Code and release documentation are Apache-2.0. The checkpoint remains subject to the Qwen3-VL base-model license and any upstream terms. Users must review upstream terms before redistribution or commercial deployment.
|