BranchOne / README.md
ndemirdemir's picture
Add Colab example timing ten questions with one model load
d3ce9ae verified
|
Raw History Blame Contribute Delete
14.1 kB
---
license: apache-2.0
library_name: transformers
base_model: Qwen/Qwen3.5-0.8B-Base
base_model_relation: adapter
tags:
- branchone
- decision-model
- multimodal
- text-classification
- custom-code
- peft
---
# BranchOne
BranchOne chooses among candidate descriptions supplied with each request. Give
it a situation, a question, and two or more choices, optionally with one image.
It returns one probability per choice and the selected candidate's ID.
This release contains trained LoRA adapters, a shared decision head, a processor,
calibration, and the inference code. The loader downloads the pinned public
Qwen3.5-0.8B-Base backbone separately on first use. No reproduction repository,
training scripts, or GitHub access is required.
## Install
The supported setup is Linux x86-64, Python 3.13, and an NVIDIA CUDA GPU.
The reference environment uses an A100, PyTorch 2.11.0 with CUDA 12.8,
Transformers 5.17.0, and FP16 backbone weights. The head computes in FP32.
CPU inference, other precisions, and quantization are not qualified for this
release. Leave several GB of GPU memory available; requirements increase with
input length. A maximum-memory benchmark is not provided.
On a fresh Colab GPU runtime or a Linux CUDA machine:
```bash
python3 -m pip install uv==0.12.17
python3 -m uv tool run --from huggingface-hub==1.33.0 hf download ndemirdemir/BranchOne --local-dir BranchOne
python3 BranchOne/install.py --venv .venv
```
The installer creates an isolated environment, installs the locked dependencies,
and compiles `causal-conv1d==1.7.0`. It needs CUDA 12.8 development tools. On
Colab it can install those tools through the configured NVIDIA apt repository;
on a machine without root access, install the toolkit first and set `CUDA_HOME`.
The initial dependency installation and kernel build can take several minutes.
Run the examples below with `.venv/bin/python`, or select that environment in
your notebook. These setup scripts install inference dependencies only.
For a complete Colab cell using this environment, see
[Ten questions with timing in Colab](#ten-questions-with-timing-in-colab).
Creating `.venv` does not switch the notebook's own Python environment.
## Text example
```python
import os
from transformers import AutoModel
# Defaults to the Hub. A downloaded directory works with the same interface.
model = AutoModel.from_pretrained(
os.environ.get("BRANCHONE_MODEL", "ndemirdemir/BranchOne"),
trust_remote_code=True,
device="cuda",
)
result = model.predict({
"state": "The parcel arrived, but the item inside is broken.",
"question": "What should happen next?",
"candidates": [
{"id": "replace", "description": "Arrange a replacement for the damaged item."},
{"id": "track", "description": "Track the parcel because it has not arrived."},
],
})
print(result["selected_id"])
print(dict(zip(result["candidate_ids"], result["probabilities"])))
```
`trust_remote_code=True` loads the BranchOne inference implementation included
in this model repository. For reproducible deployments, add
`revision="<release commit SHA>"` once a release is published. To use the directory
downloaded above, set `BRANCHONE_MODEL=./BranchOne`. Both the model package and the
base model must already be cached when using `local_files_only=True`.
## Image example
Use the same loaded `model`. This complete example creates a sample picture;
replace `picture` with a PIL image or a local image path for your own input.
```python
from PIL import Image, ImageDraw
picture = Image.new("RGB", (512, 512), "white")
ImageDraw.Draw(picture).ellipse((128, 128, 384, 384), fill="blue")
image_result = model.predict({
"state": "Look at the supplied image.",
"question": "Which shape is shown?",
"image": picture,
"candidates": [
{"id": "circle", "description": "A circle."},
{"id": "square", "description": "A square."},
{"id": "triangle", "description": "A triangle."},
],
})
print(image_result["selected_id"])
print(dict(zip(image_result["candidate_ids"], image_result["probabilities"])))
```
## Ten questions with timing in Colab
After installation finishes, paste this entire block into one Colab code cell.
It assumes the environment is at `/content/.venv`, as with the setup above run
from Colab's default `/content` directory. The first line must be `%%bash`.
The model loads once, runs a separate warm-up example, then answers ten different
text questions with three choices each. Each prediction's timer includes input
preparation and inference, with CUDA synchronization before and after the call.
Model loading, warm-up, and printing are excluded from those prediction times.
The summary reports loading and warm-up separately, plus the total, mean,
median, fastest, and slowest prediction times.
```bash
%%bash
set -e
/content/.venv/bin/python -u - <<'PY'
import os
import time
import statistics
import torch
from transformers import AutoModel
def make_request(state, question, choices):
return {
"state": state,
"question": question,
"candidates": [
{"id": str(i), "description": choice}
for i, choice in enumerate(choices)
],
}
# Load once.
started = time.perf_counter()
model = AutoModel.from_pretrained(
os.environ.get("BRANCHONE_MODEL", "ndemirdemir/BranchOne"),
trust_remote_code=True,
device="cuda",
)
torch.cuda.synchronize()
loading_seconds = time.perf_counter() - started
print("GPU:", torch.cuda.get_device_name(0))
print(f"Model loading: {loading_seconds:.3f} seconds")
questions = [
(
"The parcel arrived, but the item inside is broken.",
"What should happen next?",
[
"Arrange a replacement for the damaged item.",
"Track the parcel because it has not arrived.",
"Send a receipt for a new purchase.",
],
),
(
"Buenos días. ¿A qué hora abre la tienda?",
"Which language is this text written in?",
["Spanish.", "French.", "English."],
),
(
"Alex cannot attend on Tuesday. Alex is available Thursday after 2 PM.",
"Which appointment fits Alex's availability?",
["Tuesday at 10 AM.", "Thursday at 3 PM.", "Thursday at 9 AM."],
),
(
"Refunds are allowed within 30 days with a receipt. "
"The customer purchased the item 10 days ago and has the receipt.",
"Does this request meet the stated refund conditions?",
[
"Yes, both conditions are satisfied.",
"No, the purchase was too long ago.",
"No, the receipt is missing.",
],
),
(
"Ingredients: flour, eggs, and milk. Mix together, "
"pour into a pan, and bake for 25 minutes.",
"What kind of document is this?",
["A recipe.", "An invoice.", "Meeting minutes."],
),
(
"A Python program reports: FileNotFoundError: "
"No such file or directory: '/data/input.csv'.",
"What problem does this error describe?",
[
"The requested file cannot be found at that path.",
"The network connection timed out.",
"The password was rejected.",
],
),
(
"Maya will send the draft on Friday. Leon will review it on Monday.",
"Who is responsible for sending the draft?",
["Maya.", "Leon.", "Both Maya and Leon."],
),
(
"The rule says: reorder when stock falls below five items. "
"There are currently two items in stock.",
"What action follows the rule?",
[
"Place a replenishment order.",
"Wait because stock is above the threshold.",
"Permanently stop selling the item.",
],
),
(
"The picnic will be indoors if it rains and outdoors otherwise. "
"It is raining.",
"Where should the picnic take place?",
["Indoors.", "Outdoors.", "The location cannot be determined."],
),
(
"This product broke after one day. I regret buying it "
"and would not recommend it.",
"What is the overall sentiment?",
["Positive.", "Negative.", "Neutral."],
),
]
# Separate warm-up: excluded from the ten measured questions.
warmup = make_request(
"The delivery arrived on time and everything is intact.",
"Which description matches the delivery?",
["On time and undamaged.", "Missing.", "Damaged."],
)
torch.cuda.synchronize()
started = time.perf_counter()
model.predict(warmup)
torch.cuda.synchronize()
warmup_seconds = time.perf_counter() - started
print(f"Warm-up: {warmup_seconds:.3f} seconds")
print("\nRunning 10 timed predictions...")
# Reuse the same loaded model for every question.
timings = []
for number, (state, question, choices) in enumerate(questions, start=1):
request = make_request(state, question, choices)
torch.cuda.synchronize()
started = time.perf_counter()
result = model.predict(request)
torch.cuda.synchronize()
elapsed = time.perf_counter() - started
timings.append(elapsed)
descriptions = {
candidate["id"]: candidate["description"]
for candidate in request["candidates"]
}
print(f"\n{number}. {question}")
print("Selected:", descriptions[result["selected_id"]])
print(f"Prediction time: {elapsed:.3f} seconds")
for candidate_id, probability in zip(
result["candidate_ids"], result["probabilities"]
):
print(f" {probability:.1%} — {descriptions[candidate_id]}")
print("\n--- Timing summary ---")
print(f"Model loading: {loading_seconds:.3f} seconds")
print(f"Warm-up: {warmup_seconds:.3f} seconds")
print(f"10 predictions: {sum(timings):.3f} seconds")
print(f"Mean per prediction: {statistics.mean(timings):.3f} seconds")
print(f"Median: {statistics.median(timings):.3f} seconds")
print(f"Fastest: {min(timings):.3f} seconds")
print(f"Slowest: {max(timings):.3f} seconds")
PY
```
All ten predictions share the loaded model within this process. Re-running the
whole cell starts a new Python process and reloads the model; separate `%%bash`
cells cannot share the `model` variable. On first use, loading can also include
downloading the Qwen backbone. New input shapes may still trigger additional
kernel initialization after the warm-up.
This is a short-text timing example for your runtime, not a representative
accuracy benchmark or a measurement of long-context or image performance.
It does not establish sub-second latency. Save any results you need and stop
your Colab runtime when finished.
## Inputs and outputs
- `state`: situation or evidence as text.
- `question`: the decision to make.
- `candidates`: 2–150 unique string IDs with nonempty text descriptions. IDs are
external labels; the model reads the descriptions. This is an input guard,
not a claim of equal quality or speed at all candidate counts.
- `image`: optional single PIL image or local path. Image URLs, multiple images,
video, and audio are unsupported.
The limit is **8,192 expanded tokens per candidate branch**, including the
situation, question, prompt formatting, that candidate's description, and image
tokens. Images are resized to a budget of at most 256 visual tokens. Oversized
inputs raise an error; they are not silently truncated. Increasing the context
setting is not a supported extension of this trained release.
`predict()` returns `selected_id`, input-aligned `candidate_ids`, `probabilities`,
raw `logits`, `temperature`, token counts, and calibration/policy metadata.
The probability distribution is conditional on the supplied choices. It does
not establish that a correct choice is present.
By default, `action` is `"defer"` and `deferral_status` is `"no validated policy"`:
the release has no validated automatic-acceptance threshold. The predicted
candidate and probabilities are still returned. An optional caller-supplied
`threshold` changes the action rule; callers must validate their own threshold.
## Architecture and training
Each candidate is encoded against the same evidence on an independent Qwen
branch. A shared head maps each 1,024-wide candidate representation through a
256-wide projection and residual MLP to one scalar. Softmax over the scores,
with the supplied temperature, gives the candidate probabilities. This
checkpoint uses the independent scorer without a candidate-comparison layer.
The vision backbone is frozen. The text backbone has trained rank-8 LoRA
adapters; the decision head is trained. The vocabulary generation head is not
used. Prefix-cache reuse is disabled for this release. Additional candidates
and longer inputs still require additional computation.
Training used supervised single-label choice examples from procedural synthetic
text and rendered images. The checkpoint was selected using development loss;
temperature scaling was fitted on a separate calibration split. This release
does not establish general-purpose decision accuracy, natural-image competence,
or calibrated probabilities on a new domain. Boolean and ordered-score quality
have not been separately validated.
## Limitations
Evaluate on representative data before deploying. Missing evidence, ambiguous
choices, unsupported image content, and distribution shifts can produce wrong
answers with high confidence. Temperature scaling does not guarantee that a
particular prediction is correct. Duplicate descriptions share evidence and
can change the probability distribution.
This is an experimental inference release. It does not claim sub-second
latency for the trained checkpoint, fast long-context CPU inference, or
superiority to other decision models. The package is intended for
`AutoModel.from_pretrained(...).predict(...)`; standard generation pipelines,
hosted inference-provider support, and `save_pretrained()` are not provided.
## License
Apache-2.0. See `LICENSE` and `NOTICE`. The pinned Qwen backbone is
`Qwen/Qwen3.5-0.8B-Base` at revision
`dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68` and retains its upstream attribution.