Looking good! Have a few questions.

#2
by pylotlight - opened

Looking into MPS optimisations,

  • Thoughts on a non-CUDA fast path. FastOptions without fla or CUDA graphs: bucketed static shapes + compiled submodules? your fast path is several times faster than the reference path, and the reference path is the only thing Apple Silicon can run.
  • any way to further distill/quant model even further with minimal loss ? still hoping to get an effective super small model in background for all kinds of tasks on device.
  • max_pixels as a request parameter? vision callers can only trade detail for latency by editing a JSON file currently.

There is a limit to how much you can quantize, I think at such speeds it is pointless for this type of LLM.

@TreeLoys I was able to quantize down to ~3.55 GB via int4 with no measured quality loss, but with some response speed loss due to on the fly decoding. so yes, if latency is primary concern leaving at bf16 is generally better etc. was trying to get some other visual processing wins as well to see what I can do with this model as a fun mini project ;p

I'm for any actions and pet projects, except fasting =) We just need more gguf for analysis, so we're waiting

Here's some testing I did fyi.
image

HAL-X org

Looking into MPS optimisations,

  • Thoughts on a non-CUDA fast path. FastOptions without fla or CUDA graphs: bucketed static shapes + compiled submodules? your fast path is several times faster than the reference path, and the reference path is the only thing Apple Silicon can run.
  • any way to further distill/quant model even further with minimal loss ? still hoping to get an effective super small model in background for all kinds of tasks on device.
  • max_pixels as a request parameter? vision callers can only trade detail for latency by editing a JSON file currently.

Hi, thanks for digging into this. These are exactly the right questions.

You're right about Apple Silicon. Today MPS only gets the reference path, and the gated-delta linear-attention layers are the main bottleneck there, because without the Triton kernels they fall back to the sequential PyTorch implementation. A non-CUDA fast path built on bucketed static shapes with compiled per-layer submodules is the direction we'd take as well. Since decision requests have a single forward pass and no generation loop, padding to a few length buckets should let torch.compile keep stable graphs on MPS.

We're going to prototype it, and if you're already experimenting, a PR or even a rough branch would be very welcome.

On going smaller, weight-only quantization (8-bit, and 4-bit with group-wise scales) should cost very little accuracy, because the readout works on probability margins rather than generated text. We'll publish measured numbers on Decision Index before recommending a specific setting. A distilled sub-1B decision model for on-device background use is also on our roadmap. Until then, THX-01 (0.3B) is the option for lightweight on-device tasks that don't need thinking or vision.

Have done some fast path testing which shows promising results. Not convinced you'd want my noob PR, but can submit something for reference for at least what I've tested on my m5.

image

Have done some fast path testing which shows promising results. Not convinced you'd want my noob PR, but can submit something for reference for at least what I've tested on my m5.

image

Good, just a public ggufs on you page, pls. Great job.

HAL-X org

Have done some fast path testing which shows promising results. Not convinced you'd want my noob PR, but can submit something for reference for at least what I've tested on my m5.

image

Hello! You can create PR with GGUF files, we will approve it, and also if you can, make it in folder, for example: (/gguf/model1.gguf), something like this.

Quick update from MPS testing (M5 Pro, 48 GB, bf16), focused on an MPS fast path and GGUF exports.

MPS fast path

Quant / speed-quality-size (same harness)

  • bf16 647 ms / 0.9727, bf16+fast 515 / 0.9731, int8 849 / 0.9731, int8+fast 650 / 0.9731, int4 g32+fast 849 / 0.9742 (132/132), int4 g128+fast 862 / 0.9658 (130/132).
  • int4 g128 saved ~174 MB vs g32 but regressed on real_chart/macro, so int4 g32 + fast looks like the better small option here.

Speed / accuracy / size

Runtime findings (local PoCs)

  • Resolution/pixels dominate e2e latency. Adaptive 384->512->raw (skip 768) gave 194 ms mean, 22/22 vs 1282 ms raw on 22 vision tasks (needs validation on real/small-text docs).
  • Stage timing (bf16+fast): raw 917 ms = vision 230 + LM/other 687; 384 px 165 ms = 39 + 126.
  • Persistent Inductor cache: 126 s -> ~30 s warmup for 5 buckets.

Per-request resolution

Stage timing

Inductor cache warmup

GGUF exports (analysis)

Happy to adjust or re-run specific cases.

T

Quick update from MPS/Apple Silicon testing (M5 Pro, 48 GB, MPS, bf16 unless noted), with a focus on a narrow, PR-able fast path and analysis GGUFs:

MPS fast path (code)

  • PR (draft): https://huggingface.co/HAL-X-AI/THX-02/discussions/3
  • Branch/code: https://huggingface.co/pylotlight/THX-02/tree/mps-fast-path (thx02/mps_fast.py, minimal wiring in thx02/model.py)
  • Approach: compile only backbone.language_model with precomputed inputs_embeds (eager embed + vision + M-RoPE), fixed-length buckets (batch=1, marker readout). Vision stays eager (full-backbone compile fails on image inputs / Metal codegen). dynamic=True unsupported on Metal → static buckets.
  • Validated (AC, 13 req / 10 buckets): 1.31× mean (314→245 ms), eager control 0.98×, 18/18 top-1, max |Δlogit| 0.125, compile ~63 s warmup; short-text 124/160 tok 1.46×/1.36× (bucket spacing 1.25× avoids short-request regressions).
  • Notes: CUDA packed/FLA path unchanged; vision-compile and other knobs measured as small/neutral.

Quantization / speed–quality–size (consistent harness)

  • Refreshed eval (22 tasks × 3 conditions × 2 passes, mean p(truth), interleaved timing): bf16 647 ms / 0.9727, bf16+fast 515 / 0.9731, int8 849 / 0.9731, int8+fast 650 / 0.9731, int4 g32 + fast 849 / 0.9742 (132/132), int4 g128 + fast 862 / 0.9658 (130/132).
  • int4 g128 saves ~174 MB vs g32 at similar latency but regressed on borderline real_chart/macro → int4 g32 + fast is the recommended small option from these runs.

Speed / accuracy / size

Runtime/consumption findings (local PoCs)

  • Resolution/pixels dominate end-to-end latency (vision tower + token-dependent LM). Per-request/adaptive downscaling (384→512→raw, escalate on low confidence/margin, skip 768) gave 194 ms mean, 22/22 vs 1282 ms raw on 22 vision tasks in this harness (needs validation on real/small-text docs before defaulting aggressively).
  • Stage timing (bf16+fast): raw 917 ms = vision 230 + LM/other 687; 384 px 165 ms = 39 + 126 (readout ~0).
  • Persistent Inductor cache makes fast-path warmup predictable: 126 s → ~30 s for 5 buckets when cache is reused/persisted.

Per-request resolution

Stage timing

Inductor cache warmup

GGUF exports (analysis, as requested)

  • Branch: https://huggingface.co/pylotlight/THX-02/tree/add-gguf-lite — gguf/ with Q8_0 (4.17 GB), Q5_K_M (2.87 GB), Q4_K_M (2.54 GB) + gguf/README.md (conversion/provenance, k-quant notes). F16 also built (7.85 GB) and ready to add.
  • PR (draft): https://huggingface.co/HAL-X-AI/THX-02/discussions/4 (lite set on branch: Q8_0/Q5_K_M/Q4_K_M)
  • Backbone-only for analysis (qwen35, 378 tensors incl. SSM). Not a drop-in runnable Decider.decide() in llama.cpp: excludes readout.safetensors, thinker/ (adapter/MTP), and marker-readout decision protocol. Conversion via llama.cpp @ 3ba1ce0 / gguf 0.19.0 (torch 2.15.0, transformers 5.22.2) from prepared backbone copy.

Happy to adjust based on feedback (quant set/layout, bucket defaults, or PR split). Happy to re-run/extend evals if there are specific cases you want covered.

That would be great PR for Apple!
I've just launched THX-02 on my MacBook Neo, and it still runs really well but yeah needs some optimizations

Sign up or log in to comment