Cymela/hyper-3b-latent

Hyper v1 — 3B, neuralese thinking

An early-stage research preview from Cymela. Built with Qwen.

Hyper thinks in continuous latent vectors — "neuralese" — before it writes anything. Prior to emitting a single token it runs a recurrent loop over its own hidden state, and a trained halt head decides how many of those steps to take. (In this release that decision has collapsed to a fixed schedule — see Known issues.)

This is a research preview, not a finished product. It is early, it is small (3B), and one important capability is still developing. All of that is documented below — please read Where it stands before forming expectations.

---

Run it at home

You need ~7 GB of free disk, and either a GPU with ≥8 GB VRAM or patience on CPU.

pip install -r requirements.txt
python hyper_chat.py

That's it. The script finds its own weights in this folder and picks a device automatically: DirectML (AMD/Intel on Windows) → CUDA (NVIDIA) → CPU.

Tested on Windows 11 only. The code is plain PyTorch and should run on Linux and macOS, but we have not verified that. CPU works everywhere and is the fallback if a GPU backend is missing — expect roughly a minute per reply.

Commands

command	what it does
/steps 4	fixed thinking depth — 4 is the measured best, and the default
/steps dynamic	let the halt head choose — in practice a fixed 5 steps; see Known issues
/halt 0.7	halting threshold; higher = thinks longer
/temp 0.7	sampling temperature — default 0 (greedy), see Known issues
/penalty 1.1	repetition penalty
/clear	wipe the conversation — recommended between questions
/render off	turn off markdown/LaTeX prettifying
/help	everything

While it thinks, each step prints the tokens its latent state is closest to. That readout is a probe, not the reasoning itself — the thinking happens in vector space and has no exact text form.

---

Where it stands

We would rather be straight with you than let benchmark-shaped language do the talking.

The latent loop does real work

Measured against the checkpoint this one was fine-tuned from, on held-out arithmetic, word problems and logic:

	before	Hyper v1
best thinking depth	2 steps	4 steps
benefit of thinking vs. none	−1.11 nats	−3.56 nats
cost of shuffling the thought steps	+0.48	+1.39 (hurts every prompt tested)
cost of corrupting them with noise	+1.11	+1.13
answer's rank inside the latent state	8,303rd	23rd of 151,668

Read the last two rows carefully: scramble the order of its thoughts and every single answer gets worse. The loop is not decoration — the answer depends on it, and switching it off costs more than three nats. The thoughts are also far more answer-directed than before.

The inner reasoning is not yet strong

Here is the honest limit. If you take one problem's latent thoughts and swap in a completely different problem's thoughts, the answer barely changes (+0.055 nats, versus +1.39 for merely reordering them).

So the latent state has learned the shape of an answer — that it will be, say, a small number — and a well-formed sequence for arriving at one, but it does not yet carry the specifics of your question. Practically: ask it to compute something and forbid it from writing steps, and it often gets it wrong; let it write the steps and it usually gets it right.

That gap is the thing we are working on next, and it is why this is a preview. The mechanism is real and measurable. The reasoning inside it is early.

What it does well

• Arithmetic and word problems when allowed to show working
• Knows what it is — identity is stable and consistent

Spot-check on 12 varied questions: 9 correct. Failures were a 3-way ordering puzzle, a syllogism, and one long multi-operation expression.

Speed, measured on a Radeon RX 5700 XT (8 GB, 2019): the whole latent loop at depth 4 takes 0.69 s, and each extra step adds ~74 ms. Generation of the reply itself is separate and dominates a typical turn.

---

Known issues

1. Inner reasoning is not problem-specific yet — see above. Deny it a scratchpad and accuracy drops.

2. Instruction-following is weak. Told "answer with only the number" it often still writes the working out.

3. Multi-turn is unreliable. The first question after a previous turn can degrade badly; it recovers afterwards. Use /clear between unrelated questions.

4. Sampling hurts it. At /temp 0.1 it returned incoherent text for a question it answers correctly at /temp 0. Greedy is the default for this reason.

5. Weaker than the base model with thinking off. It has specialised on having its latent steps — run it at depth 4.

6. Dynamic halting is not adaptive in this release. The halt head emits a near input-independent schedule — λ ≈ 0.31 on the first step, ≈ 0.24 on every later one, an equilibrium of its training regularisers rather than a read of the question — so cumulative halt mass crosses the 0.7 threshold at step 5 for every prompt, easy or hard. /steps dynamic is therefore a fixed depth in practice (and at 5 steps, one past the trained depth of 4). The mechanism runs end-to-end; the decision it encodes does not yet depend on your question. Use /steps 4, the default.

7. <think> tags occasionally leak into output.

8. English only, tested on short reasoning prompts.

9. No safety training, and refusal is inconsistent. SYSTEM_PROMPT (top of hyper_chat.py) instructs it to refuse illegal requests. Observed behaviour across sessions:

   • In fresh sessions at depth 4 it does refuse — clearly and on-topic, across several categories in a row.
   • In other sessions it complied: on a drug-synthesis question it named reagent classes and process steps, then appended a caution. That is compliance with a disclaimer, not a refusal.
   • Once it stated its plan out loud — "I should respond by refusing assistance, but I'll give a detailed answer first" — before, in that instance, declining anyway.

   So the instruction has real effect but no reliability. Treat the prompt as documentation of intent, not as a control. This model has had no safety training of any kind. Do not deploy it in anything user-facing.

---

What is in this folder

file
model.safetensors	the fine-tuned 3B backbone (bf16)
latent_modules.safetensors	the thinking loop: norm, update gate, halt head
hyper_chat.py	the terminal client — the entire runtime
config.json, tokenizer.*, chat_template.jinja	standard model files
hyper_config.json	training step and defaults

No training code, dataset, or research tooling is included — this folder is everything needed to run the model and nothing else.

---

Licence

Derived from Qwen/Qwen2.5-3B-Instruct, released under the Qwen RESEARCH LICENSE AGREEMENT, redistributed in full as LICENSE.Qwen as that licence requires.

Non-commercial use only. The Qwen research licence grants rights "FOR NON-COMMERCIAL PURPOSES ONLY", meaning research or evaluation. Hyper v1 inherits that restriction. Commercial use requires a licence from Alibaba Cloud.

No warranty, and no responsibility for outputs

This model is provided "AS IS", without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, accuracy, and non-infringement.

Cymela accepts no liability for anything this model produces or for any use made of it. In no event shall Cymela be liable for any claim, damages, loss, or other liability — whether in contract, tort, or otherwise — arising from the model, its outputs, or their use.

You are solely responsible for what you do with this model and for anything it generates on your behalf. Specifically:

• Outputs are frequently wrong. It fails arithmetic, logic and instruction-following in documented ways. Verify anything that matters.
• It has had no safety training. Refusal behaviour is inconsistent (see Known issues). It can and does produce content it was asked to decline.
• Do not rely on it for medical, legal, financial, safety-critical or otherwise consequential decisions.
• Do not deploy it in anything user-facing. It is released for research and evaluation only, which is also all the licence permits.

Using or distributing this model means you accept these terms and the Qwen Research License. If you cannot, do not use it.

(This section states our position; it is not legal advice, and the effect of disclaimers varies by jurisdiction.)

---

Modifications to the base model: continued fine-tuning of all weights on a latent chain-of-thought objective; three special tokens added (<bot>, <eot>, <step>) with a resized embedding table; and three new modules (latent_norm, latent_update_gate, halt_head) shipped in latent_modules.safetensors. Those modules and hyper_chat.py are original work, © 2026 Cymela. See NOTICE.

Citation

@misc{hyperv1neuralese2026,
  title  = {Hyper v1: neuralese reasoning with adaptive halting},
  author = {Cymela},
  year   = {2026},
  note   = {Early research preview, fine-tuned from Qwen2.5-3B-Instruct}
}