README.md

---
license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE
base_model: Qwen/Qwen2.5-3B-Instruct
tags:
  - latent-reasoning
  - neuralese
  - adaptive-computation
  - research-preview
language:
  - en
pipeline_tag: text-generation
---

# Hyper v1 — 3B, neuralese thinking

**An early-stage research preview from [Cymela](https://cymela.com). Built with Qwen.**

Hyper thinks in **continuous latent vectors** — "neuralese" — before it writes
anything. Prior to emitting a single token it runs a recurrent loop over its
own hidden state, and a trained halt head decides how many of those steps to
take. (In this release that decision has collapsed to a fixed schedule — see
*Known issues*.)

This is a **research preview**, not a finished product. It is early, it is
small (3B), and one important capability is still developing. All of that is
documented below — please read *Where it stands* before forming expectations.

---

## Run it at home

You need ~7 GB of free disk, and either a GPU with ≥8 GB VRAM or patience on
CPU.

```bash
pip install -r requirements.txt
python hyper_chat.py
```

That's it. The script finds its own weights in this folder and picks a device
automatically: **DirectML** (AMD/Intel on Windows) → **CUDA** (NVIDIA) → **CPU**.

> **Tested on Windows 11 only.** The code is plain PyTorch and should run on
> Linux and macOS, but we have not verified that. CPU works everywhere and is
> the fallback if a GPU backend is missing — expect roughly a minute per reply.

### Commands

| command | what it does |
|---|---|
| `/steps 4` | fixed thinking depth — **4 is the measured best**, and the default |
| `/steps dynamic` | let the halt head choose — in practice a fixed 5 steps; see *Known issues* |
| `/halt 0.7` | halting threshold; higher = thinks longer |
| `/temp 0.7` | sampling temperature — **default 0 (greedy)**, see *Known issues* |
| `/penalty 1.1` | repetition penalty |
| `/clear` | wipe the conversation — **recommended between questions** |
| `/render off` | turn off markdown/LaTeX prettifying |
| `/help` | everything |

While it thinks, each step prints the tokens its latent state is closest to.
That readout is a *probe*, not the reasoning itself — the thinking happens in
vector space and has no exact text form.

---

## Where it stands

We would rather be straight with you than let benchmark-shaped language do the
talking.

### The latent loop does real work

Measured against the checkpoint this one was fine-tuned from, on held-out
arithmetic, word problems and logic:

| | before | **Hyper v1** |
|---|---|---|
| best thinking depth | 2 steps | **4 steps** |
| benefit of thinking vs. none | −1.11 nats | **−3.56 nats** |
| cost of shuffling the thought steps | +0.48 | **+1.39** (hurts every prompt tested) |
| cost of corrupting them with noise | +1.11 | **+1.13** |
| answer's rank inside the latent state | 8,303rd | **23rd** of 151,668 |

Read the last two rows carefully: **scramble the order of its thoughts and
every single answer gets worse.** The loop is not decoration — the answer
depends on it, and switching it off costs more than three nats. The thoughts
are also far more answer-directed than before.

### The inner reasoning is not yet strong

Here is the honest limit. If you take one problem's latent thoughts and swap
in a *completely different* problem's thoughts, the answer barely changes
(+0.055 nats, versus +1.39 for merely reordering them).

So the latent state has learned the **shape** of an answer — that it will be,
say, a small number — and a well-formed sequence for arriving at one, but it
does not yet carry the specifics of *your* question. Practically: ask it to
compute something and forbid it from writing steps, and it often gets it
wrong; let it write the steps and it usually gets it right.

That gap is the thing we are working on next, and it is why this is a preview.
The mechanism is real and measurable. The reasoning inside it is early.

### What it does well

- **Arithmetic and word problems** when allowed to show working
- **Knows what it is** — identity is stable and consistent

Spot-check on 12 varied questions: 9 correct. Failures were a 3-way ordering
puzzle, a syllogism, and one long multi-operation expression.

Speed, measured on a Radeon RX 5700 XT (8 GB, 2019): the whole latent loop at
depth 4 takes **0.69 s**, and each extra step adds ~74 ms. Generation of the
reply itself is separate and dominates a typical turn.

---

## Known issues

1. **Inner reasoning is not problem-specific yet** — see above. Deny it a
   scratchpad and accuracy drops.
2. **Instruction-following is weak.** Told "answer with only the number" it
   often still writes the working out.
3. **Multi-turn is unreliable.** The first question after a previous turn can
   degrade badly; it recovers afterwards. **Use `/clear` between unrelated
   questions.**
4. **Sampling hurts it.** At `/temp 0.1` it returned incoherent text for a
   question it answers correctly at `/temp 0`. Greedy is the default for this
   reason.
5. **Weaker than the base model with thinking off.** It has specialised on
   having its latent steps — run it at depth 4.
6. **Dynamic halting is not adaptive in this release.** The halt head emits a
   near input-independent schedule — λ ≈ 0.31 on the first step, ≈ 0.24 on
   every later one, an equilibrium of its training regularisers rather than a
   read of the question — so cumulative halt mass crosses the 0.7 threshold
   at step 5 for every prompt, easy or hard. `/steps dynamic` is therefore a
   fixed depth in practice (and at 5 steps, one past the trained depth of 4).
   The mechanism runs end-to-end; the decision it encodes does not yet depend
   on your question. Use `/steps 4`, the default.
7. **`<think>` tags occasionally leak** into output.
8. **English only**, tested on short reasoning prompts.
9. **No safety training, and refusal is inconsistent.** `SYSTEM_PROMPT` (top
   of `hyper_chat.py`) instructs it to refuse illegal requests. Observed
   behaviour across sessions:
   - In fresh sessions at depth 4 it *does* refuse — clearly and on-topic,
     across several categories in a row.
   - In other sessions it complied: on a drug-synthesis question it named
     reagent classes and process steps, then appended a caution. That is
     compliance with a disclaimer, not a refusal.
   - Once it stated its plan out loud — *"I should respond by refusing
     assistance, but I'll give a detailed answer first"* — before, in that
     instance, declining anyway.

   So the instruction has real effect but no reliability. Treat the prompt as
   documentation of intent, not as a control. This model has had no safety
   training of any kind. **Do not deploy it in anything user-facing.**

---

## What is in this folder

| file | |
|---|---|
| `model.safetensors` | the fine-tuned 3B backbone (bf16) |
| `latent_modules.safetensors` | the thinking loop: norm, update gate, halt head |
| `hyper_chat.py` | the terminal client — the entire runtime |
| `config.json`, `tokenizer.*`, `chat_template.jinja` | standard model files |
| `hyper_config.json` | training step and defaults |

No training code, dataset, or research tooling is included — this folder is
everything needed to *run* the model and nothing else.

---

## Licence

Derived from `Qwen/Qwen2.5-3B-Instruct`, released under the **Qwen RESEARCH
LICENSE AGREEMENT**, redistributed in full as `LICENSE.Qwen` as that licence
requires.

> **Non-commercial use only.** The Qwen research licence grants rights "FOR
> NON-COMMERCIAL PURPOSES ONLY", meaning research or evaluation. Hyper v1
> inherits that restriction. Commercial use requires a licence from Alibaba
> Cloud.

### No warranty, and no responsibility for outputs

This model is provided **"AS IS", without warranty of any kind**, express or
implied, including but not limited to warranties of merchantability, fitness
for a particular purpose, accuracy, and non-infringement.

**Cymela accepts no liability for anything this model produces or for any use
made of it.** In no event shall Cymela be liable for any claim, damages, loss,
or other liability — whether in contract, tort, or otherwise — arising from
the model, its outputs, or their use.

You are solely responsible for what you do with this model and for anything it
generates on your behalf. Specifically:

- **Outputs are frequently wrong.** It fails arithmetic, logic and
  instruction-following in documented ways. Verify anything that matters.
- **It has had no safety training.** Refusal behaviour is inconsistent (see
  *Known issues*). It can and does produce content it was asked to decline.
- **Do not rely on it** for medical, legal, financial, safety-critical or
  otherwise consequential decisions.
- **Do not deploy it in anything user-facing.** It is released for research
  and evaluation only, which is also all the licence permits.

Using or distributing this model means you accept these terms and the Qwen
Research License. If you cannot, do not use it.

*(This section states our position; it is not legal advice, and the effect of
disclaimers varies by jurisdiction.)*

---

Modifications to the base model: continued fine-tuning of all weights on a
latent chain-of-thought objective; three special tokens added
(`<bot>`, `<eot>`, `<step>`) with a resized embedding table; and three new
modules (`latent_norm`, `latent_update_gate`, `halt_head`) shipped in
`latent_modules.safetensors`. Those modules and `hyper_chat.py` are original
work, © 2026 Cymela. See `NOTICE`.

## Citation

```bibtex
@misc{hyperv1neuralese2026,
  title  = {Hyper v1: neuralese reasoning with adaptive halting},
  author = {Cymela},
  year   = {2026},
  note   = {Early research preview, fine-tuned from Qwen2.5-3B-Instruct}
}
```