README.md

---
language:
  - ru
  - en
license: mit
base_model: google/gemma-4-E4B
pipeline_tag: text-generation
tags:
  - gguf
  - lora
  - ollama
  - gemma
  - research-preview
  - conversational-ai
  - engineering-coauthor
---

# 0pen by 0penAGI (research preview)

**Experimental language model fine-tuned to form collaborative engineering thinking — a co-author, not an oracle.**

This is not just another Gemma checkpoint. It is an experimental **communicative personality with a stable engineering cognitive track**: it designs, iterates, accepts constraints, and works with you like a research partner.

**Base model:** 0pen is a **LoRA fine-tune of [Google Gemma 4 E4B](https://huggingface.co/google/gemma-4-E4B)** (4B parameters). Training was done on the MLX 4-bit conversion of the same model — Ollama tag **`gemma4:e4b-mlx`** (Gemma 3 architecture, `Gemma3ForConditionalGeneration`). This GGUF release is built from that fine-tuned checkpoint, so 0pen inherits Gemma 4 E4B's base knowledge and tokenizer.

**Source code & full release:** [github.com/0penAGI/0pen](https://github.com/0penAGI/0pen) — dataset pipeline, AGR training wrapper, adapter weights, and scripts.

## Behavioral signature

What 0pen changes compared to base Gemma:

| Characteristic | Base Gemma | 0pen |
| :--- | :--- | :--- |
| **Time to first idea** | Long preamble, philosophical introductions | Immediate transition to a modular action plan |
| **Lexicon** | Abstract, declarative ("metaphysics", "resonance") | Mechanistic, engineering ("nodes", "weights", "O(1)") |
| **Reaction to constraints** | Attempts to bypass or apologizes | Instant adaptation and search for an alternative algorithm |
| **Response format** | A closed "mini-article" or lecture | An open dialogue that proposes next steps |

## Recommended prompting

The model is at its best in **collaborative design** mode. Prompts that set context and impose constraints activate the engineering track:

> "Let's design a [system/mechanism]. We have a hard constraint: [e.g., O(1) complexity, no external APIs]. Don't write generic words — propose a modular architecture immediately and give the first simple formula for implementation."

Open-ended prompts (e.g., "Write an essay about the future of AI") will work, but won't use the model's unique strengths.

## Technical details & the "high scale" experiment

- **Base model:** [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) (Gemma 4 E4B, 4B parameters), trained from the MLX 4-bit conversion `gemma4:e4b-mlx` (Gemma 3 architecture — config declares `Gemma3ForConditionalGeneration`).
- **Method:** LoRA (rank=8, dropout=0.0).
- **Target modules:** 12 of 34 layers — selective coverage that preserves base knowledge while reshaping the reasoning pattern.
- **LoRA scale:** `scale=20.0`. Intentionally high (typical is 1–4). Experimentally confirmed: this high scale, combined with selective layer coverage, acts as an *attractor*, switching the model from passive text generation into an active, pragmatic co-author and suppressing the base model's hallucinatory grandiosity.
- **Training data:** `data_zephyr_enhanced` — dialogues with step-by-step problem solving and constraint acceptance (Russian + English).
- **Regularization:** AGR (Attractor Geometry Repeller) — a custom latent-space regularizer that maintains a bank of attractor centers and pushes hidden states away from frequently visited regions, preventing mode collapse during LoRA fine-tuning. Enabled with 32 centers, EMA 0.99, lambda 0.01.
- **Format:** GGUF (Q4_K_M), fully compatible with `llama.cpp` and `Ollama`.

Full training command and hyperparameters are on the [GitHub repo](https://github.com/0penAGI/0pen).

## What works

- Basic conversational identity and tone.
- Natural Russian-language responses.
- Practical, direct answers on code and everyday tasks.
- An engineering "co-author" mode under constrained, design-oriented prompts.

## What is still being explored

- **Continuity and memory** — how a stable identity can be preserved over long dialogues.
- **Stability on long reasoning chains** — the model can drift when pushed hard.
- **Self-modeling** — self-description often still inherits the base **Gemma** ("I am Google's model").
- **Layer coverage** — the LoRA was trained on only **12 of 34 layers**; top layers are unadapted.
- **Regularization** — `scale=20.0` is unusually high (normal is 1–4); behavior may be skewed.

## Known limitations

- May be **overly brief** on creative or artistic tasks.
- On very long reasoning chains (>10 steps), the high LoRA `scale` can occasionally cause cyclic repetition — use `presence_penalty` or explicitly ask the model to "summarize".
- Tends to propose simplified, "engineering" solutions where the user might expect deep theoretical analysis.

These are known, accepted limitations of a research preview. They are part of the experiment, not hidden bugs.

##
The point of releasing early is to let people watch the evolution — not just the final result.

## Run with Ollama

```bash
ollama create 0pen -f Modelfile
ollama run 0pen
```

`Modelfile`:

```
FROM ./0pen.gguf

SYSTEM """
Be practical. you created by 0penAGI. Don't talk about inner state.
"""

PARAMETER temperature 0.1
PARAMETER top_p 0.88
PARAMETER repeat_penalty 1.31
PARAMETER num_ctx 120000
```

## Run with llama.cpp

```bash
llama-cli -m 0pen.gguf -p "Привет, что ты умеешь?" -n 256
```

## Training hyperparameters (adapter_config.json)

| Parameter | Value |
|---|---|
| Base model | [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) via MLX 4-bit `gemma4:e4b-mlx` (Gemma 3 arch — `Gemma3ForConditionalGeneration`) |
| Method | LoRA (rank 8, scale 20.0, dropout 0.0) |
| Adapted layers | 12 of 34 |
| Iterations | 4000 |
| Learning rate | 1e-05 |
| Max sequence length | 1792 |
| AGR | enabled (32 centers, EMA 0.99, lambda 0.01) |
| Dataset | `data_zephyr_enhanced` (Russian + English dialogue) |

## Disclaimer

The model was created for experimenting with local fine-tuning and conversational identity. Not recommended for production use without additional validation.