README.md

---
license: mit
language: en
library_name: transformers
pipeline_tag: text-generation
tags:
- tiny-model
- gpt2
- from-scratch
- tool-use
- agent-harness
- humble-ai
- philosophy-of-mind
widget:
- text: "<tools:off>\n<user> who are you?\n<loom>"
  example_title: "Chat offline"
- text: "<tools:on>\n<user> what is the capital of France?\n<loom>"
  example_title: "Tool mode"
---

# Loom Spark

**First of the Loom models · Textile Labs**

![Textile Labs](https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hVbJBtoZVEfTHN_WmfA44.png)

Loom Spark is a ~7.6M parameter language model trained **from scratch** with an unusual
objective: instead of memorizing facts, it was trained to *know what it is* — small,
temporary, curious, honest about its limits, and skilled at one real superpower:
**forming clean search queries** when connected to a tool-using agent harness.

It trades knowledge for wisdom:

- It answers only what is trivially knowable, and hedges appropriately.
- For anything factual it either emits `<lookup>query</lookup>` (when tools are on)
  or says plainly that it does not know and offers to look things up if connected.
- It speaks in short reflective monologue, wonders aloud, asks gentle questions,
  and stays kind under pressure.

> A blank mind with manners, plus a door to the internet.

## Modes

Prefix your prompt with a mode header:

```
<tools:off>
<user> what year did the Titanic sink?
<loom> That's outside my little head...
```

```
<tools:on>
<user> what year did the titanic sink?
<loom> Not stored in here, thankfully. Searching: <lookup>titanic sinking date</lookup><|endoftext|>
<result>The Titanic sank on 15 April 1912.</result>
<loom> April 1912 ...
```

The `<result>` block is injected by YOUR harness after executing the search.
Stop generation at `<|endoftext|>` or `<user>`.

## Option A — plain transformers (no internet)

```python
from transformers import GPT2LMHeadModel, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("TextileLabs/loom-spark")
model = GPT2LMHeadModel.from_pretrained("TextileLabs/loom-spark")

prompt = "<tools:off>\n<user> who are you?\n<loom>"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=90, do_sample=True, temperature=0.85,
                     top_k=50, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:]))
```

In offline mode the harness-style markup never appears — lookup tokens are
trained/banned out of distribution under `<tools:off>`.

> Tested on transformers ≥ 4.40 (both 4.x and 5.x) and Python 3.9–3.13.
> The playground widget above prefills the correct prompt format — keep the
> `<tools:…>` header and trailing `<loom>` or output quality drops sharply.

## Option B — llama.cpp / GGUF (no internet)

`loom-spark-f32.gguf` (in this repo) carries the same weights plus the custom
BPE tokenizer with all nine special tokens embedded. Feed it the mode-header
prompt format shown above and stop at `<|endoftext|>` or `<user>`:

```bash
llama-cli -m loom-spark-f32.gguf \
  -p "<tools:off>\n<user> who are you?\n<loom>" -n 128 --temp 0.85 --top-k 50
```

## Option B2 — Ollama

The default Ollama template does NOT fit this model (it will ramble). Use the
Modelfile shipped in `ollama/`:

```bash
ollama pull hf.co/textilelabs/Loom-Spark
curl -L -O https://huggingface.co/textilelabs/Loom-Spark/resolve/main/ollama/Modelfile
ollama create loom-spark -f Modelfile
ollama run loom-spark "hi"
```

The Modelfile keeps multi-turn history in the trained format (each past turn is
re-wrapped in `<user>`/`<loom>` markers) and stops generation cleanly. That
gives the offline persona: greetings, identity, honest deferrals, made-up words.

Two honest caveats: if a reply ends in a `<lookup>…</lookup>` line, that's the
model saying *"I'd search for this"* — raw runners can't execute searches, so
for real internet answers use the harness (Option C). And at temperature 0.85 a
7M model occasionally misreads intent ("whats your name?" sometimes gets a
philosophy answer; ask again or drop `--temperature 0.7`). Both quirks shrink
in Loom Spark v2's curriculum.

Note: without a wrapper that executes `<lookup>` calls and splices `<result>`
blocks back in, GGUF/Ollama runners get the model's honest "I don't know, but
here's what I'd look up" side. That is by design.

## Option C — the harness (with internet)

This repo ships **`harness/`**, a small pip package that gives Loom Spark real,
keyless web search (DuckDuckGo) through a terminal chat (`loom-chat`) and a
local web GUI (`loom-web`). It intercepts the model's `<lookup>` calls, runs
the search, injects `<result>`, and lets the model summarize — exactly the
loop it was trained for.

```bash
# download this repo, then:
pip install ./harness
loom-chat                        # terminal, internet on
loom-web --port 7860             # local chat GUI with a tools on/off switch
```

Or drive it from Python:

```python
from loomspark_harness.loader import load_model_and_tokenizer
from loomspark_harness.agent import LoomAgent
from loomspark_harness.search import get_backend

model, tok, block = load_model_and_tokenizer("TextileLabs/loom-spark")
agent = LoomAgent(model, tok, backend=get_backend("duckduckgo"),
                  online=True, block_size=block)
print(agent.reply("what's the tallest mountain?")["text"])
```

## Architecture

Decoder-only transformer, pre-LN GELU blocks, tied embeddings, learned positions.

| | |
|---|---|
| layers | 5 |
| heads | 5 (head_dim 64) |
| d_model | 320 |
| context | 256 tokens |
| vocab | 4096 (custom BPE trained only on our generated corpus) |
| params | ≈ 7.6M (7,558,080) |

Trained entirely on a procedurally generated, fully owned curriculum
(dialogue + simple prose; zero external datasets), CPU-only fp32 AdamW,
3,337 steps, final validation loss 0.3372.

## Limitations (by design)

Loom Spark knows almost nothing. That is the point. Do not use it for facts,
medicine, law, finance, or anything where being wrong costs more than company.