textilelabs/Loom-Spark

Loom Spark

First of the Loom models · Textile Labs

Loom Spark is a ~7.6M parameter language model trained from scratch with an unusual objective: instead of memorizing facts, it was trained to know what it is — small, temporary, curious, honest about its limits, and skilled at one real superpower: forming clean search queries when connected to a tool-using agent harness.

It trades knowledge for wisdom:

• It answers only what is trivially knowable, and hedges appropriately.
• For anything factual it either emits <lookup>query</lookup> (when tools are on) or says plainly that it does not know and offers to look things up if connected.
• It speaks in short reflective monologue, wonders aloud, asks gentle questions, and stays kind under pressure.

A blank mind with manners, plus a door to the internet.

Modes

Prefix your prompt with a mode header:

<tools:off>
<user> what year did the Titanic sink?
<loom> That's outside my little head...

<tools:on>
<user> what year did the titanic sink?
<loom> Not stored in here, thankfully. Searching: <lookup>titanic sinking date</lookup><|endoftext|>
<result>The Titanic sank on 15 April 1912.</result>
<loom> April 1912 ...

The <result> block is injected by YOUR harness after executing the search. Stop generation at <|endoftext|> or <user>.

Option A — plain transformers (no internet)

from transformers import GPT2LMHeadModel, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("TextileLabs/loom-spark")
model = GPT2LMHeadModel.from_pretrained("TextileLabs/loom-spark")

prompt = "<tools:off>\n<user> who are you?\n<loom>"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=90, do_sample=True, temperature=0.85,
                     top_k=50, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:]))

In offline mode the harness-style markup never appears — lookup tokens are trained/banned out of distribution under <tools:off>.

Tested on transformers ≥ 4.40 (both 4.x and 5.x) and Python 3.9–3.13. The playground widget above prefills the correct prompt format — keep the <tools:…> header and trailing <loom> or output quality drops sharply.

Option B — llama.cpp / GGUF (no internet)

loom-spark-f32.gguf (in this repo) carries the same weights plus the custom BPE tokenizer with all nine special tokens embedded. Feed it the mode-header prompt format shown above and stop at <|endoftext|> or <user>:

llama-cli -m loom-spark-f32.gguf \
  -p "<tools:off>\n<user> who are you?\n<loom>" -n 128 --temp 0.85 --top-k 50

Option B2 — Ollama

The default Ollama template does NOT fit this model (it will ramble). Use the Modelfile shipped in ollama/:

ollama pull hf.co/textilelabs/Loom-Spark
curl -L -O https://huggingface.co/textilelabs/Loom-Spark/resolve/main/ollama/Modelfile
ollama create loom-spark -f Modelfile
ollama run loom-spark "hi"

The Modelfile keeps multi-turn history in the trained format (each past turn is re-wrapped in <user>/<loom> markers) and stops generation cleanly. That gives the offline persona: greetings, identity, honest deferrals, made-up words.

Two honest caveats: if a reply ends in a <lookup>…</lookup> line, that's the model saying "I'd search for this" — raw runners can't execute searches, so for real internet answers use the harness (Option C). And at temperature 0.85 a 7M model occasionally misreads intent ("whats your name?" sometimes gets a philosophy answer; ask again or drop --temperature 0.7). Both quirks shrink in Loom Spark v2's curriculum.

Note: without a wrapper that executes <lookup> calls and splices <result> blocks back in, GGUF/Ollama runners get the model's honest "I don't know, but here's what I'd look up" side. That is by design.

Option C — the harness (with internet)

This repo ships harness/, a small pip package that gives Loom Spark real, keyless web search (DuckDuckGo) through a terminal chat (loom-chat) and a local web GUI (loom-web). It intercepts the model's <lookup> calls, runs the search, injects <result>, and lets the model summarize — exactly the loop it was trained for.

# download this repo, then:
pip install ./harness
loom-chat                        # terminal, internet on
loom-web --port 7860             # local chat GUI with a tools on/off switch

Or drive it from Python:

from loomspark_harness.loader import load_model_and_tokenizer
from loomspark_harness.agent import LoomAgent
from loomspark_harness.search import get_backend

model, tok, block = load_model_and_tokenizer("TextileLabs/loom-spark")
agent = LoomAgent(model, tok, backend=get_backend("duckduckgo"),
                  online=True, block_size=block)
print(agent.reply("what's the tallest mountain?")["text"])

Architecture

Decoder-only transformer, pre-LN GELU blocks, tied embeddings, learned positions.

layers	5
heads	5 (head_dim 64)
d_model	320
context	256 tokens
vocab	4096 (custom BPE trained only on our generated corpus)
params	≈ 7.6M (7,558,080)

Trained entirely on a procedurally generated, fully owned curriculum (dialogue + simple prose; zero external datasets), CPU-only fp32 AdamW, 3,337 steps, final validation loss 0.3372.

Limitations (by design)

Loom Spark knows almost nothing. That is the point. Do not use it for facts, medicine, law, finance, or anything where being wrong costs more than company.