LF-01-320M

LF-01-320M is a small chat-tuned checkpoint of LoomFormer-Paraplex, an experimental decoder-only language model with causal GQA, Paraplex FFNs, DepthAttn residual reads, and a recurrent 3×3 Tria operator carry.

This is a proof of concept, not a polished small language model. It is undertrained because I am limited by compute and, honestly, by laziness. That is not a euphemism. The checkpoint is published because the architecture is already alive enough to inspect: it can hold the chat format, produce coherent text, and react observably to Tria inference controls. Its knowledge, factuality, reasoning, and resistance to repetition are still weak.

Important: this is custom LoomFormer code, not a drop-in transformers.AutoModelForCausalLM checkpoint. Use the LoomFormer-Paraplex repository and its loomchat.py runtime. The Hub inference widget is intentionally disabled.

Model files

File	Purpose
LF01-320M.bf16.aio	Recommended chat package: bf16 weights, tokenizer, chat template, and model configuration in one Loom archive
LF01-320M.bf16.safetensors	Weight-only export for inspection and custom integration
config.json	Architecture and training configuration
tokenizer32768.json	Exact 32,768-token tokenizer
chat_template.jinja	ChatML-style chat and tool-call template
config/	Published PT and SFT configurations

The .aio archive is the easiest and least ambiguous way to run the model. The Safetensors file is safe weight serialization, but it does not by itself provide a standard Transformers implementation. Do not silently substitute a different tokenizer, chat template, or model class.

Quick start

Clone the runtime:

git clone https://github.com/srose69/Loomformer-Paraplex.git
cd Loomformer-Paraplex
./setup.sh

Download the model:

hf download srs6901/LF-01-320M \
  LF01-320M.bf16.aio \
  --local-dir checkpoints/LF-01-320M

Start an interactive chat:

python loomchat.py \
  checkpoints/LF-01-320M/LF01-320M.bf16.aio \
  --device cuda:0 \
  --dtype bf16 \
  --temperature 0.7 \
  --top-p 0.9 \
  --max-new 512

Inside LoomChat, /help lists the runtime controls. The most interesting ones for this checkpoint are:

/window 256
/alpha 0.1
/beta 0.5

These are the training values. Changing them is an experiment, not a recommended production tuning recipe. Changing window, alpha, or beta invalidates cached state, which LoomChat handles for you.

If model weights fit on one GPU but a long KV cache does not, persistent KV can be kept in pinned CPU memory:

python loomchat.py \
  checkpoints/LF-01-320M/LF01-320M.bf16.aio \
  --device cuda:0 \
  --dtype bf16 \
  --kvstorage cpu

What the checkpoint can and cannot do

The model has completed supervised chat fine-tuning and generally recognizes the assistant format. It can write locally coherent English, follow simple instructions, and sometimes keep a scene or subject stable across a response. It also has <think> and </think> tokens and can produce a recognizable planning register when that register is forced by a prefilled prefix.

At its current maturity:

• factual content is unreliable;
• reasoning is shallow and often becomes a performance of reasoning rather than a solution;
• a short prompt with one salient word can become a strong lexical attractor;
• repetition and semantic drift are common;
• instruction following is fragile;
• ordinary instructions such as “think step by step” do not reliably activate the think register;
• no broad capability, safety, bias, or standard academic benchmark suite is reported.

For qualitative experiments, use a longer prompt with several semantic anchors and generate enough tokens to cross multiple temporal boundaries. Short questions such as “What causes rain?” mostly measure lexical attraction in this checkpoint and are poor probes of Tria cadence.

Architecture

Property	Value
Unique model parameters	317,850,314
Public size class	320M
Decoder blocks	20
Model width	576
Query heads	24 × 24
KV heads	6
GQA group size	4
Paraplex hidden width	3,456
Vocabulary	32,768
Embeddings	tied
Phase sectors	open
Activation	PvPowLU
PT sequence length	2,048
SFT sequence length	4,096 with YaRN
Context length	8,192 with YaRN
Tria temporal window during training	256
Tria carrier alpha during training	0.1
PolARM beta during training	0.5
Weight precision in published files	bf16

The exact parameter count refers to unique model parameters. Serialized state files may contain an explicit copy of tied weights and therefore appear larger when tensor entries are counted naively.

LoomFormer in one paragraph

LoomFormer remains a causal Transformer: token mixing is still performed by causal grouped-query attention and autoregressive inference still uses a KV cache. The experiment begins around that backbone:

1. Paraplex replaces the ordinary scalar FFN activation with coupled real, phase, and phase-conditioned output coordinates.
2. DepthAttn reads learned mixtures of states produced earlier in network depth instead of using only a fixed residual source.
3. Tria builds a small operator from the Paraplex coordinates, composes it through layers and selected temporal boundaries, and feeds learned summaries back into later computation.

The full equations, implementation notes, CUDA paths, and memory accounting are in the LoomFormer-Paraplex README.

Training

LF-01 saw 5,980,160,000 pretraining tokens (~19 TPP) across two PT phases:

Phase	Dataset	Steps	Sequence length	Temporal window	Achieved bpb
PT 1	fw-edu-10bt	35,000	2,048	256	1.08
PT 2	FSS1STR	15,000 additional steps, ending at PT step 50,000	2,048	256	1.24

It was then supervised-fine-tuned on FSSFT1:

Phase	Dataset	Steps	Tokens	Sequence length	Temporal window
SFT	FSSFT1	10,000	327,680,000	4,096 with YaRN	256

The SFT budget is about one token per model parameter. No explicit <CARRY> tokens were inserted into the training data. Temporal refeed was trained using the fixed W=256 boundaries: eight possible fixed-window refeeds in a 2,048-token PT stream and sixteen in a 4,096-token SFT stream.

These dataset names identify the actual local corpora used for the run. They are not declared as Hub dataset IDs, and this model card does not pretend to provide a complete public reconstruction of their provenance or filtering.

The training optimizer was ATOM. The published checkpoint is the result of the SFT stage, not the raw pretrained base.

Tria: mechanics and current observations

Tria is an auxiliary operator path, not a second token memory with a readable text decoder. For every token and Paraplex hidden channel, it constructs a local 3×3 operator from the pairwise relations between the real, phase, and activated coordinates. Those operators are composed through depth. Finished depth carriers are then composed across a temporal segment and, when a boundary fires, reused as:

• a seed for the next segment's first Tria layer; and
• a sparse final cross-attention key/value source.

The local construction uses a bounded skew-symmetric generator related to infinitesimal rotations in so(3), fixed axis rotations, and local normalization. This gives the route a mathematically controlled way to compose small operator updates, but it does not prove what semantic information a trained network will store there.

Tria has no auxiliary semantic target and no independent LM head. It learns only through how its state changes the ordinary next-token loss. Mechanically, I know how the carrier is built, propagated, stabilized, and read. Semantically, I do not yet know exactly what the trained model uses it for. Direct probes have not produced a stable token-level dictionary.

If I anthropomorphize it, the closest image is not a readable notebook but a tiny hippocampus (7+-2 unno): distributed state that carries context across boundaries and can bias a later trajectory without offering a clean symbolic transcript. This is an analogy, not a neuroscience claim. The less romantic description is “opaque recurrent operator carry.”

Inference controls

LoomChat exposes three useful Tria controls:

Control	Mechanical role	Preliminary observation on LF-01
window / W	Maximum number of generated token steps between fixed temporal boundaries	Smaller values often hold the response closer to the current subject; larger values often permit looser continuation
alpha	Strength of the local Tria carrier update/injection	alpha=0.1 held subject and prompt intent better than alpha=0 in the tested long prompts
beta	PolARM Gram correction at a fired boundary	Higher values appeared smoother and more stable on long runs, but this result is less isolated and less certain

These are empirical interpretations from a small, undertrained checkpoint, not universal laws of the architecture.

window matters only when autoregressive generation actually crosses temporal boundaries. Changing it while generating a short answer, or judging it from a prompt that never reaches a boundary, says very little. Its effect also depends strongly on alpha: with alpha=0, large changes in W produced much less change in the overall generation trajectory.

On richer prompts, tested values 48, 64, 128, 256, and 512 did not simply break the model. They changed the character of generation. A useful working hypothesis from these tests is:

• smaller W: stronger topical hold;
• larger W: more freedom and associative drift;
• larger alpha: stronger influence from the accumulated carrier;
• larger beta: stronger geometric correction and smoothing, potentially improving long-run stability at the cost of blur.

The beta interpretation still needs controlled repeated tests with the same long prompt, several real boundaries, and no simultaneous parameter changes. I have not completed those tests yet because, again, I am lazy.

One qualitative sample

Prompt:

Imagine a small town by the sea where people have lived for generations. Describe what daily life there might feel like across a single day, from morning to night.

With W=48 and a 160-token generation cap, one run began:

What a lovely day to visit a small town. I've been doing this for years, and I'm happy to share that with you. As a small town, I can tell you that the daily life you're facing is truly unforgettable. The sun is shining down on your skin, casting a warm glow over the sand, and the air is thick with the smell of damp earth and wind.

To get started, I might ask you a few questions. What kind of lifestyle do you think you'll be able to support? Are you looking for a quiet, tranquil place to relax or a quiet, peaceful space? Are there any specific activities or events that you'd like to include in your daily life?

This is intentionally not presented as an impressive answer. It shows both sides of the checkpoint: it can maintain a scene and tone, while still misreading perspective, falling into assistant habits, and producing thin content.

Intended use

Reasonable uses:

• architecture research and ablations;
• studying Paraplex, DepthAttn, and Tria dynamics;
• qualitative experiments with temporal carry;
• testing custom inference runtimes;
• toy chat, creative writing, and role-play experiments where unpredictability is acceptable.

This checkpoint is not suitable for:

• factual or high-stakes assistance;
• medical, legal, financial, or safety-critical decisions;
• production agents;
• reliable tool use;
• evaluating the general capability of mature 300M-class models;
• claims about long-context memory without controlled experiments.

The model can emit false, biased, repetitive, unsafe, or nonsensical text. There was no dedicated RLHF stage and no comprehensive safety evaluation. Users are responsible for validation, filtering, and deployment constraints.

Reproducibility and caveats

• The source code is pre-release research code. Configuration names, checkpoint migration, and CUDA kernels may change.
• Causal attention remains quadratic in sequence length. Tria is not a replacement for attention.
• The architecture carries more runtime state than a minimal decoder: attention KV, depth state, phase traces, Tria carriers, and sparse boundary keys.
• The published inference observations are qualitative and prompt-sensitive.
• A small model's lexical attractors can dominate the effects being studied.
• The theoretical ability of a recurrent operator path to propagate state over very long dialogues is not evidence that this checkpoint has learned to do so reliably.
• No license metadata is currently declared for this checkpoint. Check the repository terms before redistribution or downstream release.

Links

• Model: https://huggingface.co/srs6901/LF-01-320M
• Source and architecture documentation: https://github.com/srose69/Loomformer-Paraplex
• Recommended AIO package: https://huggingface.co/srs6901/LF-01-320M/blob/main/LF01-320M.bf16.aio

Citation

Until there is a paper or archived release, cite the source repository and this checkpoint:

@software{lf01320m,
  author = {srose69},
  title  = {LF-01-320M},
  year   = {2026},
  url    = {https://huggingface.co/srs6901/LF-01-320M}
}

@software{loomformer_paraplex,
  author = {srose69},
  title  = {LoomFormer-Paraplex},
  year   = {2026},
  url    = {https://github.com/srose69/Loomformer-Paraplex}
}