lowdown-labs/fela-autocomplete

FELA LLM 1.5: On device code autocomplete (fill in the middle)

FELA Autocomplete is a code completion model. You give it the code before your cursor, and optionally more codebase context, and it writes the piece that goes in between. This is the same fill in the middle idea that powers the gray autocomplete text in a modern editor. It is a 1.79 billion parameter model that runs on a plain CPU with no GPU, so it can sit inside a local IDE helper or an on premises code tool and complete code without sending your source to the cloud.

It was warm started from Qwen2.5-Coder 1.5B and clean room distilled, then annealed and supervised fine tuned for fill in the middle. Our key niche insight is to provide for inline single line completion, not full text generation. We feel this a great compromise to have a speedy development experience while still having a product that doesn't contribute to cognitive atrophy and feels better to use.

Headline result: fill in the middle

We wanted to apply the FELA methodology as an LLM. Given a tight training budget while we study how to scale this new architecture, we thought a smaller model would be a great autocomplete tool to demonstrate the power of efficient computing.

We measure functional pass@1 with the HumanEval-SingleLineInfilling methodology, so a completion counts only when it makes the code run correctly, not when it matches the original character for character.

Stage	FIM Pass@1	FIM EM	FIM Edit Sim	MBPP	HumanEval
Base (15B token distill)	49.6%	42.0	73.6	13.0	11.0
Plus fill in the middle anneal	54.4%	47.4	77.6	16.0	9.8
Plus fill in the middle SFT (final)	54.6%	46.8	78.4	18.0	11.0

Over half of single line completions functionally work. We lead with FIM pass@1 because that is the product metric. The Rust server auto formats the output in microseconds, so we optimize for semantic correctness (pass@1 and edit similarity), not whitespace exact match.

MBPP and HumanEval are shown for context only. They ask the model to write a whole function from a description, which is the wrong lens for an autocomplete model and low by design at this size. We provide it to compare with other language models at size ~1.5B. Do not read the HumanEval number as the quality of the product; read the FIM pass@1.

How it got here

1. Base: clean room distill from Qwen2.5-Coder 1.5B to 15 billion tokens, on a permissive code mix with fill in the middle in the objective.
2. Fill in the middle anneal (the big lever): a further billion tokens with a line aware fill in the middle objective (mask whole lines, matching how the eval scores lines) across Python, JavaScript, TypeScript, Java, Go, Rust, and C plus plus. This train and eval alignment gave the largest jump.
3. Fill in the middle SFT: a further 1.2 billion tokens with the native Qwen fill in the middle tokens and the loss on the middle span. This consolidated the gains and lifted general code (MBPP 16 to 18).

What goes in, what comes out

• Input: your code as text, split into a prefix (before the cursor) and an optional suffix (after the cursor). The text is turned into tokens with the Qwen2.5-Coder tokenizer, which carries the fill in the middle markers <|fim_prefix|>, <|fim_suffix|>, and <|fim_middle|>.
• Output: the code that belongs between the prefix and the suffix. By default it writes one line and stops, which is what an inline autocomplete wants. With no suffix it simply continues the prefix.
• In plain terms: you type import numpy as and it finishes the line with np; you give it the body of def add(a, b): with the call add(2, 3) waiting below it, and it fills in return a + b.

Why we built it this way

The model has no growing cache of past tokens, which is the usual reason memory creeps up as context gets longer. Its sequence mixers are Fourier Neural Operators (a filter the model learns and applies in the frequency domain) for global context, Gated-DeltaNet recall layers that overwrite an old memory when a new one arrives (so keys do not interfere), and a Landmark routing layer that lets a token look back at summaries of earlier chunks. None of these keep a per token attention cache, so the working memory stays small and fixed no matter how much code is in the window. That is what lets it decode on a low power CPU next to you, with the source never leaving the machine.

The full stack is 28 layers at width 1536, in the repeating pattern of three Fourier mixers then one Gated-DeltaNet recall layer, with a Landmark routing layer every seventh layer.

Performance

The model runs on CPU with no GPU. Two paths run these weights:

Path	Format	Size on disk	Single core decode
Reference (this repo, modeling.py)	bf16	3.58 GB	about 4 tokens per second
Production (FELA server, separate Rust runtime)	int8	about 2.08 GB	about 22 tokens per second

The modeling.py loader here is the plain, readable reference path in pure PyTorch. It loads the weights, runs a real forward, and decodes on CPU, which is enough for the quickstart and for verifying the model. It is a correctness path, not the fast path, so do not read its speed as the product speed. Loading model_int8.safetensors in Python is smaller in memory but not faster, because the pure torch int8 path dequantizes the weights on the fly.

The fast path is the CPU native FELA server, a separate Rust runtime that runs these weights as int8 with a constant state streaming decode and no KV cache. On one core it decodes at about 22 tokens per second, roughly 20 times the plain f32 path, measured at this model's shape with a steady state O(1) decode step. Single stream decode is bound by memory bandwidth, since it streams the full int8 weight set once per token, so adding cores does not speed up one stream (measured flat from one to eight cores); extra cores serve more streams at once and speed up the prefill. The server's Rust int8 forward is checked against the PyTorch model with a golden test, so the fast serving path produces the same model, not a lookalike.

What to expect

Some real completions from the final model:

• import numpy as gives np
• prefix def add(a, b): with the call add(2, 3) in the suffix gives return a + b
• prefix def is_even(n):\n return n % 2 == gives 0

Where it earns its keep is single line fill in the middle on the patterns developers write all day - and it can be made more useful with prefilled repository context. It is a 1.5B class model, so it is not a whole function generator or a reasoning model: multi line blocks, novel algorithms, and anything that needs deep reasoning about your specific code are outside what it is built for. Read completions before you use them.

How to run it

It runs on CPU; no GPU is required. See quickstart/ for a runnable example. The short version, using the self contained modeling.py, model_cpu_gpt2.py, and config.json in this repo:

from modeling import load_model
m = load_model("/path/to/this/repo")

# Fill in the middle: code before and after the cursor -> the middle
r = m.complete("def add(a, b):\n    ", suffix="\nresult = add(2, 3)\n")
print(r["middle"])          # e.g. " return a + b"

# Plain autocomplete: continue a single line
print(m.complete("import numpy as ")["middle"])   # e.g. " np"

# Load the smaller int8 weights instead (same completions, smaller on disk)
m8 = load_model("/path/to/this/repo", quant="int8")

It also loads with transformers directly. The code is custom, so pass trust_remote_code=True:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("lowdown-labs/fela-autocomplete", trust_remote_code=True, dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("lowdown-labs/fela-autocomplete", trust_remote_code=True)
prompt = "<|fim_prefix|>def add(a, b):\n    <|fim_suffix|>\nresult = add(2, 3)\n<|fim_middle|>"
enc = tok(prompt, return_tensors="pt")
out = model.generate(**enc, max_new_tokens=8, use_cache=False)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

The loader reads the bf16 model.safetensors by default. Pass quant="int8" to load the smaller model_int8.safetensors instead; it dequantizes the linear weights with their per channel scales and keeps the Fourier filters in bf16, giving the same completions. Pass quant="auto" to prefer bf16 when present and fall back to int8.

complete is greedy by default, so a skeptic gets the same real output twice. For an interactive playground, see the Hugging Face Space in space/. For production serving, use the CPU native FELA server (https://github.com/Lowdown-Labs/fela_server).

Loading, exports, and verification

• Weights ship as model.safetensors in bf16 (no pickle code execution risk). config.json holds the architecture hyperparameters. tokenizer.json is the Qwen2.5-Coder tokenizer with the fill in the middle tokens. The architecture is defined in the self contained model_cpu_gpt2.py, and the pure torch CPU paths for the Gated-DeltaNet and Landmark layers are in cpu_delta.py, cpu_landmark.py, cpu_swa.py, and cpu_patch.py, which ship beside it. modeling.py wires them into a loader with a complete fill in the middle entry point.
• verify.py builds the model, loads the weights, runs a fixed input forward, checks the output shape and the vocabulary size, confirms the logits are finite, and compares the top predicted tokens against a captured reference.

Formats

• bf16: the reference weights and the default load path in this repo (model.safetensors, 3.58 GB).
• int8: the smaller on device format (model_int8.safetensors, about 2.08 GB), weight only int8 per channel on the large linear layers, with the Fourier filters kept in bf16. Load it in Python with load_model(dir, quant="int8"), or serve it with the FELA server. The int8 forward matches the bf16 model by the golden test.

Model details

• Parameters: 1,788,379,536 (about 1.79 billion).
• Layers: 28, width 1536, 12 heads, feed forward hidden 8960.
• Sequence mixers: Fourier Neural Operator (512 modes), Gated-DeltaNet recall, and Landmark global routing, in the SSSL pattern (three Fourier mixers then one recall layer), with a Landmark layer every seventh layer.
• Tokenizer: Qwen2.5-Coder (vocabulary 151936), which carries the fill in the middle tokens.
• Warm start: Qwen2.5-Coder 1.5B, then clean room distilled, annealed, and fill in the middle supervised fine tuned.

Intended use, limitations, and safety

What it is for: inline code autocomplete and fill in the middle inside a local or on premises developer tool, where running on CPU with no GPU and keeping the source on the machine matter. It is a research preview for that use.

What it is not for: it is not a whole function generator, not a chat model, and not a reasoning model. Do not rely on its completions for correctness or security without reading them, and do not ship it into a product without your own evaluation. It is a 1.5B class model and will be wrong on anything past common single line patterns.

Evaluated conditions and known failure modes:

• Strong on single line, high frequency fill in the middle; weaker on multi line blocks and novel logic.
• The reference modeling.py path is bf16 and slow; the int8 FELA server is the fast path.
• No safety or license filtering of generated code is claimed. Review generated code before you use it.

Privacy: the model runs on the device or on your own CPU, so your source does not have to leave the machine and there is no per call API. That is the on device privacy angle.

Model family

This is part of the FELA family from Lowdown Labs: one Fourier Neural Operator architecture across many modalities, all CPU native and subquadratic. This repo is pushed as lowdown-labs/FELA-autocomplete. Sibling repos are trained independently per modality and share no weights, so none carries a base_model link.

Acknowledgements and references

• Fourier Neural Operator: Li, Z., et al. (2021). Fourier Neural Operator for Parametric Partial Differential Equations. ICLR. https://arxiv.org/abs/2010.08895
• Gated Linear Attention: Yang, S., et al. (2023). Gated Linear Attention Transformers with Hardware-Efficient Training. https://arxiv.org/abs/2312.06635
• Gated DeltaNet: Yang, S., et al. (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. https://arxiv.org/abs/2412.06464
• Landmark Attention: Mohtashami, A., Jaggi, M. (2023). Landmark Attention: Random-Access Infinite Context Length for Transformers. https://arxiv.org/abs/2305.16300
• Qwen2.5-Coder: Hui, B., et al. (2024). Qwen2.5-Coder Technical Report. https://arxiv.org/abs/2409.12186
• PyTorch: Paszke, A., et al. (2019). NeurIPS. https://arxiv.org/abs/1912.01703
• flash-linear-attention: the reference kernels for the Gated-DeltaNet layer. https://github.com/fla-org/flash-linear-attention

How to cite

@misc{lowdownlabs_fela_llm,
  title  = {FELA LLM 1.5: An on device Fourier Neural Operator code autocomplete model},
  author = {Lowdown Labs},
  year   = {2026},
  note   = {Model card}
}

License

Released under the Lowdown Labs Lovely License 1.0 (CC BY-NC 4.0 plus Hippocratic License 3.0). See LICENSE. For most LL models, a commercial license may be available; contact Lowdown Labs.