README.md

---
license: other
license_name: lowdown-labs-lovely-license-1.0
license_link: LICENSE
tags:
  - fela
  - fourier-neural-operator
  - fno
  - gated-deltanet
  - cpu
  - on-device
  - code
  - autocomplete
  - fill-in-the-middle
  - constant-memory
library_name: transformers
pipeline_tag: text-generation
language:
  - code
model-index:
- name: fela-autocomplete
  results:
  - task:
      type: text-generation
      name: Fill in the middle single line infilling
    dataset:
      type: humaneval-singlelineinfilling
      name: HumanEval SingleLineInfilling
    metrics:
    - type: pass@1
      value: 54.6
      name: FIM pass@1
---

# FELA LLM 1.5: On device code autocomplete (fill in the middle)

FELA Autocomplete is a code completion model. You give it the code before your cursor, and
optionally more codebase context, and it writes the piece that goes in between. This is
the same fill in the middle idea that powers the gray autocomplete text in a modern editor.
It is a 1.79 billion parameter model that runs on a plain CPU with no GPU, so it can sit
inside a local IDE helper or an on premises code tool and complete code without sending your
source to the cloud.

It was warm started from Qwen2.5-Coder 1.5B and clean room distilled, then annealed and
supervised fine tuned for fill in the middle. Our key niche insight is to provide for inline single line
completion, not full text generation. We feel this a great compromise to have a speedy
development experience while still having a product that doesn't contribute to cognitive
atrophy and feels better to use.

# Headline result: fill in the middle

We wanted to apply the FELA methodology as an LLM. Given a tight training budget while we
study how to scale this new architecture, we thought a smaller model would be a great
autocomplete tool to demonstrate the power of efficient computing.

We measure functional pass@1 with the HumanEval-SingleLineInfilling methodology, so
a completion counts only when it makes the code run correctly, not when it matches the
original character for character.

| Stage | FIM Pass@1 | FIM EM | FIM Edit Sim | MBPP | HumanEval |
|---|---|---|---|---|---|
| Base (15B token distill) | 49.6% | 42.0 | 73.6 | 13.0 | 11.0 |
| Plus fill in the middle anneal | 54.4% | 47.4 | 77.6 | 16.0 | 9.8 |
| Plus fill in the middle SFT (final) | **54.6%** | 46.8 | **78.4** | **18.0** | 11.0 |

Over half of single line completions functionally work. We lead with FIM pass@1 because that
is the product metric. The Rust server auto formats the output in microseconds, so we optimize
for semantic correctness (pass@1 and edit similarity), not whitespace exact match.

MBPP and HumanEval are shown for context only. They ask the model to write a whole function
from a description, which is the wrong lens for an autocomplete model and low by design at this
size. We provide it to compare with other language models at size ~1.5B. Do not read the HumanEval number as the quality of the product; read the FIM pass@1.

# How it got here

1. Base: clean room distill from Qwen2.5-Coder 1.5B to 15 billion tokens, on a permissive code
   mix with fill in the middle in the objective.
2. Fill in the middle anneal (the big lever): a further billion tokens with a line aware fill
   in the middle objective (mask whole lines, matching how the eval scores lines) across
   Python, JavaScript, TypeScript, Java, Go, Rust, and C plus plus. This train and eval
   alignment gave the largest jump.
3. Fill in the middle SFT: a further 1.2 billion tokens with the native Qwen fill in the middle
   tokens and the loss on the middle span. This consolidated the gains and lifted general code
   (MBPP 16 to 18).

# What goes in, what comes out

- Input: your code as text, split into a prefix (before the cursor) and an optional suffix
  (after the cursor). The text is turned into tokens with the Qwen2.5-Coder tokenizer, which
  carries the fill in the middle markers `<|fim_prefix|>`, `<|fim_suffix|>`, and
  `<|fim_middle|>`.
- Output: the code that belongs between the prefix and the suffix. By default it writes one
  line and stops, which is what an inline autocomplete wants. With no suffix it simply
  continues the prefix.
- In plain terms: you type `import numpy as ` and it finishes the line with ` np`; you give it
  the body of `def add(a, b):` with the call `add(2, 3)` waiting below it, and it fills in
  ` return a + b`.

# Why we built it this way

The model has no growing cache of past tokens, which is the usual reason memory creeps up as
context gets longer. Its sequence mixers are Fourier Neural Operators (a filter the model
learns and applies in the frequency domain) for global context, Gated-DeltaNet recall layers
that overwrite an old memory when a new one arrives (so keys do not interfere), and a Landmark
routing layer that lets a token look back at summaries of earlier chunks. None of these keep a
per token attention cache, so the working memory stays small and fixed no matter how much code
is in the window. That is what lets it decode on a low power CPU next to you, with the source
never leaving the machine.

The full stack is 28 layers at width 1536, in the repeating pattern of three Fourier mixers
then one Gated-DeltaNet recall layer, with a Landmark routing layer every seventh layer.

# Performance

The model runs on CPU with no GPU. Two paths run these weights:

| Path | Format | Size on disk | Single core decode |
|---|---|---|---|
| Reference (this repo, `modeling.py`) | bf16 | 3.58 GB | about 4 tokens per second |
| Production (FELA server, separate Rust runtime) | int8 | about 2.08 GB | about 22 tokens per second |

The `modeling.py` loader here is the plain, readable reference path in pure PyTorch. It loads
the weights, runs a real forward, and decodes on CPU, which is enough for the quickstart and for
verifying the model. It is a correctness path, not the fast path, so do not read its speed as the
product speed. Loading `model_int8.safetensors` in Python is smaller in memory but not faster,
because the pure torch int8 path dequantizes the weights on the fly.

The fast path is the CPU native FELA server, a separate Rust runtime that runs these weights as
int8 with a constant state streaming decode and no KV cache. On one core it decodes at about 22
tokens per second, roughly 20 times the plain f32 path, measured at this model's shape with a
steady state O(1) decode step. Single stream decode is bound by memory bandwidth, since it
streams the full int8 weight set once per token, so adding cores does not speed up one stream
(measured flat from one to eight cores); extra cores serve more streams at once and speed up the
prefill. The server's Rust int8 forward is checked against the PyTorch model with a golden test,
so the fast serving path produces the same model, not a lookalike.

# What to expect

Some real completions from the final model:

- `import numpy as ` gives ` np`
- prefix `def add(a, b):` with the call `add(2, 3)` in the suffix gives ` return a + b`
- prefix `def is_even(n):\n    return n % 2 == ` gives `0`

Where it earns its keep is single line fill in the middle on the patterns developers write all
day - and it can be made more useful with prefilled repository context.
It is a 1.5B class model, so it is not a whole function generator or a reasoning model:
multi line blocks, novel algorithms, and anything that needs deep reasoning about your specific
code are outside what it is built for. Read completions before you use them.

# How to run it

It runs on CPU; no GPU is required. See `quickstart/` for a runnable example. The short
version, using the self contained `modeling.py`, `model_cpu_gpt2.py`, and `config.json` in
this repo:

```python
from modeling import load_model
m = load_model("/path/to/this/repo")

# Fill in the middle: code before and after the cursor -> the middle
r = m.complete("def add(a, b):\n    ", suffix="\nresult = add(2, 3)\n")
print(r["middle"])          # e.g. " return a + b"

# Plain autocomplete: continue a single line
print(m.complete("import numpy as ")["middle"])   # e.g. " np"

# Load the smaller int8 weights instead (same completions, smaller on disk)
m8 = load_model("/path/to/this/repo", quant="int8")
```

It also loads with transformers directly. The code is custom, so pass `trust_remote_code=True`:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("lowdown-labs/fela-autocomplete", trust_remote_code=True, dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("lowdown-labs/fela-autocomplete", trust_remote_code=True)
prompt = "<|fim_prefix|>def add(a, b):\n    <|fim_suffix|>\nresult = add(2, 3)\n<|fim_middle|>"
enc = tok(prompt, return_tensors="pt")
out = model.generate(**enc, max_new_tokens=8, use_cache=False)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
```

The loader reads the bf16 `model.safetensors` by default. Pass `quant="int8"` to load the
smaller `model_int8.safetensors` instead; it dequantizes the linear weights with their per
channel scales and keeps the Fourier filters in bf16, giving the same completions. Pass
`quant="auto"` to prefer bf16 when present and fall back to int8.

`complete` is greedy by default, so a skeptic gets the same real output twice. For an
interactive playground, see the Hugging Face Space in `space/`. For production serving, use
the CPU native FELA server (`https://github.com/Lowdown-Labs/fela_server`).

## Loading, exports, and verification

- Weights ship as `model.safetensors` in bf16 (no pickle code execution risk). `config.json`
  holds the architecture hyperparameters. `tokenizer.json` is the Qwen2.5-Coder tokenizer with
  the fill in the middle tokens. The architecture is defined in the self contained
  `model_cpu_gpt2.py`, and the pure torch CPU paths for the Gated-DeltaNet and Landmark layers
  are in `cpu_delta.py`, `cpu_landmark.py`, `cpu_swa.py`, and `cpu_patch.py`, which ship beside
  it. `modeling.py` wires them into a loader with a `complete` fill in the middle entry point.
- `verify.py` builds the model, loads the weights, runs a fixed input forward, checks the
  output shape and the vocabulary size, confirms the logits are finite, and compares the top
  predicted tokens against a captured reference.

## Formats

- bf16: the reference weights and the default load path in this repo (`model.safetensors`,
  3.58 GB).
- int8: the smaller on device format (`model_int8.safetensors`, about 2.08 GB), weight only int8
  per channel on the large linear layers, with the Fourier filters kept in bf16. Load it in
  Python with `load_model(dir, quant="int8")`, or serve it with the FELA server. The int8
  forward matches the bf16 model by the golden test.

## Model details

- Parameters: 1,788,379,536 (about 1.79 billion).
- Layers: 28, width 1536, 12 heads, feed forward hidden 8960.
- Sequence mixers: Fourier Neural Operator (512 modes), Gated-DeltaNet recall, and Landmark
  global routing, in the SSSL pattern (three Fourier mixers then one recall layer), with a
  Landmark layer every seventh layer.
- Tokenizer: Qwen2.5-Coder (vocabulary 151936), which carries the fill in the middle tokens.
- Warm start: Qwen2.5-Coder 1.5B, then clean room distilled, annealed, and fill in the middle
  supervised fine tuned.

# Intended use, limitations, and safety

What it is for: inline code autocomplete and fill in the middle inside a local or on premises
developer tool, where running on CPU with no GPU and keeping the source on the machine matter.
It is a research preview for that use.

What it is not for: it is not a whole function generator, not a chat model, and not a reasoning
model. Do not rely on its completions for correctness or security without reading them, and do
not ship it into a product without your own evaluation. It is a 1.5B class model and will be
wrong on anything past common single line patterns.

Evaluated conditions and known failure modes:

- Strong on single line, high frequency fill in the middle; weaker on multi line blocks and
  novel logic.
- The reference `modeling.py` path is bf16 and slow; the int8 FELA server is the fast path.
- No safety or license filtering of generated code is claimed. Review generated code before you
  use it.

Privacy: the model runs on the device or on your own CPU, so your source does not have to leave
the machine and there is no per call API. That is the on device privacy angle.

# Model family

This is part of the FELA family from Lowdown Labs: one Fourier Neural Operator architecture
across many modalities, all CPU native and subquadratic. This repo is pushed as
`lowdown-labs/FELA-autocomplete`. Sibling repos are trained independently per modality and
share no weights, so none carries a `base_model` link.

# Acknowledgements and references

- Fourier Neural Operator: Li, Z., et al. (2021). Fourier Neural Operator for Parametric
  Partial Differential Equations. ICLR. https://arxiv.org/abs/2010.08895
- Gated Linear Attention: Yang, S., et al. (2023). Gated Linear Attention Transformers with
  Hardware-Efficient Training. https://arxiv.org/abs/2312.06635
- Gated DeltaNet: Yang, S., et al. (2024). Gated Delta Networks: Improving Mamba2 with Delta
  Rule. https://arxiv.org/abs/2412.06464
- Landmark Attention: Mohtashami, A., Jaggi, M. (2023). Landmark Attention: Random-Access
  Infinite Context Length for Transformers. https://arxiv.org/abs/2305.16300
- Qwen2.5-Coder: Hui, B., et al. (2024). Qwen2.5-Coder Technical Report.
  https://arxiv.org/abs/2409.12186
- PyTorch: Paszke, A., et al. (2019). NeurIPS. https://arxiv.org/abs/1912.01703
- flash-linear-attention: the reference kernels for the Gated-DeltaNet layer.
  https://github.com/fla-org/flash-linear-attention

# How to cite

```bibtex
@misc{lowdownlabs_fela_llm,
  title  = {FELA LLM 1.5: An on device Fourier Neural Operator code autocomplete model},
  author = {Lowdown Labs},
  year   = {2026},
  note   = {Model card}
}
```

# License

Released under the Lowdown Labs Lovely License 1.0 (CC BY-NC 4.0 plus Hippocratic License 3.0). See LICENSE. For most LL models, a commercial license may be available; contact Lowdown Labs.