FLCMind

A small Llama-style language model family implemented from scratch in plain PyTorch — no pretrained weights, no peft, no trl. Only the Hugging Face base classes (PreTrainedModel, PretrainedConfig) and tokenizer are used; the architecture itself is hand-written. It ships in two variants: a ~65M-parameter dense model and a ~200M-parameter MoE model (4 experts, Top-1 routing, ~65M active parameters per token).

FLCMind is an end-to-end LLM project built for learning: custom BPE tokenizer → pretraining → full-parameter SFT → OpenAI-compatible serving. This repository hosts both SFT checkpoints together with the model code and tokenizer, so the weights can be used without cloning anything else.

Training scripts, data pipeline and serving code: github.com/FLCgeigei/flcmind

FLCMind architecture

Highlights

  • Llama-style architecture, written from scratch: RMSNorm, RoPE, GQA, SwiGLU, plus an optional Mixture-of-Experts FFN.
  • Two variants: dense (65M params) and MoE (200M total, ~65M active per token, 4 experts Top-1).
  • Tiny and cheap to run: inference works on CPU; training fits in ~6 GB of GPU memory.
  • OpenAI-compatible API out of the box (/v1/chat/completions, streaming supported).
  • Custom tokenizer (BPE, vocab size 6400) trained together with the model.

Which checkpoint should I use?

Checkpoint Params Size Use it for
full_sft_768.pth ~65M (dense) ~138 MB Chat / inference — recommended
moe_sft_768_moe.pth ~200M total, ~65M active ~407 MB Chat, if you want to try the MoE variant

Both are SFT weights and can hold multi-turn conversations. A pretraining-only checkpoint is not included: it only continues text and produces incoherent replies in chat.

Benchmarks

Measured on a single RTX 4090 in fp16, 5-shot on C-Eval / CMMLU with a completion-style prompt. Every model — including the baselines — was run through the same prompt format and the same letter-logprob scoring rule, so the comparison is internally consistent; the baseline numbers are not vendor-reported figures.

Model Params (total / active) C-Eval 5-shot CMMLU 5-shot PPL↓ Decode (tok/s) VRAM (GB)
FLCMind-Dense-SFT 63.9M 26.3 ± 2.3 24.7 ± 2.3 170.8 161 0.2
FLCMind-MoE-SFT 198.4M / 63.9M 25.2 ± 2.3 23.8 ± 2.3 112.2 111 0.4
GPT2-Chinese-Distil-60M‡ 59.5M 23.0 ± 2.2 24.4 ± 2.3 — 398 0.2
Wenzhong-GPT2-110M‡ 124.5M 27.9 ± 2.4 24.6 ± 2.3 — 218 0.4
SmolLM2-135M§ 134.5M 22.7 ± 2.2 25.0 ± 2.3 — 68 0.3
SmolLM2-1.7B§ 1.71B 33.2 ± 2.5 32.8 ± 2.5 — 85 3.5

n = 1346 for C-Eval (full val split) and 1340 for CMMLU; ± is the half-width of the Wilson 95% confidence interval. PPL is computed per token on C-Eval prompts and is only comparable between models that share a tokenizer. Decode is single-request throughput (batch size 1, 32 generated tokens, fp16) — not a batched serving number. § English-dominant model. ‡ BertTokenizer-based: cannot encode A/B/C/D as single tokens, so the score is a floor — GPT2-Chinese predicts one fixed letter 100% of the time and its accuracy is just that letter's share of the gold answers.

Four-way random guessing is 25.0%. Every model in the 60–135M range lands in 22–28%, so FLCMind is on par with its size class rather than "knowing 26% of the exam" — at 64M parameters with a 6400-token vocabulary, knowledge-heavy multiple-choice questions are close to unanswerable.

The MoE variant earns its keep in language modelling rather than in answering: at the same active parameter count, PPL drops 34% (170.8 → 112.2) while C-Eval stays flat, and decode throughput falls from 161 to 111 tok/s.

Loading the weights

The checkpoints are plain PyTorch state_dicts, and loading is intentionally explicit: no config.json, no auto_map, no trust_remote_code. This is a teaching project — you can see exactly how the config and the weights line up.

from model_flcmind import FLCMindConfig, FLCMindForCausalLM

config = FLCMindConfig(hidden_size=768, num_hidden_layers=8, use_moe=False)
model = FLCMindForCausalLM(config)
model.load_state_dict(torch.load("full_sft_768.pth", map_location="cpu"), strict=True)

AutoModelForCausalLM.from_pretrained(...) is not supported: the weight files keep their original .pth names, which the Hugging Face auto-loader does not look for. Use the snippet above instead.

Quick start

The FLCMind scripts resolve paths relative to their own location, so the files must sit in the directories the project expects:

git clone https://github.com/FLCgeigei/flcmind
hf download FLCgeigeigei/flcmind --local-dir flcmind_hf

cp flcmind_hf/model_flcmind.py     flcmind/model/
cp flcmind_hf/tokenizer*.json      flcmind/model/
cp flcmind_hf/full_sft_768.pth     flcmind/out/
cp flcmind_hf/moe_sft_768_moe.pth  flcmind/out/

Chat from the command line

# Dense — from the project root
python eval_llm.py --weight full_sft

# MoE — --use_moe must match how the weights were trained
python eval_llm.py --use_moe 1 --weight moe_sft

--weight is a prefix, not a filename. The script appends _768.pth for dense and _768_moe.pth for MoE, so moe_sft resolves to moe_sft_768_moe.pth.

OpenAI-compatible server

cd scripts
python serve_openai_api.py --weight full_sft             # Dense
python serve_openai_api.py --use_moe 1 --weight moe_sft  # MoE

Then call it like any OpenAI endpoint:

from openai import OpenAI
client = OpenAI(api_key="sk-any", base_url="http://localhost:8998/v1")
resp = client.chat.completions.create(
    model="flcmind-local:latest",
    messages=[{"role": "user", "content": "Hello"}],
    temperature=0.7,
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

Use a positive temperature (e.g. 0.7). temperature=0 triggers a divide-by-zero in the sampler and can leave the CUDA context corrupted until the server is restarted.

Architecture

Item Dense MoE
hidden_size 768 768
num_hidden_layers 8 8
Attention GQA GQA
FFN SwiGLU 4 experts, Top-1
Total params ~65M ~200M
Active params / token ~65M ~65M
Vocab size 6400 6400

--hidden_size, --num_hidden_layers and --use_moe must match the weights at load time; otherwise you get shape-mismatch errors.

Training

Both checkpoints were trained on FLCgeigeigei/flcmind-datasets (~1.2 GB pretraining text, ~1.7 GB SFT conversations):

  1. Pretraining — next-token prediction on the pretraining split.
  2. SFT — full-parameter fine-tuning on multi-turn conversations, with the loss masked to assistant turns.

Limitations

  • Very small model. Expect limited world knowledge, weak reasoning, and frequent factual errors. Do not rely on it for factual answers. The benchmark table above is the quantitative version of this statement.
  • Short context. Training used max_length of 512 (pretraining) and 1024 (SFT) tokens; long conversations degrade quickly.
  • Chinese + English only, with a 6400-token vocabulary — other languages are effectively unsupported.
  • Not instruction-safe by default. No RLHF or safety tuning was applied; outputs should be treated as raw model behavior.
  • For learning and experimentation, not production use.

License

The weights and code are released for learning and research purposes. The training data comes from public sources; please also comply with the licenses of those original datasets.

Downloads last month
22
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for FLCgeigeigei/flcmind

Finetunes
1 model

Collection including FLCgeigeigei/flcmind