config.json

{
  "model_type": "looped_transformer",
  "seq_len": 256,
  "vocab_size": 16384,
  "d_model": 736,
  "n_heads": 8,
  "d_ff": 1472,
  "dropout": 0.05,
  "mtp_depth": 1,
  "mtp_lambda": 0.3,
  "min_loops": 2,
  "max_loops": 8,
  "eval_loops": 6,
  "tokenizer_dir": "tokenizer",
  "special_tokens": {
    "pad_id": 0,
    "bos_id": 1,
    "eos_id": 2
  },
  "total_params": 24176128,
  "tied_weights": [
    {
      "kept": "tok_emb.weight",
      "dropped": "head.weight"
    }
  ],
  "notes": "Custom looped-transformer architecture (shared CoreBlock iterated min_loops-max_loops times) with MTP auxiliary head, trained with the Muon optimizer. NOT a GGUF/llama.cpp-compatible architecture -- load with this repo's LoopedTransformer class + safetensors.torch.load_file(), not with llama.cpp or similar. IMPORTANT: 'tied_weights' above lists parameter pairs that share the same underlying tensor in the original model (weight tying) and were de-duplicated for safetensors export (which refuses to save aliased tensors). After loading, manually re-tie them, e.g. for the default case: model.head.weight = model.tok_emb.weight -- do this BEFORE calling load_state_dict with strict=True, or load with strict=False and assign the tied tensor afterward."
}