Rorical/logos-1b-base

Logos 1B Base

Rorical/logos-1b-base is a 1.1B-parameter base causal language model using the Logos architecture. It is released as sharded safetensors weights with Hugging Face trust_remote_code support.

This is a base pretrained checkpoint, not an instruction-tuned or chat-aligned model.

Model Details

• Architecture: Logos causal language model
• Parameters: 1,107,983,696
• Weights: bf16, sharded safetensors
• Context length: 4096 tokens
• Tokenizer: cl100k_base via tiktoken
• Training data: HuggingFaceFW/fineweb-edu, sample-100BT
• Training objective: next-token prediction
• License: Apache-2.0

The released checkpoint uses a looped Logos topology with 2 entry layers, 6 recurrent body layers over 3 loops, and 2 exit layers. Attention schedules combine HCA, CSA, SWA, and KDA attention variants. The model also uses sparse MoE feed-forward layers with 2 shared experts, 32 sparse experts, and top-k routing.

Installation

pip install -U torch transformers safetensors tiktoken einops torchao

Because this repository contains custom model and tokenizer code, load it with trust_remote_code=True. As usual, inspect remote code before enabling it in production environments.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Rorical/logos-1b-base"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device)
model.eval()

prompt = "In a recent study, researchers found that"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.8,
        top_k=50,
        do_sample=True,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Pipeline

import torch
from transformers import pipeline

device = 0 if torch.cuda.is_available() else -1
dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

generator = pipeline(
    "text-generation",
    model="Rorical/logos-1b-base",
    tokenizer="Rorical/logos-1b-base",
    trust_remote_code=True,
    dtype=dtype,
    device=device,
)

print(generator(
    "In a recent study, researchers found that",
    max_new_tokens=120,
    do_sample=True,
    temperature=0.8,
    top_k=50,
)[0]["generated_text"])

Files

• model-00001-of-00010.safetensors ... model-00010-of-00010.safetensors: sharded bf16 model weights
• model.safetensors.index.json: safetensors shard index
• config.json: Hugging Face model configuration
• generation_config.json: default generation IDs and cache setting
• configuration_logos.py, modeling_logos.py, tokenization_logos.py, models/: custom code required by trust_remote_code=True

Training Configuration

The training run was configured for a 20B-token pretraining budget on FineWeb-Edu with 4096-token sequences, bf16 precision, gradient checkpointing, Muon/AdamW optimization, WSD learning-rate scheduling, and streaming data loading.

Key architecture settings from the released config:

• d_model: 1024
• num_heads: 16
• head_dim: 64
• d_ff: 2730
• num_entry_layers: 2
• num_body_layers: 6
• num_exit_layers: 2
• num_loops: 3
• num_shared_experts: 2
• num_sparse_experts: 32
• top_k: 6
• expert_d_ff: 832
• csa_compression: 4
• hca_compression: 128
• swa_window: 256

Intended Use

This checkpoint is intended for research, architecture exploration, continued pretraining, evaluation, and downstream fine-tuning experiments.

It is not intended to be used directly as a safety-aligned assistant. For assistant-style applications, fine-tune and evaluate the model with task-specific data, safety mitigations, and deployment monitoring.

Limitations

• The model is a base LM and may produce toxic, biased, private, false, or otherwise unsafe text.
• The model is not instruction tuned and may not follow user requests reliably.
• Outputs are not fact-checked.
• The training data is web-derived and may contain undesirable or copyrighted material.
• The tokenizer is based on cl100k_base; behavior differs from byte-level BPE tokenizers used by many open models.
• Loading requires trust_remote_code=True because Logos is not a built-in Transformers architecture.

License

The model weights and accompanying code are released under the Apache License 2.0.