README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- logos
- causal-lm
- text-generation
- safetensors
- custom-code
- pytorch
- base-model
datasets:
- HuggingFaceFW/fineweb-edu
---

# Logos 1B Base

`Rorical/logos-1b-base` is a 1.1B-parameter base causal language model using the Logos architecture. It is released as sharded `safetensors` weights with Hugging Face `trust_remote_code` support.

This is a base pretrained checkpoint, not an instruction-tuned or chat-aligned model.

## Model Details

- **Architecture:** Logos causal language model
- **Parameters:** 1,107,983,696
- **Weights:** bf16, sharded `safetensors`
- **Context length:** 4096 tokens
- **Tokenizer:** `cl100k_base` via `tiktoken`
- **Training data:** `HuggingFaceFW/fineweb-edu`, `sample-100BT`
- **Training objective:** next-token prediction
- **License:** Apache-2.0

The released checkpoint uses a looped Logos topology with 2 entry layers, 6 recurrent body layers over 3 loops, and 2 exit layers. Attention schedules combine HCA, CSA, SWA, and KDA attention variants. The model also uses sparse MoE feed-forward layers with 2 shared experts, 32 sparse experts, and top-k routing.

## Installation

```bash
pip install -U torch transformers safetensors tiktoken einops torchao
```

Because this repository contains custom model and tokenizer code, load it with `trust_remote_code=True`. As usual, inspect remote code before enabling it in production environments.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Rorical/logos-1b-base"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device)
model.eval()

prompt = "In a recent study, researchers found that"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.8,
        top_k=50,
        do_sample=True,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```

### Pipeline

```python
import torch
from transformers import pipeline

device = 0 if torch.cuda.is_available() else -1
dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

generator = pipeline(
    "text-generation",
    model="Rorical/logos-1b-base",
    tokenizer="Rorical/logos-1b-base",
    trust_remote_code=True,
    dtype=dtype,
    device=device,
)

print(generator(
    "In a recent study, researchers found that",
    max_new_tokens=120,
    do_sample=True,
    temperature=0.8,
    top_k=50,
)[0]["generated_text"])
```

## Files

- `model-00001-of-00010.safetensors` ... `model-00010-of-00010.safetensors`: sharded bf16 model weights
- `model.safetensors.index.json`: safetensors shard index
- `config.json`: Hugging Face model configuration
- `generation_config.json`: default generation IDs and cache setting
- `configuration_logos.py`, `modeling_logos.py`, `tokenization_logos.py`, `models/`: custom code required by `trust_remote_code=True`

## Training Configuration

The training run was configured for a 20B-token pretraining budget on FineWeb-Edu with 4096-token sequences, bf16 precision, gradient checkpointing, Muon/AdamW optimization, WSD learning-rate scheduling, and streaming data loading.

Key architecture settings from the released config:

- `d_model`: 1024
- `num_heads`: 16
- `head_dim`: 64
- `d_ff`: 2730
- `num_entry_layers`: 2
- `num_body_layers`: 6
- `num_exit_layers`: 2
- `num_loops`: 3
- `num_shared_experts`: 2
- `num_sparse_experts`: 32
- `top_k`: 6
- `expert_d_ff`: 832
- `csa_compression`: 4
- `hca_compression`: 128
- `swa_window`: 256

## Intended Use

This checkpoint is intended for research, architecture exploration, continued pretraining, evaluation, and downstream fine-tuning experiments.

It is not intended to be used directly as a safety-aligned assistant. For assistant-style applications, fine-tune and evaluate the model with task-specific data, safety mitigations, and deployment monitoring.

## Limitations

- The model is a base LM and may produce toxic, biased, private, false, or otherwise unsafe text.
- The model is not instruction tuned and may not follow user requests reliably.
- Outputs are not fact-checked.
- The training data is web-derived and may contain undesirable or copyrighted material.
- The tokenizer is based on `cl100k_base`; behavior differs from byte-level BPE tokenizers used by many open models.
- Loading requires `trust_remote_code=True` because Logos is not a built-in Transformers architecture.

## License

The model weights and accompanying code are released under the Apache License 2.0.