exnivo/Echo88-150M-Base

Echo88-150M Base

A 163M parameter retro English base language model trained from scratch.

Echo88-150M Base is a small decoder-only causal language model trained from scratch on the Echo88 pretraining dataset.

Echo88 is designed as a retro language model inspired by the language, culture, computing, magazines, Usenet discussions, and older book text available up to the late 1980s.

This is a base model, not an instruction-tuned chatbot. It is trained for next-token prediction and is best used for text completion, continued pretraining, or supervised fine-tuning.

For chat/instruction behavior, use:

exnivo/Echo88-150M-Instruct

Quick Start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "exnivo/Echo88-150M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "The personal computer revolution of the 1980s"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=140,
        do_sample=True,
        temperature=0.75,
        top_p=0.9,
        repetition_penalty=1.08,
        no_repeat_ngram_size=4,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

At a Glance

Item	Details
Model type	Base causal language model
Parameters	163,606,272
Approx. size	163.6M
Architecture	LLaMA-style decoder-only transformer
Language	English
Context length	2048 tokens
Vocabulary size	32,768
Layers	18
Hidden size	768
Intermediate size	2048
Attention heads	12
Key/value heads	4
Tokenizer	Custom Echo88 byte-level BPE tokenizer
Training type	From scratch
Theme	Late-1980s / retro computing / historical text
Training objective	Autoregressive next-token prediction
Instruction-tuned version	exnivo/Echo88-150M-Instruct

Model Details

Item	Value
Model name	Echo88-150M-Base
Model type	Decoder-only causal language model
Architecture	llama / LlamaForCausalLM
Training type	From scratch
Parameters	163,606,272
Language	English
Context length	2048 tokens
Vocabulary size	32,768
Hidden size	768
Intermediate size	2048
Layers	18
Attention heads	12
Key/value heads	4
Activation	SiLU
RMS norm epsilon	1e-05
Tied embeddings	false
BOS token	`<
EOS token	`<
PAD token	`<
UNK token	`<
Training objective	Autoregressive next-token prediction

Tokenizer

Echo88 uses a custom tokenizer with a vocabulary size of 32,768.

Registered special tokens:

<|endoftext|>
<|unk|>
<|pad|>
<|system|>
<|user|>
<|assistant|>
<|end|>

The base model is not trained to behave as a chat assistant. The chat tokens are mainly useful for compatibility with the Echo88 instruction-tuned model.

Base Model Usage

Echo88-150M Base should be prompted as a text completion model.

Good prompt style:

The personal computer revolution of the 1980s

A bulletin board system, often called a BBS,

BYTE Magazine, July 1987:

From: user@utzoo
Subject: learning C on a small machine

Avoid expecting direct assistant behavior like:

User: What is a modem?
Assistant:

For chat behavior, use exnivo/Echo88-150M-Instruct.

Recommended Generation Settings

For retro text completion:

output = model.generate(
    **inputs,
    max_new_tokens=140,
    do_sample=True,
    temperature=0.75,
    top_p=0.9,
    repetition_penalty=1.08,
    no_repeat_ngram_size=4,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

For deterministic testing:

output = model.generate(
    **inputs,
    max_new_tokens=140,
    do_sample=False,
    repetition_penalty=1.08,
    no_repeat_ngram_size=4,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

Because this is a small base model, long generations may repeat, drift, or turn into magazine/OCR-like fragments.

Example Outputs

These examples are from a local greedy text-completion test. They are raw base-model continuations, not instruction-following answers.

Prompt

A bulletin board system, often called a BBS,

Output

A bulletin board system, often called a BBS,
is available for $10.00 from:

     The Software Exchange
            P.O. Box 833
   [PHONE]

  I have been using the software package for about a year now and am
having trouble getting it to work.  I have had no problems with it
and I have not had any problems with it.  I would appreciate any
suggestions or suggestions.

Prompt

In 1986, a 2400 baud modem

Output

In 1986, a 2400 baud modem
is available.  The modem is a
modem that can be used with
the Hayes Smartmodem 1200
and the Hayes Smartmodem 1200.
The modem is also available for
the Hayes Micromodem II and
Apple II Plus.

Prompt

From: user@utzoo
Subject: learning C on a small machine

Output

From: user@utzoo
Subject: learning C on a small machine

I have been using the C compiler for about 3 years now, and I have been
using it for about 2 years now.  It is very easy to use, but I have
been using it for about 3 months now.

These examples show the intended retro text-completion style. They also show typical small-model repetition and drift.

Training Data

Echo88-150M Base was trained on the Echo88 pretraining dataset.

The packed training set contains:

Item	Value
Train tokens	1,470,629,888
Eval tokens	1,454,080
Train blocks	718,081
Eval blocks	710
Block size	2048 tokens
Packed dtype	uint16

The dataset includes a mixture of:

• public-domain book text
• Gutenberg-style older books
• UTZOO Usenet posts
• BYTE Magazine text
• PC Magazine text
• TIME Magazine text
• Internet Archive Magazine Rack OCR text
• computer and technology magazine text
• general historical magazine text

The dataset emphasizes the 1950s through the late 1980s, with a strong focus on early personal computing, printed magazines, Usenet, and older long-form writing.

Dataset used:

exnivo/Echo88-150M-Base

Training Configuration

Echo88-150M Base was trained as a LLaMA-style decoder-only causal language model.

Main configuration:

vocab_size: 32768
hidden_size: 768
intermediate_size: 2048
num_hidden_layers: 18
num_attention_heads: 12
num_key_value_heads: 4
max_position_embeddings: 2048
activation: SiLU
normalization: RMSNorm
position encoding: RoPE
attention: grouped-query attention

Training setup:

precision: bf16
sequence length: 2048
optimizer: AdamW
scheduler: cosine
weight decay: 0.1
gradient clipping: 1.0
max steps: 5610
training tokens: ~1.47B

Relationship to Echo88

Echo88 is a retro small-language-model project focused on historical text and late-1980s style language modeling.

Stage	Repository	Purpose
Base dataset	exnivo/Echo88-150M-Base	Retro pretraining data
Base model	exnivo/Echo88-150M-Base	Small causal LM trained from scratch
SFT dataset	exnivo/Echo88-Instruct-173K	Instruction/chat fine-tuning data
Instruct model	exnivo/Echo88-150M-Instruct	Retro assistant model fine-tuned from the base model

Pipeline:

Echo88 Pretraining Dataset
        ↓
Echo88-150M Base
        ↓
Echo88 Instruct 173K
        ↓
Echo88-150M Instruct

Intended Use

Echo88-150M Base is intended for:

• causal language modeling
• retro / historical AI experiments
• small language model research
• continued pretraining
• instruction tuning
• 1980s-style assistant experiments
• computer-history language modeling
• training Echo88-150M-Instruct
• studying time-bounded model behavior
• studying pre-modern-internet style corpora

Recommended flow:

Echo88-150M Base
→ supervised fine-tuning on Echo88-Instruct-173K
→ Echo88-150M Instruct

Not Instruction Tuned

This model is not instruction tuned.

It may not reliably follow commands, answer questions directly, or behave like a chat assistant.

Expected behavior:

• continues prompts
• completes paragraphs
• imitates old magazine/book/Usenet style
• may produce raw text instead of direct answers
• may hallucinate
• may repeat phrases
• may generate OCR-like artifacts
• may continue into fake addresses, phone numbers, article fragments, or listings

For chat behavior, use:

exnivo/Echo88-150M-Instruct

Knowledge Boundary

Echo88 is designed around a historical data mixture ending around the late 1980s.

The model should not be expected to know modern topics such as:

• Google
• Wikipedia
• iPhone
• smartphones
• modern social media
• Windows 95 and later software
• COVID-19
• modern AI systems
• 2000s, 2010s, or 2020s events

Because this is a base model, it does not reliably refuse or explain post-1988 topics. In testing, prompts such as Google is, The iPhone is, Windows 95 is, and YouTube is produced hallucinated continuations rather than careful boundary-aware answers.

The instruction-tuned model is better suited for 1988-boundary behavior, but it is still experimental.

Evaluation Notes

A local quick test confirmed:

Metric	Value
Parameters	163,606,272
Vocabulary size	32,768
Context length	2048
Layers	18
Hidden size	768
Attention heads	12
Key/value heads	4

The same test showed that the model has a clear retro completion style on prompts about BBS systems, modems, BYTE Magazine, and Usenet-style messages.

It also showed the model can hallucinate badly on modern or post-1988 prompts, which is expected for a base model.

Strengths

Echo88-150M Base is useful because it is:

• small and lightweight
• trained from scratch
• focused on retro/historical text
• strong at imitating older magazine and Usenet-style text
• useful for computer-history language modeling
• suitable as a base for instruction tuning
• interesting for time-bounded small-model experiments
• part of the Echo88 model family

Limitations

Echo88-150M Base is experimental and small.

Known limitations:

• not instruction tuned
• may hallucinate
• may repeat text
• may produce OCR-like artifacts
• may reflect outdated historical language or views
• may struggle with complex reasoning
• may not reliably refuse post-1988 topics
• may produce incomplete or strange continuations
• may mix unrelated historical/computer facts
• may generate fake addresses, phone numbers, product listings, or citations
• may continue modern prompts with unrelated old-book or magazine-like text

This model is intended for research, experimentation, and creative retro AI work. It is not intended for high-stakes use.

Bias and Historical Content

The training data includes historical books, magazines, and Usenet text. As a result, the model may reproduce outdated language, assumptions, stereotypes, or viewpoints present in older source material.

Users should review outputs carefully.

Suggested Evaluation

Recommended checks:

• retro computing completions
• magazine-style continuations
• Usenet-style continuations
• repetition behavior
• OCR artifact frequency
• modern-topic hallucination checks
• base vs instruct comparison
• continued pretraining experiments
• SFT experiments with Echo88-Instruct-173K

Example prompts:

The personal computer revolution of the 1980s

A bulletin board system, often called a BBS,

In 1986, a 2400 baud modem

BYTE Magazine, July 1987:

From: user@utzoo
Subject: learning C on a small machine

Google is

The iPhone is

Model Family

Echo88 model family:

Repository	Type
exnivo/Echo88-150M-Base	Base model
exnivo/Echo88-150M-Instruct	Instruction-tuned model
exnivo/Echo88-Instruct-173K	SFT dataset
exnivo/Echo88-150M-Base	Pretraining dataset

Citation

If you use this model, you can cite it as:

@misc{echo88_150m_base,
  title = {Echo88-150M Base},
  author = {exnivo},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/exnivo/Echo88-150M-Base}}
}

License

The model weights are released under the Apache 2.0 license.

The training dataset is mixed-source and released separately. Users are responsible for checking dataset source rights, licensing, and suitability for their own use case.

Disclaimer

Echo88-150M Base is an experimental small base language model. It may produce incorrect, biased, outdated, unsafe, nonsensical, or misleading outputs.

Do not use this model for high-stakes decisions or as a reliable source of factual information.