Continuum-0.1B / README.md
11-47's picture
Upload README.md with huggingface_hub
35773de verified
|
Raw
History Blame Contribute Delete
2.09 kB
metadata
license: apache-2.0
language:
  - en
tags:
  - continuum
  - hybrid-mind
  - self-evolving
  - causal-lm
  - meta-learning
  - continual-learning
  - emotional-intelligence
  - multimodal
  - memory
  - reinforcement-learning
model-index:
  - name: Continuum-0.1B
    results: []
pipeline_tag: text-generation
library_name: transformers

Continuum-0.1B — The Hybrid Mind

Continuum-0.1B is a ~111M-parameter self-evolving Small Language Model built from scratch by 11-47 / WithInUsAI. It unifies 25 autonomous Hybrid-Mind subsystems into a single decoder-only transformer forward pass.

Architecture

Component Spec
Type Decoder-only transformer
Parameters ~111M (tied embeddings)
Context 64 096 tokens (64K + seed)
Layers 12
Hidden size 768
FFN SwiGLU, intermediate=2048
Attention GQA (12Q / 4KV heads)
Position NTK-aware RoPE θ=500 000
Norm Pre-norm RMSNorm
Dtype BFloat16

Training

  • Hardware: 2x T4 GPUs (Kaggle) via DataParallel
  • Steps: 3000 (effective batch 16 × 2048 tokens)
  • Resume: Auto-resumes from HF Hub checkpoint if kernel restarts

Datasets

Dataset Source
Claude Opus Mythos 5K WithinUsAI/claude_opus_mythos_5k
Claude Opus 4.8 Distill WithinUsAI/claude_opus_4.8_distill
Claude Mythos Distill WithinUsAI/claude_mythos_distill
Opus 4.7 Thinking Max Distill (25K) WithinUsAI/Opus4.7_thinking_max_distill_god_seed_25k
Claude Opus 4.7 Distilled WithinUsAI/claude_Opus_4.7_Distilled
Mythos Preview 5K v2 11-47/cluade_mythos_preview_5k_v2
Claude Opus 4.8 Max Thinking 5K v2 11-47/claude_opus_4.8_max_thinking_5k_v2

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("11-47/Continuum-0.1B", trust_remote_code=True)
model     = AutoModelForCausalLM.from_pretrained(
    "11-47/Continuum-0.1B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

License

Apache 2.0