Continuum-0.1B / README.md
11-47's picture
Upload README.md with huggingface_hub
35773de verified
|
Raw
History Blame Contribute Delete
2.09 kB
---
license: apache-2.0
language:
- en
tags:
- continuum
- hybrid-mind
- self-evolving
- causal-lm
- meta-learning
- continual-learning
- emotional-intelligence
- multimodal
- memory
- reinforcement-learning
model-index:
- name: Continuum-0.1B
results: []
pipeline_tag: text-generation
library_name: transformers
---
# Continuum-0.1B — The Hybrid Mind
**Continuum-0.1B** is a ~111M-parameter self-evolving Small Language Model built from
scratch by **11-47 / WithInUsAI**. It unifies 25 autonomous Hybrid-Mind subsystems
into a single decoder-only transformer forward pass.
## Architecture
| Component | Spec |
|---|---|
| Type | Decoder-only transformer |
| Parameters | ~111M (tied embeddings) |
| Context | 64 096 tokens (64K + seed) |
| Layers | 12 |
| Hidden size | 768 |
| FFN | SwiGLU, intermediate=2048 |
| Attention | GQA (12Q / 4KV heads) |
| Position | NTK-aware RoPE θ=500 000 |
| Norm | Pre-norm RMSNorm |
| Dtype | BFloat16 |
## Training
- **Hardware**: 2x T4 GPUs (Kaggle) via DataParallel
- **Steps**: 3000 (effective batch 16 × 2048 tokens)
- **Resume**: Auto-resumes from HF Hub checkpoint if kernel restarts
## Datasets
| Dataset | Source |
|---------|--------|
| Claude Opus Mythos 5K | `WithinUsAI/claude_opus_mythos_5k` |
| Claude Opus 4.8 Distill | `WithinUsAI/claude_opus_4.8_distill` |
| Claude Mythos Distill | `WithinUsAI/claude_mythos_distill` |
| Opus 4.7 Thinking Max Distill (25K) | `WithinUsAI/Opus4.7_thinking_max_distill_god_seed_25k` |
| Claude Opus 4.7 Distilled | `WithinUsAI/claude_Opus_4.7_Distilled` |
| Mythos Preview 5K v2 | `11-47/cluade_mythos_preview_5k_v2` |
| Claude Opus 4.8 Max Thinking 5K v2 | `11-47/claude_opus_4.8_max_thinking_5k_v2` |
## Quick Start
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("11-47/Continuum-0.1B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
"11-47/Continuum-0.1B",
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
```
## License
Apache 2.0