README.md

---
language:
- zh
- en
- code
license: apache-2.0
library_name: cpp
tags:
- neuroflow
- causal-lm
- sn
- ecn
- dmn
- memory-augmented
- transformer
- cpp
- llm
pipeline_tag: text-generation
---

# NeuroFlow C++ LLM

NeuroFlow is a **memory-augmented causal language model** implemented entirely in **C++17**, featuring a three-brain architecture (SN/ECN/DMN) inspired by cognitive neuroscience. Designed for efficient training and inference on consumer GPUs.

## Model Architecture

| Parameter | Value | Description |
|-----------|-------|-------------|
| `d_model` | 512 | Model dimension |
| `hidden_dim` | 2048 | FFN hidden dimension |
| `memory_dim` | 512 | Memory dimension |
| `num_layers` | 12 | ECN layers |
| `memory_slots` | 64 | Memory slots |
| `num_associations` | 8 | DMN association heads |
| `vocab_size` | 128,000 | 128K multilingual BPE tokenizer |
| `max_seq_len` | 512 | Maximum sequence length |
| `causal_window_size` | 64 | Causal attention window |
| `lm_num_attn_layers` | 2 | Causal LM attention layers |
| `params` | ~120M | Total parameters |

### Three-Brain Architecture
- **SN (Sensory Network)**: Input encoding and feature extraction
- **ECN (Executive Control Network)**: Core reasoning and processing layers
- **DMN (Default Mode Network)**: Memory-augmented association and retrieval

## Files

| File | Size | Description |
|------|------|-------------|
| `output/checkpoint_step1000/model.nfv1` | 431 MB | Checkpoint at 1000 steps |
| `output/checkpoint_step2000/model.nfv1` | 431 MB | Checkpoint at 2000 steps |
| `output/lm_head_lmh1.nfv1` | 255 MB | LM head weights (native format v1) |
| `configs/config.json` | — | Model architecture configuration |
| `configs/tokenizer_128k.json` | — | 128K BPE tokenizer |
| `configs/huggingface/` | — | HuggingFace-compatible tokenizer files (vocab.json, merges.txt) |

### Source Code
The full C++ source is included under `src/` and `include/` directories:
- **Core**: `tensor.hpp/cpp`, `model.hpp/cpp`, `tokenizer.hpp/cpp`
- **Architecture**: `causal_lm.hpp/cpp`, `generative_model.hpp/cpp`, `networks.hpp`
- **Training**: `train_lm.hpp/cpp`, `train_v2.cpp`, `sft_train.cpp`, `dpo_train.cpp`
- **CUDA**: `cuda_context.hpp/cpp`, `cuda_kernels.hpp`, `tensor_ops.cpp`
- **Optimizers**: `adamw.hpp/cpp`, `scheduler.hpp/cpp`, `grad_scaler.hpp/cpp`

## Build & Train

### Prerequisites
- CMake ≥ 3.15
- C++17 compiler (GCC ≥ 9, MSVC 2019+)
- CUDA Toolkit ≥ 11.4 (optional, for GPU training)
- BLAS (OpenBLAS recommended)

### Build
```bash
# CPU only
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

# With CUDA
mkdir build_cuda && cd build_cuda
cmake .. -DNEUROFLOW_USE_CUDA=ON -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
```

### Train
```bash
./build_cuda/neuroflow_train_v2 \
  --config configs/config_distill.json \
  --data data/distill_train.txt \
  --output output \
  --epochs 20 \
  --batch-size 64 \
  --lr 0.0003 \
  --use-cuda --adam
```

## Training Scripts

Key Python scripts in `scripts/`:
- `train_distill.py` — Knowledge distillation training pipeline
- `preprocess_distill.py` — Data preprocessing for distillation
- `deploy_dsw.sh` — One-click deployment for Alibaba Cloud DSW (A10 GPU)
- `train_optimized.sh` — Optimized multi-stage training

## License

Apache 2.0

## Links

- [GitHub Repository](https://github.com/chenzhiwenhphp12-afk/neuroflow-model)
- [HuggingFace Mirror](https://hf-mirror.com/cwenzi/neuroflow-cpp)