File size: 3,471 Bytes
26d5b81 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | ---
language:
- zh
- en
- code
license: apache-2.0
library_name: cpp
tags:
- neuroflow
- causal-lm
- sn
- ecn
- dmn
- memory-augmented
- transformer
- cpp
- llm
pipeline_tag: text-generation
---
# NeuroFlow C++ LLM
NeuroFlow is a **memory-augmented causal language model** implemented entirely in **C++17**, featuring a three-brain architecture (SN/ECN/DMN) inspired by cognitive neuroscience. Designed for efficient training and inference on consumer GPUs.
## Model Architecture
| Parameter | Value | Description |
|-----------|-------|-------------|
| `d_model` | 512 | Model dimension |
| `hidden_dim` | 2048 | FFN hidden dimension |
| `memory_dim` | 512 | Memory dimension |
| `num_layers` | 12 | ECN layers |
| `memory_slots` | 64 | Memory slots |
| `num_associations` | 8 | DMN association heads |
| `vocab_size` | 128,000 | 128K multilingual BPE tokenizer |
| `max_seq_len` | 512 | Maximum sequence length |
| `causal_window_size` | 64 | Causal attention window |
| `lm_num_attn_layers` | 2 | Causal LM attention layers |
| `params` | ~120M | Total parameters |
### Three-Brain Architecture
- **SN (Sensory Network)**: Input encoding and feature extraction
- **ECN (Executive Control Network)**: Core reasoning and processing layers
- **DMN (Default Mode Network)**: Memory-augmented association and retrieval
## Files
| File | Size | Description |
|------|------|-------------|
| `output/checkpoint_step1000/model.nfv1` | 431 MB | Checkpoint at 1000 steps |
| `output/checkpoint_step2000/model.nfv1` | 431 MB | Checkpoint at 2000 steps |
| `output/lm_head_lmh1.nfv1` | 255 MB | LM head weights (native format v1) |
| `configs/config.json` | — | Model architecture configuration |
| `configs/tokenizer_128k.json` | — | 128K BPE tokenizer |
| `configs/huggingface/` | — | HuggingFace-compatible tokenizer files (vocab.json, merges.txt) |
### Source Code
The full C++ source is included under `src/` and `include/` directories:
- **Core**: `tensor.hpp/cpp`, `model.hpp/cpp`, `tokenizer.hpp/cpp`
- **Architecture**: `causal_lm.hpp/cpp`, `generative_model.hpp/cpp`, `networks.hpp`
- **Training**: `train_lm.hpp/cpp`, `train_v2.cpp`, `sft_train.cpp`, `dpo_train.cpp`
- **CUDA**: `cuda_context.hpp/cpp`, `cuda_kernels.hpp`, `tensor_ops.cpp`
- **Optimizers**: `adamw.hpp/cpp`, `scheduler.hpp/cpp`, `grad_scaler.hpp/cpp`
## Build & Train
### Prerequisites
- CMake ≥ 3.15
- C++17 compiler (GCC ≥ 9, MSVC 2019+)
- CUDA Toolkit ≥ 11.4 (optional, for GPU training)
- BLAS (OpenBLAS recommended)
### Build
```bash
# CPU only
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
# With CUDA
mkdir build_cuda && cd build_cuda
cmake .. -DNEUROFLOW_USE_CUDA=ON -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
```
### Train
```bash
./build_cuda/neuroflow_train_v2 \
--config configs/config_distill.json \
--data data/distill_train.txt \
--output output \
--epochs 20 \
--batch-size 64 \
--lr 0.0003 \
--use-cuda --adam
```
## Training Scripts
Key Python scripts in `scripts/`:
- `train_distill.py` — Knowledge distillation training pipeline
- `preprocess_distill.py` — Data preprocessing for distillation
- `deploy_dsw.sh` — One-click deployment for Alibaba Cloud DSW (A10 GPU)
- `train_optimized.sh` — Optimized multi-stage training
## License
Apache 2.0
## Links
- [GitHub Repository](https://github.com/chenzhiwenhphp12-afk/neuroflow-model)
- [HuggingFace Mirror](https://hf-mirror.com/cwenzi/neuroflow-cpp)
|