File size: 2,189 Bytes
31dc8dc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | .. _preparation:
===========
Preparation
===========
Last updated: 2025-11-08
With the environment and dependencies installed, the final step is to prepare the necessary assets for training. This guide covers the two main prerequisites: downloading and converting the model weights, and formatting the training dataset.
Download and Merge Model Weights
================================
Our training scripts require model weights in a “merged-expert” format
for optimal performance. Before starting, you must download the standard
weights and convert them.
Step 1: Download Original Model
-------------------------------
We provide a helper script to download the weights from Hugging Face:
.. code-block:: bash
# Choose a destination for the original model files
python ./scripts/download_hf_model.py \
--repo_id inclusionAI/LLaDA2.0-mini-preview \
--local_dir /path/to/separate_expert_model
Step 2: Convert to Merged Format
--------------------------------
Run the following script to create the merged checkpoint required for training:
.. code-block:: bash
# Use the path from the previous step as the source
python scripts/moe_convertor.py \
--input-path /path/to/separate_expert_model \
--output-path /path/to/save/merged_model \
--mode merge
The directory ``/path/to/save/merged_model`` is what you will use for
the training script. For more details, see `MoE Expert Merging and
Splitting Utilities <#moe-expert-merging-and-splitting-utilities>`__
Prepare Training Data
=====================
This tutorial uses the ``openai/gsm8k`` dataset and demonstrates how to convert it into the conversational format.
Provided Script
---------------
We provide an example script ``./scripts/build_gsm8k_dataset.py`` for this purpose. You can adapt this script or write your own to process other datasets.
The script converts the "question" and "answer" fields into a conversational messages field. The processed dataset is saved to the ``./gsm8k_datasets/`` directory, split into:
- ``train.jsonl`` - Training data
- ``test.jsonl`` - Evaluation data
Run the script:
.. code-block:: bash
python ./scripts/build_gsm8k_dataset.py
|