File size: 4,199 Bytes
094778b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 | {
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 🎓 Lab 00: The Simplest Starting Point\n",
"\n",
"Welcome! Before we spin up massive cloud GPUs or dive into complex AI loops, let's start with the simplest, most fundamental step of training any AI model: **The Data and The Tokenizer**.\n",
"\n",
"We can run this entire notebook locally on your laptop's CPU. No cloud required!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 1: Where is our Data?\n",
"For our autonomous research agent, we are using the **TinyStories** dataset. These are short stories generated by GPT-4 designed to teach small models english grammar and reasoning without needing billions of parameters.\n",
"\n",
"Let's look at the data prep script we already ran."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import pandas as pd\n",
"\n",
"data_dir = os.path.expanduser(\"~/.cache/autoresearch/data\")\n",
"print(f\"Checking {data_dir}...\")\n",
"\n",
"# List the downloaded parquet files\n",
"files = sorted(os.listdir(data_dir))\n",
"for f in files[:5]:\n",
" print(f\"Found shard: {f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 2: Read a Tiny Story\n",
"Let's actually open one of these data shards and read what a \"Tiny Story\" looks like. This is exactly what the AI will read over and over again to learn how to speak."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Read the first shard using pandas (pyarrow required)\n",
"df = pd.read_parquet(os.path.join(data_dir, files[0]))\n",
"\n",
"# Print the very first story!\n",
"story = df.iloc[0]['text']\n",
"print(\"--- STORY ---\")\n",
"print(story)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 3: The Tokenizer (How the AI Reads)\n",
"AI doesn't read letters. It reads \"Tokens\" (groups of characters). Earlier, we trained a custom **Byte-Pair Encoding (BPE)** tokenizer specifically on this dataset.\n",
"\n",
"Let's load the tokenizer and see how it converts the story above into numbers."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"sys.path.append(\"autoresearch\")\n",
"from autoresearch.prepare import Tokenizer\n",
"\n",
"# Load our trained tokenizer\n",
"tokenizer = Tokenizer.from_directory()\n",
"print(f\"Our AI has a vocabulary of {tokenizer.get_vocab_size()} unique tokens.\")\n",
"\n",
"# Let's translate a sentence into tokens!\n",
"sentence = \"Once upon a time, there was a little dog.\"\n",
"tokens = tokenizer.encode(sentence)\n",
"\n",
"print(f\"\\nSentence: {sentence}\")\n",
"print(f\"Tokens: {tokens}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 4: The Mojo Compiler (Our Secret Weapon)\n",
"Instead of writing the neural network in Python (which is slow), we pivoted to writing it in **Mojo**.\n",
"Let's execute the Mojo code we just wrote right here in the notebook using the shell!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"cd autoresearch/mojo && ~/.pixi/bin/pixi run mojo train.mojo"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🎉 Congratulations!\n",
"You just:\n",
"1. Loaded raw AI training data.\n",
"2. Tokenized text like a neural network does.\n",
"3. Compiled a blazing-fast Mojo file.\n",
"\n",
"You are now officially building an AI lab from the ground up."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
|