README.md

<div id="ax-org-profile" style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; width: 100%; margin: 30px 0;">
<style>
#ax-org-profile * { box-sizing: border-box; }
.ix-outer { position: relative; overflow: hidden; background: linear-gradient(135deg, #f97316 0%, #22c55e 100%); border-radius: 16px; padding: 35px; box-shadow: 0 10px 30px rgba(0,0,0,0.08); text-align: left; }
@keyframes diagonalShimmer { 0% { transform: translateX(-150%) skewX(-15deg); } 50% { transform: translateX(150%) skewX(-15deg); } 100% { transform: translateX(150%) skewX(-15deg); } }
.ix-outer::before { content: ""; position: absolute; top: 0; left: 0; width: 100%; height: 100%; background: linear-gradient(90deg, rgba(255, 255, 255, 0) 0%, rgba(255, 255, 255, 0.2) 40%, rgba(255, 255, 255, 0.4) 50%, rgba(255, 255, 255, 0.2) 60%, rgba(255, 255, 255, 0) 100%); animation: diagonalShimmer 5s infinite ease-in-out; pointer-events: none; z-index: 1; }
.ix-label { position: relative; z-index: 2; font-size: 13px; font-weight: 800; color: rgba(255, 255, 255, 0.99); text-transform: uppercase; letter-spacing: 1.5px; text-shadow: 1px 1px 2px rgba(0,0,0,0.1); margin-bottom: 6px; }
.ix-title { position: relative; z-index: 2; font-size: 36px; font-weight: 800; color: #ffffff; margin: 0 0 5px 0; text-shadow: 1px 3px 5px rgba(0,0,0,0.5); border: none; padding: 0; background: none; }
.ix-subtitle { position: relative; z-index: 2; font-size: 15px; font-weight: 700; color: #1e293b; margin: 0 0 25px 0; }
.ix-inner { position: relative; z-index: 2; background: #0f172a; border-radius: 12px; padding: 25px; color: #f8fafc; box-shadow: 0 8px 15px rgba(0,0,0,0.55); max-height: 320px; overflow-y: auto; }
.ix-inner::-webkit-scrollbar { width: 8px; }
.ix-inner::-webkit-scrollbar-track { background: #1e293b; border-radius: 4px; }
.ix-inner::-webkit-scrollbar-thumb { background: #475569; border-radius: 4px; }
.ix-inner::-webkit-scrollbar-thumb:hover { background: #64748b; }
.ix-inner h3 { font-size: 18px; font-weight: 800; color: #93c5fd; margin: 25px 0 10px 0; border-bottom: 1px solid #334155; padding-bottom: 5px; text-transform: uppercase; letter-spacing: 0.5px;}
.ix-inner h3:first-child { margin-top: 0; }
.ix-inner p { font-size: 14px; line-height: 1.6; margin: 0 0 12px 0; color: #cbd5e1; }
.ix-inner ul { margin: 0 0 16px 0; padding-left: 20px; color: #cbd5e1; font-size: 14px; line-height: 1.6; list-style-type: square; }
.ix-inner li { margin-bottom: 6px; }
.ix-highlight { color: #f472b6; font-weight: 700; }
.ix-link { color: #bfdbfe; text-decoration: none; font-weight: 600; }
.ix-link:hover { color: #ffffff; text-decoration: underline; }
.ix-author-box { background: #1e293b; padding: 18px; border-radius: 8px; margin-bottom: 20px; border: 1px solid #334155; border-left: 4px solid #f472b6; }
.ix-author-box p { margin: 0 0 8px 0; }
.ix-author-box p:last-child { margin: 0; }
</style>
<div class="ix-outer">
<div class="ix-label">Organization Profile</div>
<h1 class="ix-title">AutonomousX</h1>
<p class="ix-subtitle">Open Source Research for Building Large Language Models from Scratch and Finetuning on TPUs & GPUs</p>
<div class="ix-inner">
<div class="ix-author-box">
<p><strong style="color: #ffffff; font-size: 16px;">Rohit Yadav</strong> <span style="color: #94a3b8; font-size: 13px;">| B.Tech 3rd Year</span></p>
<p style="font-size: 13px; color: #94a3b8;">Dr. B.R. Ambedkar National Institute of Technology (NIT) Jalandhar, India</p>
<p>
<span style="margin-right: 10px;">📧 <a href="mailto:yrohit1825@gmail.com" class="ix-link">yrohit1825@gmail.com</a></span>
<span style="margin-right: 10px;">🔗 <a href="https://www.linkedin.com/in/rohit-yadav-25535b256/" class="ix-link" target="_blank">LinkedIn</a></span>
<span>💻 <a href="https://github.com/YADAV1825" class="ix-link" target="_blank">GitHub</a></span>
</p>
</div>
<h3>Mission</h3>
<p>
<span class="ix-highlight">AutonomousX</span> aims to make LLM training infrastructure accessible and reproducible for researchers, students, and developers. While modern language models are widely used, complete end-to-end guides for training LLMs from scratch on TPUs remain scarce, particularly for beginners working with <strong>JAX and distributed TPU training</strong>. AutonomousX focuses on filling this gap by publishing fully reproducible open-source pipelines that demonstrate how to train language models from scratch using limited compute resources.
</p>
<p><em>Compute supporting the development of this organization and its models was provided by <strong>Google's TRC Program (TPU Research Cloud)</strong>.</em></p>
<h3>Research Focus</h3>
<p>The organization explores multiple aspects of efficient LLM training on TPUs, including:</p>
<ul>
<li>Custom transformer architectures</li>
<li>Variants with and without <strong>RoPE (Rotary Positional Embeddings)</strong></li>
<li>Memory-efficient training techniques</li>
<li>Custom optimizer experiments</li>
<li>Training pipeline optimization using <strong>JAX + pmap</strong></li>
<li>Efficient dataset streaming and preprocessing</li>
</ul>
<p>The goal is to demonstrate how meaningful LLM research can be conducted even with compute-limited environments.</p>
<h3>Instinct Model Family</h3>
<p>AutonomousX develops the <strong>Instinct</strong> family of language models. These models are built entirely from scratch, including tokenizer, architecture, training pipeline, and TPU training infrastructure. Instinct models explore different configurations such as:</p>
<ul>
<li>Transformer architectures with and without RoPE</li>
<li>Custom training optimizers</li>
<li>TPU-optimized training pipelines using JAX + pmap</li>
<li>Memory-efficient training for limited hardware environments</li>
</ul>
<p>The models are designed to demonstrate how modern language models can be trained on small TPU pods such as TPU v4-8.</p>
<h3>Compute Strategy</h3>
<p>One of the core goals of AutonomousX is to explore efficient training on limited compute resources. Research focuses on training models:</p>
<ul>
<li>Up to <strong>~1.5B parameters</strong></li>
<li>On <strong>small TPU v4-8 pods</strong></li>
<li>Across <strong>hundreds of billions of tokens</strong></li>
</ul>
<p>By optimizing training pipelines and architecture design, AutonomousX investigates how far efficient training can scale without access to massive GPU clusters.</p>
<h3>Open Source Philosophy</h3>
<p>AutonomousX publishes complete reproducible implementations including Dataset pipelines, Tokenizer training, Model architectures, TPU training scripts, Checkpointing systems, and Inference pipelines. All repositories aim to provide transparent and educational implementations so the open-source community can learn how large language models are trained from the ground up.</p>
<h3>Why This Matters</h3>
<p>Many tutorials focus only on using pretrained models, but very few resources explain:</p>
<ul>
<li>How to train LLMs from scratch</li>
<li>How to run training pipelines on TPUs</li>
<li>How distributed JAX training works</li>
<li>How datasets like <strong>The PILE</strong> are processed at scale</li>
</ul>
<p>AutonomousX aims to make these processes accessible, understandable, and reproducible.</p>
</div>
</div>
</div>