| ---
|
| language:
|
| - code
|
| license: apache-2.0
|
| library_name: transformers
|
| tags:
|
| - code
|
| - python
|
| - javascript
|
| - cpp
|
| - sql
|
| - html
|
| - code-generation
|
| - codebharat
|
| - llama
|
| - PyTorch
|
| - byte-level-bpe
|
| pipeline_tag: text-generation
|
| widget:
|
| - text: "def quicksort(arr):"
|
| example_title: "Python QuickSort"
|
| - text: "function debounce(func, wait) {"
|
| example_title: "JavaScript Debounce"
|
| - text: "int binarySearch(const std::vector<int>& arr, int target) {"
|
| example_title: "C++ Binary Search"
|
| ---
|
|
|
| # CodeBharat-100M
|
|
|
| **CodeBharat-100M** is a 100.68M parameter decoder-only Transformer pretrained from scratch on code (Python, JavaScript, TypeScript, C++, SQL, HTML/CSS, and synthetic textbooks).
|
|
|
| ## Model Details
|
| - **Architecture:** Decoder-Only Transformer (Llama/Qwen-style: RMSNorm, RoPE, SwiGLU, Grouped-Query Attention)
|
| - **Parameters:** 100,679,424 (100.68M)
|
| - **Vocabulary:** 49,152 tokens (Byte-level BPE)
|
| - **Context Window:** 1,024 tokens
|
| - **Training Device:** NVIDIA GeForce RTX 5050 GPU
|
| - **Final Validation Loss:** 1.3891
|
|
|
| ## Quickstart Usage
|
|
|
| ### Native PyTorch / Tokenizers Usage
|
|
|
| ```python
|
| import torch
|
| from tokenizers import Tokenizer
|
| from pathlib import Path
|
|
|
| # Load tokenizer and model weights
|
| tokenizer = Tokenizer.from_file("tokenizer.json")
|
| weights = torch.load("pytorch_model.bin", map_location="cuda" if torch.cuda.is_available() else "cpu")
|
| ```
|
|
|
| ### Run Inference via CLI
|
| ```bash
|
| python 100m-codebharat/scripts/11_generate.py --prompt "def binary_search(arr, target):"
|
| ```
|
|
|
| ## Model Card Info
|
| - **Developed by:** CodeBharat Team
|
| - **Model Type:** Causal Language Model for Code
|
| - **License:** Apache 2.0
|
|
|
