File size: 1,680 Bytes
6af0329 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
language:
- code
license: apache-2.0
library_name: transformers
tags:
- code
- python
- javascript
- cpp
- sql
- html
- code-generation
- codebharat
- llama
- PyTorch
- byte-level-bpe
pipeline_tag: text-generation
widget:
- text: "def quicksort(arr):"
example_title: "Python QuickSort"
- text: "function debounce(func, wait) {"
example_title: "JavaScript Debounce"
- text: "int binarySearch(const std::vector<int>& arr, int target) {"
example_title: "C++ Binary Search"
---
# CodeBharat-100M
**CodeBharat-100M** is a 100.68M parameter decoder-only Transformer pretrained from scratch on code (Python, JavaScript, TypeScript, C++, SQL, HTML/CSS, and synthetic textbooks).
## Model Details
- **Architecture:** Decoder-Only Transformer (Llama/Qwen-style: RMSNorm, RoPE, SwiGLU, Grouped-Query Attention)
- **Parameters:** 100,679,424 (100.68M)
- **Vocabulary:** 49,152 tokens (Byte-level BPE)
- **Context Window:** 1,024 tokens
- **Training Device:** NVIDIA GeForce RTX 5050 GPU
- **Final Validation Loss:** 1.3891
## Quickstart Usage
### Native PyTorch / Tokenizers Usage
```python
import torch
from tokenizers import Tokenizer
from pathlib import Path
# Load tokenizer and model weights
tokenizer = Tokenizer.from_file("tokenizer.json")
weights = torch.load("pytorch_model.bin", map_location="cuda" if torch.cuda.is_available() else "cpu")
```
### Run Inference via CLI
```bash
python 100m-codebharat/scripts/11_generate.py --prompt "def binary_search(arr, target):"
```
## Model Card Info
- **Developed by:** CodeBharat Team
- **Model Type:** Causal Language Model for Code
- **License:** Apache 2.0
|