README.md

---
library_name: transformers
tags:
- bitnet
- 1.58-bit
- ternary
- custom-architecture
- text-generation
---

# Model Card for BitNet_LLM_Project

This is a custom, character-level causal language model trained from scratch to validate and implement the mechanics of **BitNet b1.58** extreme quantization. The architecture replaces standard continuous 16-bit floating-point linear layers with custom ternary layers, restricting effective model weights strictly to the discrete states of -1, 0, and 1.

## Model Details

### Model Description

This model card belongs to a custom Hugging Face implementation of a 1.58-bit Ternary NanoGPT model. It employs a Straight-Through Estimator (STE) during the backpropagation phase to bypass the non-differentiable nature of discrete quantization step functions. While the forward pass utilizes sharp ternary weight states to run highly efficient matrix operations, the backward pass continuously routes gradients to high-precision latent weights, allowing micro-adjustments to accumulate over time.

- **Developed by:** Mudunk
- **Funded by [optional]:** Self-funded open-source research project
- **Shared by [optional]:** Mudunk
- **Model type:** Custom NanoGPT Causal LM
- **Language(s) (NLP):** English
- **License:** MIT
- **Finetuned from model [optional]:** None (Trained completely from scratch)

### Model Sources [optional]

- **Repository:** Mudunk/BitNet_LLM_Project
- **Paper [optional]:** Microsoft BitNet b1.58 Research Specification
- **Demo [optional]:** Not applicable

## Uses

### Direct Use

This model is intended for structural research, inference testing, and hardware efficiency benchmarking of extreme quantization mechanics. It can be used to generate character-by-character text streams based on an input prompt.

### Downstream Use [optional]

Can be integrated into localized, ultra-low-power edge computing prototypes evaluating discrete weight models or custom hardware inference engines.

### Out-of-Scope Use

This architecture is not intended for commercial applications, long-form coherent text generation, or complex reasoning tasks. Due to its unique quantization constraints and small operational footprint, it will not function reliably outside of controlled character-level generation scripts.

## Bias, Risks, and Limitations

The primary technical limitation of this model is its compressed representation capacity. At a miniature scale of roughly 10 million parameters, a 1.58-bit model possesses a tight informational ceiling. Because it operates on a character level rather than using a sub-word tokenization framework (like Tiktoken), the model expends a significant portion of its internal parameters learning basic spelling and word boundaries rather than mastering complex grammatical syntax.

### Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. To achieve the best possible text output and limit structural degradation during inference, it is highly recommended to override standard text-generation defaults. Apply strict sampling constants to minimize noise in the token distribution: Set Temperature to a low value (between 0.1 and 0.3) to keep predictions deterministic, and enforce Top-K Sampling capped at 10 tokens to aggressively remove the long tail of low-probability character selections.

## How to Get Started with the Model

Use the code below to initialize the custom architecture and load the open-source weights directly through the Hugging Face library:

```python
from transformers import AutoModelForCausalLM

# Load the custom 1.58-bit model architecture and configuration
model = AutoModelForCausalLM.from_pretrained(
    "Mudunk/BitNet_LLM_Project", 
    trust_remote_code=True
)
```

## Training Details

### Training Data

The model was trained entirely on the Tiny Shakespeare raw text corpus. The data pipeline treats the text as a continuous sequence of individual character characters rather than standard dictionary words, operating with a character vocabulary size of 65.

### Training Procedure

#### Preprocessing [optional]

Text inputs were mapped directly to unique character integers and broken down into sequence blocks of 256 characters per batch training instance.

#### Training Hyperparameters

- **Training regime:** Extreme Quantization (Ternary Weight States: -1, 0, 1) running with continuous latent weight tracking.
- **Learning Rate:** 3e-3 (Optimized from the baseline 1e-3 configuration to pull latent weights across discrete quantization thresholds more effectively).
- **Total Iterations:** 5,000 steps
- **Model Architecture Options:** bias=False across attention blocks and linear components, with bias=False explicitly passed to the final Layer Normalization (ln_f) layer to ensure strict weight matrix alignment.

#### Speeds, Sizes, Times [optional]

- **Hardware Type:** Single NVIDIA T4 Cloud GPU
- **Hours used:** ~1 Hour
- **Throughput / Convergence Behavior:** The optimization path exhibits a classic "staircase convergence" pattern, where the model loss flatlines into highly active noise plateaus for several hundred steps before experiencing sudden structural drops as clusters of background weights cross their activation thresholds simultaneously.

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

Evaluation was continuously performed against an isolated validation split taken directly from the primary character text corpus (Tiny Shakespeare).

#### Factors

The evaluation is disaggregated by sequence lengths up to 256 characters, tracking model capacity limits under extreme compression constraints.

#### Metrics

Progress was monitored using cross-entropy training loss and validation loss tracked at regular step intervals.

### Results

The optimized ternary model successfully broke through the validation limits set by initial control experiments, accelerating convergence speeds by a factor of three in the early training stages.

- **Final Training Loss achieved:** 2.3900
- **Final Validation Loss achieved:** 2.4500

#### Summary

Despite the brutal information bottleneck introduced by 1.58-bit constraints, the final model demonstrates clear structural learning. The generated outputs accurately replicate script formatting conventions (such as character names in all-caps followed by clean newlines), maintain correct English vowel-to-consonant ratios, and reliably spell essential short vocabulary words (the, thou, and, my, me).

## Model Examination [optional]

Weights were structurally inspected post-training to confirm a proper ternary distribution across the custom BitLinear layers, ensuring effective parameter allocation without zero-gradient stalling.

## Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

- **Hardware Type:** NVIDIA T4 Tensor Core GPU
- **Hours used:** 1 Hour
- **Cloud Provider:** Google Colab Cloud Infrastructure
- **Compute Region:** Dynamic Cloud Allocation
- **Carbon Emitted:** Minimal (localized experimental research scale)

## Technical Specifications [optional]

### Model Architecture and Objective

The model structure is built as a custom causal autoregressive transformer containing 6 hidden layers and 6 attention heads with an embedding dimension of 384. Standard floating-point matrix multiplications are substituted with optimized BitLinear blocks that apply absolute mean scales to weights and absolute maximum scales to activations before performing quantized computations.

### Compute Infrastructure

#### Hardware

A single cloud-allocated NVIDIA T4 GPU was utilized for both the core optimization run and the universal serialization conversions.

#### Software

PyTorch 2.0+ paired with the Hugging Face transformers and safetensors engine wrappers.

## Citation [optional]

**BibTeX:**
```bibtex
@misc{mudunk2026bitnetllm,
  author = {Mudunk},
  title = {BitNet_LLM_Project: A 1.58-Bit Ternary NanoGPT Implementation},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub}
}
```

**APA:**
Mudunk. (2026). *BitNet_LLM_Project: A 1.58-Bit Ternary NanoGPT Implementation*. Hugging Face Model Hub.

## Glossary [optional]

- **Straight-Through Estimator (STE):** An optimization technique that passes gradients unmodified through a non-differentiable step function during backpropagation.
- **Ternary Quantization:** Restricting parameters to exactly three states (-1, 0, 1) to eliminate floating-point multiplications entirely.

## More Information [optional]

For inquiries regarding the underlying custom BitLinear implementation or layer configuration extensions, please refer to the primary GitHub project space.

## Model Card Authors [optional]

- Mudunk

## Model Card Contact

Available via the Hugging Face repository discussion boards.