---
license: apache-2.0
tags:
  - custom-architecture
  - from-scratch
  - language-model
  - non-transformer
  - tensorflow
---

# TERA V2

A language model built entirely from scratch. No pretrained weights. No standard transformers.

## Architecture

TERA V2 uses a custom non-transformer architecture with the following components:

- **Time Mix** for sequence mixing
- **Token Shift** for position encoding
- **GroupNorm** for normalization
- **Channel Mix** with **Squared ReLU** for feed-forward
- **Stochastic Depth** for regularization
- **Untied Embeddings**

## Model Specifications

| Specification | Value |
|---------------|-------|
| Parameters | ~726K |
| Vocabulary Size | 510 |
| Context Length | 32 tokens |
| Hidden Size (d_model) | 128 |
| Attention Heads | 4 |
| Layers | 3 |
| Framework | TensorFlow / Keras |

## Training Details

- Trained from scratch on clean question-answer pairs
- No pretrained weights were used at any stage
- Custom BPE-lite tokenizer trained on the same data
- Loss function: Sigmoid cross-entropy
- Optimizer: Adam with cosine learning rate schedule
- Training format: Q: question / A: answer

## How To Use

1. Download all files from this repository
2. Install TensorFlow
3. Load the tokenizer from tokenizer.json
4. Build the model using model_config.json
5. Load weights from model.weights.h5
6. Format input as: Q: your question here / A:

## Example Input and Output

Input: Q: What is the sun?

Output: The sun is a star at the center of our solar system.

Input: Q: Hello

Output: Hello! How can I help you today?

## Files Included

| File | Description |
|------|-------------|
| model.py | Model architecture code |
| tokenizer.py | Tokenizer class code |
| model_config.json | Model hyperparameters |
| tokenizer.json | Trained tokenizer vocabulary |
| model.weights.h5 | Trained model weights |
| training_data.py | Training data used |
| loss_history.json | Training loss over epochs |
| training_state.json | Final training stats |

## Live Demo

Try TERA V2 live at: https://huggingface.co/spaces/vedaco/tera.v2

## Created By

**Vedaco Team**

## License

Apache 2.0