README.md

---
license: apache-2.0
language:
- en
tags:
- tiny
- fromzero
- regression
- api_pricing
- research
- token_estimation
- prediction
- token_counting
- dataset_labeling_tokens
datasets:
- fromziro/jetoncount_corpus
---

# JetonCount

## Summary
```
Task: regression
Total training time: 111 minutes
Params: 7009
Final MAE: 192
Framework: PyTorch
Authors: Paul Courneya, Jonathon Ly
```

## Description

JetonCount is a 7k-parameter MLP regression model trained to predict the number of tokens a piece of text might contain using only six input features.

## Model Details

- Architecture: MLP
- Input Feature Dimension: 19
- Raw Input Features: 7
- Engineered Features: True
- Log1p Engineered Features: True
- Hidden Size: 32
- Number of Layers: 8
- Activation: SiLU
- Dropout: 0.005
- Total Parameters: 7,009

### Input Features

- `chars`
- `words`
- `avg_chars_per_word`
- `longest_word_chars`
- `symbol_ratio`
- `punctuation_ratio`
- `vocab_size`
  
## Training

### Dataset

22M rows. 28 tokenizers. 9 sources.

- Tokenizers: [`tokenizers_used.txt`](https://huggingface.co/fromziro/JetonCount/blob/main/tokenizers_used.txt)
- Datasets: [`datasets_used.txt`](https://huggingface.co/fromziro/JetonCount/edit/main/datasets_used.txt)

The sources include math, code, educational text, general web, social media posts (reddit), and instruct data.
The tokenizers range from GPT2 to DeepSeek-v4 and have varying vocabulary sizes (250 to 256,000).

### Training Details

- Maximum Learning Rate: 6e-3
- Minimum Learning Rate: 3e-6
- Number of Epochs: 3
- Batch Size: 32000
- Eval Split Ratio: 0.005
- Gradient Accumulation Steps: 1
- Gradient Clipping: 1.0
- AdamW Betas: `(0.9, 0.95)`
- DType: `float32`

### Final Eval and Train Results

- Train:
  - R²: 0.951257
  - MSE: 938621.647477
  - RMSE: 968.824880
  - MAE: 192.055944
  - MRE: 0.137838
  - Explained Variance: 0.951305
  - Loss: 938621.647477

- Eval:
  - R²: 0.9738627018499254
  - MSE: 480722.18035314884
  - RMSE: 693.3413159138498
  - MAE: 163.19862499318103
  - MRE: 0.10729957442834033
  - Explained Variance: 0.9738670065468997
  - Loss: 480722.18035314884

- Test:
  - R²: 0.9717820439277628
  - MSE: 388793.8423401673
  - RMSE: 623.5333530294649
  - MAE: 157.27345487387672
  - MRE: 0.10509939610167868
  - Explained Variance: 0.9717854117493441
  - Loss: 388793.8423401673

### Hardware

- CPU: Ryzen 5 2600 (data preparation and training)

### Predictions

| Actual Tokens | Model Prediction |
| ------------- | ---------------- |
| 197           | 239              | 
| 1333          | 1395             |
| 5973          | 6609             |
| 18569         | 20423            |

Note: Rounded to nearest integer.

#### Example

- Input Text (taken from wikipedia):
```
Artificial intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making. It is a field of research in engineering, mathematics and computer science that develops and studies methods and software that enable machines to perceive their environment and use learning and intelligence to take actions that maximize their chances of achieving defined goals.[1]
```
- Input vocab size (matching tokenizer): `2560`
- Tokenizer (baseline): FromZero/Er-Tiny-1.3M

Out:
```
{
  "actual_token_count": 139,
  "prediction" "190.5435028076172",
  "model_latency_ms": "0.24457614858252544",
  "tokenizer_latency_ms": 0.3174110000009023
}
```

### Is it faster than a tokenizer, though?

We came across a dilemma: why build this model if a tokenizer is more accurate anyway?

The answer is **speed**.

In our tests, especially on long texts, the model is significantly faster than a tokenizer.

| Tokens | Model Latency (ms) | Tokenizer Latency (ms)   |
|--------|--------------------|--------------------------|
| 197    | 0.2429             | 0.4134                   |
| 1333   | 0.3409             | 1.8775                   |
| 5973   | 0.9827             | 7.6504                   |
| 18569  | 5.2890             | 28.8244                  |

Note: Model latency changes based on hardware.

## Use Cases

1. Educational work and research
2. API Pricing Estimation
3. Dataset labeling
4. Or just for fun.

## Limitations

1. The model is an approximation and can produce errors on out-of-distrubtion texts.
2. Prediction accuracy heavily depends on the corectness of the input features.
3. It does not perform actual tokenization and therefore is much less accurate than an actual tokenizer.
4. Vocabulary sizes larger than 128k can result in performance degradation.
5. Short text (under 32 tokens) can result in performance degradation.
## License

Before using, distributing, selling, or modifying this software, you must read the license [here](https://huggingface.co/fromziro/Er-Tiny-1.3M/blob/main/LICENSE.txt).

## Inference

```python
from __future__ import annotations

import json
import re
import time
from dataclasses import dataclass
from typing import Tuple

import torch
from transformers import AutoModel, AutoTokenizer

MODEL_ID = "fromziro/JetonCount"
TOKENIZER_ID = "fromziro/Er-Tiny-1.3M"

FEATURE_MEAN = None
FEATURE_STD = None
TARGET_OFFSET = 0.0

DEFAULT_VOCAB_SIZE = 2564

TEXT = "Put your text here."
TOKENIZER_ROUNDS = 100
MODEL_ROUNDS = 1000

PUNCTUATION_CHARS = set(r""".,!?;:'"`~@#$%^&*()-_=+[]{}<>/\|""")
SYMBOL_CHARS = set(r"""@#$%^&*()-_=+[]{}<>/\|~`""")


@dataclass
class TextStats:
    chars: float
    words: float
    avg_chars_per_word: float
    punctuation_ratio: float
    symbol_ratio: float
    longest_word_chars: float
    vocab_size: float


def compute_text_stats(text: str, vocab_size: int) -> TextStats:
    chars = len(text)
    words_list = re.findall(r"\b\w+\b", text, flags=re.UNICODE)
    words = len(words_list)

    total_word_chars = sum(len(w) for w in words_list)
    avg_chars_per_word = (total_word_chars / words) if words else 0.0
    longest_word_chars = max((len(w) for w in words_list), default=0)

    if chars:
        punctuation_count = sum(1 for ch in text if ch in PUNCTUATION_CHARS)
        symbol_count = sum(1 for ch in text if ch in SYMBOL_CHARS)
        punctuation_ratio = punctuation_count / chars
        symbol_ratio = symbol_count / chars
    else:
        punctuation_ratio = 0.0
        symbol_ratio = 0.0

    return TextStats(
        chars=float(chars),
        words=float(words),
        avg_chars_per_word=float(avg_chars_per_word),
        punctuation_ratio=float(punctuation_ratio),
        symbol_ratio=float(symbol_ratio),
        longest_word_chars=float(longest_word_chars),
        vocab_size=float(vocab_size),
    )


def build_feature_tensor(stats: TextStats) -> torch.Tensor:
    base = torch.tensor(
        [
            stats.chars,
            stats.words,
            stats.avg_chars_per_word,
            stats.punctuation_ratio,
            stats.symbol_ratio,
            stats.longest_word_chars,
            stats.vocab_size,
        ],
        dtype=torch.float32,
    )

    chars, words, avg_chars_per_word, punctuation_ratio, symbol_ratio, longest_word_chars, vocab_size = base
    eps = 1e-6

    extra = torch.tensor(
        [
            chars / max(words.item(), 1.0),
            words / max(chars.item(), 1.0),
            torch.log1p(torch.clamp(chars, min=0.0)).item(),
            torch.log1p(torch.clamp(words, min=0.0)).item(),
            torch.log1p(torch.clamp(vocab_size, min=0.0)).item(),
            (chars * punctuation_ratio).item(),
            (chars * symbol_ratio).item(),
            (words * avg_chars_per_word).item(),
            (words * punctuation_ratio).item(),
            (longest_word_chars * punctuation_ratio).item(),
            ((avg_chars_per_word + longest_word_chars) * (1.0 + punctuation_ratio + symbol_ratio)).item(),
            ((chars + eps) * (punctuation_ratio + symbol_ratio + eps)).item(),
        ],
        dtype=torch.float32,
    )

    return torch.cat([base, extra], dim=0)


def standardize_features(x: torch.Tensor) -> torch.Tensor:
    if FEATURE_MEAN is None or FEATURE_STD is None:
        return x
    mean = torch.tensor(FEATURE_MEAN, dtype=x.dtype, device=x.device)
    std = torch.tensor(FEATURE_STD, dtype=x.dtype, device=x.device)
    safe_std = torch.where(torch.isfinite(std) & (std != 0), std, torch.ones_like(std))
    safe_mean = torch.where(torch.isfinite(mean), mean, torch.zeros_like(mean))
    return (x - safe_mean) / safe_std


def benchmark_tokenizer(tokenizer, text: str, rounds: int = 100) -> Tuple[int, float]:
    tokenizer(text)
    start = time.perf_counter()
    actual_count = 0
    for _ in range(rounds):
        ids = tokenizer(text, add_special_tokens=False).input_ids
        actual_count = len(ids)
    elapsed_ms = (time.perf_counter() - start) * 1000.0 / rounds
    return actual_count, elapsed_ms


@torch.inference_mode()
def benchmark_model(model, feature_tensor: torch.Tensor, rounds: int = 1000) -> Tuple[float, float]:
    x = standardize_features(feature_tensor).unsqueeze(0)

    _ = model(input_features=x)

    start = time.perf_counter()
    pred = 0.0
    for _ in range(rounds):
        out = model(input_features=x)
        pred = float(out.logits.squeeze().item())
    elapsed_ms = (time.perf_counter() - start) * 1000.0 / rounds
    return pred, elapsed_ms


def main() -> None:
    tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_ID, use_fast=True)
    model = AutoModel.from_pretrained(MODEL_ID, trust_remote_code=True)
    model.eval()

    stats = compute_text_stats(TEXT, DEFAULT_VOCAB_SIZE)
    feature_tensor = build_feature_tensor(stats)

    actual_count, tokenizer_latency_ms = benchmark_tokenizer(tokenizer, TEXT, rounds=TOKENIZER_ROUNDS)
    prediction, model_latency_ms = benchmark_model(model, feature_tensor, rounds=MODEL_ROUNDS)

    result = {
        "actual_token_count": actual_count,
        "prediction": prediction,
        "model_latency_ms": model_latency_ms,
        "tokenizer_latency_ms": tokenizer_latency_ms,
        "model_id": MODEL_ID,
        "tokenizer_id": TOKENIZER_ID,
        "vocab_size": DEFAULT_VOCAB_SIZE,
        "features": {
            "chars": stats.chars,
            "words": stats.words,
            "avg_chars_per_word": stats.avg_chars_per_word,
            "punctuation_ratio": stats.punctuation_ratio,
            "symbol_ratio": stats.symbol_ratio,
            "longest_word_chars": stats.longest_word_chars,
            "vocab_size": stats.vocab_size,
        },
    }

    print(json.dumps(result, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()
```

### Demo (HF Space)

[JetonCount](https://huggingface.co/spaces/fromziro/JetonCount)

## Copyright

```
Copyright (c) 2026 FromZero  
Copyright (c) 2026 Paul Courneya
Copyright (c) 2026 Jonathon LY
```

## Citation

```bibtex
@misc{jetoncount,
  title        = {JetonCount},
  organization = [FromZero],
  authors      = {Paul Courneya, Jonathon LY},
  year         = {2026},
  url          = {https://huggingface.co/fromziro/JetonCount}
}
```