---
license: apache-2.0
tags:
- russian
- english
- causal-lm
- frozen-embeddings
- conceptual-demo
- transformer
pipeline_tag: text-generation
library_name: transformers
---

# demo_bvv_ru

This repository contains the model and associated resources from the papers

[📚 Paper (Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations)](https://huggingface.co/papers/2507.04886) - 

[📚 Paper (Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate)](https://huggingface.co/papers/2507.07129) - 

[💻 Code](https://github.com/AVBochkov/Embeddings)

---

## Model summary

**Proof-of-concept Transformer LM with frozen, non-semantic token embeddings trained on a small English-Russian corpus.**

**This model is part of a series of models designed to demonstrate:**
- The viability of transformer language models where the embedding layer is precomputed from non-semantic (Unicode/visual) features and entirely _frozen_ during training.
- The possibility of modular/federated model fusion (MoE) by combining models with a shared token embedding matrix, without any additional retraining or alignment.

- **Parameters:** 0.5B  
- **Architecture:** 16-layer transformer, rotary attention, 1024 context, 32 heads.
- **Embedding:** Precomputed, _frozen_ visual/Unicode-based.
- **Training corpus:** Small-scale, <10B tokens, ~10% SFT-mixed (for metric tracking, not strong performance).
- **Languages:** Russian, English.
- **MoE compatibility:** Embedding space is shared with other `bvv` models (e.g. `Bochkov/demo_bvv_zh`) enabling seamless MoE or model fusion at output head level.

## Key points
This model was trained on a small corpus and is intended only to demonstrate the viability of frozen, visual/Unicode-derived embeddings for training and transfer between languages.

Performance is not comparable to SOTA but shows competitive compositional skills versus a fully trainable embedding baseline.

For direct benchmarking, see also [Bochkov/demo_bvv_unfrozen_ru] — an identical architecture and dataset, but with standard trainable token embeddings.
Enables seamless fusion/MoE with Bochkov/demo_bvv_zh and Bochkov/demo_bvv_moe (merged model) due to shared embedding space.

## Key results

- **MMLU avg**: 22.3% ±0.1
- **ARC-e**: 23.0%
- **ARC-c**: 24.6%
- **CommonsenseQA**: 20.1%
- **SQUAD**: 14.8%
- **BLEU [en-ru]**: 6.4%
- **BLEU [ru-en]**: 8.8%

This work demonstrates that transformer blocks, not token embeddings, carry the semantic burden in LLMs — a step toward modular, fusable, multilingual LMs.

## Example Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained('Bochkov/demo_bvv_ru', trust_remote_code=True).to('cuda')
tokenizer = AutoTokenizer.from_pretrained('Bochkov/demo_bvv_ru')
inputs = tokenizer("Hello, мир! ", return_tensors="pt").to('cuda')
outputs = model.generate(
    **inputs, 
    max_new_tokens=100, 
    temperature=0.8, 
    top_k=50, 
    top_p=0.95, 
    do_sample=True
)
print(tokenizer.decode(outputs[0]))
```

## 🧑‍🔬 Citation & Concept

If you find this work helpful or inspiring, please consider citing the associated papers:

```
@article{
      bochkov2025emergent,
      title={Emergent Semantics Beyond Token Embeddings: Transformer {LM}s with Frozen Visual Unicode Representations},
      author={Andrey Bochkov},
      journal={Transactions on Machine Learning Research},
      issn={2835-8856},
      year={2025},
      url={https://openreview.net/forum?id=Odh8IynO1o},
      note={}
}

@misc{bochkov2025growingtransformersmodularcomposition,
      title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.07129},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.07129}, 
}
```