---
license: apache-2.0
---
## Markov vs Transformer: Text Generation Experiment

This project compares classic statistical models (Markov chains & n-grams) with modern transformers (GPT-2) on the same text corpus (Pushkin’s poetry).
It highlights the jump from local memory to long-context reasoning in text generation.

## Motivation

In 1906, Andrey Markov showed how letter dependencies could model text.
Fast forward a century — transformers now handle hundreds of tokens of context with attention.
This repo recreates that evolution, side by side.

## Models Compared

Markov Chains (n=1,3,5)

Generate text from local character windows.

Capture letter frequencies, small fragments of fluency.

Fail on long-term dependencies → loops & gibberish.

GPT-2 (medium)

Pretrained transformer with 345M parameters.

Extends prompts into fluent, poetic lines.

Still prone to degeneration without sampling controls.

## Results

Prompt: “I loved you” (Pushkin corpus)

Markov vs GPT-2 on Pushkin (Prompt: “I loved you”)
Model	Sample Output (excerpt)
Markov (n=3)	“I longer trouble you so tenderly, sorrow. I loved in my so sincerely extinguished in my shyness, no loved now by jealousy. I loved you may be loved you may be loved you so tenderly…”
Markov (n=5)	“I loved you silently, without hope, Tormented now by jealousy. I loved you: and perhaps this flame Has not entirely extinguished in my soul; But let it no longer trouble you; I do not entirely extinguished in my soul; But let it…”
GPT-2 medium	“I love you with all my heart, without reserve. I am in love with you now, and have never been. I am in love with you now, and will never be… I love you with all my heart, without reserve…”

## Key Observations

Markov chains: good for local coherence, but collapse quickly.

Transformers: sustain global structure, more creative continuations.

Both models show failure modes — repetition loops highlight why sampling strategies matter.

Demonstrates the leap from statistical modeling → neural networks → generative AI.

## How to Run

Markov chains

mc = NGramMarkov(n=5)
mc.train(corpus)
print(mc.generate("<Pushkin Poetry Corpus of Choice>", 200))


## GPT-2 (via 🤗 Transformers)

from transformers import pipeline

generator = pipeline("text-generation", model="gpt2-medium")

print(generator("<Pushkin Poetry Corpus of Choice>", max_length=80, do_sample=True))

✨ Author

## Developed by [Naga Adithya Kaushik (GenAIDevTOProd)], assisted with AI(Debug, text corpus generation only)

For research, debug, and teaching purposes.