README.md

---
license: mit
language:
  - en
  - multilingual
library_name: transformers
tags:
  - password-generator
  - password-cracking
  - cybersecurity
  - red-team
  - password-analysis
  - llm
  - penetration-testing
  - passguess
  - pml-22l-o
pipeline_tag: text-generation
---

# PML-22L-O -- Password Modeling Language

**Part of the Password Modeling Language (PML) series alongside [PML-6L](https://huggingface.co/K0D3IN/PML-6L) and [PML-12L-O](https://huggingface.co/K0D3IN/PML-12L-O).**

This model is a **separate architecture trained from scratch** (not a finetune of PML-6L or PML-12L-O) on **57 million USER+COUNTRY→password pairs** extracted from stealer logs. PML-22L-O is the **deep variant** (22 layers, 384-dim embeddings) — following the MobileLLM hypothesis that deeper-and-thinner architectures outperform wider-and-shallower ones at the same parameter budget.

Its sibling [PML-12L-O](https://huggingface.co/K0D3IN/PML-12L-O) is the **wide variant** (12 layers, 512-dim embeddings). Both share approximately 46M parameters, enabling a controlled deep-versus-wide comparison.

## Architecture

| Parameter | Value |
|-----------|-------|
| Layers | 22 |
| Hidden dimension | 384 |
| Attention heads | 6 |
| Head dimension | 64 |
| FFN hidden dimension | 1024 (8/3 × 384) |
| **Total parameters** | **45,237,120** |
| Vocabulary size | 8192 (BBPE) |
| Max sequence length | 48 |
| Position encoding | RoPE |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Attention | Scaled Dot-Product (PyTorch SDPA) |
| Training precision | FP16 |

### Conditioning Tags

```
[USER:john_doe][COUNTRY:TR][LEN:12][CHARS:LOWER+UPPER+DIGIT]:myPassword1
```

Five augmentation modes during training: 40% full tag, 20% user-only, 15% context, 15% unconditional, 10% constraint-only.

### Loss Masking

Only the password portion (after `]:`) contributes to loss. Tag tokens masked with −100. EOS weighted 5× to encourage complete generation.

## Training Data

57 M unique USER+COUNTRY→password pairs from 200+ GB of publicly available stealer log archives, filtered through a strict gold-quality pipeline:

| Filter | Rule |
|--------|------|
| Unicode letter | Username must contain ≥1 letter |
| $ sign | Reject if present in username |
| Digit ratio | ≥8 chars + >60% digits → reject |
| File extensions | `.pdf`, `.exe`, `.html`, … |
| URL paths | ≥2 slashes → reject |
| Generic names | `null`, `none`, `undefined`, `n/a`, … |
| Template markers | `xxx` patterns |
| Length | User 2–64, password ≥4 chars |

## Training Configuration

| Hyperparameter | Value |
|----------------|-------|
| Optimizer | AdamW (β₁=0.9, β₂=0.95) |
| Peak LR / min LR | 2×10⁻⁴ → 2×10⁻⁵ (cosine, 70 % schedule) |
| Warmup steps | 5 000 |
| Label smoothing | 0.05 |
| Batch size | 256 (eff. 2 048 with 8× grad accum) |
| Weight decay | 0.1 (projections), 0.0 (biases, norms, embedding) |
| Gradient clipping | 1.0 |
| EOS weight | 5× |
| Hardware | RTX 4060 Laptop (8 GB VRAM) |
| Training time | ~13 h/epoch |

## Loss Progression

| Epoch | Train Loss | Val Loss | Val Perplexity |
|-------|-----------|----------|----------------|
| 1 | 5.717 | 5.150 | 172.5 |
| 2 | 5.337 | 5.079 | 160.5 |
| 3 | 5.284 | 5.042 | 154.8 |
| 4 | **5.225** | **5.016** | **150.8** |

## Deep vs. Wide: PML-22L-O vs. PML-12L-O

| Metric | PML-12L-O (wide) | PML-22L-O (deep) |
|--------|-------------------|-------------------|
| Architecture | 12L-512E-8H | 22L-384E-6H |
| Parameters | 46.1M | 45.2M |
| Val Loss (Ep4) | 5.006 | 5.016 |
| Cultural patterns by Ep1 | username-only | **username + 1907** |
| Cover @1000 guesses | 1.40 % | **1.65 %** |
| Speed (beam=40, FP16) | **1013 pw/s** | 990 pw/s |

Key finding: the deeper model learns culturally-aware password patterns **3× faster** than the wider model (produces `fb1907` by epoch 1 vs. epoch 2–3 for the 12L variant). Cover rate is also **18 % higher** at the 1 000‑guess budget.

## Generation Speed

Benchmarked on RTX 4060 with FP16 autocast:

| Strategy | Speed |
|----------|-------|
| `generate_fast()` (batch=256) | **1386 pw/s** |
| `generate_beam()` (beam=40) | **990 pw/s** |
| `generate()` (single) | 5–10 pw/s |

## Cover Rate

2 000 random validation samples, 1 000 beam-search guesses per credential:

| Budget | Cover |
|--------|-------|
| @1 guess | 0.10 % |
| @10 guesses | 0.55 % |
| @100 guesses | 1.30 % |
| @1 000 guesses | **1.65 %** |

## Usage

### Fast generation

```python
import torch
from model_v5 import PasswordLLaMA
from tokenizers import Tokenizer

model = PasswordLLaMA(vocab_size=8192, n_layer=22, n_embd=384, n_head=6, max_seq_len=48)
state = torch.load("model.safetensors", weights_only=True)
model.load_state_dict(state, strict=True)
model.eval()

toker = Tokenizer.from_file("tokenizer.json")
prefix_ids = toker.encode("[USER:john][COUNTRY:TR]:").ids

with torch.no_grad():
    pws = model.generate_fast(toker, temperature=0.8, top_k=50,
                               max_len=48, min_len=4,
                               prefix_ids=prefix_ids, batch_size=256, device="cuda")
print(pws[:5])
```

### Beam search (higher quality)

```python
pws = model.generate_beam(toker, temperature=0.8, top_k=50,
                           max_len=48, min_len=6,
                           prefix_ids=prefix_ids,
                           n_passwords=100, beam=40,
                           length_penalty=0.85, device="cuda")
```

### With Transformers pipeline

```python
from transformers import pipeline
pipe = pipeline("text-generation", model="K0D3IN/PML-22L-O")
result = pipe("[USER:john][COUNTRY:TR]:", max_new_tokens=16)
```

## Intended Use

| ✅ Appropriate | ❌ Inappropriate |
|---------------|-----------------|
| Authorized penetration testing | Credential stuffing against live systems |
| Red team engagements with written authorization | Account takeover |
| Password policy research | Illegal access to systems |
| Security awareness training | Privacy violations |
| Academic password research | Harassment or doxxing |
| Internal security audits | Any use without explicit authorization |

## Ethical Notice

**This model generates password candidates that may match real user credentials.**

Passwords generated are statistical patterns derived from stealer log archives. They do not contain actual training examples. However, due to the nature of password generation, some outputs may coincidentally match real passwords currently in use.

### Users are solely responsible for:
- Obtaining proper authorization before any security testing
- Complying with all applicable laws (CFAA, GDPR, KVKK, etc.)
- Ensuring testing is conducted within authorized scope only
- Secure handling and disposal of generated password lists

### This model should NOT be used for:
- Any illegal activity
- Attacking systems without explicit written permission
- Mass credential stuffing or account takeover attempts
- Harassing individuals or organizations

## Files

| File | Size | Description |
|------|------|-------------|
| `model.safetensors` | ~181 MB | Model weights |
| `config.json` | ~1 KB | Model configuration |
| `tokenizer.json` | ~2 MB | BBPE tokenizer (vocab=8192) |
| `model_v5.py` | ~12 KB | Model definition |
| `demo/` | — | Local Gradio app (run with `python demo/app.py`) |

## Citation

```bibtex
@software{PML-22L-O,
  author = {Mübeşşir Yusuf Akhan},
  title = {PML-22L-O: Conditional Password Generation via USER+COUNTRY},
  year = {2026},
  url = {https://huggingface.co/K0D3IN/PML-22L-O}
}
```

## License

This project is licensed under the **MIT License** — see the [LICENSE](LICENSE) file for details.

## Acknowledgments

This model is part of the PML series alongside [PML-6L](https://huggingface.co/K0D3IN/PML-6L) and [{sibling}](https://huggingface.co/K0D3IN/{sibling}).

- To my mother, who has always supported my ideas
- To my father, who bought me the computer that made this project possible
- To my friend Özüm, who supports me in realizing my ideals

## Support Open Source AI Research

Every donation helps maintain and improve this project:

**Monero (XMR):**
```
83iqXtvVu28ZiL9bsATMerSgbFFiD1J1jc96CcxJLEnAW3KBmBKedWnUAeLvLvEA9aBiUBpHQJs1iNHYtkTLZbNUEymobSS
```

**Bitcoin (BTC):**
```
bc1qmnlvpukcgl0hsr7nje0x8555mhtxjt80wtmlxm
```