kennethge123/qwen3-bart-inversion-noisy

Qwen3 BART Inversion Model (Noise-Robust)

Noise-robust variant of the Vec2Text BART inversion model. Fine-tuned with embedding perturbations (cosine similarity 0.65-0.95) for robustness to noisy/approximate embeddings.

Model Details

• Base model: facebook/bart-base
• Embedding model: Qwen3-Embedding-8B (4096-dim)
• Training data: 1M sentences from ClimbMix corpus
• Architecture: Vec2Text embedding inversion (embedding -> text)
• Noise training: Cosine similarity perturbations (0.65-0.95)
• Fine-tuning steps: 4641
• Base: Clean BART inversion model

Evaluation Results (200 held-out samples)

Metric	Score
Token F1	0.6315 (+/-0.1727)
BLEU-4	0.2413 (+/-0.2564)
ROUGE-L	0.5289 (+/-0.2069)
Exact Match	5/200 (2.5%)

Example Reconstructions

Original	Reconstructed	Token F1
"This is important because childhood sets the stage for the robustness of the im...	"This is important because it sets the foundation for robust immune system robus...	0.789
Chapter 8: Made in the USA - A Statement About Quality and Pride

Have you ever ... | Chapter 8: A Symbol of Quality in America Have you ever made a garment inside yo... | 0.481 | | Less than 30% of the 36 million cat owners in the U.S. know that the beautiful s... | According to the U.S.G.A.S., 30% of cat owners know that their cats can kill mor... | 0.545 | | The key ingredients of Turkish meals are meat, vegetables, and legumes.... | The main ingredients of Turkish meals include meat, vegetables, and legumes.... | 0.867 | | The platform will then generate a list of available options with prices, travel ... | The platform will then provide available options such as travel time, amenities,... | 0.708 |

Usage

import torch, torch.nn as nn, transformers
from safetensors.torch import load_file

# Load model
model = transformers.AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-base")
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/bart-base")
hidden = model.config.hidden_size

embedding_transform = nn.Sequential(
    nn.Linear(4096, 4096), nn.LayerNorm(4096), nn.Dropout(0.1), nn.GELU(),
    nn.Linear(4096, hidden * 16),
)

# Load weights (download from this repo)
state = load_file("model.safetensors")  # or torch.load("bart_noisy.pt")
et_state = {k.replace("embedding_transform.", ""): v for k, v in state.items() if k.startswith("embedding_transform.")}
embedding_transform.load_state_dict(et_state)
ed_state = {k.replace("encoder_decoder.", ""): v for k, v in state.items() if k.startswith("encoder_decoder.")}
model.load_state_dict(ed_state, strict=False)

# Invert a Qwen3-Embedding-8B embedding (4096-dim)
device = torch.device("cuda")
model, embedding_transform = model.to(device).eval(), embedding_transform.to(device).eval()

with torch.no_grad():
    emb = torch.tensor(your_embedding, dtype=torch.float32).unsqueeze(0).to(device)
    proj = embedding_transform(emb).reshape(1, 16, hidden)
    out = model.generate(inputs_embeds=proj, attention_mask=torch.ones(1, 16, device=device),
                         max_length=128, num_beams=4, early_stopping=True)
    text = tokenizer.decode(out[0], skip_special_tokens=True)