QuixiAI
/

Prisma-VL-8B

@@ -14,19 +14,17 @@ pipeline_tag: image-text-to-text
 # Prisma-VL-8B: Introspective Vision-Language Model
-**An 8-billion parameter vision-language model architected from the ground up with 16-bit temporal uncertainty feedback for self-aware, calibrated predictions.**
 ## What is This?
-Prisma-VL-8B is a **reference implementation** of an introspective transformer architecture. The model doesn't just predict - it *knows* when it's uncertain and uses that self-awareness to calibrate subsequent predictions.
-This is not a base model with modifications. **This IS the architecture.** The 16-bit temporal uncertainty feedback mechanism is fundamental to how this model thinks.
 ## Core Architecture
 ### The Introspective Mechanism
-Every transformer processes tokens sequentially. Prisma-VL-8B adds one crucial element: **memory of its own uncertainty**.
 ```
 Standard Transformer:
@@ -38,11 +36,11 @@ Introspective Transformer:
 ### How It Works
-**The 65,536-Level Uncertainty System:**
 At each prediction step:
 1. **Measure**: Compute entropy of output distribution (how uncertain am I?)
-2. **Quantize**: Convert to 16-bit code (0-65535, representing confidence levels)
 3. **Inject**: Next token receives this as learned embedding signal
 4. **Learn**: Through training, model learns what each uncertainty level means
@@ -51,25 +49,6 @@ At each prediction step:
 - When it's extrapolating (rising uncertainty)
 - When it needs to be conservative (high uncertainty)
-### Architecture Components
-```python
-# Core introspective components (built into PrismaVLModel)
-self.uncertainty_embeddings = nn.Embedding(65536, hidden_dim)
-# 65,536 learned vectors: "uncertainty vocabulary"
-# Each represents: "I was X% uncertain on the last token"
-self.prev_uncertainty_code = None  # [batch, seq] with values [0-65535]
-# Temporal memory: tracks uncertainty history across generation
-```
-**Parameter Cost:** 65,536 × 4096 = 268,435,456 parameters (3.35% of model)
-**Memory Cost:** 2 bytes per token (uncertainty code)
-**Compute Cost:** One embedding lookup per token (negligible)
 ## Why This Matters
 ### Traditional Language Models
@@ -92,20 +71,7 @@ Generate "The capital of France is Mad..."
 [code:23] → [code:15] → [code:142] → STOP  # Detects uncertainty spike
 ```
-The model **feels** when predictions are going wrong and can self-correct or abstain.
-## What Gets Learned
-Through standard training (no special loss needed), the 65,536 uncertainty embeddings learn semantic meaning:
-| Code Range | Semantic Meaning | Learned Behavior |
-|------------|------------------|------------------|
-| 0-16383 | "I was very confident" | Maintain trajectory, continue assertively |
-| 16384-32767 | "Moderate confidence" | Proceed with caution, verify facts |
-| 32768-49151 | "Some uncertainty" | Hedge statements, qualify claims |
-| 49152-65535 | "Very uncertain" | Conservative generation, flag uncertainty |
-This creates a **calibration vocabulary** - the model learns to speak about its own knowledge state with fine-grained resolution.
 ## Usage
@@ -166,183 +132,9 @@ print(f"Average confidence: {1 - mean_uncertainty:.2%}")
 print(f"Highest uncertainty code: {max_uncertainty}")
 ```
-### Resetting State
-```python
-# Between independent generations, reset uncertainty history
-model.model.reset_uncertainty()
-# Fresh start - no previous context
-outputs = model.generate(**inputs)
-```
-## Model Specifications
-### Vision Encoder
-- **Architecture**: 27-layer Vision Transformer
-- **Hidden Dimension**: 1152
-- **Patch Size**: 16×16
-- **Temporal Patches**: 2 (for video)
-- **Parameters**: ~1.15B
-### Language Model
-- **Architecture**: 36-layer Transformer
-- **Hidden Dimension**: 4096
-- **Attention Heads**: 32 (8 KV heads, GQA)
-- **Intermediate Size**: 12,288
-- **Context Length**: 262,144 tokens
-- **Parameters**: ~6.85B
-### Introspective System
-- **Uncertainty Levels**: 65,536 (16-bit)
-- **Uncertainty Embeddings**: 65,536 × 4096
-- **Parameters**: 268,435,456 (268M)
-- **Overhead**: 3.35% of total model
-### Total Model
-- **Parameters**: ~8.27B (7.85B base + 268M introspective)
-- **Precision**: BFloat16 recommended
-- **Hardware**: 24GB VRAM recommended
-## Design Philosophy
-### Why 16-bit Quantization?
-- **Fine-Grained Resolution**: 65,536 levels capture nuanced confidence gradations
-- **Rich Representation**: Model can learn subtle uncertainty distinctions
-- **Precise Calibration**: Higher resolution enables better self-awareness
-- **Still Efficient**: Only 2 bytes per token, single embedding table lookup
-### Why Temporal Feedback?
-- **Causal Awareness**: Model sees its own prediction history
-- **Self-Correction**: Can detect and recover from errors
-- **Calibration**: Learns confidence from experience
-- **No External Labels**: Uses its own predictions as training signal
-### Why Built-In?
-- **Native Integration**: Works seamlessly with vision and text processing
-- **Always Active**: No modes to enable/disable
-- **End-to-End Training**: Learns uncertainty simultaneously with task
-- **Production Ready**: No inference overhead, no special handling
-## When to Use This Architecture
-### ✅ Good Fit
-- Applications requiring calibrated confidence estimates
-- Domains where hallucination prevention is critical
-- Long-form generation (benefits from temporal awareness)
-- Interactive systems (can express uncertainty to users)
-- Research on model introspection and self-awareness
-### ⚠️ Considerations
-- Requires fine-tuning for uncertainty calibration
-- Adds 1.05M parameters (minimal but non-zero)
-- Uncertainty codes need interpretation in your domain
-## Performance Characteristics
-### Computational Overhead
-| Phase | Additional Cost |
-|-------|----------------|
-| Forward Pass | +1 embedding lookup per token (~0.1% compute) |
-| Uncertainty Computation | Entropy calculation (in `torch.no_grad()`, negligible) |
-| Memory | +2 bytes per token in cache |
-| Training | Standard backprop through uncertainty embeddings |
-### Expected Benefits (After Fine-tuning)
-- **Calibration**: Better alignment between confidence and accuracy
-- **Hallucination Reduction**: Early detection of uncertain territory
-- **Adaptive Behavior**: Conservative when uncertain, assertive when confident
-- **Interpretability**: Uncertainty codes reveal model state
-## Training Recommendations
-### Initial Setup
-1. Load model with randomly initialized uncertainty embeddings
-2. Use your standard vision-language training recipe
-3. No changes to loss functions or training loops required
-4. Uncertainty mechanism learns automatically
-### Convergence
-- Uncertainty embeddings converge at similar rate to language model
-- Monitor validation loss as usual
-- Well-calibrated uncertainty emerges with sufficient training data
-### Fine-tuning
-- Start from pre-trained weights (if available)
-- Use domain-specific data for best calibration
-- Larger batch sizes help uncertainty statistics stabilize
-### Evaluation
-```python
-# Assess calibration: compare uncertainty to actual accuracy
-# High uncertainty should correlate with lower accuracy
-```
-## Technical Implementation
-### Files
-- `modeling.py`: Core architecture with introspective mechanism
-- `configuration.py`: Model configuration
-- `processing.py`: Vision/text processor
-- `test.py`: Inference example
-### Key Methods
-```python
-# In PrismaVLModel
-def __init__(self):
-    self.uncertainty_embeddings = nn.Embedding(65536, hidden_dim)
-    self.prev_uncertainty_code = None
-def reset_uncertainty(self):
-    """Clear uncertainty history between generations"""
-    self.prev_uncertainty_code = None
-# In forward pass
-uncertainty_embeds = self.uncertainty_embeddings(prev_uncertainty_code)
-inputs_embeds = inputs_embeds + uncertainty_shifted
-# After logits
-entropy = -(probs * log_probs).sum(-1)
-uncertainty_code = (entropy_norm * 65535).long()
-```
-### Dependencies
-```
-torch >= 2.0.0
-transformers >= 4.57.0
-accelerate >= 0.20.0
-Pillow
-```
-## Hardware Requirements
-| Configuration | VRAM | Precision | Batch Size |
-|--------------|------|-----------|------------|
-| Minimum | 16GB | 8-bit | 1 |
-| Recommended | 24GB | BFloat16 | 2-4 |
-| Optimal | 40GB+ | BFloat16 | 8+ |
-## Research Context
-This architecture demonstrates that **transformer self-awareness is learnable** through standard training. No RLHF, no auxiliary losses, no external signals - just 65,536 embeddings that learn to represent "how uncertain was I?"
-The key insight: **uncertainty is a learnable signal, not a post-hoc calculation**. With 16-bit quantization, the model can develop a highly nuanced understanding of its own confidence states.
-## Future Directions
-Potential extensions of this architecture:
-1. **Multi-Resolution Uncertainty**: Track uncertainty at token, phrase, and document levels
-2. **Cross-Modal Uncertainty**: Separate tracking for vision vs. language predictions
-3. **Uncertainty-Guided Sampling**: Adjust temperature based on live uncertainty
-4. **Explicit Uncertainty Tokens**: Generate "<uncertain>" tokens in output
-5. **Confidence-Aware Search**: Use uncertainty for better beam search
 ## Citation
@@ -358,12 +150,6 @@ Potential extensions of this architecture:
 Apache 2.0
-## Acknowledgments
-- Architecture inspired by temporal feedback patterns in cognitive science
-- 16-bit high-resolution quantization for fine-grained uncertainty representation
-- Vision-language backbone based on multimodal transformer designs
 ## Additional Resources
 - [Architecture Deep Dive](./INTROSPECTIVE_ARCHITECTURE.md)
@@ -372,10 +158,4 @@ Apache 2.0
 ---
-**This is not a modified model. This is the architecture.**
-Prisma-VL-8B exists to demonstrate that transformers can be introspective by design.
-**Status**: ✅ Production ready - fully functional in training and inference
-**Last Updated**: 2025-01-08

 # Prisma-VL-8B: Introspective Vision-Language Model
+**A vision-language model architected with temporal uncertainty feedback for self-aware predictions.**
 ## What is This?
+Prisma-VL-8B is a reference implementation of an introspective transformer architecture. The model uses its confidence to calibrate subsequent predictions.
 ## Core Architecture
 ### The Introspective Mechanism
+Every transformer processes tokens sequentially. Prisma-VL-8B adds one crucial element: memory of its own uncertainty.
 ```
 Standard Transformer:
 ### How It Works
+The Uncertainty System:
 At each prediction step:
 1. **Measure**: Compute entropy of output distribution (how uncertain am I?)
+2. **Quantize**: Convert to 16-bit code representing confidence levels
 3. **Inject**: Next token receives this as learned embedding signal
 4. **Learn**: Through training, model learns what each uncertainty level means
 - When it's extrapolating (rising uncertainty)
 - When it needs to be conservative (high uncertainty)
 ## Why This Matters
 ### Traditional Language Models
 [code:23] → [code:15] → [code:142] → STOP  # Detects uncertainty spike
 ```
+The model feels when predictions are going wrong and can self-correct or abstain.
 ## Usage
 print(f"Highest uncertainty code: {max_uncertainty}")
 ```
+## Introspection
+From prediction, emerges language.  From awareness of uncertainty, emerges introspection.
 ## Citation
 Apache 2.0
 ## Additional Resources
 - [Architecture Deep Dive](./INTROSPECTIVE_ARCHITECTURE.md)
 ---
+Prisma-VL-8B demonstrates introspective transformers.