Upload 207 files

.gitattributes

@@ -33,3 +33,157 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+sample_task/0001.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0002.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0003.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0004.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0005.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0006.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0007.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0008.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0009.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0010.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0011.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0012.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0013.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0014.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0015.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0016.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0017.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0018.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0019.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0020.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0021.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0022.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0023.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0024.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0025.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0026.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0027.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0028.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0029.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0030.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0031.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0032.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0033.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0034.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0035.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0036.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0037.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0038.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0039.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0043.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0044.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0045.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0046.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0047.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0048.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0049.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0050.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0051.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0052.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0053.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0054.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0055.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0056.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0057.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0059.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0060.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0061.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0062.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0063.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0064.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0065.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0066.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0067.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0068.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0069.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0070.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0072.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0073.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0074.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0075.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0076.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0077.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0078.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0079.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0080.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0081.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0082.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0083.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0084.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0085.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0086.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0087.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0088.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0089.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0090.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0091.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0092.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0093.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0094.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0095.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0096.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0097.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0098.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0099.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0100.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0101.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0102.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0103.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0104.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0105.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0107.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0108.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0109.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0110.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0111.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0112.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0113.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0114.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0115.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0116.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0117.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0118.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0119.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0120.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0121.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0122.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0123.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0124.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0125.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0126.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0127.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0128.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0129.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0130.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0131.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0132.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0133.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0134.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0135.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0136.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0137.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0138.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0139.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0140.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0141.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0142.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0143.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0144.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0145.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0146.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0147.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0148.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0149.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0150.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0151.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0152.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0153.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0154.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0155.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0156.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0157.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0158.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0159.png filter=lfs diff=lfs merge=lfs -text
+sample_task/0160.png filter=lfs diff=lfs merge=lfs -text

ai_http.py

@@ -1,5 +1,6 @@
 import numpy as np
 import time
+import torch
 from typing import Dict, Any, Optional, Tuple, Union, List
 from enum import Enum
 from tensor_core import TensorCoreArray
@@ -96,27 +97,56 @@ class AIAccelerator:
             return False
         return model_id in self.model_registry and self.storage.is_model_loaded(model_id)
         
-    def load_model(self, model_id: str, model: Dict[str, Any], processor: Any = None) -> bool:
-        """Load a model into the accelerator"""
+    def load_model(self, model_id: str, model: Dict[str, Any], 
+                   processor: Any = None, model_config: Dict[str, Any] = None) -> bool:
+        """Load a model into the virtual GPU accelerator
+        
+        Args:
+            model_id: Unique identifier for the model
+            model: Model dictionary containing layer weights and architecture
+            processor: Optional preprocessing/postprocessing functions
+            model_config: Optional model configuration
+        """
         try:
             if not self.storage:
                 raise RuntimeError("No storage available")
                 
-            # Store model in local storage
-            if not self.storage.load_model(model_id, model_data=model):
-                raise RuntimeError("Failed to store model in local storage")
+            # Extract and store model weights in virtual VRAM
+            weights = {}
+            for layer_name, layer_data in model.get("layers", {}).items():
+                # Store weights and biases in virtual VRAM
+                weight_id = f"{model_id}/{layer_name}/weight"
+                if not self.storage.store_tensor(weight_id, layer_data["weight"]):
+                    raise RuntimeError(f"Failed to store weights for layer {layer_name}")
+                weights[layer_name] = {"weight": weight_id}
                 
-            # Update model registry and resource monitor
+                # Store bias if present
+                if "bias" in layer_data:
+                    bias_id = f"{model_id}/{layer_name}/bias"
+                    if not self.storage.store_tensor(bias_id, layer_data["bias"]):
+                        raise RuntimeError(f"Failed to store bias for layer {layer_name}")
+                    weights[layer_name]["bias"] = bias_id
+            
+            # Update model registry with weight references and config
             self.model_registry[model_id] = {
-                'config': model,
+                'weights': weights,
+                'config': model_config or {},
+                'architecture': model.get("architecture", {}),
                 'loaded_at': time.time(),
                 'processor': processor
             }
             
+            # Pre-allocate VRAM if using size limits
+            if hasattr(self.vram, 'pre_allocate_vram'):
+                total_size = sum(
+                    np.prod(layer["weight"].shape) * 4  # Assuming float32
+                    for layer in model.get("layers", {}).values()
+                )
+                if not self.vram.pre_allocate_vram(total_size):
+                    raise RuntimeError("Insufficient VRAM for model weights")
+            
             # Update resource monitoring
             self.resource_monitor['loaded_models'].add(model_id)
-            
-            # Update storage monitoring if available
             if hasattr(self.storage, 'resource_monitor'):
                 self.storage.resource_monitor['loaded_models'].add(model_id)
                 
@@ -126,16 +156,16 @@ class AIAccelerator:
             print(f"Error loading model {model_id}: {str(e)}")
             return False
         
-        # Model registries
-        self.model_registry: Dict[str, Any] = {}
-        self.tokenizer_registry: Dict[str, Any] = {}
-        self.model_configs: Dict[str, Any] = {}  # Store model architectures
-        self.model_loaded = False
+        # # Model registries
+        # self.model_registry: Dict[str, Any] = {}
+        # self.tokenizer_registry: Dict[str, Any] = {}
+        # self.model_configs: Dict[str, Any] = {}  # Store model architectures
+        # self.model_loaded = False
         
-        # Batch processing configuration
-        self.max_batch_size = 64
-        self.min_batch_size = 4
-        self.dynamic_batching = True  # Enable automatic batch size adjustment
+        # # Batch processing configuration
+        # self.max_batch_size = 64
+        # self.min_batch_size = 4
+        # self.dynamic_batching = True  # Enable automatic batch size adjustment
         
     def _serialize_model_config(self, config: Any) -> dict:
         """Convert model config to a serializable format."""
@@ -511,22 +541,56 @@ class AIAccelerator:
             return False
 
     def inference(self, model_id: str, input_tensor_id: str) -> Optional[np.ndarray]:
-        """Run inference using HTTP storage"""
+        """Run PyTorch model inference using virtual GPU acceleration"""
         try:
-            # Load input tensor
+            # Load input tensor from storage
             input_data = self.storage.load_tensor(input_tensor_id)
             if input_data is None:
                 print(f"Could not load input tensor {input_tensor_id}")
                 return None
             
-            # Run inference via HTTP API
-            result = self.storage.start_inference(model_id, input_data)
+            # Convert to PyTorch tensor and move to vGPU
+            from torch_vgpu import to_vgpu
+            input_tensor = to_vgpu(torch.from_numpy(input_data), vram=self.vram)
             
-            if result and result.get('output') is not None:
-                return result['output']
-            else:
-                print(f"Inference failed for model {model_id}")
+            # Get model from registry
+            if not self.has_model(model_id):
+                print(f"Model {model_id} not loaded")
+                return None
+                
+            model_info = self.model_registry[model_id]
+            model = model_info.get("model")
+            if not isinstance(model, torch.nn.Module):
+                print(f"Invalid model type for {model_id}")
                 return None
+            
+            # Move model to vGPU device
+            model = model.to(input_tensor.device)
+            model.eval()
+            
+            # Run inference
+            with torch.no_grad():
+                # Apply any preprocessing from model config
+                if "preprocess" in model_info:
+                    input_tensor = model_info["preprocess"](input_tensor)
+                
+                # Forward pass through model on vGPU
+                output = model(input_tensor)
+                
+                # Apply any postprocessing from model config
+                if "postprocess" in model_info:
+                    output = model_info["postprocess"](output)
+            
+            # Convert output to numpy and store in VRAM
+            output_np = output.cpu().numpy()
+            output_id = f"{model_id}_output_{time.time()}"
+            self.storage.store_tensor(output_id, output_np)
+            
+            # Track compute statistics
+            self.total_compute_time += time.time()
+            self.operations_performed += 1
+            
+            return output_np
                 
         except Exception as e:
             print(f"Error during inference: {str(e)}")

http_storage.py

@@ -46,10 +46,34 @@ class LocalStorage:
         for path in [self.vram_path, self.models_path, self.cache_path, self.state_path]:
             path.mkdir(parents=True, exist_ok=True)
             
+        # Basic state management
         self.lock = threading.Lock()
         self._closing = False
         self._connected = True
         
+        # Resource monitoring
+        self.resource_monitor = {
+            'vram_used': 0,
+            'active_tensors': 0,
+            'loaded_models': set(),
+            'last_updated': time.time()
+        }
+        
+        # Storage statistics
+        self.stats = {
+            'total_size': 0,
+            'available_size': float('inf'),
+            'model_count': 0,
+            'tensor_count': 0
+        }
+        
+        # Initialize registries
+        self.model_registry = {}
+        self.tensor_registry = {}
+        
+        self.initialized = True
+        self._connected = True
+        
         # Initialize monitoring
         self.resource_monitor = {
             'vram_used': 0,
@@ -115,31 +139,25 @@ class LocalStorage:
         The actual initialization happens in __new__ and _init_singleton"""
         pass
 
-    def _create_session(self):
-        """Initialize local storage session"""
+    def _check_storage_ready(self) -> bool:
+        """Check if local storage is ready for use"""
         try:
-            # Create status file to track session
-            status_path = self.base_path / "session_status.json"
-            status_data = {
-                "session_id": self.session_id,
-                "created_at": time.time(),
-                "resource_limits": {
-                    "max_vram_gb": 40,  # A100 size
-                    "max_models": 5,
-                    "max_batch_size": 32
-                }
-            }
-            
-            with open(status_path, 'w') as f:
-                json.dump(status_data, f, indent=2)
+            # Verify all required directories exist and are accessible
+            for path in [self.vram_path, self.models_path, self.cache_path, self.state_path]:
+                if not path.exists() or not os.access(str(path), os.R_OK | os.W_OK):
+                    return False
+                    
+            # Update storage statistics
+            self.stats.update({
+                'total_size': sum(f.stat().st_size for f in self.base_path.rglob('*') if f.is_file()),
+                'model_count': len(list(self.models_path.glob('*'))),
+                'tensor_count': len(list(self.vram_path.glob('*.npy')))
+            })
             
-            logging.info(f"Local storage session created: {self.session_id}")
             return True
             
         except Exception as e:
-            logging.error(f"Failed to create HTTP session: {e}")
-            self.error_count += 1
-            self.last_error_time = time.time()
+            logging.error(f"Storage check failed: {e}")
             return False
 
     def _check_storage(self) -> Dict[str, Any]:
@@ -359,15 +377,9 @@ class LocalStorage:
 
     def ping(self) -> bool:
         """Check if local storage is accessible"""
-        try:
-            # Check if all storage directories exist and are accessible
-            for path in [self.vram_path, self.models_path, self.cache_path, self.state_path]:
-                if not path.exists() or not os.access(str(path), os.R_OK | os.W_OK):
-                    return False
-            return True
-        except Exception as e:
-            logging.error(f"Storage check failed: {e}")
+        if self._closing:
             return False
+        return self._check_storage_ready()
 # Compatibility aliases for existing code
 HTTPGPUStorage = LocalStorage
 WebSocketGPUStorage = LocalStorage

model_inference_flow.txt

@@ -0,0 +1,95 @@
+Model Inference Flow on Virtual GPU
+================================
+
+1. Storage and VRAM Setup
+-------------------------
+[HTTPGPUStorage]  
+      │     ╲
+      │      ╲    Zero-Copy
+      │       ╲   Memory Mapping
+      ▼        ▼
+[Local Storage]──>[Virtual VRAM]
+ (Memory Pages)     (Page Tables)
+      │                  │
+      └──────────────┐  │
+                     ▼  ▼
+                [vGPU Device]
+                     │
+                     ▼
+2. Model Loading and Device Movement
+----------------------------------
+[Florence-2-Large] ---load---> [PyTorch Model]
+         │                          │
+         │                          ▼
+         │                   [to_vgpu() conversion]
+         │                          │
+         └─────────────────┐       │
+                          ▼       ▼
+                    [Model on vGPU Device]
+                           │
+                           ▼
+3. Input Processing and Inference
+--------------------------------
+[Input Text] -----> [Tokenizer] -----> [Tensor]
+                                         │
+                                         ▼
+                              [to_vgpu() conversion]
+                                         │
+                                         ▼
+                               [Tensor on vGPU]
+                                         │
+                                         ▼
+4. Model Inference Flow
+----------------------
+[Model Forward Pass]
+       │
+       ▼
+[vGPU Computation]
+       │
+       ▼
+[PyTorch Output Tensor]
+       │
+       ▼
+[Last Hidden State]
+(Shape: [batch_size, seq_length, hidden_size])
+
+Data Flow and Memory Management:
+-----------------------------
+1. Storage Layer:   
+   - HTTPGPUStorage ──> Local Storage (Memory Pages)
+   - Local Storage ──> Virtual VRAM (Zero-Copy)
+   - Virtual VRAM manages page tables pointing to local storage
+
+2. Memory Architecture:
+   - Local Storage: Physical memory pages
+   - Virtual VRAM: Page tables and memory mappings
+   - Zero-copy between Local Storage and VRAM
+   - Direct memory access for GPU operations
+
+3. Processing Flow:
+   - Model Layer:   HF Model ──> PyTorch ──> vGPU
+   - Input Layer:   Text ──> Tokens ──> Tensor ──> vGPU
+   - Output Layer:  vGPU ──> PyTorch Tensor ──> Results
+
+Key Components:
+--------------
+- HTTP Storage:  HTTPGPUStorage (Network interface)
+- Local Store:   Memory pages (Physical storage)
+- Virtual VRAM:  Page tables (Memory management)
+- Device:        vGPU (Computation)
+- Model:         Florence-2-Large (transformer)
+- Framework:     PyTorch (ML operations)
+- Interface:     to_vgpu() (Zero-copy transfer)
+
+Memory Management Details:
+------------------------
+1. Local Storage:
+   - Manages physical memory pages
+   - Direct mapping to virtual VRAM
+   - Zero-copy access for GPU ops
+
+2. Virtual VRAM:
+   - Page table management
+   - Memory mapping to local storage
+   - No physical copying of data
+   - Direct GPU access to memory