Upload folder using huggingface_hub

README_HF.md

@@ -0,0 +1,39 @@
+# Advanced Tokenizer System for LiMp
+
+## 🧠 Overview
+Sophisticated multi-modal tokenization system with semantic awareness, mathematical processing, and fractal-based tokenization.
+
+## 🚀 Key Features
+- **Multi-Modal Tokenization**: Traditional, semantic, mathematical, and fractal
+- **High Capacity Processing**: Handles unlimited character counts
+- **Intelligent Chunking**: Semantic-aware with context preservation
+- **Batch Processing**: High-performance parallel processing
+- **Training Data Generation**: Creates high-quality training datasets
+- **Mathematical AI**: Advanced mathematical expression processing
+
+## 🛠 Quick Start
+```python
+from advanced_tokenizer_system import AdvancedTokenizer, TokenizerConfig
+
+config = TokenizerConfig()
+tokenizer = AdvancedTokenizer(config)
+
+import asyncio
+result = await tokenizer.tokenize("Hello world! x^2 + y^2 = z^2")
+print(f"Tokens: {result.total_tokens}")
+```
+
+## 📁 Files
+- `advanced_tokenizer_system.py` - Main tokenizer
+- `batch_processing_system.py` - Batch processing
+- `high_capacity_input_processor.py` - Large text processing
+- `intelligent_chunking_processor.py` - Smart chunking
+- `advanced_training_data_generator.py` - Training data
+- `matrix_training_data.jsonl` - Sample data
+
+## 🧪 Test
+```bash
+python3 working_test.py
+```
+
+Ready for advanced AI tokenization! 🚀

requirements.txt

@@ -1,58 +1,14 @@
-# Numbskull - Advanced AI Embedding Pipeline Requirements
-# Core dependencies for the sophisticated multi-modal embedding system
-# Updated: October 2024 - Pinned to latest secure versions
-
-# Core scientific computing
-numpy==2.3.3  # Updated from >=1.24.0
-scipy==1.16.2  # Updated from >=1.10.0
-
-# Mathematical processing
-sympy==1.14.0  # Updated from >=1.12
-matplotlib==3.10.7  # Updated from >=3.7.0
-
-# Machine learning
-scikit-learn==1.7.2  # Updated from >=1.3.0
-
-# Async HTTP and networking
-httpx==0.28.1  # Updated from >=0.24.0 - includes security fixes
-aiofiles==25.1.0  # Updated from >=23.2.1
-
-# Database connectivity
-asyncpg==0.30.0  # Updated from >=0.28.0
-psycopg2-binary==2.9.11  # Updated from >=2.9.0 - includes security patches
-
-# Data processing
-pandas==2.3.3  # Updated from >=2.0.0
-pydantic==2.12.0  # Updated from >=2.0.0 - includes validation improvements
-
-# Web framework (for API endpoints)
-fastapi==0.118.3  # Updated from >=0.100.0 - includes security fixes
-uvicorn==0.37.0  # Updated from >=0.23.0 - includes security updates
-
-# Utilities
-python-dateutil==2.9.0.post0  # Updated from >=2.8.0
-python-multipart==0.0.20  # Updated from >=0.0.6
-
-# Development and testing
-pytest==8.4.2  # Updated from >=7.4.0
-pytest-asyncio==1.2.0  # Updated from >=0.21.0
-black==25.9.0  # Updated from >=23.0.0
-flake8==7.3.0  # Updated from >=6.0.0
-
-# Graph/complex networks for emergent modules
-networkx==3.5  # Updated from >=3.1
-
-# Optional dependencies (install separately if needed)
-# sentence-transformers>=2.2.0
-# transformers>=4.30.0
-# torch>=2.0.0
-# faiss-cpu>=1.7.4
-# annoy>=1.17.0
-# hnswlib>=0.7.0
-
-# Numbskull integration - Advanced embedding pipeline
-# Install as editable package from local path
--e /home/kill/numbskull
-
-# Additional dependency for HTTP requests in dual orchestrator
-requests>=2.31.0
+numpy>=1.21.0
+torch>=1.9.0
+asyncio
+pathlib
+dataclasses
+typing
+datetime
+json
+hashlib
+re
+multiprocessing
+threading
+queue
+psutil