USER_GUIDE.md

# NullAI Phi-4 14B (v2) - Complete User Guide

**Complete guide to using the NullAI knowledge management system**

[English](#english) | [日本語](#japanese)

---

<a name="english"></a>
# English Guide

## Table of Contents

1. [Installation](#installation)
2. [Quick Start](#quick-start)
3. [Knowledge Tiles](#knowledge-tiles)
4. [3D Spatial Memory](#3d-spatial-memory)
5. [API Reference](#api-reference)
6. [Fine-Tuning & Training](#fine-tuning)
7. [Web Editor](#web-editor)
8. [Data Import & Export](#data-import-export) 📦 **NEW**
9. [Advanced Features](#advanced-features)
10. [Troubleshooting](#troubleshooting)
11. [Best Practices](#best-practices)

---

## 1. Installation {#installation}

### One-Command Setup

```bash
curl -sSL https://huggingface.co/kofdai/nullai-phi-4-14b-v2/raw/main/setup.sh | bash
```

This will:
- Download the 27GB model file
- Install all dependencies
- Setup backend and frontend
- Initialize the database
- Create startup scripts

### Manual Installation

```bash
# Clone repository
git clone https://huggingface.co/kofdai/nullai-phi-4-14b-v2
cd nullai-phi-4-14b-v2

# Run setup script
chmod +x setup.sh
./setup.sh
```

### System Requirements

- **RAM**: 32GB minimum, 64GB recommended
- **Disk**: 40GB free space
- **OS**: macOS, Linux, or WSL2 on Windows
- **Python**: 3.9 or higher
- **Node.js**: 18 or higher (optional, for frontend)

---

## 2. Quick Start {#quick-start}

### Starting the System

```bash
cd ~/nullai/nullai-phi-4-14b-v2

# Start everything
./start_nullai.sh

# Or start components separately
./start_backend.sh    # API server on port 8000
./start_frontend.sh   # Web UI on port 5173
```

### Your First Knowledge Tile

#### Using Python API

```python
import requests

# Create a knowledge tile
tile_data = {
    "domain_id": "medical",
    "topic": "Synaptic Plasticity",
    "content": "Synaptic plasticity refers to the ability of synapses to strengthen or weaken over time, in response to increases or decreases in their activity.",
    "coordinates": {
        "x": 0.6,  # Abstraction (0=concrete, 1=abstract)
        "y": 0.7,  # Expertise (0=basic, 1=advanced)
        "z": 0.3   # Temporality (0=timeless, 1=current)
    },
    "verification_type": "community_verified"
}

response = requests.post(
    "http://localhost:8000/api/knowledge/tiles",
    json=tile_data,
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

print(response.json())
```

#### Using Web Editor

1. Open https://dendritic-memory-editor.pages.dev
2. Fill in tile information
3. Visualize in 3D space
4. Export as .iath file
5. Import via API: `POST /api/knowledge/import/iath`

### Testing the Model

```python
from llama_cpp import Llama

llm = Llama(
    model_path="~/nullai/nullai-phi-4-14b-v2/phi-4-f16.gguf",
    n_ctx=16384,
    n_threads=8,
    n_gpu_layers=-1  # Use GPU if available
)

response = llm(
    "Explain synaptic plasticity in simple terms.",
    max_tokens=256,
    temperature=0.7
)

print(response['choices'][0]['text'])
```

---

## 3. Knowledge Tiles {#knowledge-tiles}

### Tile Structure

```json
{
  "id": "tile_abc123",
  "domain_id": "medical",
  "topic": "Neuroplasticity",
  "content": "Detailed explanation...",
  "coordinates": {
    "x": 0.5,  // Abstraction axis
    "y": 0.8,  // Expertise axis
    "z": 0.2   // Temporality axis
  },
  "certainty_score": 0.95,
  "verification_type": "expert_verified",
  "orcid_verified": true,
  "expert_id": "0000-0002-1234-5678",
  "reasoning_chain": [
    {
      "step": 1,
      "content": "Based on research...",
      "certainty": 0.9
    }
  ],
  "citations": [
    {
      "title": "Neural Mechanisms...",
      "authors": "Smith et al.",
      "year": 2023,
      "doi": "10.1234/example"
    }
  ],
  "created_at": "2025-12-03T10:00:00Z",
  "updated_at": "2025-12-03T10:00:00Z"
}
```

### Creating Tiles

#### API Endpoint

```bash
POST /api/knowledge/tiles
Content-Type: application/json
Authorization: Bearer YOUR_TOKEN

{
  "domain_id": "medical",
  "topic": "Your Topic",
  "content": "Your detailed content...",
  "coordinates": {"x": 0.5, "y": 0.5, "z": 0.5},
  "verification_type": "community_verified"
}
```

#### Response

```json
{
  "success": true,
  "tile_id": "tile_abc123",
  "message": "Tile created successfully",
  "coordinates": {"x": 0.5, "y": 0.5, "z": 0.5}
}
```

### Retrieving Tiles

#### Get All Tiles

```bash
GET /api/knowledge/tiles?domain=medical&limit=50&offset=0
```

#### Get Specific Tile

```bash
GET /api/knowledge/tiles/{tile_id}
```

#### Spatial Search

```python
# Find tiles near a coordinate
response = requests.post(
    "http://localhost:8000/api/knowledge/search/spatial",
    json={
        "center": {"x": 0.5, "y": 0.7, "z": 0.3},
        "radius": 0.2,
        "domain": "medical"
    }
)
```

### Updating Tiles

```bash
PUT /api/knowledge/tiles/{tile_id}
Content-Type: application/json

{
  "content": "Updated content...",
  "coordinates": {"x": 0.6, "y": 0.8, "z": 0.3}
}
```

### Deleting Tiles

```bash
DELETE /api/knowledge/tiles/{tile_id}
```

---

## 4. 3D Spatial Memory {#3d-spatial-memory}

### Understanding the 3D Coordinate System

```
         Z (Temporality)
         ↑
         |     ╱ z=1.0 (Current, rapidly changing)
         |   ╱
         | ╱______ z=0.0 (Timeless, unchanging)
         |╱
         O────────→ Y (Expertise)
        ╱|          y=0.0 (Basic) → y=1.0 (Advanced)
      ╱  |
    ╱    ↓
   X (Abstraction)
   x=0.0 (Concrete) → x=1.0 (Abstract)
```

### Coordinate Guidelines

#### X-Axis: Abstraction Level
- **0.0 - 0.3**: Concrete (specific examples, case studies)
- **0.3 - 0.7**: Moderate (general principles with examples)
- **0.7 - 1.0**: Abstract (theoretical concepts, philosophical)

**Example:**
- `x=0.1`: "The heart pumps blood through arteries"
- `x=0.5`: "Cardiovascular system function and regulation"
- `x=0.9`: "Principles of biological fluid dynamics"

#### Y-Axis: Expertise Level
- **0.0 - 0.3**: Beginner (introductory, basic concepts)
- **0.3 - 0.7**: Intermediate (requires background knowledge)
- **0.7 - 1.0**: Expert (advanced, specialized knowledge)

**Example:**
- `y=0.2`: "What is machine learning?"
- `y=0.5`: "Gradient descent optimization"
- `y=0.9`: "Novel architectures in attention mechanisms"

#### Z-Axis: Temporality
- **0.0 - 0.3**: Timeless (fundamental truths, unchanging)
- **0.3 - 0.7**: Moderate (evolving but stable)
- **0.7 - 1.0**: Current (latest research, rapidly changing)

**Example:**
- `z=0.1`: "Water is H2O"
- `z=0.5`: "Current understanding of quantum computing"
- `z=0.9`: "Latest COVID-19 variant information"

### Spatial Search Strategies

#### Proximity Search
Find tiles near a specific coordinate:

```python
def search_nearby(center, radius=0.2):
    response = requests.post(
        "http://localhost:8000/api/knowledge/search/spatial",
        json={
            "center": center,
            "radius": radius,
            "limit": 20
        }
    )
    return response.json()

# Example: Find intermediate medical knowledge
results = search_nearby(
    center={"x": 0.5, "y": 0.5, "z": 0.3},
    radius=0.2
)
```

#### Progressive Learning Path
Navigate from beginner to expert:

```python
# Start with basics
basic_tiles = search_nearby({"x": 0.3, "y": 0.2, "z": 0.3})

# Progress to intermediate
intermediate_tiles = search_nearby({"x": 0.5, "y": 0.5, "z": 0.3})

# Advance to expert
expert_tiles = search_nearby({"x": 0.7, "y": 0.8, "z": 0.3})
```

---

## 5. API Reference {#api-reference}

### Authentication

#### Register User

```bash
POST /api/auth/register
Content-Type: application/json

{
  "username": "your_username",
  "email": "your@email.com",
  "password": "secure_password"
}
```

#### Login

```bash
POST /api/auth/login
Content-Type: application/json

{
  "username": "your_username",
  "password": "your_password"
}
```

Response:
```json
{
  "access_token": "eyJ0eXAiOiJKV1QiLCJhbGc...",
  "token_type": "bearer"
}
```

### Knowledge Tile Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/knowledge/tiles` | List all tiles |
| GET | `/api/knowledge/tiles/{id}` | Get specific tile |
| POST | `/api/knowledge/tiles` | Create new tile |
| PUT | `/api/knowledge/tiles/{id}` | Update tile |
| DELETE | `/api/knowledge/tiles/{id}` | Delete tile |
| POST | `/api/knowledge/search/spatial` | Spatial search |
| POST | `/api/knowledge/import/iath` | Import .iath file |
| GET | `/api/knowledge/export/iath` | Export as .iath |

### Domain Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/domains` | List all domains |
| GET | `/api/domains/{id}` | Get domain info |
| POST | `/api/domains` | Create domain |
| PUT | `/api/domains/{id}` | Update domain |

### Training Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/training/start` | Start fine-tuning |
| GET | `/api/training/status` | Check training status |
| GET | `/api/training/history` | Training history |
| POST | `/api/training/export` | Export trained model |

### Apprentice Model Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/api/apprentice/status` | Get apprentice status |
| POST | `/api/apprentice/train` | Train apprentice model |
| GET | `/api/apprentice/threshold` | Check threshold status |
| POST | `/api/apprentice/export` | Export apprentice model |

---

## 6. Fine-Tuning & Training {#fine-tuning}

### Overview

NullAI uses LoRA (Low-Rank Adaptation) for efficient fine-tuning of the Phi-4 model on your knowledge tiles.

### Training Workflow

```
1. Create Knowledge Tiles
   ↓
2. Generate Training Data
   ↓
3. Start LoRA Fine-Tuning
   ↓
4. Monitor Training Progress
   ↓
5. Export Adapted Model
   ↓
6. Deploy Apprentice Model
```

### Starting Training

```python
import requests

# Prepare training configuration
training_config = {
    "domain": "medical",
    "epochs": 3,
    "learning_rate": 0.0003,
    "batch_size": 4,
    "lora_rank": 8,
    "lora_alpha": 16,
    "target_modules": ["q_proj", "v_proj"]
}

# Start training
response = requests.post(
    "http://localhost:8000/api/training/start",
    json=training_config,
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

training_id = response.json()["training_id"]
print(f"Training started: {training_id}")
```

### Monitoring Progress

```python
import time

while True:
    response = requests.get(
        f"http://localhost:8000/api/training/status",
        params={"training_id": training_id}
    )

    status = response.json()
    print(f"Progress: {status['progress']}%")
    print(f"Loss: {status['current_loss']}")

    if status['status'] == 'completed':
        print("Training completed!")
        break

    time.sleep(30)  # Check every 30 seconds
```

### Apprentice Model Threshold System

The apprentice model automatically tracks when it has accumulated enough training to "self-identify" as a specialized model.

```python
# Check threshold status
response = requests.get("http://localhost:8000/api/apprentice/status")
status = response.json()

print(f"Training examples: {status['training_examples']}")
print(f"Threshold: {status['threshold']}")
print(f"Can self-identify: {status['can_self_identify']}")

if status['can_self_identify']:
    # Export as independent model
    export_response = requests.post(
        "http://localhost:8000/api/apprentice/export",
        json={
            "model_name": "nullai-medical-specialist",
            "format": "gguf",
            "quantization": "Q4_K_M"
        }
    )
    print(f"Model exported: {export_response.json()['output_file']}")
```

---

## 7. Web Editor {#web-editor}

### Using the Dendritic Memory Editor

The web-based editor provides a visual interface for creating and managing knowledge tiles.

**URL**: https://dendritic-memory-editor.pages.dev

### Features

#### 1. Visual Tile Creation
- Form-based interface
- Real-time validation
- Preview before saving

#### 2. 3D Coordinate Visualizer
- Interactive 3D space
- Drag-and-drop tile positioning
- Visual understanding of spatial relationships

#### 3. Import/Export
- Load existing .iath files
- Export to .iath format
- Batch operations

#### 4. API Integration
- Direct upload to NullAI backend
- One-click import
- Automatic coordinate adjustment

### Workflow

```
1. Open Web Editor
   ↓
2. Create New Tile or Import
   ↓
3. Fill in Content
   - Topic
   - Content
   - Domain
   - Coordinates
   ↓
4. Visualize in 3D Space
   ↓
5. Adjust Coordinates (optional)
   ↓
6. Export as .iath
   ↓
7. Import to NullAI Backend
```

### Example: Creating a Tile in Web Editor

1. **Open Editor**: Navigate to https://dendritic-memory-editor.pages.dev

2. **Select Domain**: Choose "Medical"

3. **Enter Information**:
   - Topic: "Dendritic Spine Function"
   - Content: "Dendritic spines are small membranous protrusions..."
   - Coordinates: x=0.6, y=0.7, z=0.3

4. **Preview in 3D**: See where your tile appears in the knowledge space

5. **Export**: Click "Export .iath"

6. **Import to NullAI**:
   ```bash
   curl -X POST http://localhost:8000/api/knowledge/import/iath \
     -H "Authorization: Bearer YOUR_TOKEN" \
     -F "file=@dendritic_spine.iath"
   ```

---

## 8. Data Import & Export {#data-import-export}

### Overview

NullAI supports importing and exporting knowledge tiles in `.iath` format (Ilm-Athens format), enabling seamless data exchange between different instances and the dendritic-memory-editor application.

### Quick Start with Import Tools

We provide three methods for importing `.iath` files:

#### Method 1: Python Script (Recommended)

```bash
# Single file
python import_iath.py /path/to/Medical_tiles.iath

# Multiple files
python import_iath.py /path/to/*.iath
```

#### Method 2: Batch Import Script

```bash
# Import all .iath files from Downloads folder
./batch_import_iath.sh

# Import from custom directory
./batch_import_iath.sh /path/to/directory
```

#### Method 3: cURL (Manual)

```bash
# Get auth token
TOKEN=$(curl -X POST http://localhost:8000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "your@email.com", "password": "password"}' \
  | jq -r '.access_token')

# Import file
curl -X POST http://localhost:8000/api/knowledge/import/iath \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@Medical_tiles.iath"
```

### Export Data

```bash
# Export all domains
curl -X GET "http://localhost:8000/api/knowledge/export/iath" \
  -H "Authorization: Bearer $TOKEN" \
  -o exported_all.iath

# Export specific domain
curl -X GET "http://localhost:8000/api/knowledge/export/iath?domain_id=medical" \
  -H "Authorization: Bearer $TOKEN" \
  -o exported_medical.iath
```

### Database Configuration

To change the database used by the import system:

**Create `backend/.env` file:**

```bash
# SQLite (default)
DATABASE_URL=sqlite:///./sql_app.db

# PostgreSQL
DATABASE_URL=postgresql://username:password@localhost:5432/nullai_db

# MySQL
DATABASE_URL=mysql://username:password@localhost:3306/nullai_db
```

### Detailed Documentation

📖 **For complete documentation, see:** [IATH_IMPORT_GUIDE.md](./IATH_IMPORT_GUIDE.md)

This comprehensive guide includes:
- Detailed setup instructions
- All import methods with examples
- Database switching procedures
- Troubleshooting common issues
- Frontend integration examples

---

## 9. Advanced Features {#advanced-features}

### ORCID Expert Verification

Link your tiles to your ORCID profile for expert verification.

```python
# Register ORCID
response = requests.post(
    "http://localhost:8000/api/oauth/orcid/register",
    json={
        "orcid_id": "0000-0002-1234-5678",
        "access_token": "your_orcid_token"
    }
)

# Create expert-verified tile
tile_data = {
    "domain_id": "medical",
    "topic": "Advanced Neuroscience",
    "content": "Expert knowledge...",
    "verification_type": "expert_verified",
    "orcid_verified": True
}
```

### Reasoning Chains

Add transparent reasoning to your tiles:

```python
tile_with_reasoning = {
    "domain_id": "logic_reasoning",
    "topic": "Logical Deduction",
    "content": "If A implies B, and B implies C...",
    "reasoning_chain": [
        {
            "step": 1,
            "content": "Given: A → B (if A then B)",
            "certainty": 1.0,
            "reasoning_type": "premise"
        },
        {
            "step": 2,
            "content": "Given: B → C (if B then C)",
            "certainty": 1.0,
            "reasoning_type": "premise"
        },
        {
            "step": 3,
            "content": "Therefore: A → C (if A then C)",
            "certainty": 1.0,
            "reasoning_type": "conclusion"
        }
    ]
}
```

### Multi-Stage Judge System

NullAI implements a three-stage verification system:

#### Alpha Lobe (Logic Judge)
```python
# Evaluate logical consistency
response = requests.post(
    "http://localhost:8000/api/judge/alpha",
    json={
        "tile_id": "tile_abc123",
        "check_type": "logical_consistency"
    }
)
# Returns: logical_score, inconsistencies, suggestions
```

#### Beta Lobe Basic (Domain Judge)
```python
# Check domain expertise
response = requests.post(
    "http://localhost:8000/api/judge/beta-basic",
    json={
        "tile_id": "tile_abc123",
        "domain": "medical"
    }
)
# Returns: domain_accuracy, expert_level, corrections
```

#### Beta Lobe Advanced (Reasoning Judge)
```python
# Validate reasoning chain
response = requests.post(
    "http://localhost:8000/api/judge/beta-advanced",
    json={
        "tile_id": "tile_abc123",
        "check_reasoning": True
    }
)
# Returns: reasoning_validity, confidence, improvements
```

### Database Enrichment

Automatically enhance your knowledge base:

```python
# Start enrichment process
response = requests.post(
    "http://localhost:8000/api/enrichment/start",
    json={
        "domain": "medical",
        "target_tile_count": 10000,
        "use_master_model": True,
        "auto_verify": True
    }
)

enrichment_id = response.json()["enrichment_id"]

# Monitor progress
status = requests.get(
    f"http://localhost:8000/api/enrichment/status/{enrichment_id}"
)
```

### Workspace Management

Organize tiles into workspaces for different projects:

```python
# Create workspace
workspace = requests.post(
    "http://localhost:8000/api/workspaces",
    json={
        "name": "Neuroscience Research",
        "description": "Collaborative neuroscience knowledge",
        "domains": ["medical", "ai_fundamentals"]
    }
)

workspace_id = workspace.json()["id"]

# Add tiles to workspace
requests.post(
    f"http://localhost:8000/api/workspaces/{workspace_id}/tiles",
    json={"tile_ids": ["tile_1", "tile_2", "tile_3"]}
)
```

---

## 10. Troubleshooting {#troubleshooting}

### Common Issues

#### Model Loading Errors

**Problem**: "Out of memory" error when loading model

**Solution**:
```python
# Reduce context window
llm = Llama(
    model_path="phi-4-f16.gguf",
    n_ctx=4096,  # Instead of 16384
    n_gpu_layers=0  # Use CPU only if GPU memory insufficient
)
```

#### API Connection Refused

**Problem**: Cannot connect to `localhost:8000`

**Solution**:
```bash
# Check if backend is running
curl http://localhost:8000/health

# If not, start it
cd ~/nullai/nullai-phi-4-14b-v2
./start_backend.sh
```

#### Authentication Errors

**Problem**: "401 Unauthorized" on API calls

**Solution**:
```python
# Get fresh token
login_response = requests.post(
    "http://localhost:8000/api/auth/login",
    json={"username": "your_user", "password": "your_pass"}
)
token = login_response.json()["access_token"]

# Use in requests
headers = {"Authorization": f"Bearer {token}"}
```

#### Database Locked

**Problem**: "Database is locked" error

**Solution**:
```bash
# Stop all instances
pkill -f "uvicorn"

# Restart
./start_backend.sh
```

#### Training Fails

**Problem**: LoRA training crashes

**Solution**:
```python
# Reduce batch size and parameters
training_config = {
    "batch_size": 1,  # Reduce from 4
    "lora_rank": 4,   # Reduce from 8
    "gradient_accumulation_steps": 4  # Compensate for smaller batch
}
```

### Performance Optimization

#### Speed Up Inference

```python
# Use GPU acceleration
llm = Llama(
    model_path="phi-4-f16.gguf",
    n_gpu_layers=35,  # Offload all layers to GPU
    n_batch=512,      # Larger batch for GPU
    n_threads=1       # Use fewer CPU threads when using GPU
)
```

#### Reduce Memory Usage

```python
# Use quantized model (when available)
# Or reduce context window
llm = Llama(
    model_path="phi-4-f16.gguf",
    n_ctx=2048,  # Smaller context = less memory
    use_mmap=True,  # Use memory mapping
    use_mlock=False  # Don't lock in RAM
)
```

---

## 11. Best Practices {#best-practices}

### Knowledge Tile Creation

#### ✅ DO:
- Be specific and factual
- Include sources and citations
- Use appropriate coordinates
- Add reasoning chains for complex topics
- Update tiles when information changes

#### ❌ DON'T:
- Mix multiple topics in one tile
- Use vague or ambiguous language
- Claim certainty without evidence
- Duplicate existing tiles
- Skip coordinate assignment

### Coordinate Assignment

**General Guidelines:**

1. **Start Conservative**: Begin with moderate coordinates (0.4-0.6) and adjust

2. **Test Placement**: Use spatial search to see nearby tiles

3. **Consider Audience**:
   - Teaching beginners? y=0.2-0.4
   - Research papers? y=0.7-0.9

4. **Time Sensitivity**:
   - Historical facts: z=0.1-0.3
   - Latest research: z=0.7-0.9

### Training Best Practices

1. **Data Quality Over Quantity**
   - 100 high-quality tiles > 1000 mediocre tiles
   - Ensure diverse coverage of domain

2. **Incremental Training**
   - Start with small epochs (1-2)
   - Evaluate before continuing
   - Save checkpoints frequently

3. **Threshold Management**
   - Monitor apprentice model progress
   - Don't rush self-identification
   - Validate before exporting

### API Usage

1. **Authentication**
   - Store tokens securely
   - Refresh before expiration
   - Use environment variables

2. **Rate Limiting**
   - Implement exponential backoff
   - Batch operations when possible
   - Cache frequently accessed tiles

3. **Error Handling**
   ```python
   import time

   def api_call_with_retry(url, max_retries=3):
       for attempt in range(max_retries):
           try:
               response = requests.get(url, timeout=10)
               response.raise_for_status()
               return response.json()
           except requests.exceptions.RequestException as e:
               if attempt == max_retries - 1:
                   raise
               wait_time = 2 ** attempt
               time.sleep(wait_time)
   ```

### Security

1. **Protect Credentials**
   ```bash
   # Use environment variables
   export NULLAI_TOKEN="your_token_here"
   export NULLAI_API_URL="http://localhost:8000"
   ```

2. **HTTPS in Production**
   - Never use HTTP for production
   - Use SSL certificates
   - Enable CORS properly

3. **Input Validation**
   - Sanitize all inputs
   - Validate coordinates (0-1 range)
   - Check file uploads

### Collaboration

1. **Workspace Organization**
   - Create project-specific workspaces
   - Assign clear roles
   - Document conventions

2. **Version Control**
   - Export workspaces regularly
   - Keep .iath files in git
   - Document changes

3. **Expert Verification**
   - Use ORCID for credentials
   - Peer review important tiles
   - Maintain audit trail

---

## Support & Community

- **Documentation**: This guide + README.md
- **API Docs**: http://localhost:8000/docs (when running)
- **Web Editor**: https://dendritic-memory-editor.pages.dev
- **Model Hub**: https://huggingface.co/kofdai/nullai-phi-4-14b-v2

---

<a name="japanese"></a>
# 日本語ガイド

## 目次

1. [インストール](#インストール)
2. [クイックスタート](#クイックスタート-ja)
3. [知識タイル](#知識タイル)
4. [3D空間記憶](#3d空間記憶)
5. [APIリファレンス](#apiリファレンス)
6. [ファインチューニング](#ファインチューニング)
7. [ウェブエディター](#ウェブエディター)
8. [データのインポート&エクスポート](#データのインポートエクスポート) 📦 **NEW**
9. [高度な機能](#高度な機能)
10. [トラブルシューティング](#トラブルシューティング-ja)
11. [ベストプラクティス](#ベストプラクティス-ja)

---

## 1. インストール {#インストール}

### ワンコマンドセットアップ

```bash
curl -sSL https://huggingface.co/kofdai/nullai-phi-4-14b-v2/raw/main/setup.sh | bash
```

以下が自動実行されます：
- 27GBモデルファイルのダウンロード
- 全依存関係のインストール
- バックエンドとフロントエンドのセットアップ
- データベースの初期化
- 起動スクリプトの作成

### 手動インストール

```bash
# リポジトリのクローン
git clone https://huggingface.co/kofdai/nullai-phi-4-14b-v2
cd nullai-phi-4-14b-v2

# セットアップスクリプトの実行
chmod +x setup.sh
./setup.sh
```

### システム要件

- **RAM**: 最小32GB、推奨64GB
- **ディスク**: 40GB以上の空き容量
- **OS**: macOS、Linux、またはWindows WSL2
- **Python**: 3.9以上
- **Node.js**: 18以上（フロントエンド用、オプション）

---

## 2. クイックスタート {#クイックスタート-ja}

### システムの起動

```bash
cd ~/nullai/nullai-phi-4-14b-v2

# すべてを起動
./start_nullai.sh

# または個別に起動
./start_backend.sh    # ポート8000でAPIサーバー
./start_frontend.sh   # ポート5173でWeb UI
```

### 最初の知識タイル

#### Python APIを使用

```python
import requests

# 知識タイルの作成
tile_data = {
    "domain_id": "medical",
    "topic": "シナプス可塑性",
    "content": "シナプス可塑性とは、活動の増減に応じてシナプスが時間とともに強化または弱化する能力を指します。",
    "coordinates": {
        "x": 0.6,  # 抽象度 (0=具体的, 1=抽象的)
        "y": 0.7,  # 専門性 (0=基礎, 1=高度)
        "z": 0.3   # 時間性 (0=普遍的, 1=最新)
    },
    "verification_type": "community_verified"
}

response = requests.post(
    "http://localhost:8000/api/knowledge/tiles",
    json=tile_data,
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

print(response.json())
```

#### ウェブエディターを使用

1. https://dendritic-memory-editor.pages.dev を開く
2. タイル情報を入力
3. 3D空間で可視化
4. .iathファイルとしてエクスポート
5. APIで インポート: `POST /api/knowledge/import/iath`

### モデルのテスト

```python
from llama_cpp import Llama

llm = Llama(
    model_path="~/nullai/nullai-phi-4-14b-v2/phi-4-f16.gguf",
    n_ctx=16384,
    n_threads=8,
    n_gpu_layers=-1  # GPUがあれば使用
)

response = llm(
    "シナプス可塑性を簡単に説明してください。",
    max_tokens=256,
    temperature=0.7
)

print(response['choices'][0]['text'])
```

---

## 3. 知識タイル {#知識タイル}

### タイル構造

```json
{
  "id": "tile_abc123",
  "domain_id": "medical",
  "topic": "神経可塑性",
  "content": "詳細な説明...",
  "coordinates": {
    "x": 0.5,  // 抽象度軸
    "y": 0.8,  // 専門性軸
    "z": 0.2   // 時間性軸
  },
  "certainty_score": 0.95,
  "verification_type": "expert_verified",
  "orcid_verified": true,
  "expert_id": "0000-0002-1234-5678",
  "reasoning_chain": [
    {
      "step": 1,
      "content": "研究に基づき...",
      "certainty": 0.9
    }
  ],
  "citations": [
    {
      "title": "神経メカニズム...",
      "authors": "Smith et al.",
      "year": 2023,
      "doi": "10.1234/example"
    }
  ],
  "created_at": "2025-12-03T10:00:00Z",
  "updated_at": "2025-12-03T10:00:00Z"
}
```

### タイルの作成

#### APIエンドポイント

```bash
POST /api/knowledge/tiles
Content-Type: application/json
Authorization: Bearer YOUR_TOKEN

{
  "domain_id": "medical",
  "topic": "あなたのトピック",
  "content": "詳細な内容...",
  "coordinates": {"x": 0.5, "y": 0.5, "z": 0.5},
  "verification_type": "community_verified"
}
```

### 座標ガイドライン

#### X軸：抽象度レベル
- **0.0 - 0.3**: 具体的（具体例、ケーススタディ）
- **0.3 - 0.7**: 中程度（例を含む一般原則）
- **0.7 - 1.0**: 抽象的（理論的概念、哲学的）

#### Y軸：専門性レベル
- **0.0 - 0.3**: 初心者（入門、基本概念）
- **0.3 - 0.7**: 中級者（予備知識が必要）
- **0.7 - 1.0**: 専門家（高度、専門知識）

#### Z軸：時間性
- **0.0 - 0.3**: 普遍的（基本真理、不変）
- **0.3 - 0.7**: 中程度（進化中だが安定）
- **0.7 - 1.0**: 最新（最新研究、急速に変化）

---

## 4. 3D空間記憶 {#3d空間記憶}

### 3D座標系の理解

```
         Z (時間性)
         ↑
         |     ╱ z=1.0 (最新、急速に変化)
         |   ╱
         | ╱______ z=0.0 (普遍的、不変)
         |╱
         O────────→ Y (専門性)
        ╱|          y=0.0 (基礎) → y=1.0 (高度)
      ╱  |
    ╱    ↓
   X (抽象度)
   x=0.0 (具体的) → x=1.0 (抽象的)
```

### 空間検索戦略

#### 近接検索

```python
def search_nearby(center, radius=0.2):
    response = requests.post(
        "http://localhost:8000/api/knowledge/search/spatial",
        json={
            "center": center,
            "radius": radius,
            "limit": 20
        }
    )
    return response.json()

# 例：中級医療知識を検索
results = search_nearby(
    center={"x": 0.5, "y": 0.5, "z": 0.3},
    radius=0.2
)
```

---

## 5. APIリファレンス {#apiリファレンス}

### 認証

#### ユーザー登録

```bash
POST /api/auth/register
Content-Type: application/json

{
  "username": "your_username",
  "email": "your@email.com",
  "password": "secure_password"
}
```

#### ログイン

```bash
POST /api/auth/login
Content-Type: application/json

{
  "username": "your_username",
  "password": "your_password"
}
```

### 知識タイルエンドポイント

| メソッド | エンドポイント | 説明 |
|----------|---------------|------|
| GET | `/api/knowledge/tiles` | 全タイル一覧 |
| GET | `/api/knowledge/tiles/{id}` | 特定タイル取得 |
| POST | `/api/knowledge/tiles` | 新規タイル作成 |
| PUT | `/api/knowledge/tiles/{id}` | タイル更新 |
| DELETE | `/api/knowledge/tiles/{id}` | タイル削除 |
| POST | `/api/knowledge/search/spatial` | 空間検索 |
| POST | `/api/knowledge/import/iath` | .iathファイルインポート |
| GET | `/api/knowledge/export/iath` | .iath形式でエクスポート |

---

## 6. ファインチューニング {#ファインチューニング}

### 概要

NullAIはLoRA（Low-Rank Adaptation）を使用して、Phi-4モデルを知識タイルで効率的にファインチューニングします。

### トレーニングの開始

```python
import requests

# トレーニング設定の準備
training_config = {
    "domain": "medical",
    "epochs": 3,
    "learning_rate": 0.0003,
    "batch_size": 4,
    "lora_rank": 8,
    "lora_alpha": 16,
    "target_modules": ["q_proj", "v_proj"]
}

# トレーニング開始
response = requests.post(
    "http://localhost:8000/api/training/start",
    json=training_config,
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

training_id = response.json()["training_id"]
print(f"トレーニング開始: {training_id}")
```

### 進捗のモニタリング

```python
import time

while True:
    response = requests.get(
        f"http://localhost:8000/api/training/status",
        params={"training_id": training_id}
    )

    status = response.json()
    print(f"進捗: {status['progress']}%")
    print(f"損失: {status['current_loss']}")

    if status['status'] == 'completed':
        print("トレーニング完了！")
        break

    time.sleep(30)  # 30秒ごとにチェック
```

### 弟子モデル閾値システム

弟子モデルは、特化モデルとして「自己識別」するのに十分なトレーニングが蓄積されたときを自動的に追跡します。

```python
# 閾値ステータスの確認
response = requests.get("http://localhost:8000/api/apprentice/status")
status = response.json()

print(f"トレーニング例: {status['training_examples']}")
print(f"閾値: {status['threshold']}")
print(f"自己識別可能: {status['can_self_identify']}")

if status['can_self_identify']:
    # 独立モデルとしてエクスポート
    export_response = requests.post(
        "http://localhost:8000/api/apprentice/export",
        json={
            "model_name": "nullai-medical-specialist",
            "format": "gguf",
            "quantization": "Q4_K_M"
        }
    )
    print(f"モデルエクスポート済み: {export_response.json()['output_file']}")
```

---

## 7. ウェブエディター {#ウェブエディター}

### Dendritic Memory Editorの使用

ウェブベースのエディターは、知識タイルを作成・管理するための視覚的インターフェースを提供します。

**URL**: https://dendritic-memory-editor.pages.dev

### ワークフロー

```
1. ウェブエディターを開く
   ↓
2. 新規タイル作成またはインポート
   ↓
3. コンテンツ入力
   - トピック
   - 内容
   - ドメイン
   - 座標
   ↓
4. 3D空間で可視化
   ↓
5. 座標調整（オプション）
   ↓
6. .iathとしてエクスポート
   ↓
7. NullAIバックエンドにインポート
```

---

## 8. データのインポート&エクスポート {#データのインポートエクスポート}

### 概要

NullAIは`.iath`形式（Ilm-Athens形式）での知識タイルのインポート・エクスポートをサポートし、異なるインスタンスやdendritic-memory-editorアプリケーションとのシームレスなデータ交換を可能にします。

### インポートツールのクイックスタート

`.iath`ファイルをインポートする3つの方法を提供しています:

#### 方法1: Pythonスクリプト（推奨）

```bash
# 単一ファイル
python import_iath.py /path/to/Medical_tiles.iath

# 複数ファイル
python import_iath.py /path/to/*.iath
```

#### 方法2: バッチインポートスクリプト

```bash
# Downloadsフォルダから全ての.iathファイルをインポート
./batch_import_iath.sh

# カスタムディレクトリから指定
./batch_import_iath.sh /path/to/directory
```

#### 方法3: cURL（手動）

```bash
# 認証トークンを取得
TOKEN=$(curl -X POST http://localhost:8000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "your@email.com", "password": "password"}' \
  | jq -r '.access_token')

# ファイルをインポート
curl -X POST http://localhost:8000/api/knowledge/import/iath \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@Medical_tiles.iath"
```

### データのエクスポート

```bash
# 全てのドメインをエクスポート
curl -X GET "http://localhost:8000/api/knowledge/export/iath" \
  -H "Authorization: Bearer $TOKEN" \
  -o exported_all.iath

# 特定のドメインをエクスポート
curl -X GET "http://localhost:8000/api/knowledge/export/iath?domain_id=medical" \
  -H "Authorization: Bearer $TOKEN" \
  -o exported_medical.iath
```

### データベース設定

インポートシステムで使用するデータベースを変更するには:

**`backend/.env` ファイルを作成:**

```bash
# SQLite（デフォルト）
DATABASE_URL=sqlite:///./sql_app.db

# PostgreSQL
DATABASE_URL=postgresql://username:password@localhost:5432/nullai_db

# MySQL
DATABASE_URL=mysql://username:password@localhost:3306/nullai_db
```

### 詳細ドキュメント

📖 **完全なドキュメントは以下を参照:** [IATH_IMPORT_GUIDE.md](./IATH_IMPORT_GUIDE.md)

この包括的なガイドには以下が含まれます:
- 詳細なセットアップ手順
- 例を含む全てのインポート方法
- データベース切り替え手順
- 一般的な問題のトラブルシューティング
- フロントエンド統合の例

---

## 9. 高度な機能 {#高度な機能}

### ORCID専門家認証

タイルをORCIDプロファイルにリンクして専門家認証を行います。

### 推論チェーン

タイルに透明な推論を追加：

```python
tile_with_reasoning = {
    "domain_id": "logic_reasoning",
    "topic": "論理的推論",
    "content": "AがBを含意し、BがCを含意する場合...",
    "reasoning_chain": [
        {
            "step": 1,
            "content": "前提：A → B（AならばB）",
            "certainty": 1.0,
            "reasoning_type": "premise"
        },
        {
            "step": 2,
            "content": "前提：B → C（BならばC）",
            "certainty": 1.0,
            "reasoning_type": "premise"
        },
        {
            "step": 3,
            "content": "結論：A → C（AならばC）",
            "certainty": 1.0,
            "reasoning_type": "conclusion"
        }
    ]
}
```

---

## 10. トラブルシューティング {#トラブルシューティング-ja}

### よくある問題

#### モデル読み込みエラー

**問題**: "Out of memory"エラー

**解決策**:
```python
# コンテキストウィンドウを縮小
llm = Llama(
    model_path="phi-4-f16.gguf",
    n_ctx=4096,  # 16384の代わりに
    n_gpu_layers=0  # GPUメモリ不足の場合はCPUのみ使用
)
```

#### API接続拒否

**問題**: `localhost:8000`に接続できない

**解決策**:
```bash
# バックエンドが実行中か確認
curl http://localhost:8000/health

# 実行されていない場合、起動
cd ~/nullai/nullai-phi-4-14b-v2
./start_backend.sh
```

---

## 11. ベストプラクティス {#ベストプラクティス-ja}

### 知識タイル作成

#### ✅ すべきこと:
- 具体的で事実に基づく
- 出典と引用を含める
- 適切な座標を使用
- 複雑なトピックには推論チェーンを追加
- 情報が変わったらタイルを更新

#### ❌ すべきでないこと:
- 1つのタイルに複数のトピックを混在させる
- 曖昧または不明確な言葉を使用
- 証拠なしに確実性を主張
- 既存のタイルを重複させる
- 座標割り当てをスキップ

### 座標割り当て

**一般的なガイドライン:**

1. **保守的に開始**: 中程度の座標(0.4-0.6)から始めて調整

2. **配置をテスト**: 空間検索を使用して近くのタイルを確認

3. **対象者を考慮**:
   - 初心者に教える? y=0.2-0.4
   - 研究論文? y=0.7-0.9

4. **時間的感受性**:
   - 歴史的事実: z=0.1-0.3
   - 最新研究: z=0.7-0.9

---

## サポート & コミュニティ

- **ドキュメント**: このガイド + README.md
- **API ドキュメント**: http://localhost:8000/docs （実行時）
- **ウェブエディター**: https://dendritic-memory-editor.pages.dev
- **モデルHub**: https://huggingface.co/kofdai/nullai-phi-4-14b-v2

---

*Last Updated: December 2025*
*Version: 2.0*