Spaces:

kofdai
/

nullai-knowledge-system

Sleeping

App Files Files Community

nullai-knowledge-system / USER_GUIDE.md

kofdai

Deploy NullAI Knowledge System to Spaces

075a2b6 verified 2 months ago

preview code

raw

history blame contribute delete

39.5 kB

NullAI Phi-4 14B (v2) - Complete User Guide

Complete guide to using the NullAI knowledge management system

English | 日本語

English Guide

Installation
Quick Start
Knowledge Tiles
3D Spatial Memory
API Reference
Fine-Tuning & Training
Web Editor
Data Import & Export 📦 NEW
Advanced Features
Troubleshooting
Best Practices

1. Installation {#installation}

One-Command Setup

curl -sSL https://huggingface.co/kofdai/nullai-phi-4-14b-v2/raw/main/setup.sh | bash

This will:

Download the 27GB model file
Install all dependencies
Setup backend and frontend
Initialize the database
Create startup scripts

Manual Installation

# Clone repository
git clone https://huggingface.co/kofdai/nullai-phi-4-14b-v2
cd nullai-phi-4-14b-v2

# Run setup script
chmod +x setup.sh
./setup.sh

System Requirements

RAM: 32GB minimum, 64GB recommended
Disk: 40GB free space
OS: macOS, Linux, or WSL2 on Windows
Python: 3.9 or higher
Node.js: 18 or higher (optional, for frontend)

2. Quick Start {#quick-start}

Starting the System

cd ~/nullai/nullai-phi-4-14b-v2

# Start everything
./start_nullai.sh

# Or start components separately
./start_backend.sh    # API server on port 8000
./start_frontend.sh   # Web UI on port 5173

Your First Knowledge Tile

Using Python API

import requests

# Create a knowledge tile
tile_data = {
    "domain_id": "medical",
    "topic": "Synaptic Plasticity",
    "content": "Synaptic plasticity refers to the ability of synapses to strengthen or weaken over time, in response to increases or decreases in their activity.",
    "coordinates": {
        "x": 0.6,  # Abstraction (0=concrete, 1=abstract)
        "y": 0.7,  # Expertise (0=basic, 1=advanced)
        "z": 0.3   # Temporality (0=timeless, 1=current)
    },
    "verification_type": "community_verified"
}

response = requests.post(
    "http://localhost:8000/api/knowledge/tiles",
    json=tile_data,
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

print(response.json())

Using Web Editor

Open https://dendritic-memory-editor.pages.dev
Fill in tile information
Visualize in 3D space
Export as .iath file
Import via API: POST /api/knowledge/import/iath

Testing the Model

from llama_cpp import Llama

llm = Llama(
    model_path="~/nullai/nullai-phi-4-14b-v2/phi-4-f16.gguf",
    n_ctx=16384,
    n_threads=8,
    n_gpu_layers=-1  # Use GPU if available
)

response = llm(
    "Explain synaptic plasticity in simple terms.",
    max_tokens=256,
    temperature=0.7
)

print(response['choices'][0]['text'])

3. Knowledge Tiles {#knowledge-tiles}

Tile Structure

{
  "id": "tile_abc123",
  "domain_id": "medical",
  "topic": "Neuroplasticity",
  "content": "Detailed explanation...",
  "coordinates": {
    "x": 0.5,  // Abstraction axis
    "y": 0.8,  // Expertise axis
    "z": 0.2   // Temporality axis
  },
  "certainty_score": 0.95,
  "verification_type": "expert_verified",
  "orcid_verified": true,
  "expert_id": "0000-0002-1234-5678",
  "reasoning_chain": [
    {
      "step": 1,
      "content": "Based on research...",
      "certainty": 0.9
    }
  ],
  "citations": [
    {
      "title": "Neural Mechanisms...",
      "authors": "Smith et al.",
      "year": 2023,
      "doi": "10.1234/example"
    }
  ],
  "created_at": "2025-12-03T10:00:00Z",
  "updated_at": "2025-12-03T10:00:00Z"
}

Creating Tiles

API Endpoint

POST /api/knowledge/tiles
Content-Type: application/json
Authorization: Bearer YOUR_TOKEN

{
  "domain_id": "medical",
  "topic": "Your Topic",
  "content": "Your detailed content...",
  "coordinates": {"x": 0.5, "y": 0.5, "z": 0.5},
  "verification_type": "community_verified"
}

Response

{
  "success": true,
  "tile_id": "tile_abc123",
  "message": "Tile created successfully",
  "coordinates": {"x": 0.5, "y": 0.5, "z": 0.5}
}

Retrieving Tiles

Get All Tiles

GET /api/knowledge/tiles?domain=medical&limit=50&offset=0

Get Specific Tile

GET /api/knowledge/tiles/{tile_id}

Spatial Search

# Find tiles near a coordinate
response = requests.post(
    "http://localhost:8000/api/knowledge/search/spatial",
    json={
        "center": {"x": 0.5, "y": 0.7, "z": 0.3},
        "radius": 0.2,
        "domain": "medical"
    }
)

Updating Tiles

PUT /api/knowledge/tiles/{tile_id}
Content-Type: application/json

{
  "content": "Updated content...",
  "coordinates": {"x": 0.6, "y": 0.8, "z": 0.3}
}

Deleting Tiles

DELETE /api/knowledge/tiles/{tile_id}

4. 3D Spatial Memory {#3d-spatial-memory}

Understanding the 3D Coordinate System

         Z (Temporality)
         ↑
         |     ╱ z=1.0 (Current, rapidly changing)
         |   ╱
         | ╱______ z=0.0 (Timeless, unchanging)
         |╱
         O────────→ Y (Expertise)
        ╱|          y=0.0 (Basic) → y=1.0 (Advanced)
      ╱  |
    ╱    ↓
   X (Abstraction)
   x=0.0 (Concrete) → x=1.0 (Abstract)

Coordinate Guidelines

X-Axis: Abstraction Level

0.0 - 0.3: Concrete (specific examples, case studies)
0.3 - 0.7: Moderate (general principles with examples)
0.7 - 1.0: Abstract (theoretical concepts, philosophical)

Example:

x=0.1: "The heart pumps blood through arteries"
x=0.5: "Cardiovascular system function and regulation"
x=0.9: "Principles of biological fluid dynamics"

Y-Axis: Expertise Level

0.0 - 0.3: Beginner (introductory, basic concepts)
0.3 - 0.7: Intermediate (requires background knowledge)
0.7 - 1.0: Expert (advanced, specialized knowledge)

Example:

y=0.2: "What is machine learning?"
y=0.5: "Gradient descent optimization"
y=0.9: "Novel architectures in attention mechanisms"

Z-Axis: Temporality

0.0 - 0.3: Timeless (fundamental truths, unchanging)
0.3 - 0.7: Moderate (evolving but stable)
0.7 - 1.0: Current (latest research, rapidly changing)

Example:

z=0.1: "Water is H2O"
z=0.5: "Current understanding of quantum computing"
z=0.9: "Latest COVID-19 variant information"

Spatial Search Strategies

Proximity Search

Find tiles near a specific coordinate:

def search_nearby(center, radius=0.2):
    response = requests.post(
        "http://localhost:8000/api/knowledge/search/spatial",
        json={
            "center": center,
            "radius": radius,
            "limit": 20
        }
    )
    return response.json()

# Example: Find intermediate medical knowledge
results = search_nearby(
    center={"x": 0.5, "y": 0.5, "z": 0.3},
    radius=0.2
)

Progressive Learning Path

Navigate from beginner to expert:

# Start with basics
basic_tiles = search_nearby({"x": 0.3, "y": 0.2, "z": 0.3})

# Progress to intermediate
intermediate_tiles = search_nearby({"x": 0.5, "y": 0.5, "z": 0.3})

# Advance to expert
expert_tiles = search_nearby({"x": 0.7, "y": 0.8, "z": 0.3})

5. API Reference {#api-reference}

Authentication

Register User

POST /api/auth/register
Content-Type: application/json

{
  "username": "your_username",
  "email": "your@email.com",
  "password": "secure_password"
}

Login

POST /api/auth/login
Content-Type: application/json

{
  "username": "your_username",
  "password": "your_password"
}

Response:

{
  "access_token": "eyJ0eXAiOiJKV1QiLCJhbGc...",
  "token_type": "bearer"
}

Knowledge Tile Endpoints

Method	Endpoint	Description
GET	`/api/knowledge/tiles`	List all tiles
GET	`/api/knowledge/tiles/{id}`	Get specific tile
POST	`/api/knowledge/tiles`	Create new tile
PUT	`/api/knowledge/tiles/{id}`	Update tile
DELETE	`/api/knowledge/tiles/{id}`	Delete tile
POST	`/api/knowledge/search/spatial`	Spatial search
POST	`/api/knowledge/import/iath`	Import .iath file
GET	`/api/knowledge/export/iath`	Export as .iath

Domain Endpoints

Method	Endpoint	Description
GET	`/api/domains`	List all domains
GET	`/api/domains/{id}`	Get domain info
POST	`/api/domains`	Create domain
PUT	`/api/domains/{id}`	Update domain

Training Endpoints

Method	Endpoint	Description
POST	`/api/training/start`	Start fine-tuning
GET	`/api/training/status`	Check training status
GET	`/api/training/history`	Training history
POST	`/api/training/export`	Export trained model

Apprentice Model Endpoints

Method	Endpoint	Description
GET	`/api/apprentice/status`	Get apprentice status
POST	`/api/apprentice/train`	Train apprentice model
GET	`/api/apprentice/threshold`	Check threshold status
POST	`/api/apprentice/export`	Export apprentice model

6. Fine-Tuning & Training {#fine-tuning}

Overview

NullAI uses LoRA (Low-Rank Adaptation) for efficient fine-tuning of the Phi-4 model on your knowledge tiles.

Training Workflow

1. Create Knowledge Tiles
   ↓
2. Generate Training Data
   ↓
3. Start LoRA Fine-Tuning
   ↓
4. Monitor Training Progress
   ↓
5. Export Adapted Model
   ↓
6. Deploy Apprentice Model

Starting Training

import requests

# Prepare training configuration
training_config = {
    "domain": "medical",
    "epochs": 3,
    "learning_rate": 0.0003,
    "batch_size": 4,
    "lora_rank": 8,
    "lora_alpha": 16,
    "target_modules": ["q_proj", "v_proj"]
}

# Start training
response = requests.post(
    "http://localhost:8000/api/training/start",
    json=training_config,
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

training_id = response.json()["training_id"]
print(f"Training started: {training_id}")

Monitoring Progress

import time

while True:
    response = requests.get(
        f"http://localhost:8000/api/training/status",
        params={"training_id": training_id}
    )

    status = response.json()
    print(f"Progress: {status['progress']}%")
    print(f"Loss: {status['current_loss']}")

    if status['status'] == 'completed':
        print("Training completed!")
        break

    time.sleep(30)  # Check every 30 seconds

Apprentice Model Threshold System

The apprentice model automatically tracks when it has accumulated enough training to "self-identify" as a specialized model.

# Check threshold status
response = requests.get("http://localhost:8000/api/apprentice/status")
status = response.json()

print(f"Training examples: {status['training_examples']}")
print(f"Threshold: {status['threshold']}")
print(f"Can self-identify: {status['can_self_identify']}")

if status['can_self_identify']:
    # Export as independent model
    export_response = requests.post(
        "http://localhost:8000/api/apprentice/export",
        json={
            "model_name": "nullai-medical-specialist",
            "format": "gguf",
            "quantization": "Q4_K_M"
        }
    )
    print(f"Model exported: {export_response.json()['output_file']}")

7. Web Editor {#web-editor}

Using the Dendritic Memory Editor

The web-based editor provides a visual interface for creating and managing knowledge tiles.

URL: https://dendritic-memory-editor.pages.dev

Features

1. Visual Tile Creation

Form-based interface
Real-time validation
Preview before saving

2. 3D Coordinate Visualizer

Interactive 3D space
Drag-and-drop tile positioning
Visual understanding of spatial relationships

3. Import/Export

Load existing .iath files
Export to .iath format
Batch operations

4. API Integration

Direct upload to NullAI backend
One-click import
Automatic coordinate adjustment

Workflow

1. Open Web Editor
   ↓
2. Create New Tile or Import
   ↓
3. Fill in Content
   - Topic
   - Content
   - Domain
   - Coordinates
   ↓
4. Visualize in 3D Space
   ↓
5. Adjust Coordinates (optional)
   ↓
6. Export as .iath
   ↓
7. Import to NullAI Backend

Example: Creating a Tile in Web Editor

Open Editor: Navigate to https://dendritic-memory-editor.pages.dev
Select Domain: Choose "Medical"
Enter Information:
- Topic: "Dendritic Spine Function"
- Content: "Dendritic spines are small membranous protrusions..."
- Coordinates: x=0.6, y=0.7, z=0.3
Preview in 3D: See where your tile appears in the knowledge space
Export: Click "Export .iath"

Import to NullAI:

curl -X POST http://localhost:8000/api/knowledge/import/iath \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -F "file=@dendritic_spine.iath"

8. Data Import & Export {#data-import-export}

Overview

NullAI supports importing and exporting knowledge tiles in .iath format (Ilm-Athens format), enabling seamless data exchange between different instances and the dendritic-memory-editor application.

Quick Start with Import Tools

We provide three methods for importing .iath files:

Method 1: Python Script (Recommended)

# Single file
python import_iath.py /path/to/Medical_tiles.iath

# Multiple files
python import_iath.py /path/to/*.iath

Method 2: Batch Import Script

# Import all .iath files from Downloads folder
./batch_import_iath.sh

# Import from custom directory
./batch_import_iath.sh /path/to/directory

Method 3: cURL (Manual)

# Get auth token
TOKEN=$(curl -X POST http://localhost:8000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "your@email.com", "password": "password"}' \
  | jq -r '.access_token')

# Import file
curl -X POST http://localhost:8000/api/knowledge/import/iath \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@Medical_tiles.iath"

Export Data

# Export all domains
curl -X GET "http://localhost:8000/api/knowledge/export/iath" \
  -H "Authorization: Bearer $TOKEN" \
  -o exported_all.iath

# Export specific domain
curl -X GET "http://localhost:8000/api/knowledge/export/iath?domain_id=medical" \
  -H "Authorization: Bearer $TOKEN" \
  -o exported_medical.iath

Database Configuration

To change the database used by the import system:

Create backend/.env file:

# SQLite (default)
DATABASE_URL=sqlite:///./sql_app.db

# PostgreSQL
DATABASE_URL=postgresql://username:password@localhost:5432/nullai_db

# MySQL
DATABASE_URL=mysql://username:password@localhost:3306/nullai_db

Detailed Documentation

📖 For complete documentation, see: IATH_IMPORT_GUIDE.md

This comprehensive guide includes:

Detailed setup instructions
All import methods with examples
Database switching procedures
Troubleshooting common issues
Frontend integration examples

9. Advanced Features {#advanced-features}

ORCID Expert Verification

Link your tiles to your ORCID profile for expert verification.

# Register ORCID
response = requests.post(
    "http://localhost:8000/api/oauth/orcid/register",
    json={
        "orcid_id": "0000-0002-1234-5678",
        "access_token": "your_orcid_token"
    }
)

# Create expert-verified tile
tile_data = {
    "domain_id": "medical",
    "topic": "Advanced Neuroscience",
    "content": "Expert knowledge...",
    "verification_type": "expert_verified",
    "orcid_verified": True
}

Reasoning Chains

Add transparent reasoning to your tiles:

tile_with_reasoning = {
    "domain_id": "logic_reasoning",
    "topic": "Logical Deduction",
    "content": "If A implies B, and B implies C...",
    "reasoning_chain": [
        {
            "step": 1,
            "content": "Given: A → B (if A then B)",
            "certainty": 1.0,
            "reasoning_type": "premise"
        },
        {
            "step": 2,
            "content": "Given: B → C (if B then C)",
            "certainty": 1.0,
            "reasoning_type": "premise"
        },
        {
            "step": 3,
            "content": "Therefore: A → C (if A then C)",
            "certainty": 1.0,
            "reasoning_type": "conclusion"
        }
    ]
}

Multi-Stage Judge System

NullAI implements a three-stage verification system:

Alpha Lobe (Logic Judge)

# Evaluate logical consistency
response = requests.post(
    "http://localhost:8000/api/judge/alpha",
    json={
        "tile_id": "tile_abc123",
        "check_type": "logical_consistency"
    }
)
# Returns: logical_score, inconsistencies, suggestions

Beta Lobe Basic (Domain Judge)

# Check domain expertise
response = requests.post(
    "http://localhost:8000/api/judge/beta-basic",
    json={
        "tile_id": "tile_abc123",
        "domain": "medical"
    }
)
# Returns: domain_accuracy, expert_level, corrections

Beta Lobe Advanced (Reasoning Judge)

# Validate reasoning chain
response = requests.post(
    "http://localhost:8000/api/judge/beta-advanced",
    json={
        "tile_id": "tile_abc123",
        "check_reasoning": True
    }
)
# Returns: reasoning_validity, confidence, improvements

Database Enrichment

Automatically enhance your knowledge base:

# Start enrichment process
response = requests.post(
    "http://localhost:8000/api/enrichment/start",
    json={
        "domain": "medical",
        "target_tile_count": 10000,
        "use_master_model": True,
        "auto_verify": True
    }
)

enrichment_id = response.json()["enrichment_id"]

# Monitor progress
status = requests.get(
    f"http://localhost:8000/api/enrichment/status/{enrichment_id}"
)

Workspace Management

Organize tiles into workspaces for different projects:

# Create workspace
workspace = requests.post(
    "http://localhost:8000/api/workspaces",
    json={
        "name": "Neuroscience Research",
        "description": "Collaborative neuroscience knowledge",
        "domains": ["medical", "ai_fundamentals"]
    }
)

workspace_id = workspace.json()["id"]

# Add tiles to workspace
requests.post(
    f"http://localhost:8000/api/workspaces/{workspace_id}/tiles",
    json={"tile_ids": ["tile_1", "tile_2", "tile_3"]}
)

10. Troubleshooting {#troubleshooting}

Common Issues

Model Loading Errors

Problem: "Out of memory" error when loading model

Solution:

# Reduce context window
llm = Llama(
    model_path="phi-4-f16.gguf",
    n_ctx=4096,  # Instead of 16384
    n_gpu_layers=0  # Use CPU only if GPU memory insufficient
)

API Connection Refused

Problem: Cannot connect to localhost:8000

Solution:

# Check if backend is running
curl http://localhost:8000/health

# If not, start it
cd ~/nullai/nullai-phi-4-14b-v2
./start_backend.sh

Authentication Errors

Problem: "401 Unauthorized" on API calls

Solution:

# Get fresh token
login_response = requests.post(
    "http://localhost:8000/api/auth/login",
    json={"username": "your_user", "password": "your_pass"}
)
token = login_response.json()["access_token"]

# Use in requests
headers = {"Authorization": f"Bearer {token}"}

Database Locked

Problem: "Database is locked" error

Solution:

# Stop all instances
pkill -f "uvicorn"

# Restart
./start_backend.sh

Training Fails

Problem: LoRA training crashes

Solution:

# Reduce batch size and parameters
training_config = {
    "batch_size": 1,  # Reduce from 4
    "lora_rank": 4,   # Reduce from 8
    "gradient_accumulation_steps": 4  # Compensate for smaller batch
}

Performance Optimization

Speed Up Inference

# Use GPU acceleration
llm = Llama(
    model_path="phi-4-f16.gguf",
    n_gpu_layers=35,  # Offload all layers to GPU
    n_batch=512,      # Larger batch for GPU
    n_threads=1       # Use fewer CPU threads when using GPU
)

Reduce Memory Usage

# Use quantized model (when available)
# Or reduce context window
llm = Llama(
    model_path="phi-4-f16.gguf",
    n_ctx=2048,  # Smaller context = less memory
    use_mmap=True,  # Use memory mapping
    use_mlock=False  # Don't lock in RAM
)

11. Best Practices {#best-practices}

Knowledge Tile Creation

✅ DO:

Be specific and factual
Include sources and citations
Use appropriate coordinates
Add reasoning chains for complex topics
Update tiles when information changes

❌ DON'T:

Mix multiple topics in one tile
Use vague or ambiguous language
Claim certainty without evidence
Duplicate existing tiles
Skip coordinate assignment

Coordinate Assignment

General Guidelines:

Start Conservative: Begin with moderate coordinates (0.4-0.6) and adjust
Test Placement: Use spatial search to see nearby tiles
Consider Audience:
- Teaching beginners? y=0.2-0.4
- Research papers? y=0.7-0.9
Time Sensitivity:
- Historical facts: z=0.1-0.3
- Latest research: z=0.7-0.9

Training Best Practices

Data Quality Over Quantity
- 100 high-quality tiles > 1000 mediocre tiles
- Ensure diverse coverage of domain
Incremental Training
- Start with small epochs (1-2)
- Evaluate before continuing
- Save checkpoints frequently
Threshold Management
- Monitor apprentice model progress
- Don't rush self-identification
- Validate before exporting

API Usage

Authentication
- Store tokens securely
- Refresh before expiration
- Use environment variables
Rate Limiting
- Implement exponential backoff
- Batch operations when possible
- Cache frequently accessed tiles

Error Handling

import time

def api_call_with_retry(url, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, timeout=10)
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            if attempt == max_retries - 1:
                raise
            wait_time = 2 ** attempt
            time.sleep(wait_time)

Security

Protect Credentials

# Use environment variables
export NULLAI_TOKEN="your_token_here"
export NULLAI_API_URL="http://localhost:8000"

HTTPS in Production
- Never use HTTP for production
- Use SSL certificates
- Enable CORS properly
Input Validation
- Sanitize all inputs
- Validate coordinates (0-1 range)
- Check file uploads

Collaboration

Workspace Organization
- Create project-specific workspaces
- Assign clear roles
- Document conventions
Version Control
- Export workspaces regularly
- Keep .iath files in git
- Document changes
Expert Verification
- Use ORCID for credentials
- Peer review important tiles
- Maintain audit trail

Support & Community

Documentation: This guide + README.md
API Docs: http://localhost:8000/docs (when running)
Web Editor: https://dendritic-memory-editor.pages.dev
Model Hub: https://huggingface.co/kofdai/nullai-phi-4-14b-v2

日本語ガイド

1. インストール {#インストール}

ワンコマンドセットアップ

curl -sSL https://huggingface.co/kofdai/nullai-phi-4-14b-v2/raw/main/setup.sh | bash

以下が自動実行されます：

27GBモデルファイルのダウンロード
全依存関係のインストール
バックエンドとフロントエンドのセットアップ
データベースの初期化
起動スクリプトの作成

手動インストール

# リポジトリのクローン
git clone https://huggingface.co/kofdai/nullai-phi-4-14b-v2
cd nullai-phi-4-14b-v2

# セットアップスクリプトの実行
chmod +x setup.sh
./setup.sh

システム要件

RAM: 最小32GB、推奨64GB
ディスク: 40GB以上の空き容量
OS: macOS、Linux、またはWindows WSL2
Python: 3.9以上
Node.js: 18以上（フロントエンド用、オプション）

2. クイックスタート {#クイックスタート-ja}

システムの起動

cd ~/nullai/nullai-phi-4-14b-v2

# すべてを起動
./start_nullai.sh

# または個別に起動
./start_backend.sh    # ポート8000でAPIサーバー
./start_frontend.sh   # ポート5173でWeb UI

最初の知識タイル

Python APIを使用

import requests

# 知識タイルの作成
tile_data = {
    "domain_id": "medical",
    "topic": "シナプス可塑性",
    "content": "シナプス可塑性とは、活動の増減に応じてシナプスが時間とともに強化または弱化する能力を指します。",
    "coordinates": {
        "x": 0.6,  # 抽象度 (0=具体的, 1=抽象的)
        "y": 0.7,  # 専門性 (0=基礎, 1=高度)
        "z": 0.3   # 時間性 (0=普遍的, 1=最新)
    },
    "verification_type": "community_verified"
}

response = requests.post(
    "http://localhost:8000/api/knowledge/tiles",
    json=tile_data,
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

print(response.json())

ウェブエディターを使用

https://dendritic-memory-editor.pages.dev を開く
タイル情報を入力
3D空間で可視化
.iathファイルとしてエクスポート
APIでインポート: POST /api/knowledge/import/iath

モデルのテスト

from llama_cpp import Llama

llm = Llama(
    model_path="~/nullai/nullai-phi-4-14b-v2/phi-4-f16.gguf",
    n_ctx=16384,
    n_threads=8,
    n_gpu_layers=-1  # GPUがあれば使用
)

response = llm(
    "シナプス可塑性を簡単に説明してください。",
    max_tokens=256,
    temperature=0.7
)

print(response['choices'][0]['text'])

3. 知識タイル {#知識タイル}

タイル構造

{
  "id": "tile_abc123",
  "domain_id": "medical",
  "topic": "神経可塑性",
  "content": "詳細な説明...",
  "coordinates": {
    "x": 0.5,  // 抽象度軸
    "y": 0.8,  // 専門性軸
    "z": 0.2   // 時間性軸
  },
  "certainty_score": 0.95,
  "verification_type": "expert_verified",
  "orcid_verified": true,
  "expert_id": "0000-0002-1234-5678",
  "reasoning_chain": [
    {
      "step": 1,
      "content": "研究に基づき...",
      "certainty": 0.9
    }
  ],
  "citations": [
    {
      "title": "神経メカニズム...",
      "authors": "Smith et al.",
      "year": 2023,
      "doi": "10.1234/example"
    }
  ],
  "created_at": "2025-12-03T10:00:00Z",
  "updated_at": "2025-12-03T10:00:00Z"
}

タイルの作成

APIエンドポイント

POST /api/knowledge/tiles
Content-Type: application/json
Authorization: Bearer YOUR_TOKEN

{
  "domain_id": "medical",
  "topic": "あなたのトピック",
  "content": "詳細な内容...",
  "coordinates": {"x": 0.5, "y": 0.5, "z": 0.5},
  "verification_type": "community_verified"
}

座標ガイドライン

X軸：抽象度レベル

0.0 - 0.3: 具体的（具体例、ケーススタディ）
0.3 - 0.7: 中程度（例を含む一般原則）
0.7 - 1.0: 抽象的（理論的概念、哲学的）

Y軸：専門性レベル

0.0 - 0.3: 初心者（入門、基本概念）
0.3 - 0.7: 中級者（予備知識が必要）
0.7 - 1.0: 専門家（高度、専門知識）

Z軸：時間性

0.0 - 0.3: 普遍的（基本真理、不変）
0.3 - 0.7: 中程度（進化中だが安定）
0.7 - 1.0: 最新（最新研究、急速に変化）

4. 3D空間記憶 {#3d空間記憶}

3D座標系の理解

         Z (時間性)
         ↑
         |     ╱ z=1.0 (最新、急速に変化)
         |   ╱
         | ╱______ z=0.0 (普遍的、不変)
         |╱
         O────────→ Y (専門性)
        ╱|          y=0.0 (基礎) → y=1.0 (高度)
      ╱  |
    ╱    ↓
   X (抽象度)
   x=0.0 (具体的) → x=1.0 (抽象的)

空間検索戦略

近接検索

def search_nearby(center, radius=0.2):
    response = requests.post(
        "http://localhost:8000/api/knowledge/search/spatial",
        json={
            "center": center,
            "radius": radius,
            "limit": 20
        }
    )
    return response.json()

# 例：中級医療知識を検索
results = search_nearby(
    center={"x": 0.5, "y": 0.5, "z": 0.3},
    radius=0.2
)

5. APIリファレンス {#apiリファレンス}

認証

ユーザー登録

POST /api/auth/register
Content-Type: application/json

{
  "username": "your_username",
  "email": "your@email.com",
  "password": "secure_password"
}

知識タイルエンドポイント

メソッド	エンドポイント	説明
GET	`/api/knowledge/tiles`	全タイル一覧
GET	`/api/knowledge/tiles/{id}`	特定タイル取得
POST	`/api/knowledge/tiles`	新規タイル作成
PUT	`/api/knowledge/tiles/{id}`	タイル更新
DELETE	`/api/knowledge/tiles/{id}`	タイル削除
POST	`/api/knowledge/search/spatial`	空間検索
POST	`/api/knowledge/import/iath`	.iathファイルインポート
GET	`/api/knowledge/export/iath`	.iath形式でエクスポート

6. ファインチューニング {#ファインチューニング}

概要

NullAIはLoRA（Low-Rank Adaptation）を使用して、Phi-4モデルを知識タイルで効率的にファインチューニングします。

トレーニングの開始

import requests

# トレーニング設定の準備
training_config = {
    "domain": "medical",
    "epochs": 3,
    "learning_rate": 0.0003,
    "batch_size": 4,
    "lora_rank": 8,
    "lora_alpha": 16,
    "target_modules": ["q_proj", "v_proj"]
}

# トレーニング開始
response = requests.post(
    "http://localhost:8000/api/training/start",
    json=training_config,
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

training_id = response.json()["training_id"]
print(f"トレーニング開始: {training_id}")

進捗のモニタリング

import time

while True:
    response = requests.get(
        f"http://localhost:8000/api/training/status",
        params={"training_id": training_id}
    )

    status = response.json()
    print(f"進捗: {status['progress']}%")
    print(f"損失: {status['current_loss']}")

    if status['status'] == 'completed':
        print("トレーニング完了！")
        break

    time.sleep(30)  # 30秒ごとにチェック

弟子モデル閾値システム

弟子モデルは、特化モデルとして「自己識別」するのに十分なトレーニングが蓄積されたときを自動的に追跡します。

# 閾値ステータスの確認
response = requests.get("http://localhost:8000/api/apprentice/status")
status = response.json()

print(f"トレーニング例: {status['training_examples']}")
print(f"閾値: {status['threshold']}")
print(f"自己識別可能: {status['can_self_identify']}")

if status['can_self_identify']:
    # 独立モデルとしてエクスポート
    export_response = requests.post(
        "http://localhost:8000/api/apprentice/export",
        json={
            "model_name": "nullai-medical-specialist",
            "format": "gguf",
            "quantization": "Q4_K_M"
        }
    )
    print(f"モデルエクスポート済み: {export_response.json()['output_file']}")

7. ウェブエディター {#ウェブエディター}

Dendritic Memory Editorの使用

ウェブベースのエディターは、知識タイルを作成・管理するための視覚的インターフェースを提供します。

URL: https://dendritic-memory-editor.pages.dev

ワークフロー

1. ウェブエディターを開く
   ↓
2. 新規タイル作成またはインポート
   ↓
3. コンテンツ入力
   - トピック
   - 内容
   - ドメイン
   - 座標
   ↓
4. 3D空間で可視化
   ↓
5. 座標調整（オプション）
   ↓
6. .iathとしてエクスポート
   ↓
7. NullAIバックエンドにインポート

8. データのインポート&エクスポート {#データのインポートエクスポート}

概要

NullAIは.iath形式（Ilm-Athens形式）での知識タイルのインポート・エクスポートをサポートし、異なるインスタンスやdendritic-memory-editorアプリケーションとのシームレスなデータ交換を可能にします。

インポートツールのクイックスタート

.iathファイルをインポートする3つの方法を提供しています:

方法1: Pythonスクリプト（推奨）

# 単一ファイル
python import_iath.py /path/to/Medical_tiles.iath

# 複数ファイル
python import_iath.py /path/to/*.iath

方法2: バッチインポートスクリプト

# Downloadsフォルダから全ての.iathファイルをインポート
./batch_import_iath.sh

# カスタムディレクトリから指定
./batch_import_iath.sh /path/to/directory

方法3: cURL（手動）

# 認証トークンを取得
TOKEN=$(curl -X POST http://localhost:8000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "your@email.com", "password": "password"}' \
  | jq -r '.access_token')

# ファイルをインポート
curl -X POST http://localhost:8000/api/knowledge/import/iath \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@Medical_tiles.iath"

データのエクスポート

# 全てのドメインをエクスポート
curl -X GET "http://localhost:8000/api/knowledge/export/iath" \
  -H "Authorization: Bearer $TOKEN" \
  -o exported_all.iath

# 特定のドメインをエクスポート
curl -X GET "http://localhost:8000/api/knowledge/export/iath?domain_id=medical" \
  -H "Authorization: Bearer $TOKEN" \
  -o exported_medical.iath

データベース設定

インポートシステムで使用するデータベースを変更するには:

backend/.env ファイルを作成:

# SQLite（デフォルト）
DATABASE_URL=sqlite:///./sql_app.db

# PostgreSQL
DATABASE_URL=postgresql://username:password@localhost:5432/nullai_db

# MySQL
DATABASE_URL=mysql://username:password@localhost:3306/nullai_db

詳細ドキュメント

📖 完全なドキュメントは以下を参照: IATH_IMPORT_GUIDE.md

この包括的なガイドには以下が含まれます:

詳細なセットアップ手順
例を含む全てのインポート方法
データベース切り替え手順
一般的な問題のトラブルシューティング
フロントエンド統合の例

9. 高度な機能 {#高度な機能}

ORCID専門家認証

タイルをORCIDプロファイルにリンクして専門家認証を行います。

推論チェーン

タイルに透明な推論を追加：

tile_with_reasoning = {
    "domain_id": "logic_reasoning",
    "topic": "論理的推論",
    "content": "AがBを含意し、BがCを含意する場合...",
    "reasoning_chain": [
        {
            "step": 1,
            "content": "前提：A → B（AならばB）",
            "certainty": 1.0,
            "reasoning_type": "premise"
        },
        {
            "step": 2,
            "content": "前提：B → C（BならばC）",
            "certainty": 1.0,
            "reasoning_type": "premise"
        },
        {
            "step": 3,
            "content": "結論：A → C（AならばC）",
            "certainty": 1.0,
            "reasoning_type": "conclusion"
        }
    ]
}

10. トラブルシューティング {#トラブルシューティング-ja}

よくある問題

モデル読み込みエラー

問題: "Out of memory"エラー

解決策:

# コンテキストウィンドウを縮小
llm = Llama(
    model_path="phi-4-f16.gguf",
    n_ctx=4096,  # 16384の代わりに
    n_gpu_layers=0  # GPUメモリ不足の場合はCPUのみ使用
)

API接続拒否

問題: localhost:8000に接続できない

解決策:

# バックエンドが実行中か確認
curl http://localhost:8000/health

# 実行されていない場合、起動
cd ~/nullai/nullai-phi-4-14b-v2
./start_backend.sh

11. ベストプラクティス {#ベストプラクティス-ja}

知識タイル作成

✅ すべきこと:

具体的で事実に基づく
出典と引用を含める
適切な座標を使用
複雑なトピックには推論チェーンを追加
情報が変わったらタイルを更新

❌ すべきでないこと:

1つのタイルに複数のトピックを混在させる
曖昧または不明確な言葉を使用
証拠なしに確実性を主張
既存のタイルを重複させる
座標割り当てをスキップ

座標割り当て

一般的なガイドライン:

保守的に開始: 中程度の座標(0.4-0.6)から始めて調整
配置をテスト: 空間検索を使用して近くのタイルを確認
対象者を考慮:
- 初心者に教える? y=0.2-0.4
- 研究論文? y=0.7-0.9
時間的感受性:
- 歴史的事実: z=0.1-0.3
- 最新研究: z=0.7-0.9

サポート & コミュニティ

ドキュメント: このガイド + README.md
API ドキュメント: http://localhost:8000/docs （実行時）
ウェブエディター: https://dendritic-memory-editor.pages.dev
モデルHub: https://huggingface.co/kofdai/nullai-phi-4-14b-v2

Last Updated: December 2025 Version: 2.0

NullAI Phi-4 14B (v2) - Complete User Guide

English Guide

Table of Contents

1. Installation {#installation}

One-Command Setup

Manual Installation

System Requirements

2. Quick Start {#quick-start}

Starting the System

Your First Knowledge Tile

Using Python API

Using Web Editor

Testing the Model

3. Knowledge Tiles {#knowledge-tiles}

Tile Structure

Creating Tiles

API Endpoint

Response

Retrieving Tiles

Get All Tiles

Get Specific Tile

Spatial Search

Updating Tiles

Deleting Tiles

4. 3D Spatial Memory {#3d-spatial-memory}

Understanding the 3D Coordinate System

Coordinate Guidelines

X-Axis: Abstraction Level

Y-Axis: Expertise Level

Z-Axis: Temporality

Spatial Search Strategies

Proximity Search

Progressive Learning Path

5. API Reference {#api-reference}

Authentication

Register User

Login

Knowledge Tile Endpoints

Domain Endpoints

Training Endpoints

Apprentice Model Endpoints

6. Fine-Tuning & Training {#fine-tuning}

Overview

Training Workflow

Starting Training

Monitoring Progress

Apprentice Model Threshold System

7. Web Editor {#web-editor}

Using the Dendritic Memory Editor

Features

1. Visual Tile Creation

2. 3D Coordinate Visualizer

3. Import/Export

4. API Integration

Workflow

Example: Creating a Tile in Web Editor

8. Data Import & Export {#data-import-export}

Overview

Quick Start with Import Tools

Method 1: Python Script (Recommended)

Method 2: Batch Import Script

Method 3: cURL (Manual)

Export Data

Database Configuration

Detailed Documentation

9. Advanced Features {#advanced-features}

ORCID Expert Verification

Reasoning Chains

Multi-Stage Judge System

Alpha Lobe (Logic Judge)

Beta Lobe Basic (Domain Judge)

Beta Lobe Advanced (Reasoning Judge)

Database Enrichment

Workspace Management

10. Troubleshooting {#troubleshooting}

Common Issues

Model Loading Errors

API Connection Refused

Authentication Errors

Database Locked