Add file existence check before upload for deduplication

- Create reusable checkFileExists() helper function
- Add file existence check in /upload endpoint to skip uploading duplicate files
- Refactor /upload/remote to use the shared checkFileExists() function
- Add CLAUDE.md with codebase architecture documentation

This prevents unnecessary uploads to HuggingFace Dataset when files with
the same hash already exist, saving bandwidth and processing time.

CLAUDE.md

@@ -0,0 +1,171 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Hugstream Upload Server is an external upload proxy for the Hugstream file storage system. It runs as a Docker container (typically on Hugging Face Spaces) and handles direct file uploads from browsers to Hugging Face Datasets, offloading bandwidth from the main VPS and keeping the HF_TOKEN secure on the server side.
+
+## Key Architecture Concepts
+
+### Upload Flow
+1. Browser sends file + sessionToken directly to this upload server (not through main VPS)
+2. Upload server validates sessionToken against PostgreSQL database (shared with main app)
+3. File is uploaded to HF Dataset using content-addressed storage (hash-based paths)
+4. Database record created with fileId, which can be used to generate download URLs via proxy
+
+### Authentication Pattern
+- Uses session token authentication (from `auth-session` cookie)
+- Session tokens are SHA-256 hashed to produce sessionId
+- Database validates session and retrieves userId
+- No separate upload server token validation for user requests (legacy UPLOAD_SERVER_TOKEN may be removed)
+
+### Content-Addressed Storage
+- Files are stored by MD5 hash for global deduplication
+- HF path format: `{hash}` (just the hash, no userId prefix)
+- Database `file` table maps fileId -> hash -> HfPath
+- Same file uploaded by different users = single HF storage + multiple DB records
+
+### Remote Upload System (URL Downloads)
+- Supports downloading files from remote URLs and uploading to HF
+- Uses aria2c (if available) for multi-connection downloads with progress tracking
+- Falls back to standard fetch if aria2c not installed
+- For large files (>5GB), uses Git LFS with `huggingface-cli lfs-enable-largefiles`
+- Progress tracked via Server-Sent Events (SSE) or polling API
+- Automatic cleanup of temporary files after upload
+
+## Development Commands
+
+```bash
+# Development server (watch mode)
+npm run dev
+
+# Build TypeScript
+npm run build
+
+# Production server
+npm start
+
+# Docker build & run
+docker build -t hugstream-upload .
+docker run -p 7860:7860 --env-file .env hugstream-upload
+```
+
+## Database Schema
+
+Uses Drizzle ORM with PostgreSQL (via pg driver). Three main tables:
+- `user`: User accounts (id, username, passwordHash, storageQuota)
+- `session`: Session tokens (id=hashed token, userId, expiresAt)
+- `file`: File metadata (id, name, userId, parentId, hash, hfPath, isUploaded, etc.)
+
+Database connection uses singleton pool pattern in `src/lib/db/index.ts` for performance.
+
+## DNS Workaround for Koyeb
+The codebase includes a DNS resolution workaround for Koyeb-hosted PostgreSQL (which uses Neon). If DATABASE_URL contains `koyeb.app`, the DNS lookup is intercepted and mapped to the underlying Neon infrastructure. See `src/lib/db/index.ts:setupDnsWorkaround()`.
+
+## File Upload Methods
+
+### 1. Direct Upload (`POST /upload`)
+- Accepts multipart/form-data with file, sessionToken, parentId
+- Validates session, calculates hash, uploads to HF with retry logic
+- Creates database record and returns downloadUrl (if PUBLIC_DOWNLOAD_PROXY_HOST configured)
+
+### 2. Remote Upload (`POST /upload/remote`)
+- Downloads file from URL, then uploads to HF
+- Returns SSE stream for real-time progress updates
+- Uses aria2c for multi-connection downloads when available
+- For files >5GB: Uses Git clone + LFS + `huggingface-cli lfs-enable-largefiles`
+- For smaller files or when aria2 unavailable: Uses HuggingFace Hub commit API
+- Supports cancellation via `DELETE /upload/remote/:uploadId`
+
+### 3. Progress Monitoring
+- SSE endpoint: `GET /upload/remote/progress/:uploadId`
+- Polling endpoint: `GET /upload/remote/progress-poll/:uploadId`
+
+## HuggingFace Upload Strategy
+
+The upload strategy varies by file size and method:
+
+1. **Small files (<5GB) in direct upload**: Uses `@huggingface/hub` uploadFile API with retry logic for concurrent conflicts
+2. **Large files in remote upload**: Uses Git clone + Git LFS + push workflow
+3. **Retry logic**: Exponential backoff (1s, 2s, 4s, 8s, 16s) for concurrent commit conflicts
+
+## Environment Variables
+
+Required:
+- `DATABASE_URL`: PostgreSQL connection string
+- `HF_TOKEN`: Hugging Face API token with write access
+- `HF_DATASET_REPO`: Dataset repository (format: `username/repo`)
+
+Optional:
+- `UPLOAD_SERVER_TOKEN`: Legacy token (may be unused)
+- `ALLOWED_ORIGIN`: CORS origin (default: `*`)
+- `PUBLIC_DOWNLOAD_PROXY_HOST`: Base URL for download links
+- `PORT`: Server port (default: 7860)
+
+## Important Implementation Details
+
+### Concurrent Upload Handling
+- Multiple users may upload the same file simultaneously
+- Retry logic with exponential backoff handles HuggingFace "commit operation in progress" errors
+- Deduplication is detected via conflict errors or pre-upload fileExists check
+
+### Git Configuration
+- Git identity configured at runtime via `start.sh` script
+- Required for Git-based uploads to HuggingFace Dataset
+- Identity: "Hugstream Upload Bot <bot@hugstream.upload>"
+
+### Aria2 Integration
+- Spawns temporary aria2c RPC server for each download
+- Uses random port (6800-7800 range) to avoid conflicts
+- 16 concurrent connections per download for speed
+- Automatic cleanup of aria2 process and temp files
+
+### Hash Calculation
+- For direct uploads: Buffer-based MD5 (in-memory)
+- For remote uploads with aria2: Streaming hash calculation (memory efficient)
+- Used for deduplication and file integrity
+
+## Testing Endpoints
+
+- `GET /health`: Health check (returns status, service name, timestamp)
+- `GET /test-db`: Database connection test (queries user and file tables)
+
+## Common Patterns
+
+### Creating Database Records
+```typescript
+const fileId = encodeBase32LowerCase(crypto.getRandomValues(new Uint8Array(15)))
+await db.insert(fileTable).values({
+  id: fileId,
+  name: filename,
+  hash: calculatedHash,
+  hfPath,
+  userId,
+  // ... other fields
+})
+```
+
+### Validating Sessions
+```typescript
+const userId = await validateSessionToken(sessionToken)
+if (!userId) {
+  return c.json({ error: 'Unauthorized' }, 401)
+}
+```
+
+### Upload with Retry
+```typescript
+const result = await uploadToHFWithRetry(hfPath, buffer, maxRetries)
+// result.deduplicated = true if file already existed
+```
+
+## Docker Image
+
+Based on `node:20-slim` with:
+- aria2 for fast downloads
+- git + git-lfs for HF uploads
+- python3 + pip for huggingface-cli (large file support)
+- ca-certificates for SSL
+
+Build cache invalidation comment in Dockerfile should be updated when dependencies change.

src/index.ts

@@ -146,6 +146,27 @@ function getMimeType(filename: string): string {
   return mimeTypes[ext || ''] || 'application/octet-stream'
 }
 
+// Helper function to check if file exists in HF dataset
+async function checkFileExists(hfPath: string): Promise<boolean> {
+  try {
+    const { fileExists } = await import('@huggingface/hub')
+    const exists = await fileExists({
+      repo: {
+        type: 'dataset',
+        name: HF_DATASET_REPO
+      },
+      path: hfPath,
+      credentials: {
+        accessToken: HF_TOKEN
+      }
+    })
+    return exists
+  } catch (error: any) {
+    console.log(`[CHECK] Error checking file existence (will attempt upload anyway): ${error.message}`)
+    return false
+  }
+}
+
 // Helper function to retry HF upload with exponential backoff
 async function uploadToHFWithRetry(
   hfPath: string,
@@ -447,11 +468,26 @@ app.post('/upload', async (c) => {
     // Build HF path using only hash (global deduplication)
     const hfPath = calculatedHash
 
-    console.log(`[UPLOAD] Hash: ${calculatedHash}, uploading to HF`)
+    console.log(`[UPLOAD] Hash: ${calculatedHash}`)
+
+    // Check if file already exists in dataset
+    const fileExistsInDataset = await checkFileExists(hfPath)
+    console.log(`[UPLOAD] File exists check: ${fileExistsInDataset ? 'yes (deduplicated)' : 'no (will upload)'}`)
 
-    // Upload to Hugging Face Dataset with retry logic
+    // Upload to Hugging Face Dataset with retry logic (skip if file already exists)
+    let deduplicated = false
     try {
-      const result = await uploadToHFWithRetry(hfPath, buffer)
+      let result: { success: boolean; deduplicated: boolean }
+
+      if (fileExistsInDataset) {
+        console.log(`[UPLOAD] Deduplication detected, skipping upload`)
+        result = { success: true, deduplicated: true }
+        deduplicated = true
+      } else {
+        console.log(`[UPLOAD] Uploading to HF`)
+        result = await uploadToHFWithRetry(hfPath, buffer)
+        deduplicated = result.deduplicated
+      }
 
       // Generate download URL only if proxy is configured
       // Note: Download proxy expects fileId, not hash
@@ -788,25 +824,9 @@ async function processRemoteUploadWithStream(
 
     console.log(`[REMOTE_UPLOAD] Hash: ${calculatedHash}`)
 
-    // Check if file already exists in dataset using @huggingface/hub
-    let fileExistsInDataset = false
-    try {
-      const { fileExists } = await import('@huggingface/hub')
-      fileExistsInDataset = await fileExists({
-        repo: {
-          type: 'dataset',
-          name: HF_DATASET_REPO
-        },
-        path: hfPath,
-        credentials: {
-          accessToken: HF_TOKEN
-        }
-      })
-
-      console.log(`[REMOTE_UPLOAD] File exists check: ${fileExistsInDataset ? 'yes (deduplicated)' : 'no (will upload)'}`)
-    } catch (checkError: any) {
-      console.log(`[REMOTE_UPLOAD] Error checking file existence (will attempt upload anyway): ${checkError.message}`)
-    }
+    // Check if file already exists in dataset
+    const fileExistsInDataset = await checkFileExists(hfPath)
+    console.log(`[REMOTE_UPLOAD] File exists check: ${fileExistsInDataset ? 'yes (deduplicated)' : 'no (will upload)'}`)
 
     // Update progress to uploading (99%)
     sendProgress({