Spaces:

lucifer925
/

rustfs-storage

Sleeping

App Files Files Community

rustfs-storage / README.md

Akshar2325

chore: update Dockerfile and README.md for improved dependencies and versioning

3d83ed9 8 months ago

preview code

Raw

History Blame Contribute Delete

9.06 kB

metadata

title: RustFS S3-Compatible Storage
emoji: 🗄️
colorFrom: yellow
colorTo: red
sdk: docker
pinned: false
license: apache-2.0
tags:
  - storage
  - s3
  - rustfs
  - object-storage
  - learning

🗄️ RustFS S3-Compatible Storage

High-performance Rust-based object storage for learning and experimentation

📋 Overview

This is a learning project demonstrating RustFS deployment on HuggingFace Spaces with:

✅ S3-Compatible API - Works with AWS SDKs and tools
✅ Public Read Access - Anyone can view/download files
✅ Authenticated Write - Only authenticated users can upload/delete
✅ Web Console - Built-in management UI
✅ Presigned URLs - Generate temporary upload/download links
⚠️ Alpha Software - Use for learning, not production!

🚀 Quick Start

Access Points

Service	URL	Authentication
S3 API	`https://YOUR_USERNAME-rustfs-storage.hf.space`	Required for writes
Console	`https://YOUR_USERNAME-rustfs-storage.hf.space/console/`	Required
Health	`https://YOUR_USERNAME-rustfs-storage.hf.space/health`	Public

Default Bucket

Name: storage
Public Read: ✅ Yes (anyone can download)
Public Write: ❌ No (authentication required)

🔐 Authentication

Set these as HuggingFace Secrets (not in code!):

RUSTFS_ACCESS_KEY=your-access-key
RUSTFS_SECRET_KEY=your-secret-key

💻 Usage Examples

1. Upload File (Authenticated)

# Using AWS CLI
aws configure set aws_access_key_id YOUR_ACCESS_KEY
aws configure set aws_secret_access_key YOUR_SECRET_KEY

aws s3 cp myfile.jpg \
  s3://storage/images/myfile.jpg \
  --endpoint-url https://YOUR_USERNAME-rustfs-storage.hf.space

2. Download File (Public - No Auth)

# Direct download
curl "https://YOUR_USERNAME-rustfs-storage.hf.space/storage/images/myfile.jpg" \
  -o downloaded.jpg

3. Generate Presigned Upload URL

import boto3
from botocore.config import Config

# Configure S3 client for RustFS
s3_client = boto3.client(
    's3',
    endpoint_url='https://YOUR_USERNAME-rustfs-storage.hf.space',
    aws_access_key_id='YOUR_ACCESS_KEY',
    aws_secret_access_key='YOUR_SECRET_KEY',
    config=Config(signature_version='s3v4'),
    region_name='us-east-1'
)

# Generate presigned upload URL (expires in 1 hour)
presigned_url = s3_client.generate_presigned_url(
    'put_object',
    Params={
        'Bucket': 'storage',
        'Key': 'uploads/file.jpg',
        'ContentType': 'image/jpeg'
    },
    ExpiresIn=3600
)

print(f"Upload URL: {presigned_url}")

# Use the URL to upload
import requests
with open('myfile.jpg', 'rb') as f:
    requests.put(presigned_url, data=f, headers={'Content-Type': 'image/jpeg'})

4. Generate Presigned Download URL

# Generate presigned download URL (expires in 1 hour)
presigned_url = s3_client.generate_presigned_url(
    'get_object',
    Params={
        'Bucket': 'storage',
        'Key': 'images/file.jpg'
    },
    ExpiresIn=3600
)

print(f"Download URL: {presigned_url}")

5. List Files

# Using AWS CLI
aws s3 ls s3://storage/ \
  --endpoint-url https://YOUR_USERNAME-rustfs-storage.hf.space

6. Delete File (Authenticated)

# Using AWS CLI
aws s3 rm s3://storage/images/myfile.jpg \
  --endpoint-url https://YOUR_USERNAME-rustfs-storage.hf.space

🎯 Use Cases

✅ Perfect For:

Learning S3 API - Experiment with object storage
Testing Applications - Develop S3-compatible apps
Prototyping - Quick storage solution for demos
Understanding Object Storage - Hands-on experience

❌ Not Recommended For:

Production Data - This is alpha software
Critical Files - No backup guarantees on free tier
Large Scale - HuggingFace Spaces has storage limits
Sensitive Data - Public space, use with caution

🏗️ Architecture

┌─────────────┐
│   Client    │
└──────┬──────┘
       │
       ▼
┌─────────────┐     ┌──────────────┐
│    Nginx    │────▶│   RustFS     │
│   (Port     │     │   S3 API     │
│    7860)    │     │  (Port 9000) │
└─────────────┘     └──────────────┘
       │                    │
       │                    ▼
       │            ┌──────────────┐
       └───────────▶│   Console    │
                    │  (Port 9001) │
                    └──────────────┘

🔧 Configuration

Environment Variables

Variable	Default	Description
`RUSTFS_ACCESS_KEY`	`admin`	S3 access key (⚠️ Set in Secrets!)
`RUSTFS_SECRET_KEY`	`verysecret`	S3 secret key (⚠️ Set in Secrets!)
`DEFAULT_BUCKET`	`storage`	Default bucket name
`CLIENT_MAX_BODY_SIZE`	`0`	Max upload size (0 = unlimited)
`ALLOWED_ORIGINS`	`*`	CORS allowed origins
`WORKER_CONNECTIONS`	`1024`	Nginx worker connections
`NGINX_LOG_LEVEL`	`error`	Nginx log level

Storage Limits

Max File Size: Unlimited (but HF Spaces has overall storage limits)
Total Storage: Depends on HuggingFace Space tier
Concurrent Connections: 1024

📊 Performance

RustFS claims 2.3x faster than MinIO for small (4KB) objects, but:

⚠️ This is alpha software with known issues
Performance varies based on workload
Not yet optimized for all scenarios

⚠️ Important Notes

Alpha Software Warning

RustFS is in alpha (v1.0.0-alpha.72 - Latest: Dec 5, 2024)
May have bugs and breaking changes
Not production-ready
Use for learning purposes only

Known Limitations

Some S3 features may be incomplete
Console UI has minor bugs
Performance not fully optimized
Limited community support compared to MinIO

Security Considerations

Public Read Access: All files in bucket are publicly readable
No Directory Listing: Bucket contents not publicly listed
Authentication Required: Upload/delete needs valid credentials
HTTPS: All traffic encrypted via HuggingFace proxy

📚 Documentation

🛠️ Troubleshooting

"403 Forbidden" on Upload

Problem: Can't upload files Solution: Make sure you're using correct credentials and authenticating properly

# Check your credentials
aws configure list

# Test with explicit credentials
aws s3 cp test.txt s3://storage/test.txt \
  --endpoint-url YOUR_ENDPOINT \
  --profile YOUR_PROFILE

CORS Errors in Browser

Problem: Browser blocks API requests Solution: Make sure ALLOWED_ORIGINS includes your domain

# In HuggingFace Secrets
ALLOWED_ORIGINS=https://yourdomain.com

Presigned URL Not Working

Problem: Presigned URLs return errors Solution:

Make sure signature version is s3v4
Check that endpoint URL matches exactly
Verify region is set (use us-east-1)

🤝 Contributing

This is a learning project! Feel free to:

Fork and modify
Report issues (but remember it's alpha software)
Share your experiments
Compare with MinIO/SeaweedFS

📄 License

This Space: Apache 2.0 (Free to use!)
RustFS: Apache 2.0
Learning Project: Use freely for education

🙏 Acknowledgments

RustFS Team: For building this awesome Rust-based storage
HuggingFace: For free Spaces hosting
Community: For testing and feedback

🎓 Learning Resources

Compare with Other Storage Solutions

I've also deployed these for learning:

MinIO: YOUR_USERNAME/minio-storage (Most mature)
SeaweedFS: YOUR_USERNAME/seaweedfs-storage (Go-based)
Garage: YOUR_USERNAME/garage-storage (Rust-based)
RustFS: This space (Newest, experimental)

Why Learn Object Storage?

Cloud Skills: S3 is the standard for cloud storage
Scalability: Learn distributed systems concepts
Modern Apps: Most apps use object storage
Career Growth: High demand skill

Made with ❤️ for learning about distributed storage systems

⚠️ This is experimental software - use MinIO for production workloads!