Upload 6 files

README.md

@@ -1,210 +1,212 @@
----
-license: apache-2.0
-title: Modelscope Migration
-sdk: gradio
-emoji: 🚀
-colorFrom: red
-short_description: Migrate your models to modelscope
-sdk_version: 6.2.0
----
-# 🚀 HuggingFace to ModelScope Migration Tool
-
-A user-friendly Gradio application for migrating models and datasets from [HuggingFace](https://huggingface.co/) to [ModelScope](https://www.modelscope.cn/).
-
-## ✨ Features
-
-- **Easy Migration**: Migrate models and datasets with a simple web interface
-- **Secure Authentication**: Supports token-based authentication for both platforms
-- **Progress Tracking**: Real-time status updates during migration
-- **Flexible Configuration**: Customize repository visibility, license, and metadata
-- **Error Handling**: Comprehensive error messages and validation
-- **Automatic Cleanup**: Temporary files are automatically cleaned up after migration
-
-## 📋 Prerequisites
-
-- Python 3.8 or higher
-- Internet connection
-- HuggingFace account and access token
-- ModelScope account and SDK token
-
-## 🔧 Installation
-
-1. Clone this repository:
-```bash
-git clone <repository-url>
-cd modelscope-migration
-```
-
-2. Install dependencies:
-```bash
-pip install -r requirements.txt
-```
-
-Or install manually:
-```bash
-pip install gradio huggingface-hub modelscope
-```
-
-## 🚀 Quick Start
-
-1. Run the application:
-```bash
-python app.py
-```
-
-2. Open your browser and navigate to the provided URL (usually `http://localhost:7860`)
-
-3. Fill in the required information:
-   - **HuggingFace Token**: Get from https://huggingface.co/settings/tokens
-   - **ModelScope Token**: Get from https://www.modelscope.cn/my/myaccesstoken
-   - **Source Repository**: The HuggingFace repository ID (e.g., `bert-base-uncased`)
-   - **Destination Repository**: Your desired ModelScope repository ID (e.g., `username/my-model`)
-
-4. Click "Start Migration" and wait for completion
-
-## 📖 Usage Guide
-
-### Getting Your Tokens
-
-#### HuggingFace Token:
-1. Go to https://huggingface.co/settings/tokens
-2. Click "New token"
-3. Give it a name and select appropriate permissions
-4. Copy the token (starts with `hf_`)
-
-#### ModelScope Token:
-1. Go to https://www.modelscope.cn/my/myaccesstoken
-2. Log in to your account
-3. Copy your SDK token
-
-### Migration Options
-
-| Option | Description | Default |
-|--------|-------------|---------|
-| **Repository Type** | Choose between `model` or `dataset` | `model` |
-| **Visibility** | Set repository as `public` or `private` | `public` |
-| **License** | Choose from Apache 2.0, MIT, GPL-3.0, or Other | `apache-2.0` |
-| **Chinese Name** | Optional Chinese name for the repository | None |
-
-### Example Repository IDs
-
-**HuggingFace:**
-- `bert-base-uncased` (official model)
-- `username/my-custom-model` (user model)
-- `datasets/squad` (dataset)
-
-**ModelScope:**
-- `username/repo-name` (always requires username prefix)
-
-## 🛠️ How It Works
-
-The migration process follows these steps:
-
-1. **Authentication**: Validates tokens for both HuggingFace and ModelScope
-2. **Download**: Uses `snapshot_download` to download the entire repository from HuggingFace
-3. **Upload**: Creates repository on ModelScope (if needed) and uploads all files
-4. **Cleanup**: Removes temporary files from local storage
-
-## 📝 Features in Detail
-
-### Model Migration
-
-Migrates complete model repositories including:
-- Model weights and configuration files
-- Tokenizer files
-- README and documentation
-- Any additional files in the repository
-
-### Dataset Migration
-
-Migrates dataset repositories including:
-- Dataset files (all formats)
-- Dataset cards and metadata
-- Scripts and configuration files
-
-### Error Handling
-
-The tool provides clear error messages for:
-- Invalid tokens
-- Repository not found
-- Permission errors
-- Network issues
-- Disk space problems
-
-## ⚠️ Important Notes
-
-- **Disk Space**: Ensure you have enough disk space for temporary storage of the repository
-- **Large Models**: Migration time depends on repository size and network speed
-- **Permissions**: Your tokens must have appropriate read/write permissions
-- **Private Repos**: To migrate private repositories, ensure your HuggingFace token has access
-- **Repository Creation**: The tool automatically creates the ModelScope repository if it doesn't exist
-
-## 🔒 Security
-
-- Tokens are handled securely and not logged
-- Tokens are entered as password fields (hidden input)
-- Temporary files are cleaned up after migration
-- No data is stored permanently on the server
-
-## 🐛 Troubleshooting
-
-### "Authentication failed"
-- Verify your tokens are correct and not expired
-- Check that you're using the right token for each platform
-
-### "Download failed"
-- Ensure the HuggingFace repository exists and is accessible
-- Check your internet connection
-- Verify you have permission to access the repository
-
-### "Upload failed"
-- Verify your ModelScope token has write permissions
-- Check that the destination repository name is valid
-- Ensure you're not exceeding ModelScope's quotas
-
-### "Disk space error"
-- Free up disk space on your system
-- Large models require significant temporary storage
-
-## 📚 API Documentation
-
-### HuggingFace Hub
-- Documentation: https://huggingface.co/docs/huggingface_hub
-- PyPI: https://pypi.org/project/huggingface-hub/
-
-### ModelScope
-- GitHub: https://github.com/modelscope/modelscope
-- PyPI: https://pypi.org/project/modelscope/
-
-## 🤝 Contributing
-
-Contributions are welcome! Please feel free to submit issues or pull requests.
-
-## 📄 License
-
-This project is open source and available under the Apache 2.0 License.
-
-## 🔗 Resources
-
-- [HuggingFace Hub](https://huggingface.co/)
-- [ModelScope Platform](https://www.modelscope.cn/)
-- [Gradio Documentation](https://gradio.app/docs/)
-
-## 💡 Tips
-
-1. **Test with small models first**: Before migrating large models, test with a smaller repository
-2. **Check licenses**: Ensure you have the right to migrate and redistribute the model/dataset
-3. **Preserve metadata**: The tool preserves all files, but review the ModelScope repository after migration
-4. **Network stability**: For large repositories, ensure a stable internet connection
-
-## 🆘 Support
-
-If you encounter issues:
-1. Check the troubleshooting section above
-2. Review the error message in the status output
-3. Verify your tokens and repository IDs
-4. Check the official documentation for HuggingFace and ModelScope
-
----
-
-Made with ❤️ using [Gradio](https://gradio.app/), [HuggingFace Hub](https://huggingface.co/), and [ModelScope](https://www.modelscope.cn/)
+---
+license: apache-2.0
+title: Modelscope Migration
+sdk: gradio
+emoji: 🚀
+colorFrom: red
+colorTo: blue
+short_description: Migrate your models to modelscope
+sdk_version: 6.2.0
+---
+
+# 🚀 HuggingFace to ModelScope Migration Tool
+
+A user-friendly Gradio application for migrating models and datasets from [HuggingFace](https://huggingface.co/) to [ModelScope](https://www.modelscope.cn/).
+
+## ✨ Features
+
+- **Easy Migration**: Migrate models and datasets with a simple web interface
+- **Secure Authentication**: Supports token-based authentication for both platforms
+- **Progress Tracking**: Real-time status updates during migration
+- **Flexible Configuration**: Customize repository visibility, license, and metadata
+- **Error Handling**: Comprehensive error messages and validation
+- **Automatic Cleanup**: Temporary files are automatically cleaned up after migration
+
+## 📋 Prerequisites
+
+- Python 3.8 or higher
+- Internet connection
+- HuggingFace account and access token
+- ModelScope account and SDK token
+
+## 🔧 Installation
+
+1. Clone this repository:
+```bash
+git clone <repository-url>
+cd modelscope-migration
+```
+
+2. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+Or install manually:
+```bash
+pip install gradio huggingface-hub modelscope
+```
+
+## 🚀 Quick Start
+
+1. Run the application:
+```bash
+python app.py
+```
+
+2. Open your browser and navigate to the provided URL (usually `http://localhost:7860`)
+
+3. Fill in the required information:
+   - **HuggingFace Token**: Get from https://huggingface.co/settings/tokens
+   - **ModelScope Token**: Get from https://www.modelscope.cn/my/myaccesstoken
+   - **Source Repository**: The HuggingFace repository ID (e.g., `bert-base-uncased`)
+   - **Destination Repository**: Your desired ModelScope repository ID (e.g., `username/my-model`)
+
+4. Click "Start Migration" and wait for completion
+
+## 📖 Usage Guide
+
+### Getting Your Tokens
+
+#### HuggingFace Token:
+1. Go to https://huggingface.co/settings/tokens
+2. Click "New token"
+3. Give it a name and select appropriate permissions
+4. Copy the token (starts with `hf_`)
+
+#### ModelScope Token:
+1. Go to https://www.modelscope.cn/my/myaccesstoken
+2. Log in to your account
+3. Copy your SDK token
+
+### Migration Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| **Repository Type** | Choose between `model` or `dataset` | `model` |
+| **Visibility** | Set repository as `public` or `private` | `public` |
+| **License** | Choose from Apache 2.0, MIT, GPL-3.0, or Other | `apache-2.0` |
+| **Chinese Name** | Optional Chinese name for the repository | None |
+
+### Example Repository IDs
+
+**HuggingFace:**
+- `bert-base-uncased` (official model)
+- `username/my-custom-model` (user model)
+- `datasets/squad` (dataset)
+
+**ModelScope:**
+- `username/repo-name` (always requires username prefix)
+
+## 🛠️ How It Works
+
+The migration process follows these steps:
+
+1. **Authentication**: Validates tokens for both HuggingFace and ModelScope
+2. **Download**: Uses `snapshot_download` to download the entire repository from HuggingFace
+3. **Upload**: Creates repository on ModelScope (if needed) and uploads all files
+4. **Cleanup**: Removes temporary files from local storage
+
+## 📝 Features in Detail
+
+### Model Migration
+
+Migrates complete model repositories including:
+- Model weights and configuration files
+- Tokenizer files
+- README and documentation
+- Any additional files in the repository
+
+### Dataset Migration
+
+Migrates dataset repositories including:
+- Dataset files (all formats)
+- Dataset cards and metadata
+- Scripts and configuration files
+
+### Error Handling
+
+The tool provides clear error messages for:
+- Invalid tokens
+- Repository not found
+- Permission errors
+- Network issues
+- Disk space problems
+
+## ⚠️ Important Notes
+
+- **Disk Space**: Ensure you have enough disk space for temporary storage of the repository
+- **Large Models**: Migration time depends on repository size and network speed
+- **Permissions**: Your tokens must have appropriate read/write permissions
+- **Private Repos**: To migrate private repositories, ensure your HuggingFace token has access
+- **Repository Creation**: The tool automatically creates the ModelScope repository if it doesn't exist
+
+## 🔒 Security
+
+- Tokens are handled securely and not logged
+- Tokens are entered as password fields (hidden input)
+- Temporary files are cleaned up after migration
+- No data is stored permanently on the server
+
+## 🐛 Troubleshooting
+
+### "Authentication failed"
+- Verify your tokens are correct and not expired
+- Check that you're using the right token for each platform
+
+### "Download failed"
+- Ensure the HuggingFace repository exists and is accessible
+- Check your internet connection
+- Verify you have permission to access the repository
+
+### "Upload failed"
+- Verify your ModelScope token has write permissions
+- Check that the destination repository name is valid
+- Ensure you're not exceeding ModelScope's quotas
+
+### "Disk space error"
+- Free up disk space on your system
+- Large models require significant temporary storage
+
+## 📚 API Documentation
+
+### HuggingFace Hub
+- Documentation: https://huggingface.co/docs/huggingface_hub
+- PyPI: https://pypi.org/project/huggingface-hub/
+
+### ModelScope
+- GitHub: https://github.com/modelscope/modelscope
+- PyPI: https://pypi.org/project/modelscope/
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit issues or pull requests.
+
+## 📄 License
+
+This project is open source and available under the Apache 2.0 License.
+
+## 🔗 Resources
+
+- [HuggingFace Hub](https://huggingface.co/)
+- [ModelScope Platform](https://www.modelscope.cn/)
+- [Gradio Documentation](https://gradio.app/docs/)
+
+## 💡 Tips
+
+1. **Test with small models first**: Before migrating large models, test with a smaller repository
+2. **Check licenses**: Ensure you have the right to migrate and redistribute the model/dataset
+3. **Preserve metadata**: The tool preserves all files, but review the ModelScope repository after migration
+4. **Network stability**: For large repositories, ensure a stable internet connection
+
+## 🆘 Support
+
+If you encounter issues:
+1. Check the troubleshooting section above
+2. Review the error message in the status output
+3. Verify your tokens and repository IDs
+4. Check the official documentation for HuggingFace and ModelScope
+
+---
+
+Made with ❤️ using [Gradio](https://gradio.app/), [HuggingFace Hub](https://huggingface.co/), and [ModelScope](https://www.modelscope.cn/)

app.py

@@ -20,42 +20,20 @@ class MigrationTool:
     """Handles migration of models and datasets between HuggingFace and ModelScope."""
 
     def __init__(self):
-        self.hf_api = None
-        self.ms_api = None
-        self.ms_token = None
         self.temp_dir = None
 
-    def authenticate_hf(self, token: str) -> Tuple[bool, str]:
-        """Authenticate with HuggingFace."""
-        try:
-            self.hf_api = HfApi(token=token)
-            # Test the token
-            self.hf_api.whoami(token=token)
-            return True, "✓ HuggingFace authentication successful"
-        except Exception as e:
-            return False, f"✗ HuggingFace authentication failed: {str(e)}"
-
-    def authenticate_ms(self, token: str) -> Tuple[bool, str]:
-        """Authenticate with ModelScope."""
-        try:
-            # Store the token and create HubApi instance
-            # We'll authenticate when actually using the API
-            self.ms_token = token
-            self.ms_api = HubApi()
-            return True, "✓ ModelScope token stored successfully"
-        except Exception as e:
-            return False, f"✗ ModelScope initialization failed: {str(e)}"
-
     def download_from_hf(
         self,
         repo_id: str,
-        repo_type: str = "model"
+        repo_type: str = "model",
+        token: Optional[str] = None
     ) -> Tuple[bool, str, Optional[str]]:
         """Download a repository from HuggingFace.
 
         Args:
             repo_id: HuggingFace repository ID (e.g., 'username/repo-name')
             repo_type: Type of repository ('model' or 'dataset')
+            token: HuggingFace authentication token
 
         Returns:
             Tuple of (success, message, local_path)
@@ -70,7 +48,7 @@ class MigrationTool:
                 repo_type=repo_type,
                 local_dir=self.temp_dir,
                 local_dir_use_symlinks=False,
-                token=self.hf_api.token if self.hf_api else None
+                token=token
             )
 
             return True, f"✓ Successfully downloaded {repo_type} from HuggingFace", local_path
@@ -81,6 +59,7 @@ class MigrationTool:
         self,
         local_path: str,
         repo_id: str,
+        token: str,
         repo_type: str = "model",
         visibility: str = "public",
         license_type: str = "apache-2.0",
@@ -91,6 +70,7 @@ class MigrationTool:
         Args:
             local_path: Local path to the repository
             repo_id: ModelScope repository ID (e.g., 'username/repo-name')
+            token: ModelScope authentication token
             repo_type: Type of repository ('model' or 'dataset')
             visibility: Repository visibility ('public' or 'private')
             license_type: License type
@@ -100,14 +80,9 @@ class MigrationTool:
             Tuple of (success, message)
         """
         try:
-            if not self.ms_api or not hasattr(self, 'ms_token'):
-                return False, "✗ ModelScope not authenticated"
-
-            # Login with the stored token
-            try:
-                self.ms_api.login(self.ms_token)
-            except Exception as login_error:
-                return False, f"✗ ModelScope login failed: {str(login_error)}"
+            # Create HubApi instance and login
+            api = HubApi()
+            api.login(token)
 
             # Determine visibility
             vis = ModelVisibility.PUBLIC if visibility == "public" else ModelVisibility.PRIVATE
@@ -124,7 +99,7 @@ class MigrationTool:
             # Create repository if it doesn't exist
             try:
                 if repo_type == "model":
-                    self.ms_api.create_model(
+                    api.create_model(
                         model_id=repo_id,
                         visibility=vis,
                         license=lic,
@@ -136,13 +111,13 @@ class MigrationTool:
 
             # Push the model/dataset
             if repo_type == "model":
-                self.ms_api.push_model(
+                api.push_model(
                     model_id=repo_id,
                     model_dir=local_path
                 )
             else:
                 # For datasets, use similar approach
-                self.ms_api.push_model(
+                api.push_model(
                     model_id=repo_id,
                     model_dir=local_path
                 )
@@ -185,23 +160,9 @@ class MigrationTool:
         if not hf_repo_id or not ms_repo_id:
             return "✗ Error: Both source and destination repository IDs are required"
 
-        # Authenticate with HuggingFace
-        output.append("🔐 Authenticating with HuggingFace...")
-        success, msg = self.authenticate_hf(hf_token)
-        output.append(msg)
-        if not success:
-            return "\n".join(output)
-
-        # Authenticate with ModelScope
-        output.append("\n🔐 Authenticating with ModelScope...")
-        success, msg = self.authenticate_ms(ms_token)
-        output.append(msg)
-        if not success:
-            return "\n".join(output)
-
         # Download from HuggingFace
-        output.append(f"\n⬇️  Downloading {repo_type} from HuggingFace: {hf_repo_id}...")
-        success, msg, local_path = self.download_from_hf(hf_repo_id, repo_type)
+        output.append(f"⬇️  Downloading {repo_type} from HuggingFace: {hf_repo_id}...")
+        success, msg, local_path = self.download_from_hf(hf_repo_id, repo_type, hf_token)
         output.append(msg)
         if not success:
             self.cleanup()
@@ -212,6 +173,7 @@ class MigrationTool:
         success, msg = self.upload_to_ms(
             local_path,
             ms_repo_id,
+            ms_token,
             repo_type,
             visibility,
             license_type,

requirements.txt

@@ -1,3 +1,7 @@
 gradio>=4.0.0
 huggingface-hub>=0.20.0
 modelscope>=1.11.0
+
+# Performance optimization for faster downloads/uploads
+hf_transfer  # Faster upload/download with Rust-based backend
+hf_xet  # XetHub integration for efficient model transfers