Linaqruf commited on
Commit
edbdeb5
Β·
verified Β·
1 Parent(s): f9583f1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +27 -212
README.md CHANGED
@@ -1,212 +1,27 @@
1
- ---
2
- license: apache-2.0
3
- title: Modelscope Migration
4
- sdk: gradio
5
- emoji: πŸš€
6
- colorFrom: red
7
- colorTo: blue
8
- short_description: Migrate your models to modelscope
9
- sdk_version: 6.2.0
10
- ---
11
-
12
- # πŸš€ HuggingFace to ModelScope Migration Tool
13
-
14
- A user-friendly Gradio application for migrating models and datasets from [HuggingFace](https://huggingface.co/) to [ModelScope](https://www.modelscope.cn/).
15
-
16
- ## ✨ Features
17
-
18
- - **Easy Migration**: Migrate models and datasets with a simple web interface
19
- - **Secure Authentication**: Supports token-based authentication for both platforms
20
- - **Progress Tracking**: Real-time status updates during migration
21
- - **Flexible Configuration**: Customize repository visibility, license, and metadata
22
- - **Error Handling**: Comprehensive error messages and validation
23
- - **Automatic Cleanup**: Temporary files are automatically cleaned up after migration
24
-
25
- ## πŸ“‹ Prerequisites
26
-
27
- - Python 3.8 or higher
28
- - Internet connection
29
- - HuggingFace account and access token
30
- - ModelScope account and SDK token
31
-
32
- ## πŸ”§ Installation
33
-
34
- 1. Clone this repository:
35
- ```bash
36
- git clone <repository-url>
37
- cd modelscope-migration
38
- ```
39
-
40
- 2. Install dependencies:
41
- ```bash
42
- pip install -r requirements.txt
43
- ```
44
-
45
- Or install manually:
46
- ```bash
47
- pip install gradio huggingface-hub modelscope
48
- ```
49
-
50
- ## πŸš€ Quick Start
51
-
52
- 1. Run the application:
53
- ```bash
54
- python app.py
55
- ```
56
-
57
- 2. Open your browser and navigate to the provided URL (usually `http://localhost:7860`)
58
-
59
- 3. Fill in the required information:
60
- - **HuggingFace Token**: Get from https://huggingface.co/settings/tokens
61
- - **ModelScope Token**: Get from https://www.modelscope.cn/my/myaccesstoken
62
- - **Source Repository**: The HuggingFace repository ID (e.g., `bert-base-uncased`)
63
- - **Destination Repository**: Your desired ModelScope repository ID (e.g., `username/my-model`)
64
-
65
- 4. Click "Start Migration" and wait for completion
66
-
67
- ## πŸ“– Usage Guide
68
-
69
- ### Getting Your Tokens
70
-
71
- #### HuggingFace Token:
72
- 1. Go to https://huggingface.co/settings/tokens
73
- 2. Click "New token"
74
- 3. Give it a name and select appropriate permissions
75
- 4. Copy the token (starts with `hf_`)
76
-
77
- #### ModelScope Token:
78
- 1. Go to https://www.modelscope.cn/my/myaccesstoken
79
- 2. Log in to your account
80
- 3. Copy your SDK token
81
-
82
- ### Migration Options
83
-
84
- | Option | Description | Default |
85
- |--------|-------------|---------|
86
- | **Repository Type** | Choose between `model` or `dataset` | `model` |
87
- | **Visibility** | Set repository as `public` or `private` | `public` |
88
- | **License** | Choose from Apache 2.0, MIT, GPL-3.0, or Other | `apache-2.0` |
89
- | **Chinese Name** | Optional Chinese name for the repository | None |
90
-
91
- ### Example Repository IDs
92
-
93
- **HuggingFace:**
94
- - `bert-base-uncased` (official model)
95
- - `username/my-custom-model` (user model)
96
- - `datasets/squad` (dataset)
97
-
98
- **ModelScope:**
99
- - `username/repo-name` (always requires username prefix)
100
-
101
- ## πŸ› οΈ How It Works
102
-
103
- The migration process follows these steps:
104
-
105
- 1. **Authentication**: Validates tokens for both HuggingFace and ModelScope
106
- 2. **Download**: Uses `snapshot_download` to download the entire repository from HuggingFace
107
- 3. **Upload**: Creates repository on ModelScope (if needed) and uploads all files
108
- 4. **Cleanup**: Removes temporary files from local storage
109
-
110
- ## πŸ“ Features in Detail
111
-
112
- ### Model Migration
113
-
114
- Migrates complete model repositories including:
115
- - Model weights and configuration files
116
- - Tokenizer files
117
- - README and documentation
118
- - Any additional files in the repository
119
-
120
- ### Dataset Migration
121
-
122
- Migrates dataset repositories including:
123
- - Dataset files (all formats)
124
- - Dataset cards and metadata
125
- - Scripts and configuration files
126
-
127
- ### Error Handling
128
-
129
- The tool provides clear error messages for:
130
- - Invalid tokens
131
- - Repository not found
132
- - Permission errors
133
- - Network issues
134
- - Disk space problems
135
-
136
- ## ⚠️ Important Notes
137
-
138
- - **Disk Space**: Ensure you have enough disk space for temporary storage of the repository
139
- - **Large Models**: Migration time depends on repository size and network speed
140
- - **Permissions**: Your tokens must have appropriate read/write permissions
141
- - **Private Repos**: To migrate private repositories, ensure your HuggingFace token has access
142
- - **Repository Creation**: The tool automatically creates the ModelScope repository if it doesn't exist
143
-
144
- ## πŸ”’ Security
145
-
146
- - Tokens are handled securely and not logged
147
- - Tokens are entered as password fields (hidden input)
148
- - Temporary files are cleaned up after migration
149
- - No data is stored permanently on the server
150
-
151
- ## πŸ› Troubleshooting
152
-
153
- ### "Authentication failed"
154
- - Verify your tokens are correct and not expired
155
- - Check that you're using the right token for each platform
156
-
157
- ### "Download failed"
158
- - Ensure the HuggingFace repository exists and is accessible
159
- - Check your internet connection
160
- - Verify you have permission to access the repository
161
-
162
- ### "Upload failed"
163
- - Verify your ModelScope token has write permissions
164
- - Check that the destination repository name is valid
165
- - Ensure you're not exceeding ModelScope's quotas
166
-
167
- ### "Disk space error"
168
- - Free up disk space on your system
169
- - Large models require significant temporary storage
170
-
171
- ## πŸ“š API Documentation
172
-
173
- ### HuggingFace Hub
174
- - Documentation: https://huggingface.co/docs/huggingface_hub
175
- - PyPI: https://pypi.org/project/huggingface-hub/
176
-
177
- ### ModelScope
178
- - GitHub: https://github.com/modelscope/modelscope
179
- - PyPI: https://pypi.org/project/modelscope/
180
-
181
- ## 🀝 Contributing
182
-
183
- Contributions are welcome! Please feel free to submit issues or pull requests.
184
-
185
- ## πŸ“„ License
186
-
187
- This project is open source and available under the Apache 2.0 License.
188
-
189
- ## πŸ”— Resources
190
-
191
- - [HuggingFace Hub](https://huggingface.co/)
192
- - [ModelScope Platform](https://www.modelscope.cn/)
193
- - [Gradio Documentation](https://gradio.app/docs/)
194
-
195
- ## πŸ’‘ Tips
196
-
197
- 1. **Test with small models first**: Before migrating large models, test with a smaller repository
198
- 2. **Check licenses**: Ensure you have the right to migrate and redistribute the model/dataset
199
- 3. **Preserve metadata**: The tool preserves all files, but review the ModelScope repository after migration
200
- 4. **Network stability**: For large repositories, ensure a stable internet connection
201
-
202
- ## πŸ†˜ Support
203
-
204
- If you encounter issues:
205
- 1. Check the troubleshooting section above
206
- 2. Review the error message in the status output
207
- 3. Verify your tokens and repository IDs
208
- 4. Check the official documentation for HuggingFace and ModelScope
209
-
210
- ---
211
-
212
- Made with ❀️ using [Gradio](https://gradio.app/), [HuggingFace Hub](https://huggingface.co/), and [ModelScope](https://www.modelscope.cn/)
 
1
+ ---
2
+ license: apache-2.0
3
+ title: Modelscope Migration
4
+ sdk: gradio
5
+ emoji: πŸš€
6
+ colorFrom: red
7
+ colorTo: blue
8
+ short_description: Migrate your models to modelscope
9
+ sdk_version: 6.2.0
10
+ ---
11
+
12
+ # πŸš€ HuggingFace to ModelScope Migration Tool
13
+
14
+ A user-friendly Gradio application for migrating models and datasets from [HuggingFace](https://huggingface.co/) to [ModelScope](https://www.modelscope.cn/).
15
+
16
+ ## ✨ Features
17
+
18
+ - **Easy Migration**: Migrate models and datasets with a simple web interface
19
+ - **Secure Authentication**: Supports token-based authentication for both platforms
20
+ - **Progress Tracking**: Real-time status updates during migration
21
+ - **Flexible Configuration**: Customize repository visibility, license, and metadata
22
+ - **Error Handling**: Comprehensive error messages and validation
23
+ - **Automatic Cleanup**: Temporary files are automatically cleaned up after migration
24
+
25
+ ---
26
+
27
+ Made with ❀️ using [Gradio](https://gradio.app/), [HuggingFace Hub](https://huggingface.co/), and [ModelScope](https://www.modelscope.cn/)