{ "cells": [ { "cell_type": "markdown", "metadata": { "jp-MarkdownHeadingCollapsed": true }, "source": [ "# \"Doro\" Welcome to the Hugging Face File Uploader! \n", "\n", "# Welcome to the Hugging Face File Uploader\n", "\n", "Use this notebook's widget to easily upload files to your Hugging Face repositories.\n", "\n", "## Getting Started\n", "\n", "- Install Libraries: Run the first code cell.\n", "- Log In: Run the notebook_login() cell with a Hugging Face write token.\n", "- Launch Uploader: Run the final cell to display the uploader widget.\n", "\n", "## Uploading Files\n", "\n", "- Repo Info: Fill in the destination repository details.\n", "- File Selection: Point to a local directory and select your files.\n", "- Upload: Click the upload button to start the process.\n", "\n", "## Key Features\n", "- Fast Uploads: Automatically uses hf_transfer to speed up large file transfers.\n", "- Live Progress: Monitor upload status and speed in the output log.\n", "- Simple Interface: All controls are contained within a single, easy-to-use widget.\n", "\n", "\n", "**Community & Support:**\n", "\n", "* **GitHub:** [HuggingFace\\_Backup Repository on GitHub](https://github.com/Ktiseos-Nyx/HuggingFace_Backup) (for the latest version, updates, bug reports, and contributions)\n", "* **Discord:**\n", " * [Ktiseos Nyx AI/ML Discord](https://discord.gg/HhBSvM9gBY)\n", " * [Earth & Dusk Media](https://discord.gg/5t2kYxt7An)\n", "\n", "This uploader is designed to simplify the process of getting your files onto the Hugging Face Hub. We hope you find it useful!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " # \"Doro\" Install Dependencies" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Installing/updating required packages...\n", "Package installation/update process complete.\n" ] } ], "source": [ "# Cell 1: Install Required Python Packages\n", "# -----------------------------------------------------------------------------\n", "print(\"Installing/updating required packages...\")\n", "!pip install -q huggingface_hub\n", "!pip install -q ipywidgets \n", "!pip install -q hf_transfer\n", "print(\"Package installation/update process complete.\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Xs1mb1VKLuUW" }, "source": [ "# โœจ \"Doro\" Connecting to Hugging Face: Authentication\n", "\n", "## ๐Ÿ”‘ Hugging Face Login\n", "You need a write-enabled token to upload files.\n", "\n", "### Instructions:\n", "\n", "- Create a Token: Go [here](https://huggingface.co/settings/tokens), click \"New token\", and give it the write role.\n", "- Copy the Token: Copy the new token to your clipboard.\n", "- Run Login Cell: Execute the code cell below, paste your token when prompted, and press Enter.\n", "\n", "\n", "Troubleshooting: If you get an error, the most common problem is that your token is missing write permissions. Double-check the token's role on the Hugging Face website." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e4bf0d699a1c403e978ece176575b5cd", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='
\"Doro\" Using the Hugging Face File Uploader\n", "\n", "## Uploader Checklist\n", "Follow these steps to upload your files.\n", "\n", "1. Fill in Repository Details\n", "- Owner (your username/org)\n", "- Repo Name\n", "- Repo Type (model, dataset, etc.)\n", "\n", "2. Select Local Files\n", "- Set the Source Directory to your local folder path.\n", "- Click the ๐Ÿ”„ List Files button.\n", "- Select your desired files from the list that appears.\n", "\n", "3. Review Upload Settings (Optional)\n", "- Add a Commit Message to describe your changes.\n", "- Choose whether to Create a Pull Request for this upload.\n", "\n", "4. Start the Upload\n", "- Click the โฌ†๏ธ Upload Selected Files button and monitor the output." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "cellView": "form", "id": "J851eLx6Ii3h" }, "outputs": [], "source": [ "# --- Essential Imports for the Uploader ---\n", "import glob\n", "import os\n", "import time\n", "from pathlib import Path\n", "import math # For math.log and math.floor in _format_size\n", "\n", "from huggingface_hub import HfApi\n", "from ipywidgets import (Text, Dropdown, Button, SelectMultiple, VBox, HBox,\n", " Output, Layout, Checkbox, HTML, Textarea, Label,\n", " FloatProgress)\n", "from IPython.display import display, clear_output\n", "\n", "# Attempt to enable hf_transfer.\n", "os.environ['HF_HUB_ENABLE_HF_TRANSFER'] = '1'\n", "\n", "class HuggingFaceUploader:\n", " \"\"\"\n", " A Jupyter widget-based tool to upload files to the Hugging Face Hub.\n", " \"\"\"\n", "\n", " def __init__(self):\n", " self.api = HfApi()\n", " self.file_types = [\n", " # AI Model Files ๐Ÿค–\n", " ('SafeTensors', 'safetensors'), ('PyTorch Models', 'pt'), ('PyTorch Legacy', 'pth'),\n", " ('ONNX Models', 'onnx'), ('TensorFlow Models', 'pb'), ('Keras Models', 'h5'),\n", " # Checkpoint Files ๐ŸŽฏ\n", " ('Checkpoints', 'ckpt'), ('Binary Files', 'bin'),\n", " # Config & Data Files ๐Ÿ“\n", " ('JSON Files', 'json'), ('YAML Files', 'yaml'), ('YAML Alt', 'yml'),\n", " ('Text Files', 'txt'), ('CSV Files', 'csv'), ('Pickle Files', 'pkl'),\n", " # Image Files ๐ŸŽจ\n", " ('PNG Images', 'png'), ('JPEG Images', 'jpg'), ('JPEG Alt', 'jpeg'),\n", " ('WebP Images', 'webp'), ('GIF Images', 'gif'),\n", " # Archive Files ๐Ÿ“ฆ\n", " ('ZIP Archives', 'zip'), ('TAR Files', 'tar'), ('GZ Archives', 'gz')\n", " ]\n", " self.current_directory = os.getcwd()\n", " self.hf_transfer_active = self._check_hf_transfer_availability()\n", " self._create_widgets()\n", " self._bind_events()\n", " self._update_files(None) # Initial file list update\n", "\n", " def _check_hf_transfer_availability(self):\n", " if os.environ.get(\"HF_HUB_ENABLE_HF_TRANSFER\") == \"1\":\n", " try:\n", " import hf_transfer\n", " return True\n", " except ImportError:\n", " return False\n", " return False\n", "\n", " def _create_widgets(self):\n", " # --- Repository Info ---\n", " self.repo_info_html = HTML(value=\"๐Ÿ“š Repository Details\")\n", " self.org_name_text = Text(placeholder='Organization or Username', description='Owner:', style={'description_width': 'initial'})\n", " self.repo_name_text = Text(placeholder='Repository Name', description='Repo:', style={'description_width': 'initial'})\n", " self.repo_type_dropdown = Dropdown(options=['model', 'dataset', 'space'], value='model', description='Repo Type:', style={'description_width': 'initial'})\n", " self.repo_folder_text = Text(placeholder='Optional: e.g., models/v1', description='Remote Folder:', style={'description_width': 'initial', \"flex\": \"1 1 auto\"}, layout=Layout(width='auto'))\n", "\n", " # --- File Selection ---\n", " self.file_section_html = HTML(value=\"๐Ÿ—‚๏ธ File Selection & Source\")\n", " self.file_type_dropdown = Dropdown(options=self.file_types, value='safetensors', description='File Type:', style={'description_width': 'initial'})\n", " self.sort_by_dropdown = Dropdown(options=['name', 'date'], value='name', description='Sort By:', style={'description_width': 'initial'})\n", " self.recursive_search_checkbox = Checkbox(value=False, description='Search Subdirectories', indent=False)\n", "\n", " self.directory_label = Label(value=\"Source Directory:\", layout=Layout(width='auto'))\n", " self.directory_text = Text(value=self.current_directory, description=\"\", style={'description_width': '0px'}, layout=Layout(width=\"auto\", flex='1 1 auto'))\n", " self.directory_update_btn = Button(description='๐Ÿ”„ List Files', button_style='info', tooltip='Change source directory and refresh file list', layout=Layout(width='auto'))\n", "\n", " # --- Commit Details ---\n", " self.commit_section_html = HTML(value=\"๐Ÿ’ญ Commit Details\")\n", " self.commit_msg_textarea = Textarea(value=\"Upload via HuggingFaceUploader Widget ๐Ÿค—\", placeholder='Enter your commit message (optional)', description='Message:', style={'description_width': 'initial'}, layout=Layout(width='98%', height='60px'))\n", "\n", " # --- Upload Settings ---\n", " self.upload_section_html = HTML(value=\"๐Ÿš€ Upload Settings\")\n", " self.create_pr_checkbox = Checkbox(value=False, description='Create Pull Request', indent=False)\n", " self.clear_after_checkbox = Checkbox(value=True, description='Clear output after upload', indent=False)\n", "\n", " # --- Action Buttons ---\n", " self.upload_button = Button(description='โฌ†๏ธ Upload Selected Files', button_style='success', tooltip='Start upload process', layout=Layout(width='auto', height='auto'))\n", " self.clear_output_button = Button(description='๐Ÿงน Clear Output Log', button_style='warning', tooltip='Clear the output log area', layout=Layout(width='auto'))\n", "\n", " # --- File Picker & Output ---\n", " self.file_picker_selectmultiple = SelectMultiple(options=[], description='Files:', layout=Layout(width=\"98%\", height=\"200px\"), style={'description_width': 'initial'})\n", " self.output_area = Output(layout=Layout(padding='10px', border='1px solid #ccc', margin_top='10px', width='98%', max_height='400px', overflow_y='auto'))\n", "\n", " # --- Progress Display Area ---\n", " self.current_file_label = Label(value=\"N/A\")\n", " self.file_count_label = Label(value=\"File 0/0\")\n", " self.progress_bar = FloatProgress(value=0, min=0, max=100, description='Overall:', bar_style='info', layout=Layout(width='85%'))\n", " self.progress_percent_label = Label(value=\"0%\")\n", "\n", " self.progress_display_box = VBox([\n", " HBox([Label(\"Current File:\", layout=Layout(width='100px')), self.current_file_label]),\n", " HBox([Label(\"File Count:\", layout=Layout(width='100px')), self.file_count_label]),\n", " HBox([self.progress_bar, self.progress_percent_label], layout=Layout(align_items='center'))\n", " ], layout=Layout(visibility='hidden', margin='10px 0', padding='10px', border='1px solid #ddd', width='98%'))\n", "\n", " def _bind_events(self):\n", " self.directory_update_btn.on_click(self._update_directory_and_files)\n", " self.upload_button.on_click(self._upload_files_handler)\n", " self.clear_output_button.on_click(lambda _: self.output_area.clear_output(wait=True))\n", " self.file_type_dropdown.observe(self._update_files, names='value')\n", " self.sort_by_dropdown.observe(self._update_files, names='value')\n", " self.recursive_search_checkbox.observe(self._update_files, names='value') # NEW BINDING\n", "\n", " def _update_directory_and_files(self, _):\n", " new_dir = self.directory_text.value.strip()\n", " if not new_dir:\n", " with self.output_area:\n", " clear_output(wait=True); print(f\"๐Ÿ“‚ Current directory remains: {self.current_directory}\")\n", " self._update_files(None)\n", " return\n", "\n", " if os.path.isdir(new_dir):\n", " self.current_directory = os.path.abspath(new_dir)\n", " self.directory_text.value = self.current_directory\n", " self._update_files(None)\n", " else:\n", " with self.output_area:\n", " clear_output(wait=True); print(f\"โŒ Invalid Directory: {new_dir}\")\n", "\n", " def _update_files(self, _):\n", " file_extension = self.file_type_dropdown.value\n", " self.output_area.clear_output(wait=True)\n", " try:\n", " # ENHANCEMENT: Use glob '**/ ' for recursive search\n", " glob_prefix = '**/' if self.recursive_search_checkbox.value else ''\n", " glob_pattern = f\"{glob_prefix}*.{file_extension}\"\n", " \n", " if not os.path.isdir(self.current_directory):\n", " with self.output_area: print(f\"โš ๏ธ Source directory '{self.current_directory}' is not valid.\")\n", " self.file_picker_selectmultiple.options = []\n", " return\n", "\n", " # Use rglob for simplicity if recursive\n", " source_path = Path(self.current_directory)\n", " found_paths = list(source_path.rglob(f'*.{file_extension}')) if self.recursive_search_checkbox.value else list(source_path.glob(f'*.{file_extension}'))\n", " \n", " valid_files_info = []\n", " for p in found_paths:\n", " if p.is_symlink() or not p.is_file(): continue\n", " sort_key = p.stat().st_mtime if self.sort_by_dropdown.value == 'date' else p.name.lower()\n", " valid_files_info.append((str(p), sort_key))\n", "\n", " if self.sort_by_dropdown.value == 'date':\n", " valid_files_info.sort(key=lambda item: item[1], reverse=True)\n", " else:\n", " valid_files_info.sort(key=lambda item: item[1])\n", " \n", " # Display relative paths in the picker for clarity, but store absolute paths\n", " # self.file_picker_selectmultiple.options now a list of (display_name, value)\n", " display_options = []\n", " for abs_path_str, _ in valid_files_info:\n", " display_name = os.path.relpath(abs_path_str, self.current_directory)\n", " display_options.append((display_name, abs_path_str))\n", " \n", " self.file_picker_selectmultiple.options = display_options\n", " \n", " with self.output_area:\n", " if not display_options:\n", " print(f\"๐Ÿคท No '.{file_extension}' files found in '{self.current_directory}'.\")\n", " else:\n", " print(f\"โœจ Found {len(display_options)} '.{file_extension}' files. Select files to upload.\")\n", "\n", " except Exception as e:\n", " with self.output_area:\n", " clear_output(wait=True); print(f\"โŒ Error listing files: {e}\"); traceback.print_exc(file=self.output_area)\n", "\n", " def _format_size(self, size_bytes):\n", " if size_bytes < 0: return \"Invalid size\"\n", " if size_bytes == 0: return \"0 B\"\n", " units = (\"B\", \"KB\", \"MB\", \"GB\", \"TB\", \"PB\", \"EB\")\n", " # CORRECTED: Ensure direct calls to math module, not Path.math\n", " i = math.floor(math.log(size_bytes, 1024)) if size_bytes > 0 else 0\n", " if i >= len(units): i = len(units) - 1\n", " s = round(size_bytes / (1024 ** i), 2)\n", " return f\"{s} {units[i]}\"\n", "\n", " def _print_file_info(self, file_path_str, index, total_files):\n", " file_path = Path(file_path_str)\n", " try:\n", " file_size = file_path.stat().st_size\n", " self.output_area.append_stdout(f\"๐Ÿ“ฆ Uploading {index}/{total_files}: {file_path.name} ({self._format_size(file_size)})\\n\")\n", " except FileNotFoundError:\n", " self.output_area.append_stdout(f\"โš ๏ธ File not found: {file_path_str}\\n\")\n", "\n", " def _upload_files_handler(self, _):\n", " org_or_user = self.org_name_text.value.strip()\n", " repo_name = self.repo_name_text.value.strip()\n", "\n", " if not org_or_user or not repo_name:\n", " with self.output_area: clear_output(wait=True); print(\"โ— Please fill in 'Owner' and 'Repo Name'.\")\n", " return\n", "\n", " repo_id = f\"{org_or_user}/{repo_name}\"\n", " selected_file_paths = list(self.file_picker_selectmultiple.value)\n", "\n", " if not selected_file_paths:\n", " with self.output_area: clear_output(wait=True); print(\"๐Ÿ“ Nothing selected for upload.\")\n", " return\n", "\n", " self.output_area.clear_output(wait=True)\n", " self.output_area.append_stdout(f\"๐ŸŽฏ Preparing to upload to: https://huggingface.co/{repo_id}\\n\")\n", " if self.hf_transfer_active: self.output_area.append_stdout(\"๐Ÿš€ HF_TRANSFER is enabled.\\n\")\n", " else: self.output_area.append_stdout(\"โ„น๏ธ For faster uploads, run `%pip install -q hf_transfer` and restart kernel.\\n\")\n", "\n", " self.progress_display_box.layout.visibility = 'visible'\n", " self.progress_bar.value = 0\n", " self.progress_percent_label.value = \"0%\"\n", " self.current_file_label.value = \"Initializing...\"\n", " \n", " total_files = len(selected_file_paths)\n", " self.file_count_label.value = f\"File 0/{total_files}\"\n", " \n", " repo_type = self.repo_type_dropdown.value\n", " repo_folder_prefix = self.repo_folder_text.value.strip().replace('\\\\', '/')\n", " base_commit_msg = self.commit_msg_textarea.value or \"Upload via HuggingFaceUploader Widget ๐Ÿค—\"\n", "\n", " success_count = 0\n", " for idx, local_file_path_str in enumerate(selected_file_paths, 1):\n", " try:\n", " current_file_path = Path(local_file_path_str)\n", " self.current_file_label.value = current_file_path.name\n", " self.file_count_label.value = f\"File {idx}/{total_files}\"\n", " self._print_file_info(local_file_path_str, idx, total_files)\n", " \n", " start_time = time.time()\n", " \n", " if not current_file_path.exists():\n", " self.output_area.append_stdout(f\"โŒ SKIPPED: File '{current_file_path.name}' not found.\\n\")\n", " continue\n", " \n", " # This logic correctly handles recursive uploads\n", " path_in_repo_base = os.path.relpath(current_file_path, self.current_directory).replace('\\\\', '/')\n", " path_in_repo = f\"{repo_folder_prefix}/{path_in_repo_base}\" if repo_folder_prefix.strip('/') else path_in_repo_base\n", " \n", " commit_message_for_file = f\"{base_commit_msg} ({current_file_path.name})\"\n", "\n", " response_url = self.api.upload_file(\n", " path_or_fileobj=str(current_file_path),\n", " path_in_repo=path_in_repo,\n", " repo_id=repo_id,\n", " repo_type=repo_type,\n", " create_pr=self.create_pr_checkbox.value,\n", " commit_message=commit_message_for_file,\n", " )\n", " duration = time.time() - start_time\n", " self.output_area.append_stdout(f\"โœ… Uploaded '{current_file_path.name}' to '{path_in_repo}' in {duration:.1f}s.\\n\")\n", " self.output_area.append_stdout(f\" View at: {response_url}\\n\")\n", " success_count += 1\n", "\n", " except Exception as e:\n", " self.output_area.append_stdout(f\"โŒ Error uploading {current_file_path.name}: {e}\\n\")\n", " import traceback\n", " with self.output_area: traceback.print_exc()\n", " self.output_area.append_stdout(\"\\n\")\n", " finally:\n", " percentage = int((idx / total_files) * 100)\n", " self.progress_bar.value = percentage\n", " self.progress_percent_label.value = f\"{percentage}%\"\n", "\n", " self.output_area.append_stdout(f\"\\nโœจ Upload complete. {success_count}/{total_files} files processed. โœจ\\n\")\n", " if self.create_pr_checkbox.value and success_count > 0:\n", " self.output_area.append_stdout(f\"๐ŸŽ‰ View Pull Request: https://huggingface.co/{repo_id}/pulls\\n\")\n", " elif success_count > 0 :\n", " repo_tree_url = f\"https://huggingface.co/{repo_id}/tree/main/{repo_folder_prefix.strip('/')}\".rstrip('/')\n", " self.output_area.append_stdout(f\"๐ŸŽ‰ View files at: {repo_tree_url}\\n\")\n", "\n", " self.current_file_label.value = \"Completed.\"\n", " if self.clear_after_checkbox.value:\n", " time.sleep(5)\n", " self.output_area.clear_output(wait=True)\n", " self.progress_display_box.layout.visibility = 'hidden'\n", "\n", " def display(self):\n", " repo_box = HBox([self.org_name_text, self.repo_name_text, self.repo_type_dropdown], layout=Layout(flex_flow='wrap', justify_content='space-between'))\n", " repo_folder_box = HBox([self.repo_folder_text], layout=Layout(width='100%'))\n", " dir_select_box = HBox([self.directory_label, self.directory_text, self.directory_update_btn], layout=Layout(width='100%', align_items='center'))\n", " file_opts_box = HBox([self.file_type_dropdown, self.sort_by_dropdown, self.recursive_search_checkbox], layout=Layout(flex_flow='wrap', justify_content='space-between', align_items='center'))\n", " upload_opts_box = HBox([self.create_pr_checkbox, self.clear_after_checkbox], layout=Layout(margin='5px 0'))\n", " action_buttons_box = HBox([self.upload_button, self.clear_output_button], layout=Layout(margin='10px 0 0 0', spacing='10px'))\n", "\n", " main_layout = VBox([\n", " self.repo_info_html, repo_box, repo_folder_box,\n", " HTML(\"
\"),\n", " self.file_section_html, file_opts_box, dir_select_box,\n", " self.file_picker_selectmultiple,\n", " HTML(\"
\"),\n", " self.commit_section_html, self.commit_msg_textarea,\n", " HTML(\"
\"),\n", " self.upload_section_html, upload_opts_box,\n", " action_buttons_box,\n", " self.progress_display_box,\n", " self.output_area\n", " ], layout=Layout(width='700px', padding='10px', border='1px solid lightgray'))\n", " \n", " display(main_layout)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿš€ \"Doro\" Uploader Widget! \n", "\n", "**Run the next cell to initiate the uploader widget!**\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "๐Ÿš€ Initializing Hugging Face Uploader...\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1e5ee45c1f814f75a8a55436d3b7b448", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='๐Ÿ“š Repository Details'), HBox(children=(Text(value='', description='Owner:', โ€ฆ" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "โœ… Uploader interface is ready. You can now select files and upload.\n" ] } ], "source": [ "# Uploader Widget Code\n", "print(\"๐Ÿš€ Initializing Hugging Face Uploader...\")\n", "uploader = HuggingFaceUploader()\n", "uploader.display()\n", "print(\"โœ… Uploader interface is ready. You can now select files and upload.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "colab": { "collapsed_sections": [ "IZ_JYwvBLrg-", "PNF2kdyeO3Dn" ], "private_outputs": true, "provenance": [] }, "kernelspec": { "display_name": "Python3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 4 }