devrim99 Claude Opus 4.6 (1M context) commited on
Commit
abd099a
·
1 Parent(s): 9840c5d

Update README for model repo with proper frontmatter

Browse files

- Add YAML frontmatter so HuggingFace shows clone widget
- Update pack list to current names
- Remove all --repo-type dataset references
- Simplify CLI examples with actual repo ID

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. README.md +37 -115
README.md CHANGED
@@ -1,25 +1,41 @@
 
 
 
 
 
 
 
 
 
1
  # ContentVault
2
 
3
  Content packs for [Lucky Robots](https://luckyrobots.com). Each pack contains a `metadata.yaml`, a `thumbnail.png`, and a `.zip` with the pack contents.
4
 
 
 
 
 
 
 
5
  ## Content Packs
6
 
7
  | Pack | Description |
8
  |------|-------------|
9
- | ExamplePack | Example content pack |
10
- | Go1VelocityTracking | Unitree Go1 velocity tracking |
 
11
  | MujocoExample | MuJoCo simulation example |
 
12
  | Oscillator | Oscillator environment |
13
  | Panda | Franka Panda robot |
14
  | Piper | Piper robot |
15
- | Piper-Pick-Place | Piper pick and place task |
 
16
  | PiperUnscrewCap | Piper cap unscrewing task |
 
17
  | Skies-Vol-1 | Sky HDRIs volume 1 |
18
- | TheBungalow | Bungalow environment |
19
- | TheLoft | Loft environment |
20
- | TheOffice | Office environment |
21
  | UnitreeG1 | Unitree G1 humanoid |
22
- | UnitreeGo1 | Unitree Go1 quadruped |
23
  | Welcome | Welcome / intro pack |
24
 
25
  ## CDN
@@ -32,17 +48,18 @@ https://contentvault.luckyrobots.com/{PackName}/{filename}
32
 
33
  For example:
34
  ```
35
- https://contentvault.luckyrobots.com/ExamplePack/metadata.yaml
36
- https://contentvault.luckyrobots.com/TheLoft/TheLoft.zip
37
  ```
38
 
39
  ## How sync works
40
 
41
- A Cloudflare Worker (`worker/`) listens for webhooks from HuggingFace. When a push is made to the [HuggingFace dataset repo](https://huggingface.co/datasets/luckyrobots/ContentVault), the Worker:
42
 
43
- 1. Lists all files in the repo via the HuggingFace API
44
  2. Streams each file directly to Cloudflare R2 (no buffering, handles large zip files)
45
- 3. R2 serves the files through Cloudflare's CDN with a 30-day cache, cached at 300+ edge locations worldwide
 
46
 
47
  ## Worker setup
48
 
@@ -64,8 +81,6 @@ npx wrangler secret put HF_TOKEN
64
  npx wrangler deploy
65
  ```
66
 
67
- ### Commands
68
-
69
  ## Setup
70
 
71
  ```bash
@@ -82,119 +97,27 @@ hf auth login
82
 
83
  ### Upload entire folder
84
  ```bash
85
- hf upload <repo-id> <local-folder> . --repo-type dataset --commit-message "Your message"
86
  ```
87
 
88
  ### Upload a single pack/subfolder
89
  ```bash
90
- hf upload <repo-id> <local-folder>/<pack-name> <pack-name> --repo-type dataset --commit-message "Add <pack-name>"
91
  ```
92
 
93
  ### Upload a single file
94
  ```bash
95
- hf upload <repo-id> <local-file> <path-in-repo> --repo-type dataset --commit-message "Add file"
96
  ```
97
 
98
  ### Upload large folder (resumable, multi-threaded, auto-retry)
99
  ```bash
100
- hf upload-large-folder <repo-id> <local-folder> --repo-type=dataset
101
- ```
102
-
103
- ### Delete files remotely (without re-uploading everything)
104
- ```bash
105
- hf upload <repo-id> . . --repo-type dataset --include="" --delete="<folder-name>/*" --commit-message "Remove <folder-name>"
106
- ```
107
-
108
- ### Create a new dataset repo
109
- ```bash
110
- hf repo create <repo-name> --type dataset
111
- ```
112
-
113
- ### Create under an organization
114
- ```bash
115
- hf repo create <repo-name> --type dataset --organization <org-name>
116
- ```
117
-
118
- ---
119
-
120
- ## Python One-Liners
121
-
122
- ### Upload large folder (resumable)
123
- ```bash
124
- python -c "from huggingface_hub import HfApi; HfApi().upload_large_folder(repo_id='<repo-id>', repo_type='dataset', folder_path='<local-folder>')"
125
  ```
126
 
127
- ### Upload a single folder/pack
128
  ```bash
129
- python -c "from huggingface_hub import HfApi; HfApi().upload_folder(folder_path='<local-folder>', path_in_repo='<path-in-repo>', repo_id='<repo-id>', repo_type='dataset', commit_message='Your message')"
130
- ```
131
-
132
- ### Upload a single file
133
- ```bash
134
- python -c "from huggingface_hub import HfApi; HfApi().upload_file(path_or_fileobj='<local-file>', path_in_repo='<path-in-repo>', repo_id='<repo-id>', repo_type='dataset', commit_message='Your message')"
135
- ```
136
-
137
- ### Delete a folder
138
- ```bash
139
- python -c "from huggingface_hub import HfApi; HfApi().delete_folder('<folder-name>', repo_id='<repo-id>', repo_type='dataset', commit_message='Remove <folder-name>')"
140
- ```
141
-
142
- ### Delete a single file
143
- ```bash
144
- python -c "from huggingface_hub import HfApi; HfApi().delete_file('<file-path>', repo_id='<repo-id>', repo_type='dataset', commit_message='Remove <file-path>')"
145
- ```
146
-
147
- ### Create a repo
148
- ```bash
149
- python -c "from huggingface_hub import HfApi; HfApi().create_repo(repo_id='<repo-id>', repo_type='dataset')"
150
- ```
151
-
152
- ---
153
-
154
- ## Python Script Examples
155
-
156
- ### Upload large folder (full script with options)
157
- ```python
158
- from huggingface_hub import HfApi
159
-
160
- api = HfApi()
161
- api.upload_large_folder(
162
- repo_id="<repo-id>",
163
- repo_type="dataset",
164
- folder_path="<local-folder>",
165
- )
166
- ```
167
-
168
- ### Upload a folder to a specific path in the repo
169
- ```python
170
- from huggingface_hub import HfApi
171
-
172
- api = HfApi()
173
- api.upload_folder(
174
- folder_path="<local-folder>",
175
- path_in_repo="<path-in-repo>",
176
- repo_id="<repo-id>",
177
- repo_type="dataset",
178
- commit_message="Your message",
179
- ignore_patterns=["*.cache", ".git/*"],
180
- )
181
- ```
182
-
183
- ### Batch delete + upload in one commit
184
- ```python
185
- from huggingface_hub import HfApi, CommitOperationAdd, CommitOperationDelete
186
-
187
- api = HfApi()
188
- operations = [
189
- CommitOperationDelete(path_in_repo="<old-folder>/"),
190
- CommitOperationAdd(path_in_repo="<file-path>", path_or_fileobj="<local-file>"),
191
- ]
192
- api.create_commit(
193
- repo_id="<repo-id>",
194
- repo_type="dataset",
195
- operations=operations,
196
- commit_message="Your message",
197
- )
198
  ```
199
 
200
  ---
@@ -202,8 +125,7 @@ api.create_commit(
202
  ## Tips
203
 
204
  - **Resumable uploads**: `upload_large_folder` and `hf upload-large-folder` are resumable. If interrupted, run the same command again.
205
- - **Don't use raw `git push`** for large files -- use the CLI/API instead. They handle LFS/Xet automatically.
206
  - **Write token required**: Tokens default to read-only. Enable write access at https://huggingface.co/settings/tokens.
207
  - **50 GB max per file** on HuggingFace. Split larger files before uploading.
208
- - **Ignore patterns**: Use `--include` and `--exclude` (CLI) or `ignore_patterns` (Python) to filter what gets uploaded.
209
- - **Delete the `.cache/huggingface/` folder** inside your local folder to reset upload state if something goes wrong with `upload_large_folder`.
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - content-packs
5
+ - robotics
6
+ - 3d-environments
7
+ - luckyrobots
8
+ ---
9
+
10
  # ContentVault
11
 
12
  Content packs for [Lucky Robots](https://luckyrobots.com). Each pack contains a `metadata.yaml`, a `thumbnail.png`, and a `.zip` with the pack contents.
13
 
14
+ ## Clone
15
+
16
+ ```bash
17
+ git clone https://huggingface.co/luckyrobots/ContentVault
18
+ ```
19
+
20
  ## Content Packs
21
 
22
  | Pack | Description |
23
  |------|-------------|
24
+ | Bunglalow | Bungalow environment |
25
+ | Go2VelocityTracking | Unitree Go2 velocity tracking |
26
+ | Loft | Loft environment |
27
  | MujocoExample | MuJoCo simulation example |
28
+ | Office | Office environment |
29
  | Oscillator | Oscillator environment |
30
  | Panda | Franka Panda robot |
31
  | Piper | Piper robot |
32
+ | PiperBlockStacking | Piper block stacking task |
33
+ | PiperPickPlace | Piper pick and place task |
34
  | PiperUnscrewCap | Piper cap unscrewing task |
35
+ | SO100PickAndPlace | SO-100 pick and place task |
36
  | Skies-Vol-1 | Sky HDRIs volume 1 |
 
 
 
37
  | UnitreeG1 | Unitree G1 humanoid |
38
+ | UnitreeGo2 | Unitree Go2 quadruped |
39
  | Welcome | Welcome / intro pack |
40
 
41
  ## CDN
 
48
 
49
  For example:
50
  ```
51
+ https://contentvault.luckyrobots.com/Loft/metadata.yaml
52
+ https://contentvault.luckyrobots.com/Loft/Loft.zip
53
  ```
54
 
55
  ## How sync works
56
 
57
+ A Cloudflare Worker (`worker/`) listens for webhooks from HuggingFace. When a push is made to this repo, the Worker:
58
 
59
+ 1. Parses the commit diff to find changed files
60
  2. Streams each file directly to Cloudflare R2 (no buffering, handles large zip files)
61
+ 3. Deletes removed files from R2
62
+ 4. R2 serves the files through Cloudflare's CDN with a 30-day cache at 300+ edge locations worldwide
63
 
64
  ## Worker setup
65
 
 
81
  npx wrangler deploy
82
  ```
83
 
 
 
84
  ## Setup
85
 
86
  ```bash
 
97
 
98
  ### Upload entire folder
99
  ```bash
100
+ hf upload luckyrobots/ContentVault <local-folder> . --commit-message "Your message"
101
  ```
102
 
103
  ### Upload a single pack/subfolder
104
  ```bash
105
+ hf upload luckyrobots/ContentVault <local-folder>/<pack-name> <pack-name> --commit-message "Add <pack-name>"
106
  ```
107
 
108
  ### Upload a single file
109
  ```bash
110
+ hf upload luckyrobots/ContentVault <local-file> <path-in-repo> --commit-message "Add file"
111
  ```
112
 
113
  ### Upload large folder (resumable, multi-threaded, auto-retry)
114
  ```bash
115
+ hf upload-large-folder luckyrobots/ContentVault <local-folder>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
  ```
117
 
118
+ ### Delete files remotely
119
  ```bash
120
+ hf upload luckyrobots/ContentVault . . --include="" --delete="<folder-name>/*" --commit-message "Remove <folder-name>"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
  ```
122
 
123
  ---
 
125
  ## Tips
126
 
127
  - **Resumable uploads**: `upload_large_folder` and `hf upload-large-folder` are resumable. If interrupted, run the same command again.
128
+ - **Don't use raw `git push`** for large files use the CLI/API instead. They handle LFS/Xet automatically.
129
  - **Write token required**: Tokens default to read-only. Enable write access at https://huggingface.co/settings/tokens.
130
  - **50 GB max per file** on HuggingFace. Split larger files before uploading.
131
+ - **Ignore patterns**: Use `--include` and `--exclude` to filter what gets uploaded.