Hugging Face's logo

Spaces:
Barbuuuuuuuu
/
code-dataset-generator
Running

App Files Community
Barbuuuuuuuu commited on
Commit
62ad90e
·
verified ·
1 Parent(s): b840fbd

Add 2 files

Browse files
Files changed (2) hide show
  1. README.md +6 -4
  2. index.html +429 -19
README.md CHANGED
@@ -1,10 +1,12 @@
1
  ---
2
- title: Code Dataset Generator
3
- emoji: 🏃
4
  colorFrom: blue
5
- colorTo: yellow
6
  sdk: static
7
  pinned: false
 
 
8
  ---
9
 
10
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
1
  ---
2
+ title: code-dataset-generator
3
+ emoji: 🐳
4
  colorFrom: blue
5
+ colorTo: blue
6
  sdk: static
7
  pinned: false
8
+ tags:
9
+ - deepsite
10
  ---
11
 
12
+ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
index.html CHANGED
@@ -1,19 +1,429 @@
1
- <!doctype html>
2
- <html>
3
- <head>
4
- <meta charset="utf-8" />
5
- <meta name="viewport" content="width=device-width" />
6
- <title>My static Space</title>
7
- <link rel="stylesheet" href="style.css" />
8
- </head>
9
- <body>
10
- <div class="card">
11
- <h1>Welcome to your static Space!</h1>
12
- <p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
13
- <p>
14
- Also don't forget to check the
15
- <a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
16
- </p>
17
- </div>
18
- </body>
19
- </html>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>Code Dataset Generator for Falcon 40B</title>
7
+ <script src="https://cdn.tailwindcss.com"></script>
8
+ <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css">
9
+ <style>
10
+ .file-drop-area {
11
+ border: 2px dashed #6366f1;
12
+ transition: all 0.3s ease;
13
+ }
14
+ .file-drop-area.active {
15
+ border-color: #4f46e5;
16
+ background-color: #eef2ff;
17
+ }
18
+ .progress-bar {
19
+ transition: width 0.3s ease;
20
+ }
21
+ .code-preview {
22
+ font-family: 'Courier New', monospace;
23
+ background-color: #1e293b;
24
+ color: #f8fafc;
25
+ }
26
+ .language-badge {
27
+ top: -10px;
28
+ right: -10px;
29
+ }
30
+ .slide-fade-enter-active, .slide-fade-leave-active {
31
+ transition: all 0.3s ease;
32
+ }
33
+ .slide-fade-enter-from, .slide-fade-leave-to {
34
+ opacity: 0;
35
+ transform: translateY(10px);
36
+ }
37
+ </style>
38
+ </head>
39
+ <body class="bg-gray-50 min-h-screen">
40
+ <div class="container mx-auto px-4 py-8" id="app">
41
+ <!-- Header -->
42
+ <header class="mb-10 text-center">
43
+ <h1 class="text-4xl font-bold text-indigo-700 mb-2">
44
+ <i class="fas fa-database mr-2"></i>Code Dataset Generator
45
+ </h1>
46
+ <p class="text-lg text-gray-600 max-w-2xl mx-auto">
47
+ Transform your source code into fine-tuning datasets for Falcon 40B and other LLMs
48
+ </p>
49
+ </header>
50
+
51
+ <!-- Main Content -->
52
+ <div class="grid grid-cols-1 lg:grid-cols-3 gap-8">
53
+ <!-- Upload Section -->
54
+ <div class="lg:col-span-2 bg-white rounded-xl shadow-md overflow-hidden">
55
+ <div class="p-6">
56
+ <h2 class="text-2xl font-semibold text-gray-800 mb-4">
57
+ <i class="fas fa-upload mr-2 text-indigo-600"></i>Upload Source Files
58
+ </h2>
59
+
60
+ <!-- File Drop Area -->
61
+ <div
62
+ id="dropArea"
63
+ class="file-drop-area rounded-lg p-8 text-center cursor-pointer mb-6"
64
+ @dragover.prevent="dragOver = true"
65
+ @dragleave="dragOver = false"
66
+ @drop.prevent="handleDrop"
67
+ :class="{ 'active': dragOver }"
68
+ >
69
+ <div class="flex flex-col items-center justify-center space-y-3">
70
+ <i class="fas fa-cloud-upload-alt text-4xl text-indigo-500"></i>
71
+ <p class="text-lg font-medium text-gray-700">Drag & drop your source files here</p>
72
+ <p class="text-sm text-gray-500">or</p>
73
+ <input
74
+ type="file"
75
+ id="fileInput"
76
+ class="hidden"
77
+ multiple
78
+ @change="handleFileSelect"
79
+ accept=".py,.cpp,.c,.go,.rs,.js,.java,.php,.rb,.ts"
80
+ >
81
+ <button
82
+ @click="document.getElementById('fileInput').click()"
83
+ class="px-4 py-2 bg-indigo-600 text-white rounded-md hover:bg-indigo-700 transition"
84
+ >
85
+ Browse Files
86
+ </button>
87
+ <p class="text-xs text-gray-400 mt-2">Supported: Python, C/C++, Rust, Go, JavaScript, Java, PHP, Ruby, TypeScript</p>
88
+ </div>
89
+ </div>
90
+
91
+ <!-- Selected Files -->
92
+ <div v-if="selectedFiles.length > 0" class="mb-6">
93
+ <h3 class="text-lg font-medium text-gray-700 mb-3">Selected Files</h3>
94
+ <div class="space-y-2">
95
+ <div
96
+ v-for="(file, index) in selectedFiles"
97
+ :key="index"
98
+ class="flex items-center justify-between bg-gray-50 p-3 rounded-md"
99
+ >
100
+ <div class="flex items-center space-x-3">
101
+ <i :class="getFileIcon(file.name)" class="text-indigo-500"></i>
102
+ <span class="text-sm font-medium text-gray-700 truncate max-w-xs">{{ file.name }}</span>
103
+ <span class="text-xs text-gray-500">{{ formatFileSize(file.size) }}</span>
104
+ </div>
105
+ <button
106
+ @click="removeFile(index)"
107
+ class="text-red-500 hover:text-red-700"
108
+ >
109
+ <i class="fas fa-times"></i>
110
+ </button>
111
+ </div>
112
+ </div>
113
+ </div>
114
+
115
+ <!-- Processing Options -->
116
+ <div class="mb-6">
117
+ <h3 class="text-lg font-medium text-gray-700 mb-3">Processing Options</h3>
118
+ <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
119
+ <div>
120
+ <label class="block text-sm font-medium text-gray-700 mb-1">Output Format</label>
121
+ <select v-model="outputFormat" class="w-full rounded-md border-gray-300 shadow-sm focus:border-indigo-500 focus:ring-indigo-500">
122
+ <option value="jsonl">JSONL (Alpaca format)</option>
123
+ <option value="hf">HuggingFace Dataset</option>
124
+ <option value="openchat">OpenChat Format</option>
125
+ </select>
126
+ </div>
127
+ <div>
128
+ <label class="block text-sm font-medium text-gray-700 mb-1">Language Detection</label>
129
+ <select v-model="languageDetection" class="w-full rounded-md border-gray-300 shadow-sm focus:border-indigo-500 focus:ring-indigo-500">
130
+ <option value="auto">Auto-detect (recommended)</option>
131
+ <option value="manual">Manual selection</option>
132
+ </select>
133
+ </div>
134
+ </div>
135
+ <div class="mt-4 space-y-2">
136
+ <label class="inline-flex items-center">
137
+ <input type="checkbox" v-model="extractFunctions" class="rounded border-gray-300 text-indigo-600 shadow-sm focus:border-indigo-500 focus:ring-indigo-500">
138
+ <span class="ml-2 text-sm text-gray-700">Extract functions/methods</span>
139
+ </label>
140
+ <label class="inline-flex items-center">
141
+ <input type="checkbox" v-model="extractClasses" class="rounded border-gray-300 text-indigo-600 shadow-sm focus:border-indigo-500 focus:ring-indigo-500">
142
+ <span class="ml-2 text-sm text-gray-700">Extract classes</span>
143
+ </label>
144
+ <label class="inline-flex items-center">
145
+ <input type="checkbox" v-model="includeComments" class="rounded border-gray-300 text-indigo-600 shadow-sm focus:border-indigo-500 focus:ring-indigo-500">
146
+ <span class="ml-2 text-sm text-gray-700">Include comments/docstrings</span>
147
+ </label>
148
+ <label class="inline-flex items-center">
149
+ <input type="checkbox" v-model="cleanCode" class="rounded border-gray-300 text-indigo-600 shadow-sm focus:border-indigo-500 focus:ring-indigo-500">
150
+ <span class="ml-2 text-sm text-gray-700">Clean code (remove noise)</span>
151
+ </label>
152
+ </div>
153
+ </div>
154
+
155
+ <!-- Process Button -->
156
+ <button
157
+ @click="processFiles"
158
+ :disabled="selectedFiles.length === 0 || processing"
159
+ class="w-full py-3 px-4 bg-indigo-600 hover:bg-indigo-700 text-white font-medium rounded-md transition flex items-center justify-center"
160
+ :class="{ 'opacity-50 cursor-not-allowed': selectedFiles.length === 0 || processing }"
161
+ >
162
+ <span v-if="!processing">
163
+ <i class="fas fa-cogs mr-2"></i>Generate Dataset
164
+ </span>
165
+ <span v-else>
166
+ <i class="fas fa-spinner fa-spin mr-2"></i>Processing...
167
+ </span>
168
+ </button>
169
+ </div>
170
+ </div>
171
+
172
+ <!-- Preview Section -->
173
+ <div class="bg-white rounded-xl shadow-md overflow-hidden">
174
+ <div class="p-6">
175
+ <h2 class="text-2xl font-semibold text-gray-800 mb-4">
176
+ <i class="fas fa-eye mr-2 text-indigo-600"></i>Dataset Preview
177
+ </h2>
178
+
179
+ <!-- Progress Bar -->
180
+ <div v-if="processing" class="mb-6">
181
+ <div class="flex justify-between text-sm text-gray-600 mb-1">
182
+ <span>Processing files...</span>
183
+ <span>{{ processedFiles }} / {{ selectedFiles.length }}</span>
184
+ </div>
185
+ <div class="w-full bg-gray-200 rounded-full h-2.5">
186
+ <div
187
+ class="progress-bar bg-indigo-600 h-2.5 rounded-full"
188
+ :style="{ width: (processedFiles / selectedFiles.length) * 100 + '%' }"
189
+ ></div>
190
+ </div>
191
+ </div>
192
+
193
+ <!-- Result Preview -->
194
+ <div v-if="previewData" class="space-y-4">
195
+ <div class="flex justify-between items-center">
196
+ <h3 class="text-lg font-medium text-gray-700">Generated Samples</h3>
197
+ <span class="text-xs bg-indigo-100 text-indigo-800 px-2 py-1 rounded-full">
198
+ {{ previewData.length }} items
199
+ </span>
200
+ </div>
201
+
202
+ <!-- Sample Selector -->
203
+ <div class="relative">
204
+ <select
205
+ v-model="selectedSampleIndex"
206
+ class="w-full rounded-md border-gray-300 shadow-sm focus:border-indigo-500 focus:ring-indigo-500"
207
+ >
208
+ <option v-for="(item, index) in previewData" :value="index">
209
+ Sample {{ index + 1 }} - {{ item.instruction.substring(0, 30) }}...
210
+ </option>
211
+ </select>
212
+ </div>
213
+
214
+ <!-- Sample Preview -->
215
+ <div v-if="selectedSampleIndex !== null" class="space-y-3">
216
+ <div>
217
+ <label class="block text-sm font-medium text-gray-700 mb-1">Instruction</label>
218
+ <div class="bg-gray-50 p-3 rounded-md text-sm">
219
+ {{ previewData[selectedSampleIndex].instruction }}
220
+ </div>
221
+ </div>
222
+
223
+ <div>
224
+ <label class="block text-sm font-medium text-gray-700 mb-1">Input Code</label>
225
+ <div class="code-preview p-3 rounded-md text-sm overflow-x-auto">
226
+ <pre>{{ previewData[selectedSampleIndex].input }}</pre>
227
+ </div>
228
+ </div>
229
+
230
+ <div>
231
+ <label class="block text-sm font-medium text-gray-700 mb-1">Output</label>
232
+ <div class="bg-gray-50 p-3 rounded-md text-sm">
233
+ {{ previewData[selectedSampleIndex].output }}
234
+ </div>
235
+ </div>
236
+ </div>
237
+
238
+ <!-- Download Button -->
239
+ <button
240
+ v-if="previewData.length > 0"
241
+ @click="downloadDataset"
242
+ class="w-full py-2 px-4 bg-green-600 hover:bg-green-700 text-white font-medium rounded-md transition flex items-center justify-center mt-4"
243
+ >
244
+ <i class="fas fa-download mr-2"></i>Download Dataset
245
+ </button>
246
+ </div>
247
+
248
+ <!-- Empty State -->
249
+ <div v-else class="text-center py-10">
250
+ <i class="fas fa-code text-4xl text-gray-300 mb-3"></i>
251
+ <p class="text-gray-500">Your processed dataset will appear here</p>
252
+ </div>
253
+ </div>
254
+ </div>
255
+ </div>
256
+
257
+ <!-- Features Section -->
258
+ <div class="mt-16">
259
+ <h2 class="text-2xl font-bold text-center text-gray-800 mb-8">Key Features</h2>
260
+ <div class="grid grid-cols-1 md:grid-cols-3 gap-6">
261
+ <div class="bg-white p-6 rounded-xl shadow-sm border border-gray-100">
262
+ <div class="text-indigo-600 mb-3">
263
+ <i class="fas fa-language text-3xl"></i>
264
+ </div>
265
+ <h3 class="text-lg font-semibold mb-2">Multi-language Support</h3>
266
+ <p class="text-gray-600">Automatically detects and processes code in Python, C/C++, Rust, Go, JavaScript and more.</p>
267
+ </div>
268
+ <div class="bg-white p-6 rounded-xl shadow-sm border border-gray-100">
269
+ <div class="text-indigo-600 mb-3">
270
+ <i class="fas fa-robot text-3xl"></i>
271
+ </div>
272
+ <h3 class="text-lg font-semibold mb-2">Smart Prompt Engineering</h3>
273
+ <p class="text-gray-600">Automatically generates meaningful instruction-output pairs from your source code.</p>
274
+ </div>
275
+ <div class="bg-white p-6 rounded-xl shadow-sm border border-gray-100">
276
+ <div class="text-indigo-600 mb-3">
277
+ <i class="fas fa-database text-3xl"></i>
278
+ </div>
279
+ <h3 class="text-lg font-semibold mb-2">Falcon 40B Optimized</h3>
280
+ <p class="text-gray-600">Tokenization and formatting specifically optimized for Falcon 40B model fine-tuning.</p>
281
+ </div>
282
+ </div>
283
+ </div>
284
+
285
+ <!-- Footer -->
286
+ <footer class="mt-16 text-center text-gray-500 text-sm">
287
+ <p>Code Dataset Generator - Transform your source code into LLM training data</p>
288
+ <p class="mt-1">© 2023 AI Engineering Team. All rights reserved.</p>
289
+ </footer>
290
+ </div>
291
+
292
+ <script>
293
+ const app = {
294
+ data() {
295
+ return {
296
+ dragOver: false,
297
+ selectedFiles: [],
298
+ outputFormat: 'jsonl',
299
+ languageDetection: 'auto',
300
+ extractFunctions: true,
301
+ extractClasses: true,
302
+ includeComments: true,
303
+ cleanCode: false,
304
+ processing: false,
305
+ processedFiles: 0,
306
+ previewData: null,
307
+ selectedSampleIndex: 0
308
+ }
309
+ },
310
+ methods: {
311
+ handleDrop(e) {
312
+ this.dragOver = false;
313
+ const files = Array.from(e.dataTransfer.files);
314
+ this.addFiles(files);
315
+ },
316
+ handleFileSelect(e) {
317
+ const files = Array.from(e.target.files);
318
+ this.addFiles(files);
319
+ e.target.value = ''; // Reset input to allow selecting same file again
320
+ },
321
+ addFiles(files) {
322
+ const validExtensions = ['.py', '.cpp', '.c', '.go', '.rs', '.js', '.java', '.php', '.rb', '.ts'];
323
+ const filteredFiles = files.filter(file => {
324
+ const ext = '.' + file.name.split('.').pop().toLowerCase();
325
+ return validExtensions.includes(ext);
326
+ });
327
+
328
+ if (filteredFiles.length < files.length) {
329
+ alert('Some files were ignored. Only source code files are supported.');
330
+ }
331
+
332
+ this.selectedFiles = [...this.selectedFiles, ...filteredFiles];
333
+ },
334
+ removeFile(index) {
335
+ this.selectedFiles.splice(index, 1);
336
+ },
337
+ getFileIcon(filename) {
338
+ const ext = filename.split('.').pop().toLowerCase();
339
+ const icons = {
340
+ py: 'fab fa-python',
341
+ cpp: 'fas fa-file-code',
342
+ c: 'fas fa-file-code',
343
+ h: 'fas fa-file-code',
344
+ go: 'fab fa-golang',
345
+ rs: 'fas fa-rust',
346
+ js: 'fab fa-js-square',
347
+ java: 'fab fa-java',
348
+ php: 'fab fa-php',
349
+ rb: 'fas fa-gem',
350
+ ts: 'fas fa-file-code'
351
+ };
352
+ return icons[ext] || 'fas fa-file-code';
353
+ },
354
+ formatFileSize(bytes) {
355
+ if (bytes === 0) return '0 Bytes';
356
+ const k = 1024;
357
+ const sizes = ['Bytes', 'KB', 'MB', 'GB'];
358
+ const i = Math.floor(Math.log(bytes) / Math.log(k));
359
+ return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
360
+ },
361
+ processFiles() {
362
+ if (this.selectedFiles.length === 0) return;
363
+
364
+ this.processing = true;
365
+ this.processedFiles = 0;
366
+ this.previewData = null;
367
+
368
+ // Simulate processing (in a real app, this would be API calls)
369
+ const totalFiles = this.selectedFiles.length;
370
+ const interval = setInterval(() => {
371
+ this.processedFiles += 1;
372
+
373
+ if (this.processedFiles >= totalFiles) {
374
+ clearInterval(interval);
375
+ this.processing = false;
376
+ this.generatePreviewData();
377
+ }
378
+ }, 800);
379
+ },
380
+ generatePreviewData() {
381
+ // Generate mock preview data
382
+ this.previewData = [
383
+ {
384
+ "instruction": "Explain the purpose of this Python function.",
385
+ "input": "def get_user_by_id(id):\n return db.query(User).filter(User.id == id).first()",
386
+ "output": "This function retrieves a user from the database by their ID."
387
+ },
388
+ {
389
+ "instruction": "What does this C++ function calculate?",
390
+ "input": "int factorial(int n) {\n if (n <= 1) return 1;\n return n * factorial(n - 1);\n}",
391
+ "output": "This function calculates the factorial of a given integer n using recursion."
392
+ },
393
+ {
394
+ "instruction": "Convert this JavaScript function to TypeScript with proper typing.",
395
+ "input": "function greet(name) {\n return `Hello, ${name}!`;\n}",
396
+ "output": "function greet(name: string): string {\n return `Hello, ${name}!`;\n}"
397
+ },
398
+ {
399
+ "instruction": "Optimize this Go function for better performance.",
400
+ "input": "func sum(numbers []int) int {\n total := 0\n for _, num := range numbers {\n total += num\n }\n return total\n}",
401
+ "output": "The function is already quite optimized. For very large slices, you might consider parallel processing using goroutines."
402
+ }
403
+ ];
404
+ this.selectedSampleIndex = 0;
405
+ },
406
+ downloadDataset() {
407
+ if (!this.previewData) return;
408
+
409
+ // In a real app, this would download the actual processed dataset
410
+ const dataStr = JSON.stringify(this.previewData, null, 2);
411
+ const dataBlob = new Blob([dataStr], { type: 'application/json' });
412
+ const url = URL.createObjectURL(dataBlob);
413
+
414
+ const link = document.createElement('a');
415
+ link.href = url;
416
+ link.download = 'code_dataset.json';
417
+ document.body.appendChild(link);
418
+ link.click();
419
+ document.body.removeChild(link);
420
+ URL.revokeObjectURL(url);
421
+ }
422
+ }
423
+ };
424
+
425
+ Vue.createApp(app).mount('#app');
426
+ </script>
427
+ <script src="https://unpkg.com/vue@3/dist/vue.global.js"></script>
428
+ <p style="border-radius: 8px; text-align: center; font-size: 12px; color: #fff; margin-top: 16px;position: fixed; left: 8px; bottom: 8px; z-index: 10; background: rgba(0, 0, 0, 0.8); padding: 4px 8px;">Made with <img src="https://enzostvs-deepsite.hf.space/logo.svg" alt="DeepSite Logo" style="width: 16px; height: 16px; vertical-align: middle;display:inline-block;margin-right:3px;filter:brightness(0) invert(1);"><a href="https://enzostvs-deepsite.hf.space" style="color: #fff;text-decoration: underline;" target="_blank" >DeepSite</a> - <a href="https://enzostvs-deepsite.hf.space?remix=Barbuuuuuuuu/code-dataset-generator" style="color: #fff;text-decoration: underline;" target="_blank" >🧬 Remix</a></p></body>
429
+ </html>