Connect all storage bucket files

README.md

@@ -1,17 +1,10 @@
 ---
-annotations_creators:
-- no-annotation
-language_creators:
-- found
-languages:
+license: apache-2.0
+language:
 - my
 - en
-licenses:
-- apache-2.0
-multilinguality:
-- multilingual
-size_categories:
-- n_1M_to_n_10M
+multilinguality: multilingual
+size_categories: n_1M_to_n_10M
 source_datasets:
 - amkyawdev/myanmar-llm-data
 - amkyawdev/mm-llm-coder-agent-dataset
@@ -40,6 +33,17 @@ This dataset combines three source datasets for training LLMs with Myanmar langu
 
 **Total Samples: 3,020,347**
 
+### Storage Files
+
+This dataset also connects to storage bucket files for additional data:
+
+| File | Location | Size |
+|------|----------|------|
+| ADP Execution Feedback | `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-adp-execution-feedback.jsonl` | 50.8 MB |
+| Myanmar LLM Clean | `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-clean.jsonl` | 2.79 GB |
+| Myanmar LLM Formatted | `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-formatted.jsonl` | 24.8 MB |
+| Myanmar LLM Data Formatted | `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-data-formatted.jsonl` | 3.7 kB |
+
 ### Features
 
 - **Myanmar Language Support**: Native Burmese (မြန်မာစာ) conversations and translations
@@ -54,43 +58,39 @@ Each sample contains:
 
 ```python
 {
-    # Core Fields
-    "messages": [
-        {"role": "system", "content": "You are a helpful assistant."},
-        {"role": "user", "content": "User input here"},
-        {"role": "assistant", "content": "Response here"}
-    ],
-    
-    # Task Definition
-    "instruction": "Task instruction",
-    "category": "Task category",
-    "language": "en or my",
-    "difficulty": "beginner, intermediate, or advanced",
-    "response": "Expected response/output",
-    "task_type": "Type of task",
-    
-    # Execution Feedback
-    "execution_feedback": {
-        "status": "completed or pending_validation",
-        "result": "Execution result",
-        "error_type": "runtime_error, syntax_error, etc.",
-        "error_message": "Error details",
-        "execution_time_ms": 1000,
-    },
-    
-    # Extended Fields
-    "framework": "React, Express, etc.",
-    "runtime": "Node.js, Python, etc.",
-    "database": "MongoDB, PostgreSQL, etc.",
-    "validated": True/False,
-    "rating": 0.0 to 1.0,
-    "complexity_score": 1 to 10,
-    
-    # Metadata
-    "metadata": {
-        "created_at": "2024-01-01T00:00:00",
-        "difficulty": "beginner/intermediate/advanced",
-    }
+# Core Fields
+"messages": [
+{"role": "system", "content": "You are a helpful assistant."},
+{"role": "user", "content": "User input here"},
+{"role": "assistant", "content": "Response here"}
+],
+# Task Definition
+"instruction": "Task instruction",
+"category": "Task category",
+"language": "en or my",
+"difficulty": "beginner, intermediate, or advanced",
+"response": "Expected response/output",
+"task_type": "Type of task",
+# Execution Feedback
+"execution_feedback": {
+"status": "completed or pending_validation",
+"result": "Execution result",
+"error_type": "runtime_error, syntax_error, etc.",
+"error_message": "Error details",
+"execution_time_ms": 1000,
+},
+# Extended Fields
+"framework": "React, Express, etc.",
+"runtime": "Node.js, Python, etc.",
+"database": "MongoDB, PostgreSQL, etc.",
+"validated": True/False,
+"rating": 0.0 to 1.0,
+"complexity_score": 1 to 10,
+# Metadata
+"metadata": {
+"created_at": "2024-01-01T00:00:00",
+"difficulty": "beginner/intermediate/advanced",
+}
 }
 ```
 
@@ -100,15 +100,35 @@ Each sample contains:
 from datasets import load_dataset
 
 # Load the entire dataset
-dataset = load_dataset("amkyawdev/combined-myanmar-llm-dataset")
+dataset = load_dataset("amkyawdev/mm-llm-code-v3")
 
 # Load specific split
-train_ds = load_dataset("amkyawdev/combined-myanmar-llm-dataset", split="train")
+train_ds = load_dataset("amkyawdev/mm-llm-code-v3", split="train")
 
 # Access a single sample
 sample = train_ds[0]
 print(sample["messages"])
 print(sample["execution_feedback"])
+
+# Load from storage bucket directly
+from datasets import load_dataset
+ds = load_dataset("hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-clean.jsonl")
+```
+
+### Storage Bucket Access
+
+Access data directly from storage bucket:
+
+```python
+# ADP Execution Feedback (50.8 MB)
+from datasets import load_dataset
+adp_ds = load_dataset("hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-adp-execution-feedback.jsonl")
+
+# Clean dataset (2.79 GB)
+clean_ds = load_dataset("hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-clean.jsonl")
+
+# Formatted dataset (24.8 MB)
+formatted_ds = load_dataset("hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-formatted.jsonl")
 ```
 
 ### Use Cases
@@ -121,11 +141,15 @@ print(sample["execution_feedback"])
 
 ### Data Format
 
-This dataset is available in two formats:
+This dataset is available in multiple formats:
 
 1. **HuggingFace Dataset**: Standard format with all fields
 2. **ADP Format**: JSONL format with execution feedback
    - Location: `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-adp-execution-feedback.jsonl`
+3. **Clean Format**: Cleaned and processed data
+   - Location: `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-clean.jsonl`
+4. **Formatted Format**: Pre-formatted for training
+   - Location: `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-formatted.jsonl`
 
 ### License
 
@@ -135,10 +159,10 @@ Apache 2.0 License
 
 ```bibtex
 @dataset{combined_myanmar_llm,
-  title={Combined Myanmar LLM Dataset},
-  author={amkyawdev},
-  year={2024},
-  url={https://huggingface.co/datasets/amkyawdev/combined-myanmar-llm-dataset}
+title={Combined Myanmar LLM Dataset},
+author={amkyawdev},
+year={2024},
+url={https://huggingface.co/datasets/amkyawdev/mm-llm-code-v3}
 }
 ```
 
@@ -158,6 +182,17 @@ Apache 2.0 License
 
 **ပါဝင်မှု စုစုပါး: ၃,၀၂၀,၃၄၇**
 
+### Storage ဖိုင်များ
+
+Storage bucket မှာရှိတဲ့ ဖိုင်များနှင့်လည်း ပါဝင်ပပါး။
+
+| ဖိုင် | တည်နေရာ | အရွယ်အစား |
+|------|----------|------|
+| ADP Execution Feedback | `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-adp-execution-feedback.jsonl` | ၅၀.၈ MB |
+| Myanmar LLM Clean | `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-clean.jsonl` | ၂.၇၉ GB |
+| Myanmar LLM Formatted | `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-formatted.jsonl` | ၂၄.၈ MB |
+| Myanmar LLM Data Formatted | `hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-data-formatted.jsonl` | ၃.၇ kB |
+
 ### ပါဝင်တဲ့ အချက်များ
 
 - **မြန်မာစာ ပါဝင်မှု**: မြန်မာစာ ပါးဆက်များနှင့် ဘာသာပြန်များ
@@ -169,26 +204,23 @@ Apache 2.0 License
 ### ဖွဲ့စည်းပါ
 
 ```python
-{
-    # အခြေခံ ဖိုင်များ
-    "messages": [...],
-    "instruction": "...",
-    "category": "...",
-    "language": "my သို့မဟုတ် en",
-    "difficulty": "beginner, intermediate, advanced",
-    "response": "...",
-    "task_type": "...",
-    
-    # Execution Feedback
-    "execution_feedback": {
-        "status": "completed သို့မဟုတ် pending_validation",
-        "result": "...",
-        "error_type": "...",
-        "error_message": "...",
-    },
-    
-    # နောက်ထပ် ဖိုင်များ
-    "metadata": {...}
+# အခြေခံ ဖိုင်များ
+"messages": [...],
+"instruction": "...",
+"category": "...",
+"language": "my သို့မဟုတ် en",
+"difficulty": "beginner, intermediate, advanced",
+"response": "...",
+"task_type": "...",
+# Execution Feedback
+"execution_feedback": {
+"status": "completed သို့မဟုတ် pending_validation",
+"result": "...",
+"error_type": "...",
+"error_message": "...",
+},
+# နောက်ထပ် ဖိုင်များ
+"metadata": {...}
 }
 ```
 
@@ -196,10 +228,14 @@ Apache 2.0 License
 
 ```python
 from datasets import load_dataset
-
-dataset = load_dataset("amkyawdev/combined-myanmar-llm-dataset")
+dataset = load_dataset("amkyawdev/mm-llm-code-v3")
 train_ds = dataset["train"]
 sample = train_ds[0]
+
+# Storage bucket မှာရှိတဲ့ ဖိုင်များကို သုံးဖို့
+from datasets import load_dataset
+adp_ds = load_dataset("hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-adp-execution-feedback.jsonl")
+clean_ds = load_dataset("hf://buckets/amkyawdev/mm-llm-storage/myanmar-llm-clean.jsonl")
 ```
 
 ### သုံးပြုနည်း အမျိုးမျိုး
@@ -216,4 +252,4 @@ Apache 2.0 License
 
 ### Dataset URL
 
-https://huggingface.co/datasets/amkyawdev/combined-myanmar-llm-dataset
+https://huggingface.co/datasets/amkyawdev/mm-llm-code-v3