README
🐢
apache-airflow-huggingface
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
lhoestq
updated a
Space 3 days ago
AINovice2005
posted an update 5 months ago
Post
143
The hf article detailing the integration between hf datasets and dagster is live 🤗 : https://huggingface.co/blog/AINovice2005/dagster-hf-datasets
AINovice2005
posted an update 5 months ago
Post
82
Excited to share the release of dagster-hf-datasets: A Dagster-native integration that brings Hugging Face Datasets into Dagster's asset-oriented orchestration model
The integration enables:
• 🤗 Dataset and DatasetDict assets
• 🐙 Dagster asset lineage and observability
• 📦 Parquet-backed materialization via HFParquetIOManager
• 🚀 Publishing curated datasets back to the Hugging Face Hub
• 📝 Automatic dataset card generation from pipeline metadata
As the Hub continues to grow beyond 1M+ datasets, orchestration, reproducibility, and observability are becoming increasingly important parts of the dataset lifecycle. I'm also working on a longer article covering the architecture and data pipelines enabled by the integration.
More Soon!
https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets
https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets/docs
The integration enables:
• 🤗 Dataset and DatasetDict assets
• 🐙 Dagster asset lineage and observability
• 📦 Parquet-backed materialization via HFParquetIOManager
• 🚀 Publishing curated datasets back to the Hugging Face Hub
• 📝 Automatic dataset card generation from pipeline metadata
As the Hub continues to grow beyond 1M+ datasets, orchestration, reproducibility, and observability are becoming increasingly important parts of the dataset lifecycle. I'm also working on a longer article covering the architecture and data pipelines enabled by the integration.
More Soon!
https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets
https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets/docs
AINovice2005
posted an update 6 months ago
Post
114
Published my first HF Article 🤗 , which discusses about how to benchmark pixel art with LLMs: https://huggingface.co/blog/AINovice2005/pixel-art-bench
AINovice2005
posted an update 7 months ago
Post
174
I've built a system to make open-source contributions easier to understand across repositories.
It:
aggregates merged external PRs (reviewed by maintainers)
structures them into a single contributions.md
adds a lightweight AI layer to query patterns and impact
The idea is to move from scattered PRs to a readable changelog of work.
Read about it: https://medium.com/@paragekbote23/from-commits-to-impact-building-an-automated-changelog-for-open-source-contributions-20cdfebcee58
It:
aggregates merged external PRs (reviewed by maintainers)
structures them into a single contributions.md
adds a lightweight AI layer to query patterns and impact
The idea is to move from scattered PRs to a readable changelog of work.
Read about it: https://medium.com/@paragekbote23/from-commits-to-impact-building-an-automated-changelog-for-open-source-contributions-20cdfebcee58
AINovice2005
posted an update 8 months ago
Post
3,519
In celebration of the new storage graph feature on the Hub, here's mine 😊 :
Post inspired by @ZennyKenny
Post inspired by @ZennyKenny
AINovice2005
posted an update 8 months ago
Post
157
I recently created my first storage bucket to store experiment data of my performance analysis of 15 tokenizers across 20 languages.
The setup is simple enough for a new product and can be scalable depending on the use-case 🤗 .
Bucket: https://huggingface.co/buckets/AINovice2005/tokenizer-benchmark
github gist: https://gist.github.com/ParagEkbote/b3877f667f84cbb9a27bdaca94ba662a
Article: https://medium.com/@paragekbote23/one-sentence-fifteen-tokenizers-a-tokenizer-benchmarking-pipeline-with-hf-storage-buckets-2e59790276fd
The setup is simple enough for a new product and can be scalable depending on the use-case 🤗 .
Bucket: https://huggingface.co/buckets/AINovice2005/tokenizer-benchmark
github gist: https://gist.github.com/ParagEkbote/b3877f667f84cbb9a27bdaca94ba662a
Article: https://medium.com/@paragekbote23/one-sentence-fifteen-tokenizers-a-tokenizer-benchmarking-pipeline-with-hf-storage-buckets-2e59790276fd
AINovice2005
posted an update 8 months ago
Post
155
I'm pleased to announce that native trackio support has landed in optuna v.4.8.0.
Now you can store HPO trial data to the 🤗 Hub as a dataset: AINovice2005/toy-optuna-hpo-dataset
or Space: https://huggingface.co/spaces/AINovice2005/optuna-dashboard
Article which talks about the integration: https://medium.com/@paragekbote23/designing-and-contributing-an-optuna-integration-that-performs-well-in-practice-21a07d77ec2c
Now you can store HPO trial data to the 🤗 Hub as a dataset: AINovice2005/toy-optuna-hpo-dataset
or Space: https://huggingface.co/spaces/AINovice2005/optuna-dashboard
Article which talks about the integration: https://medium.com/@paragekbote23/designing-and-contributing-an-optuna-integration-that-performs-well-in-practice-21a07d77ec2c
AINovice2005
posted an update 8 months ago
Post
98
Pro tip2: You can treat HF datasets as versioned repos by pinning a specific revision (tag, branch or commit) when downloading files. 🧠
This ensures your data processing pipelines always use the exact dataset state before passing the data to the model. It enables reproducible pipelines and allows for reliable outputs of your ML system.
This ensures your data processing pipelines always use the exact dataset state before passing the data to the model. It enables reproducible pipelines and allows for reliable outputs of your ML system.
from huggingface_hub import hf_hub_download
data_path = hf_hub_download(
repo_id="lysandre/arxiv-nlp",
filename="train.parquet",
repo_type="dataset",
revision="877b84a8f93f2d619faa2a6e514a32beef88ab0a"
)AINovice2005
posted an update 8 months ago
Post
391
Pro tip: If you are finetuning any model with tensorboard logs enabled, be sure to upload them to HF Hub as event artifacts, they can be viewed instantly. 🚀
I previously remembered this done in the notus model release: argilla/notus-7b-v1
Examples:
AINovice2005/ModernBERT-base-lora-cicflow-1m-r8
AINovice2005/ModernBERT-base-lora-cicflow-1m-r4
AINovice2005/ModernBERT-base-lora-cicflow-1m-r16
cc: @davidberenstein1957
I previously remembered this done in the notus model release: argilla/notus-7b-v1
Examples:
AINovice2005/ModernBERT-base-lora-cicflow-1m-r8
AINovice2005/ModernBERT-base-lora-cicflow-1m-r4
AINovice2005/ModernBERT-base-lora-cicflow-1m-r16
cc: @davidberenstein1957
AINovice2005
posted an update 8 months ago
Post
481
Just published my first cuda kernel, inspired by Sage Attention. Feel free to try it out ☺️
AINovice2005/attention-int8
AINovice2005/attention-int8
- 2 replies
lhoestq
authored a
paper almost 2 years ago
Post
3,776
Made a HF Dataset editor a la gg sheets here: lhoestq/dataset-spreadsheets
With Dataset Spreadsheets:
✏️ Edit datasets in the UI
🔗 Share link with collaborators
🐍 Use locally in DuckDB or Python
Available for the 100,000+ parquet datasets on HF :)
With Dataset Spreadsheets:
✏️ Edit datasets in the UI
🔗 Share link with collaborators
🐍 Use locally in DuckDB or Python
Available for the 100,000+ parquet datasets on HF :)
Post
4,212
Hey ! I'm working on a 100% synthetic Dataset Hub here (you can search for any kind of datasets an the app invents them). The link is here: infinite-dataset-hub/infinite-dataset-hub
Question for the Community:
Which models should I use to generate images and audio samples for those datasets ? 🤗
Question for the Community:
Which models should I use to generate images and audio samples for those datasets ? 🤗
- 4 replies
Post
3,097
✨ Easy Synthetic Dataset File Generation using LLM DataGen ! Link: https://huggingface.co/spaces/lhoestq/LLM_DataGen
features + how it works:
✍️ Generate the dataset content you want just by entering a file name
💡 Optionally specify the column names you need
💨 The dataset is streamed and generated on-the-fly in JSON Lines format
✅ Generation is constrained to always output valid JSON
How does this work ?
1/ Enter a file name
2/ The model generates column names for such a file. Using structured generation, it can generate 2 to 5 column names using lower case characters and underscores. I use a prompt that asks to generate column names for a realistic dataset and low temperature.
3/ The columns are used to update the Finite State Machine for the dataset content structured generation, so that it is used to generate JSON objects using those columns
4/ The model generates JSON objects using structured generation again, using the updated Finite State Machine. I use a prompt that asks for realistic data and a temperature of 1.
> Why update a Finite State Machine instead of re-creating one ?
Creating one can take up to 30sec, while updating one takes 0.1s (though it requires to manipulate a graph which is not easy to implement)
> Batched generation is faster, why not use it ?
Generate in batches is faster but tends to generate duplicates for this demo.
Further work can be to provide different prompts (one per sequence in the batch) to end up with a different distribution of sequences in each batch. Or implement a custom sampler that would forbid generating the same data in sequences of the same batch.
> How does structured generation work ?
I used the
Let me know what you think ! And feel free to duplicate/modify it to try other models/prompts or sampling methods :)
features + how it works:
✍️ Generate the dataset content you want just by entering a file name
💡 Optionally specify the column names you need
💨 The dataset is streamed and generated on-the-fly in JSON Lines format
✅ Generation is constrained to always output valid JSON
How does this work ?
1/ Enter a file name
2/ The model generates column names for such a file. Using structured generation, it can generate 2 to 5 column names using lower case characters and underscores. I use a prompt that asks to generate column names for a realistic dataset and low temperature.
3/ The columns are used to update the Finite State Machine for the dataset content structured generation, so that it is used to generate JSON objects using those columns
4/ The model generates JSON objects using structured generation again, using the updated Finite State Machine. I use a prompt that asks for realistic data and a temperature of 1.
> Why update a Finite State Machine instead of re-creating one ?
Creating one can take up to 30sec, while updating one takes 0.1s (though it requires to manipulate a graph which is not easy to implement)
> Batched generation is faster, why not use it ?
Generate in batches is faster but tends to generate duplicates for this demo.
Further work can be to provide different prompts (one per sequence in the batch) to end up with a different distribution of sequences in each batch. Or implement a custom sampler that would forbid generating the same data in sequences of the same batch.
> How does structured generation work ?
I used the
outlines library with transformers to to define a JSON schema that the generation has to follow. It uses a Finite State Machine with token_id as transitions.Let me know what you think ! And feel free to duplicate/modify it to try other models/prompts or sampling methods :)
lhoestq
authored 2
papers almost 3 years ago
lhoestq
authored 2
papers over 3 years ago