| ---
|
| license: apache-2.0
|
| language:
|
| - en
|
| ---
|
| ## 2 Million Bluesky Posts |
|
|
| This dataset contains 2 million public posts collected from Bluesky Social's firehose API, intended for machine learning research and experimentation with social media data. |
|
|
| The with-language-predictions config contains the same data as the default config but with language predictions added using the glotlid model. |
| Dataset Details |
| Dataset Description |
|
|
| This dataset consists of 2 million public posts from Bluesky Social, collected through the platform's firehose API. Each post contains text content, metadata, and information about media attachments and reply relationships. |
|
|
| - **Curated by**: Alpin Dale |
| - **Language(s) (NLP)**: Multiple (primarily English) |
| - **License**: Dataset usage is subject to Bluesky's Terms of Service |
|
|
|
|
| ## Uses |
|
|
| This dataset could be used for: |
|
|
| - Training and testing language models on social media content |
| - Analyzing social media posting patterns |
| - Studying conversation structures and reply networks |
| - Research on social media content moderation |
| - Natural language processing tasks using social media datas |
|
|
| ## Dataset Structure |
|
|
| The dataset is available in two configurations: |
| ### Default Configuration |
|
|
| Contains the following fields for each post: |
|
|
| - **text**: The main content of the post |
| - **created_at**: Timestamp of post creation |
| - **author**: The Bluesky handle of the post author |
| - **uri**: Unique identifier for the post |
| - **has_images**: Boolean indicating if the post contains images |
| - **reply_to**: URI of the parent post if this is a reply (null otherwise) |
| |
| ### With Language Predictions Configuration |
| |
| Contains all fields from the default configuration plus: |
| |
| - **predicted_language**: The predicted language code (e.g., eng_Latn, deu_Latn) |
| - **language_confidence**: Confidence score for the language prediction (0-1) |
| |
| Language predictions were added using the [glotlid](https://huggingface.co/cis-lmu/glotlid) model via fasttext. |
| |
| ## Bias, Risks, and Limitations |
| The goal of this dataset is for you to have fun :) |
