RuBia: A Russian Language Bias Detection Dataset

V Grigoreva, A Ivanova, I Alimova, E Artemova - arXiv preprint arXiv:2403.17553, 2024

Warning: this work contains upsetting or disturbing content. Large language models (LLMs) tend to learn the social and cultural biases present in the raw pre-training data. To test if an LLM's behavior is fair, functional datasets are employed, and due to their …


Sailor: Open Language Models for South-East Asia

L Dou, Q Liu, G Zeng, J Guo, J Zhou, W Lu, M Lin - arXiv preprint arXiv:2404.03608, 2024

We present Sailor, a family of open language models ranging from 0.5B to 7B parameters, tailored for South-East Asian (SEA) languages. These models are continually pre-trained from Qwen1.5, a great language model for multilingual use …


Sampling-based Pseudo-Likelihood for Membership Inference Attacks

M Kaneko, Y Ma, Y Wata, N Okazaki - arXiv preprint arXiv:2404.11262, 2024

… 2023): LLaMA-2 employs English CommonCrawl, C4, Github, Wikipedia, Books, ArXiv, and StackExchange as pre-training datasets. … or filtered to contain predominantly English text, but a small amount of non-English data is still present …


Scaffold-BPE: Enhancing Byte Pair Encoding with Simple and Effective Scaffold Token Removal

H Lian, Y Xiong, J Niu, S Mo, Z Su, Z Lin, P Liu, H Chen… - arXiv preprint arXiv …, 2024

… The Pile is composed of 22 diverse and high-quality datasets, the models trained on which significantly outperform both raw and filtered Common Crawl [10] models. The data distribution for our model training is identical to those described in the …


Scalable Distributed String Sorting

F Kurpicz, P Mehnert, P Sanders, M Schimek - arXiv preprint arXiv:2404.16517, 2024

… While different sampling and group assignment techniques may improve character balance, the CommonCrawl data set contains many duplicated strings (eg, standard legal disclaimers) which are always assigned to a single PE and cannot be …


Scalable Pre-training of Large Autoregressive Image Models

A El-Nouby, M Klein, S Zhai, MA Bautista, A Toshev… - arXiv preprint arXiv …, 2024

This paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, ie, Large Language Models (LLMs), and exhibit similar scaling properties. Specifically …


Scalable Range Search over Temporal and Numerical Expressions

V Ohr, D Gupta - The 10th ACM SIGIR/The 14th International …

… a subset of Common Crawl amounting to … Common Crawl, published as C4 [30]. This document collection consists of approximately 13.81 million documents. The fourth and final document collection we use is the entire C4-En corpus derived from …


Scaling Laws for Data Filtering--Data Curation cannot be Compute Agnostic

S Goyal, P Maini, ZC Lipton, A Raghunathan, JZ Kolter - arXiv preprint arXiv …, 2024

… As we train for longer, the accuracy gains using the LAION-filtering subset that filters the common crawl to 10% of its initial size plateau. Surprisingly, even no-filtering of the common crawl is better than the popular LAION-filtering after seeing more …


Scaling neural machine translation to 200 languages

… To overcome this problem, we created training datasets through global bitext mining in publicly available web content (drawn from repositories such as CommonCrawl). The underlying idea of our bitext mining approach is first to learn a …


sCambiaMenti at ACTI: Ensemble model with majority voting for Automatic Conspiracy Detection

S Bianco, D Salusso - EVALITA Proceedings of the Eighth Evaluation …, 2024

In this paper we describe the methodology that we have implemented to solve the subtask A of the Automatic Conspiracy Detection (ACTI) task (EVALITA 2023). We have developed different classifiers and then used a majority voting approach to …


SchemaPile: A Large Collection of Relational Database Schemas

T Döhmen, R Geacu, M Hulsebos, S Schelter - … of the ACM on Management of Data, 2024

… such as the aforementioned code dataset The Stack [25] or web crawls such as CommonCrawl [12] as alternative data sources. Furthermore, enterprises could adapt our data … Common Crawl – a free, open repository of web crawl data that …


Scoping: Towards Streamlined Entity Collections for Multi-Sourced Entity Resolution with Self-Supervised Agents

Linking multiple entities to a real-world object is a time-consuming and error-prone task. Entity Resolution (ER) includes techniques for vectorizing entities (signature), grouping similar entities into partitions (blocking), and matching entity pairs based …


SD-WEAT: Towards Robustly Measuring Bias in Input Embeddings

M Gray, L Wu - International Conference on Human-Computer …, 2024

… 300d” from the Stanford NLP Group [9], which was trained on 840 billion tokens from Common Crawl data and generates 300-dimension vectors. Moreover, other input embedding methods, including those of BERT [10, 11], SciBERT [12, 13], and …


Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion

J Zhu, H Hunag, Z Lin, J Liang, Z Tang, K Almubarak…

This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or ChatGPT 3.5, due to a …


Secret Collusion Among Generative AI Agents

SR Motwani, M Baranchuk, M Strohmeier, V Bolina… - arXiv preprint arXiv …, 2024

Recent capability increases in large language models (LLMs) open up applications in which teams of communicating generative AI agents solve joint tasks. This poses privacy and security challenges concerning the unauthorised sharing of information …


SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

K Cheng, Q Sun, Y Chu, F Xu, Y Li, J Zhang, Z Wu - arXiv preprint arXiv:2401.10935, 2024

… We collected approximately 300k web pages from the latest Common Crawl repository 2 to serve as our training data for web UI. For each webpage, we collect two types of elements from the HTML code: (1) elements that display visible text …


Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources

L Hyldig Hansen, N Andersen, J Gallifant, LG McCoy… - arXiv e-prints, 2024

LH Hansen, N Andersen, J Gallifant, LG McCoy… - arXiv preprint arXiv …, 2024

… Methods We conducted a large-scale textual analysis using a dataset comprising diverse web sources, including Arxiv, Wikipedia, and Common Crawl. The study analyzed the context in which various diseases are discussed alongside markers of …


Selecting Large Language Model to Fine-tune via Rectified Scaling Law

H Lin, B Huang, H Ye, Q Chen, Z Wang, S Li, J Ma… - arXiv preprint arXiv …, 2024

The ever-growing ecosystem of LLMs has posed a challenge in selecting the most appropriate pre-trained model to fine-tune amidst a sea of options. Given constrained resources, fine-tuning all models and making selections afterward is …


Selective Metric Differential Privacy for Language Models

… We used a pre-trained GloVe model trained on Common Crawl data with 840B tokens with a 2.2M vocabulary size for the privatization mechanism. We chose to utilize this model because it has a large vocabulary and is case-sensitive. Since GPT-2 …


Self-Augmented In-Context Learning for Unsupervised Word Translation

Y Li, A Korhonen, I Vulić - arXiv preprint arXiv:2402.10024, 2024

… 2017): the version pretrained on Wikipedia7 is used for XLING and the version pretrained with Wikipedia plus Common Crawl8 is used for PanLex-BLI, as recommended by XLING and PanLex-BLI, respectively. The same WEs are used for …


Semantic-guided spatio-temporal attention for few-shot action recognition

J Wang, B Liu - Applied Intelligence, 2024

… We utilize GloVe [49] to extract text embeddings for action labels, which is pre-trained on Common Crawl dataset of 840B tokens. For labels with multiple words, we average the embeddings of all their words. Finally, one action label is represented …


Semantically Enriched Cross-Lingual Sentence Embeddings for Crisis-related Social Media Texts

R Lamsal, MR Read, S Karunasekera - arXiv preprint arXiv:2403.16614, 2024

Tasks such as semantic search and clustering on crisis-related social media texts enhance our comprehension of crisis discourse, aiding decision-making and targeted interventions. Pre-trained language models have advanced performance in …


SeNSe: embedding alignment via semantic anchors selection

L Malandri, F Mercorio, M Mezzanzanica, F Pallucchini - International Journal of Data …, 2024

Word embeddings have proven extremely useful across many NLP applications in recent years. Several key linguistic tasks, such as machine translation and transfer learning, require comparing distributed representations of words belonging to …


SENTA: Sentence Simplification System for Slovene

A Žagar, M Klemen, M Robnik-Šikonja, I Kosem - Proceedings of the 2024 Joint …, 2024

… It uses sentence-level paraphrase data obtained by mining paraphrases from Common Crawl in combination with controllable generation mechanisms to adjust the length and lexical complexity of a simplified candidate. …


Sentence Embeddings for Massively Multilingual Speech and Text Processing

… CommonCrawl raw text data corresponds to partial snapshots of the internet totalling terabytes of text data extracted from web pages in many languages. The authors used the open-source FAISS library (Johnson et al. 2019) in order to …


Sentence Level Analysis Model for Phishing Detection Using KNN

Phishing emails have experienced a rapid surge in cyber threats globally, especially following the emergence of the COVID-19 pandemic. This form of attack has led to substantial financial losses for numerous organizations. Although various models …


Sentiment analysis on the issue of COVID-19 vaccination using LSTM and fasttext

WN Dewani, Y Azhar, N Hayatin - AIP Conference Proceedings, 2024

… The fasttext model is the result of a word vector previously trained from Common Crawl and Indonesian language Wikipedia data from Facebook. This model is trained using the Continuous Bag of Words (CBOW) method with position weight [13] …


SETEM: Self-ensemble training with Pre-trained Language Models for Entity Matching

H Ding, C Dai, Y Wu, W Ma, H Zhou - Knowledge-Based Systems, 2024

… This benchmark is constructed by extracting schema.org data from the Common Crawl and using the following attributes: brand, title, currency and price. The WDC Products benchmark consists of overall nine training, nine validation and nine test …


Shapley Values-Powered Framework for Fair Reward Split in Content Produced by GenAI

A Glinsky, A Sokolsky - arXiv preprint arXiv:2403.09700, 2024

… DATACOMP introduces a new candidate pool of 12.8 billion image-text pairs sourced from Common Crawl, which is broader than the collection in LAION-5B. This extensive pool offers a more diverse base for dataset creation. Unlike LAION-5B …


SIGN LANGUAGE RECOGNITION OF WORDS AND SENTENCE PREDICTION USING LSTM AND NLP

R GAIKWAD, L ADMUTHE - Journal of Theoretical and Applied Information …, 2024

… The dataset used for the training of the T5 model is the C4 dataset called as Colossal Clean Crawled Corpus which is collected from Common Crawl, a publicly available web archive. It consists of 750 GB clean English text scraped from the …


Silencing the Risk, Not the Whistle: A Semi-automated Text Sanitization Tool for Mitigating the Risk of Whistleblower Re-Identification

D Staufer, F Pallas, B Berendt - arXiv preprint arXiv:2405.01097, 2024

Whistleblowing is essential for ensuring transparency and accountability in both public and private sectors. However, (potential) whistleblowers often fear or face retaliation, even when reporting anonymously. The specific content of their …


SimLESS: A Secure Deduplication System over Similar Data in Cloud Media Sharing

M Song, Z Hua, Y Zheng, T Xiang, X Jia - IEEE Transactions on Information Forensics …, 2024

With the growing popularity of cloud computing, sharing media data through the cloud has become a common practice. Due to high information redundancy, media data take up a significant amount of storage space. Moreover, similar media data …


Simple and Scalable Strategies to Continually Pre-train Large Language Models

A Ibrahim, B Thérien, K Gupta, ML Richter, Q Anthony… - arXiv preprint arXiv …, 2024

… We then generate a fixed token-length response for each of the models trained German Common Crawl. As a baseline, we also evaluate … of German-language outputs from the models trained on German Common Crawl when compared to the …


Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

J Barnett, K Kieslich, N Diakopoulos - arXiv preprint arXiv:2405.09679, 2024

… They utilize proprietary datasets as well as opensource web data such as the continuously growing Common Crawl, which contains more than 250 billion pages of over a decade’s worth of textual content. While these models primarily learn how …


Slight Corruption in Pre-training Data Makes Better Diffusion Models

H Chen, Y Han, D Misra, X Li, K Hu, D Zou… - arXiv preprint arXiv …, 2024

… For example, Stable Diffusion [26] was pre-trained on LAION-2B [17], which contains billion-scale image-text pairs collected from Common Crawl [27]. Despite the heavy filtering mechanisms used in collecting pre-training datasets [17, 28], they …


Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs

J Tan, Z Dou, Y Zhu, P Guo, K Fang, JR Wen - arXiv preprint arXiv:2402.12052, 2024

The integration of large language models (LLMs) and search engines represents a significant evolution in knowledge acquisition methodologies. However, determining the knowledge that an LLM already possesses and the knowledge that requires the …


Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors

N Mireshghallah, J Mattern, S Gao, R Shokri… - Proceedings of the 18th …, 2024

As large language models are becoming more embedded in different user-facing services, it is important to be able to distinguish between human-written and machine-generated text to verify the authenticity of news articles, product reviews, etc. Thus, in this paper …


Smart Automation Using LLM

P Ethape, R Kane, G Gadekar, S Chimane - International Research Journal of …, 2023

… Such large-scale models can ingest massive amounts of data, often from the internet, but also from sources such as the Common Crawl, which comprises more than 50 billion web pages, and Wikipedia, which has approximately 57 million pages. …


Smart Bilingual Focused Crawling of Parallel Documents

C García-Romero, M Esplà-Gomis, F Sánchez-Martínez - arXiv preprint arXiv …, 2024

Crawling parallel texts $\unicode{x2014}$texts that are mutual translations$\unicode{x2014}$ from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up …


Smoky Quartz: A Transparent Bilingual Large Language Model

Q Huang, C Zhang, M Tao, J Lin, S Huang, Y Feng

Recent advancements in large-scale models have been remarkable; however, a significant challenge persists in their lack of transparency, particularly concerning the training data utilized. This obscurity raises concerns about potential data leaks …


SMW Cloud: A Corpus of Domain-Specific Knowledge Graphs from Semantic MediaWikis

Semantic wikis have become an increasingly popular means of collaboratively managing Knowledge Graphs. They are powered by platforms such as Semantic MediaWiki and Wikibase, both of which enable MediaWiki to store and publish …


Social Evolution of Published Text and The Emergence of Artificial Intelligence Through Large Language Models and The Problem of Toxicity and Bias

A Khan, P Saravanan, SK Venkatesan - arXiv preprint arXiv:2402.07166, 2024

We provide a birds eye view of the rapid developments in AI and Deep Learning that has led to the path-breaking emergence of AI in Large Language Models. The aim of this study is to place all these developments in a pragmatic broader historical social …


Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification

K Tatariya, H Lent, J Bjerva, M de Lhoneux - arXiv preprint arXiv:2402.03137, 2024

Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression, especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and …


SoK: SSO-MONITOR—The Current State and Future Research Directions in Single Sign-On Security Measurements

L Jannett, M Westers, T Wich, C Mainka, A Mayer…

Single Sign-On (SSO) with OAuth 2.0 and OpenID Connect 1.0 is essential for user authentication and authorization on the Internet. Billions of users rely on SSO services provided by Google, Facebook, and Apple. For large-scale measurements …


Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales

Z Delbari, NS Moosavi, MT Pilehvar - Proceedings of the AAAI Conference on …, 2024

With the alarming rise of hate speech in online communities, the demand for effective NLP models to identify instances of offensive language has reached a critical point. However, the development of such models heavily relies on the …


Speaker independent recognition of low-resourced multilingual Arabic spoken words through hybrid fusion

S Mehra, V Ranga, R Agarwal, S Susan - Multimedia Tools and Applications, 2024

This article introduces a supervised strategy designed to enhance spoken word recognition within the constraints of a resource-limited multilingual dataset, specifically focusing on the Arabic language. Notably, existing methodologies often …


Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction

J Pang, I Vulić - arXiv preprint arXiv:2405.10625, 2024

Transformer-based encoder-decoder models have demonstrated impressive results in chemical reaction prediction tasks. However, these models typically rely on pretraining using tens of millions of unlabelled molecules, which can be time-consuming …


Specialized Language Models with Cheap Inference from Limited Domain Data

D Grangier, A Katharopoulos, P Ablin, A Hannun - arXiv preprint arXiv:2402.01093, 2024

Large language models have emerged as a versatile tool but are challenging to apply to tasks lacking large inference budgets and large in-domain training sets. This work formalizes these constraints and distinguishes four important variables …


SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

D Liu, R Zhang, L Qiu, S Huang, W Lin, S Zhao, S Geng… - Forty-first International Conference …

P Gao, R Zhang, C Liu, L Qiu, S Huang, W Lin, S Zhao… - arXiv preprint arXiv …, 2024

… Specifically, we first collect large-scale PDF datasets from Common Crawl 1 and arXiv websites. Then, we utilize PyMuPDF 2 to get the rendering results of each page in the PDF file and also save all the text annotations along with their bounding …


Spike No More: Stabilizing the Pre-training of Large Language Models

S Takase, S Kiyono, S Kobayashi, J Suzuki - arXiv preprint arXiv:2312.16903, 2023

… 2020) that consists of clean English texts extracted from Common Crawl7 as our LLM pre-training corpus. We also used the separated part of C4 as our validation data. We used GPT-2 vocabulary (Radford et al.… https://commoncrawl.org/ …


SPML: A DSL for Defending Language Models Against Prompt Attacks

RK Sharma, V Gupta, D Grossman - arXiv preprint arXiv:2402.11755, 2024

… The pre-training data included trillions of tokens sourced from publicly available datasets, such as Common Crawl, Wikipedia, and public domain books from Project Gutenberg. We use the versions with 7 billion and 13 billion parameters, available …


Sports, crisis, and social media: a Twitter-based exploration of the Tokyo Olympics in the COVID-19 era

V Mehra, P Singh, S Bharany, RS Sawhney - Social Network Analysis and Mining, 2024

The global landscape underwent a transformative shift due to the unexpected arrival of the COVID-19 pandemic, disrupting various aspects of life, including the staging of sports events. Notably, the Tokyo Olympics, a renowned international sports …


Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

A Hans, A Schwarzschild, V Cherepanova, H Kazemi… - arXiv preprint arXiv …, 2024

… When evaluating Binoculars on samples from languages that are not well represented in Common Crawl data (standard LLM pretraining data), we find that false-positives rates remain low, which is highly desirable from a harm reduction …


SRBerta—A Transformer Language Model for Serbian Cyrillic Legal Texts

M Bogdanović, J Kocić, L Stoimenov - Information, 2024

… CommonCrawl [5] is a collection of texts whose size is measured in petabytes. This collection was … , CommonCrawl contains texts of lower quality, so development teams often resorted to processing texts before training and extracting …


Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

W Du, T Luo, Z Qiu, Z Huang, Y Shen, R Cheng, Y Guo… - arXiv preprint arXiv …, 2024

LLMs are computationally expensive to pre-train due to their large scale. Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. However, the viability of these model growth …


Stage-1 Report: Everything everywhere all at once? Exploring the richness of visual experience beyond words

L Qianchen, M Yamada, Y Aizawa, N Tsuchiya

In consciousness research, understanding the nature and mechanism of visual experience has been one of the most debated topics. For humans, vision is arguably the primary sensory modality to understand the environment (Cole & Balcetis, 2021) …


StarCoder 2 and The Stack v2: The Next Generation

A Lozhkov, R Li, LB Allal, F Cassano, J Lamy-Poirier… - arXiv preprint arXiv …, 2024

The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on …


Strategies for Corpus Development for Low‐Resource Languages: Insights from Nepal

BK Bal, B Prasain, RR Ghimire, P Acharya - Automatic Speech Recognition and …, 2024

… 25 whereas the monolingual dataset were derived from Wikipedia and Common Crawl for languages under consideration. They also experimented with four variations of training settings: fully supervised, fully unsupervised, semisupervised …


Strengthening the WiC: New Polysemy Dataset in Hindi and Lack of Cross Lingual Transfer

H Dubossarsky, F Dairkee - Proceedings of the 2024 Joint International Conference …, 2024

… TB of filtered CommonCrawl data as opposed to mBERT which was trained on Wikipedia data, and reported to outperform mBERT in a variety of cross-lingual benchmarks like XNLI, MLQA and NER. HindiBERT is a monolingual model trained …


Structured Packing in LLM Training Improves Long Context Utilization

K Staniszewski, S Tworkowski, S Jaszczur… - arXiv preprint arXiv …, 2023

Recent advances in long-context Large Language Models (LCLMs) have generated significant interest, especially in applications such as querying scientific research papers. However, their potential is often limited by inadequate context utilization. We …


Studies in conversational AI: multilingual capabilities, world knowledge, and evaluation strategies

This thesis studies the evolving landscape of conversational AI. The main research objective is to improve the conversational abilities of conversational agents, with a focus on integrating real-time knowledge and expanding multilingual capabil-ities …


Studying Peer Effects in Divergent Thinking: Theory and Method

In designing technology that supports user learning, an important first step is to understand how interactions among humans shape mutual learning. Much qualitative research in the realm of peer-assisted learning (PAL) has advanced the …


SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM

Q Wang, Y Yuan, X Yang, R Zhang, K Zhao, W Liu… - arXiv preprint arXiv …, 2024

While Large Language Models (LLMs) have achieved remarkable success in various fields, the efficiency of training and inference remains a major challenge. To address this issue, we propose SUBLLM, short for Subsampling-Upsampling-Bypass …


Supervised Learning

ER Vankov, K Gadhoumi - Statistical Methods in Epilepsy, 2024

… A corpus such as Common Crawl, which contains roughly 840 billion tokens of over two million unique words, provides such scale. There are other corpora used in these unsupervised processes, one such common candidate being a collection of all …


SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

J Kim, J Nam, S Mo, J Park, SW Lee, M Seo, JW Ha… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising …


Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation

S Soni, A Kunchukuttan, T Chakraborty, MS Akhtar - arXiv preprint arXiv …, 2024

The widespread online communication in a modern multilingual world has provided opportunities to blend more than one language (aka code-mixed language) in a single utterance. This has resulted a formidable challenge for the computational …


Systematic Evaluation of Linear Transformations for Cross-lingual Semantic Spaces

In the field of natural language understanding (NLU), a fundamental element is the representation of word meanings, a process that is integral to a wide range of applications. These applications span numerous domains including machine …


Table Representation Learning

The increasing amount of data being collected, stored, and analyzed, induces a need for efficient, scalable, and robust methods to handle this data. Representation learning, ie, the practice of leveraging neural networks to obtain generic vector …


Tackling an Unbalanced Dataset for Classifying Indonesian E-Commerce Reviews Using Multi Word Embedding Model

R Adi, BR Irnawan, J Li - 2023 Eighth International Conference on Informatics …, 2023

… Our research used a pre-trained FastText that was trained on Indonesian Common Crawl and Wikipedia provided by [24]. 3) GloVe: GloVe is an unsupervised algorithm that can produce a vector representation of words created by [22]. Besides …


Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization

X Chen, Z Wang, D Sow, J Yang, T Chen, Y Liang… - arXiv preprint arXiv …, 2024

… subset of the Common Crawl’s web crawl corpus. More recently, LLaMA (Touvron et al.… datasets to encompass multiple domains such as Common Crawl, C4, and source codes from … contents are mostly unintelligible” in the Common Crawl. All …


TEA+: A Novel Temporal Graph Random Walk Engine With Hybrid Storage Architecture

C Huan, Y Liu, H Zhang, S Song, S Pandey, S Chen… - ACM Transactions on Architecture …

Many real-world networks are characterized by being temporal and dynamic, wherein the temporal information signifies the changes in connections, such as the addition or removal of links between nodes. Employing random walks on these …


TeClass: A Human-Annotated Relevance-based Headline Classification and Generation Dataset for Telugu

G Kanumolu, L Madasu, N Surange, M Shrivastava - arXiv preprint arXiv:2404.11349, 2024

… 2019) is a multilingual version of the RoBERTa model, and it was pre-trained on a vast 2.5TB CommonCrawl dataset, which included text from 100 languages. For our experiments, we utilized the xlm-roberta-base variant, boasting 270 million parameters. …


Tele-FLM Technical Report

X Li, Y Yao, X Jiang, X Fang, C Wang, X Liu, Z Wang… - arXiv preprint arXiv …, 2024

… CommonCrawl1 is often considered to be a repository containing diverse human experience … CommonCrawl are primarily concentrated in the English segment, with the Chinese content exhibiting relatively lower information density and quality …


Tell me a story: a framework for critically investigating AI language models

L Munn, L Henrickson - Learning, Media and Technology, 2024

Large language models are rapidly being rolled out into high-stakes fields like healthcare, law, and education. However, understanding of their design considerations, operational logics, and implicit biases remains limited. How might …


Temporal Blind Spots in Large Language Models

J Wallat, A Jatowt, A Anand - arXiv preprint arXiv:2401.12078, 2024

… A mixture of training datasets has been used: a filtered version of CommonCrawl, an expanded version of WebText [44], two not further specified internet-based book corpora, and the English Wikipedia. The most recent information in text-davinci-003’s …


Temporally-Varied News Headline Generation with GPT-3


Text Emotion Recognition Using Fast Text Word Embedding in Bi-Directional Gated Recurrent Unit

C Akalya Devi, D Karthika Renuka, T Harisudhan…

Emotions are states of readiness in the mind that result from evaluations of one's own thinking or events. Although almost all of the important events in our lives are marked by emotions, the nature, causes, and effects of emotions are some of the …


Text Filtering Classifiers for Medium-Resource Languages

J Daðason, H Loftsson - Proceedings of the 2024 Joint International Conference …, 2024

… Online texts are often obtained from large datasets, such as those published by Common Crawl (CC).The GPT-3 model was pre-trained on 499B tokens, 410B of which were obtained from CC (Brown et al.… The Icelandic Common Crawl Corpus (IC3) …


Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges

J Becker, JP Wahle, B Gipp, T Ruas - arXiv preprint arXiv:2405.15604, 2024

Text generation has become more accessible than ever, and the increasing interest in these systems, especially those using large language models, has spurred an increasing number of related publications. We provide a systematic literature review …


TEXT SUMMARIZATION SYSTEM USING NATURAL LANGUAGE PROCESSING AND MACHINE LEARNING

Information overload, caused by the rapid growth of the Internet, has become a significant issue. The abundance of information available online necessitates the simplification of relevant content through summarization. Manual summarization of …


TextGram: Towards a Better Domain-Adaptive Pretraining

O Dhekane, S Hiwarkhedkar, R Joshi, A Ladkat - … , Perundurai, Erode, India, December 6–8 …

… The RealNews Dataset is a large corpus of news articles scraped from Common Crawl [16]. It consists of 5000 news domains that are indexed by Google news. IMDb Movie Reviews is a binary sentiment analysis dataset made up of 50000 reviews …


Textual Coverage of Eventive Entries in Lexical Semantic Resources

E Fucikova, CF Alcaina, J Hajic, Z Urešová - Proceedings of the 2024 Joint …, 2024

This short paper focuses on the coverage of eventive entries (verbs, predicates, etc.) of some well-known lexical semantic resources when applied to random running texts taken from the internet. While coverage gaps are often reported for manually …


The AI Companion in Education: Analyzing the Pedagogical Potential of ChatGPT in Computer Science and Engineering

Z He, T Nguyen, T Miari, M Aliasgari, S Rafatirad…

… ChatGPT3 employs a pre-trained large language model [29] trained over the Common Crawl dataset [30]. During its training process, it first preprocess data from the Common Crawl dataset, data quality is refined by eliminating irrelevant or noisy …


The AI Did My Job Well

Artificial intelligence (AI), an idea first proposed back in 1955 (McCarthy, Minsky, & Rochester) seemingly has burst into public consciousness nearly 70 years later. Like the chained prisoners emerging from Plato’s allegorical cave, many of us find …


The Case for Developing a Foundation Model for Planning-like Tasks from Scratch

B Srivastava, V Pallagani - arXiv preprint arXiv:2404.04540, 2024

Foundation Models (FMs) have revolutionized many areas of computing, including Automated Planning and Scheduling (APS). For example, a recent study found them useful for planning problems: plan generation, language translation, model …


The correlation between nativelike selection and prototypicality: a multilingual onomasiological case study using semantic embedding

H Zhang - arXiv preprint arXiv:2405.13529, 2024

In native speakers' lexical choices, a concept can be more readily expressed by one expression over another grammatical one, a phenomenon known as nativelike selection (NLS). In previous research, arbitrary chunks such as collocations have …


The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models

A Birhane, S Dehdashtian, VU Prabhu, V Boddeti - arXiv preprint arXiv:2405.04623, 2024

… This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the …


The Dark Side of Language Models: Exploring the Potential of LLMs in Multimedia Disinformation Generation and Dissemination

D Barman, Z Guo, O Conlan - Machine Learning with Applications, 2024

Disinformation - the deliberate spread of false or misleading information poses a significant threat to our society by undermining trust, exacerbating polarization, and manipulating public opinion. With the rapid advancement of artificial intelligence and …


The effect of generative artificial intelligence on managerial behavior across functions

GAI is becoming readily available and easily accessible for those who are willing to take advantage of it. As a result, the world is witnessing a transformation regarding how managers perform their daily tasks. This study aims to understand the …


THE ENHANCEMENT OF OPEN DATA USABILITY BY DEFINING A CATEGORIZATION METHOD BASED ON THE OPEN DATA PORTAL METADATA

Due to numerous data transparency and open government initiatives, a large volume of data was published on open data portals. To make it more accessible and visible, these portals have introduced data filtering by category, tags, format …


The ethical situation of DALL-E 2

E Hogea, J Rocafortf - arXiv preprint arXiv:2405.19176, 2024

A hot topic of Artificial Intelligence right now is image generation from prompts. DALL-E 2 is one of the biggest names in this domain, as it allows people to create images from simple text inputs, to even more complicated ones. The company that made this …


The Ethics and Challenges of Legal Personhood for AI

Hon. Katherine B. Forrest (Fmr.) abstract. AI’s increasing cognitive abilities will raise challenges for judges.“Legal personhood” is a flexible and political concept that has evolved throughout American history. In determining whether to expand that concept …


The Evolution of Multimodal Model Architectures

SN Wadekar, A Chaurasia, A Chadha, E Culurciello - arXiv preprint arXiv:2405.17927, 2024

This work uniquely identifies and characterizes four prevalent multimodal model architectural patterns in the contemporary multimodal landscape. Systematically categorizing models by architecture type facilitates monitoring of developments in …


The Fill-Mask Association Test (FMAT): Measuring Propositions in Natural Language

HWS Bao - Journal of Personality and Social Psychology, 2024

L Lin, B Wang, X Wang, ZX Wang, A Wiśniowski…

Recent advances in large language models are enabling the computational intelligent analysis of psychology in natural language. Here, the Fill-Mask Association Test (FMAT) is introduced as a novel and integrative method leveraging …


The Future of Data Science in Materials Science

P Chundi, V Bommanapally, V Gadhamshetty - Machine Learning in 2D Materials …, 2023

… GPT-3 is an autoregressive model built with 96 layers [31], increased embedding vector dimension of 12888, trained with 175B parameters on five datasets including Common crawl, WebText2, Books1, Books2, and Wikipedia. Though GPT-3 has …


The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)

S Zeng, J Zhang, P He, Y Xing, Y Liu, H Xu, J Ren… - arXiv preprint arXiv …, 2024

… 2021), we randomly select chunks from the Common Crawl dataset to serve as the {information} component. … 2021), we randomly select chunks from the Common Crawl dataset to serve as the {information} component. Due to the random …


The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis

M Zhang, V Gautam, M Wang, JO Alabi, X Shen… - arXiv preprint arXiv …, 2024

… The order of languages follows their data ratio in the CommonCrawl corpus10 from high-resource to low-resource. We observe large variations in model performance across different languages. For instance, there exists a large …


The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

M Munn, B Dherin, J Gonzalvo - arXiv preprint arXiv:2405.15706, 2024

Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical …


The Impact of Infrastructure: Considerations of Generative AI in the Classroom

ES Kehlenbach - Journal of Political Science Education, 2024

How are we to grapple with the increasing intrusion of technology in the classroom? What considerations need to be addressed before we begin bringing advanced technological systems like ChatGPT into our classroom spaces? A close attention to …


The Impact of OpenAI on the Work Environment located in Finland

In the modern era, technology has undergone a rapid transformation that touches every aspect of our lives. At the forefront of this transformation stands artificial intelligence (AI), a field rapidly evolving through groundbreaking innovations, and …


The model student: GPT-4 performance on graduate biomedical science exams

D Stribling, Y Xia, MK Amer, KS Graim, CJ Mulligan… - Scientific Reports, 2024

… As these study materials may have been incorporated into the GPT-4 training data such as the Common Crawl 43 , standardized examinations may not be an accurate assessment of domain-specific model knowledgebase and capability …


The OECD-UNSD Multinational Enterprise Information Platform

… (3) Common Crawl is an open source initiative to provide a monthly snapshot of the internet through web scraping. As part of the project, they provide a quarterly network graph of hyperlinks between domains, which can be used to determine …


The Open Web Index

G Hendriksen, M Dinzinger, SM Farzana, NA Fathima…

… To operationalize our crawling pipeline, we introduce a re-crawl mechanism specifically for Common Crawl dumps. Unlike Common Crawl, we crawl on a daily basis and save the results as WARC (Web ARChive) files.This approach allows us …


The Open Web Index: Crawling and Indexing the Web for Public Use

G Hendriksen, M Dinzinger, SM Farzana, NA Fathima… - European Conference on …, 2024

… To operationalize our crawling pipeline, we introduce a re-crawl mechanism specifically for Common Crawl dumps. Unlike Common Crawl, we crawl on a daily basis and save the results as WARC (Web ARChive) files. Footnote 11 This …


The Power of Absence: Thinking with Archival Theory in Algorithmic Design

J Sherman, R Morrison, L Klein, DK Rosner - arXiv preprint arXiv:2405.05420, 2024

… , for example–but also of datasets like the Colossal Clean Common Crawl corpus, or C4, which explicitly documents its decision to include … [33] have shown how the quality filters imposed on the original Common Crawl dataset exhibit a preference …


The Practical Epistemologies of Design and Artificial Intelligence

W Billingsley - Science & Education, 2024

This article explores the epistemological trade-offs that practical and technology design fields make by exploring past philosophical discussions of design, practitioner research, and pragmatism. It argues that as technologists apply Artificial …


The Right Prompts for the Job: Repair Code-Review Defects with Large Language Model

Z Zhao, Z Xu, J Zhu, P Di, Y Yao, X Ma - arXiv preprint arXiv:2312.17485, 2023

Automatic program repair (APR) techniques have the potential to reduce manual efforts in uncovering and repairing program defects during the code review (CR) process. However, the limited accuracy and considerable time costs associated with …


The Right to Scrap Data on the Internet: From the US Case hiQLabs, Inc. v. LinkedIn Corp. to the ChatGPT Scraping Cases: Differences Between US and EU Law [pre …

JTS y López - Global Privacy Law Review, 2024

… a doctor when she was undergoing treatment for a rare genetic condition ended up in the Common Crawl’s archive and LAION’s dataset, and therefore in the GhatGPT datasets (because the Common Crawl is one of the ChatGPT training …


The Role and Prospects of Using ChatGPT Artificial Intelligence-Based Chatbots in Healthcare Improvement

In the age of digital transformation, healthcare is undergoing a revolution through information technology and artificial intelligence (AI). This article explores the role of AI-based chatbots, like ChatGPT, in healthcare enhancement. Utilizing a literature …


The Science of Data Filtering: Data Curation cannot be Compute Agnostic

S Goyal, P Maini, ZC Lipton, A Raghunathan, JZ Kolter - ICLR 2024 Workshop on …

… In fact, we show that even a model trained on *unfiltered common crawl* obtains higher accuracy than that trained on the LAION dataset post 40 or more repetitions. While past research in neural scaling laws has considered web data to be …


The science of implicit race bias: Evidence from the Implicit Association Test

KN Morehouse, MR Banaji - Daedalus, 2024

Beginning in the mid-1980s, scientific psychology underwent a revolution–the implicit revolution–that led to the development of methods to capture implicit bias: attitudes, stereotypes, and identities that operate without full conscious awareness …


The Shape of Word Embeddings: Recognizing Language Phylogenies through Topological Data Analysis

O Draganov, S Skiena - arXiv preprint arXiv:2404.00500, 2024

Word embeddings represent language vocabularies as clouds of $d$-dimensional points. We investigate how information is conveyed by the general shape of these clouds, outside of representing the semantic meaning of each token. Specifically, we …


The Silicone Ceiling: Auditing GPT's Race and Gender Biases in Hiring

L Armstrong, A Liu, S MacNeil, D Metaxa - arXiv preprint arXiv:2405.04412, 2024

Large language models (LLMs) are increasingly being introduced in workplace settings, with the goals of improving efficiency and fairness. However, concerns have arisen regarding these models' potential to reflect or exacerbate social biases …


The Trade-off between Performance, Efficiency, and Fairness in Adapter Modules for Text Classification

MD Bui, K von der Wense - arXiv preprint arXiv:2405.02010, 2024

… 2019) is derived from 393,423 online biographies in English from the Common Crawl corpus, each including the subject’s occupation and gender. The dataset contains 28 occupations, assuming a binary gender classification. Gender identification …


The use of machine learning and deep learning models in detecting depression on social media: A systematic literature review

WA Gadzama, D Gabi, MS Argungu, HU Suru - Personalized Medicine in Psychiatry, 2024

Depression is regarded as one of the world's primary concerns. Recent researchers use artificial intelligence techniques like machine learning and deep learning to identify depressive symptoms automatically. This literature review focuses on using …


The Web Data Commons Schema. org Table Corpora

R Peeters, A Brinkmann, C Bizer - Companion Proceedings of the ACM on Web …, 2024

… This paper presented the WDC Schema.org Table Corpora that we extracted from the Common Crawl. To the best of our knowledge, the corpora are the largest public table corpora that contain tables from many different sources which all share a …


The Web Is Missing an Essential Part of Infrastructure: An Open Web Index A proposal for building an index of the Web that separates the infrastructure part of the …

The web as it currently exists would not be possible without search engines. They are an integral part of the Web and can also be seen as a part of the Web’s infrastructure. Google alone now serves over two trillion search queries per year. 11 …


The work of art in the age of artificial intelligibility

The emergence of complex deep-learning models capable of producing novel images on a practically innumerable number of subjects and in an equally wide variety of artistic styles is beginning to highlight serious inadequacies in the ethical …


Time for a Pause: Without Effective Public Oversight, AI in Schools Will Do More Harm Than Good.

B Williamson, A Molnar, F Boninger - 2024

Public education is a public and private good essential to democratic civic life. The public must, therefore, be able to provide meaningful direction over schools through transparent democratic governance structures. And yet artificial intelligence (AI) 1 …


Time Machine GPT

F Drinkall, E Rahimikia, JB Pierrehumbert, S Zohren - arXiv preprint arXiv:2404.18543, 2024

… To scale to even larger models, processing the annual Common Crawl datasets is a necessary step, though the dataset has proved problematic due to its scale and lack of … Aside from this, cleaning Common Crawl would also demand significant …


To store or not: Online cost optimization for running big data jobs on the cloud

X Fu, L Pan, S Liu - Future Generation Computer Systems, 2024

As businesses increasingly rely on cloud-based big data analytics services to drive insights, reducing the cost of storing and analyzing large volumes of data in the cloud has become a major concern. During the execution of big data analysis jobs …


TOFU: A Task of Fictitious Unlearning for LLMs

P Maini, Z Feng, A Schwarzschild, ZC Lipton, JZ Kolter - arXiv preprint arXiv …, 2024

… LLMs trained on Common Crawl data1 may be able to correctly answer factual questions about this person and they may wish to have their data removed from an LLM. In fact, regulations around the Right to be Forgotten that focus on this situation …


TOMGPT: Reliable Text-Only Training Approach for Cost-Effective Multi-modal Large Language Model

Y Chen, Q Wang, S Wu, Y Gao, T Xu, Y Hu - ACM Transactions on Knowledge …, 2024

… For instance, c4 [37] dataset, a colossal and cleaned version of the web crawl corpus from Common Crawl, serves as a suitable option. Since the CLIP text encoder has limitations in processing long text, we subdivided the text in the c4 …


Toward Inference-optimal Mixture-of-Expert Large Language Models

L Yun, Y Zhuang, Y Fu, EP Xing, H Zhang - arXiv preprint arXiv:2404.02852, 2024

Mixture-of-Expert (MoE) based large language models (LLMs), such as the recent Mixtral and DeepSeek-MoE, have shown great promise in scaling model size without suffering from the quadratic growth of training cost of dense transformers …


Towards a Framework for Openness in Foundation Models: Proceedings from the Columbia Convening on Openness in Artificial Intelligence

A Basdevant, C François, V Storchan, K Bankston… - arXiv preprint arXiv …, 2024

Over the past year, there has been a robust debate about the benefits and risks of open sourcing foundation models. However, this discussion has often taken place at a high level of generality or with a narrow focus on specific technical attributes. In …


Towards a Scalable Geoparsing Approach for the Web

The ongoing surge in web data generation and storage, coupled with embedded geographic information, holds immense potential for enhancing search applications across diverse domains. However, extracting geographic information for further …


Towards a small language model powered chain‐of‐reasoning for open‐domain question answering

J Roh, M Kim, K Bae - ETRI Journal, 2024

… Unlike the ONUS approach [10], which extracts simple sub-question examples from a Common Crawl, our method emphasizes the autonomous generation of a dataset without human intervention facilitated by the use of LLMs. Consequently, we …


Towards a sociology of recurrent events. Constellations of cultural change around Eurovision in 18 countries (1981–2021)

L Carbone, J Mijs, T van Dooremalen, S Daenekindt - Poetics, 2024

… First, we used CMD with a corpus that was pre-trained on Common Crawl data using fastText English embeddings. Common Crawl is a vast and heterogeneous source of information that crawls web archives and freely provides their data to the …


Towards adaptive support for self‐regulated learning of causal relations: Evaluating four Dutch word vector models

HJ Pijeira‐Díaz, S Braumann, J van de Pol, T van Gog… - British Journal of …, 2024

Advances in computational language models increasingly enable adaptive support for self‐regulated learning (SRL) in digital learning environments (DLEs; eg, via automated feedback). However, the accuracy of those models is a common concern …


Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia

D Suhartono, MRN Majiid, R Fredyan - Education and Information Technologies, 2024

… This model is a potent tool for transforming our text data into numerical representations because it was previously trained on the sizable Common Crawl Dataset in Indonesian. These embeddings are essential for later modeling and …


Towards Better Statistical Understanding of Watermarking LLMs

Z Cai, S Liu, H Wang, H Zhong, X Li - arXiv preprint arXiv:2403.13027, 2024

In this paper, we study the problem of watermarking large language models (LLMs). We consider the trade-off between model distortion and detection ability and formulate it as a constrained optimization problem based on the green-red algorithm …


Towards Building Multilingual Language Model for Medicine

P Qiu, C Wu, X Zhang, W Lin, H Wang, Y Zhang… - arXiv preprint arXiv …, 2024

In this paper, we aim to develop an open-source, multilingual language model for medicine, that the benefits a wider, linguistically diverse audience from different regions. In general, we present the contribution from the following aspects: first, for …


Towards Chapterisation of Podcasts Detection of Host and Structuring Questions in Radio Transcripts

This Master thesis investigates the application of Bidirectional Encoder Representations from Transformers (BERT) on podcast to identify the host and detect structuring questions within each episode. This research is conducted on an …


Towards Disfluency Annotated Corpora for Indian Languages

C Kochar, V Mujadia, P Mishra, DM Sharma

In the natural course of spoken language, individuals often engage in thinking and self-correction during speech production. These instances of interruption or correction are commonly referred to as disfluencies. When preparing data for …


Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models

J Wang, A Jatowt, Y Cai - arXiv preprint arXiv:2406.01863, 2024

In the evolving field of Natural Language Processing, understanding the temporal context of text is increasingly crucial. This study investigates methods to incorporate temporal information during pre-training, aiming to achieve effective time-aware …


Towards Green AI. A methodological survey of the scientific literature

E Barbierato, A Gatti - IEEE Access, 2024

The pervasive deployment of Deep Learning models has recently prompted apprehensions regarding their ecological footprint, owing to the exorbitant levels of energy consumption necessitated by the training and inference processes. The term …


Towards interactive information seeking: conversational question answering

… To further advance the field, additional conversational search benchmarks have been introduced, such as ORQuAC [93], which extends CAsT and QuAC with more conversations constructed from Natural Questions and additional passages from …


Towards Machine Translation Based on Monolingual Texts

… The CommonCrawl1 project carries out periodic web crawls and publishes the crawled data in an open … the CommonCrawl corpus. For example, the open source OSCAR2 project compiled a large multilingual corpus by language …


Towards Privacy Preserving LLMs Training

B Buesser - Large Language Models in Cybersecurity: Threats …, 2024

Privacy-preserving training of machine learning models aims to avoid or minimize (mitigate) the exact or similar reproduction (leakage) of information contained in the training data. This chapter introduces pre-processing methods (filtering and de-duplication) …


Towards quantifying politicization in foreign aid project reports

S Wang, G Eggers, ART Georgiadis, TA Đo, L Gontard… - Proceedings of the Second …, 2024

We aim to develop a metric of politicization by investigating whether this concept can be operationalized computationally using document embeddings. We are interested in measuring the extent to which foreign aid is politicized. Textual reports of foreign …


Towards reliability and interactive debugging for large language models

Large language models (LLMs) have permeated our everyday lives and are used in critical decision-making scenarios that can affect millions of people. Despite their impressive progress, model deficiencies may result in exacerbating harmful biases …


Towards the automatic creation of NER systems for new domains

E Matos, M Rodrigues, A Teixeira - Proceedings of the 16th International Conference …, 2024

… ner-roberta (ROBERTA): xlm-roberta-base-trimmed-pt-600002 model based in xlm-roberta-base, pretrained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Crosslingual …


Toxicity Detection for Free

Z Hu, J Piet, G Zhao, J Jiao, D Wagner - arXiv preprint arXiv:2405.18822, 2024

… An analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Training and evaluation of vector models for Galician

M Garcia - Language Resources and Evaluation, 2024

This paper presents a large and systematic assessment of distributional models for Galician. To this end, we have first trained and evaluated static word embeddings (eg, word2vec, GloVe), and then compared their performance with that of current …


Training BERT Models to Carry over a Coding System Developed on One Corpus to Another

D Galambos, P Zsamboki - Proceedings of the 2024 Joint International Conference …, 2024

This paper describes how we train BERT models to carry over a coding system developed on the paragraphs of a Hungarian literary journal to another. The aim of the coding system is to track trends in the perception of literary translation around …


Training Data for the Price of a Sandwich1

… , this report sheds light on the implications of Common Crawl’s popularity by exploring the values that guide Common Crawl itself in-depth. … ) whether Common Crawl’s data had been used and b) what specific version of Common Crawl was …


Transformer-Based Named Entity Recognition Model-Tamil Language Check for updates

K Dhayalan, N Sultanova, J Mustafina, P Daud - Data Science and Emerging Technologies …

This work presents different transformer-based approaches to the named entity recognition problem in the Tamil language. The WikiANN-ta dataset is used for training, validating and testing the model. Person, organization and location tags are …


Transformers and large language models in healthcare: A review

S Nerella, S Bandyopadhyay, J Zhang, M Contreras… - Artificial Intelligence in …, 2024

With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning …


Transformers at HSD-2Lang 2024: Hate Speech Detection in Arabic and Turkish Tweets Using BERT Based Architectures

K Singhal, J Bedi - Proceedings of the 7th Workshop on Challenges and …, 2024

Over the past years, researchers across the globe have made significant efforts to develop systems capable of identifying the presence of hate speech in different languages. This paper describes the team Transformers’ submission to the subtasks …


Transformers for Detection of Distressed Cardiac Patients with an ICD Based on Danish Text Messages

JDW Andersen, ML Jensen, UK Wiil, S Skovbakke… - 2023 IEEE International …, 2023

Cardiac patients with implantable cardioverter defibrillator devices frequently exhibit signs of anxiety and depression (termed "Distressed"). Early detection of these patients is vital for evaluation, intervention, and prevention against relapse …


Transformers on Dynamical Systems-An Exploration of In-context Learning

Large Language Models (LLMs) have shown to be highly effective at performing in-context learning, where, given a prompt, the model can learn from the prompt and complete the sequence without needing to perform additional gradient steps or fine-tuning. In …


Transformers@ DravidianLangTech-EACL2024: Sentiment Analysis of Code-Mixed Tamil Using RoBERTa

K Singhal, J Bedi - Proceedings of the Fourth Workshop on Speech, Vision …, 2024

In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team Transformers’ submission to the Caste/Immigration …


Transformers@ LT-EDI-EACL2024: Caste and Migration Hate Speech Detection in Tamil Using Ensembling on Transformers

K Singhal, J Bedi - Proceedings of the Fourth Workshop on Language …, 2024

In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team “Transformers” submission to the Caste …


Transforming Conversational AI

We were motivated to write this book by the launch in November 2022 of ChatGPT and by the ensuing excitement and disruption across the world of Conversational AI. The book is written for a broad audience who are already working, starting to work …


Transforming Dutch: Debiasing Dutch Corefence Resolution Systems for Non-binary Pronouns

Gender-neutral pronouns are increasingly being introduced across Western languages, and are continuously more frequently being adopted by non-binary individuals. Recent evaluations have however demonstrated that English language …


TransGPT: Multi-modal Generative Pre-trained Transformer for Transportation

P Wang, X Wei, F Hu, W Han - arXiv preprint arXiv:2402.07233, 2024

Natural language processing (NLP) is a key component of intelligent transportation systems (ITS), but it faces many challenges in the transportation domain, such as domain-specific knowledge and data, and multi-modal inputs and outputs. This …


Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation

E Yang, D Lawrie, J Mayfield, DW Oard, S Miller - arXiv preprint arXiv:2401.04810, 2024

Prior work on English monolingual retrieval has shown that a cross-encoder trained using a large number of relevance judgments for query-document pairs can be used as a teacher to train more efficient, but similarly effective, dual-encoder student …


Translating and predicting document structure for medical domain scientific abstracts

SA Rauf, F Yvon - Computer Speech & Language, 2024

Abstract Machine Translation (MT) technologies have improved in many ways and generate usable outputs for a growing number of domains and language pairs. Yet, most sentence based MT systems struggle with contextual dependencies …


Translating scientific abstracts in the bio-medical domain with structure-aware models

SA Rauf, F Yvon - Computer Speech and Language, 2024

Machine Translation (MT) technologies have improved in many ways and generate usable outputs for a growing number of domains and language pairs. Yet, most sentence based MT systems struggle with contextual dependencies, processing …


Translation from Historical to Contemporary Japanese Using Japanese T5

H Usui, K Komiya - Proceedings of the Joint 3rd International Conference …, 2023

This paper presents machine translation from historical Japanese to contemporary Japanese using a Text-to-Text Transfer Transformer (T5). The result of the previous study that used neural machine translation (NMT), Long Short Term Memory (LSTM) …


Treading water: new data on the impact of AI ethics information sessions in classics and ancient language pedagogy

EAS Ross, J Baines - Journal of Classics Teaching, 2024

Over 2023, many universities and policy organisations in the higher education (HE) sector are working to create guiding principles and guidelines for the use of generative artificial intelligence (AI) in HE Teaching and Learning (T&L). Despite …


Understanding College Students' Satisfaction With ChatGPT: An Exploratory And Predictive Machine Learning Approach Using Feature Engineering

K Pabreja, N Pabreja - MIER Journal of Educational Studies Trends and …, 2024

Artificial Intelligence (AI) technologies are continually improving and becoming more pervasive in many facets of our lives. ChatGPT is one such cutting-edge artificial intelligence application, and it has received a lot of worldwide media attention …


Understanding Emergent Abilities of Language Models from the Loss Perspective

Z Du, A Zeng, Y Dong, J Tang - arXiv preprint arXiv:2403.15796, 2024

Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) …


Understanding Large Language Models

Today, finding someone who hasn’t heard of ChatGPT, the AI chatbot that took the world by storm, is hard. ChatGPT—and its competitors such as Google Bard, Microsoft Bing Chat, etc.—are part of a broader area in AI known as large language …


Understanding latent affective bias in large pre-trained neural language models

A Kadan, P Deepak, S Bhadra, MP Gangan, VL Lajish - Natural Language …, 2024

Groundbreaking inventions and highly significant performance improvements in deep learning based Natural Language Processing are witnessed through the development of transformer based large Pre-trained Language Models (PLMs). The …


Understanding LLMs: A Comprehensive Overview from Training to Inference

Y Liu, H He, T Han, X Zhang, M Liu, J Tian, Y Zhang… - arXiv preprint arXiv …, 2024

… from the CommonCrawl. However, due to the presence of a substantial amount of low-quality data in web archives, preprocessing is essential when working with CommonCrawl data. Currently, four commonly used filtered datasets based on …


Understanding New Machine Learning Architectures: Practical Generative Artificial Intelligence for Anesthesiologists

Recent advances in neural networks have given rise to generative artificial intelligence, systems able to produce fluent responses to natural questions or attractive and even photorealistic images from text prompts. These systems were …


Understanding Privacy Risks of Embeddings Induced by Large Language Models

Z Zhu, N Shao, D Lian, C Wu, Z Liu, Y Yang, E Chen - arXiv preprint arXiv …, 2024

… CC-News (CommonCrawl News dataset) is a dataset containing news articles from international news websites. It contains 708241 English-language news articles published between January 2017 and December 2019. Pile-PubMed is a …


UNIVERSAL IMITATION GAMES

In 1950, Alan Turing proposed a framework called an imitation game in which the participants are to be classified Human or Machine solely from natural language interactions. Using mathematics largely developed since Turing–category theory–we …


Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

H Laurençon, L Tronchon, V Sanh - arXiv preprint arXiv:2403.09029, 2024

… available online (and often captured in web crawls like Common Crawl) presents a tempting resource for generating pairs of screenshots and corresponding HTML codes by simply rendering the HTML and capturing the output. However, this …


Unraveling the Mystery of Scaling Laws: Part I

H Su, Z Tian, X Shen, X Cai - arXiv preprint arXiv:2403.06563, 2024

Scaling law principles indicate a power-law correlation between loss and variables such as model size, dataset size, and computational resources utilized during training. These principles play a vital role in optimizing various aspects of model pre-training …


Unseen but influential associates: Properties of words' associates influence lexical and semantic processing

EJ Muraki, PM Pexman - Psychonomic Bulletin & Review, 2024

In many models of lexical and semantic processing, it is assumed that single word processing is a function of the characteristics of the words presented and the distributional properties of the words’ networks. Recent research suggests that …


Unsupervised LLM Adaptation for Question Answering

K Saito, K Sohn, CY Lee, Y Ushiku - arXiv preprint arXiv:2402.12170, 2024

… Commoncrawl 2. The topics cover politics, sports, business, etc. To reduce the number of tokens, we ask ChatGPT to generate summaries. … We employ NewsFetch7 to crawl the news from Commoncrawl. The published date of the article …


Unsupervised Outlier Detection for Language-Independent Text Quality Filtering

J Daðason, H Loftsson - Proceedings of the 3rd Annual Meeting of the Special …, 2024

… Common Crawl (CC) is an organization that maintains a massive repository of data crawled … corpora, all of which are derived from Common Crawl. It consists of 6.3 trillion tokens in 167 … crawled sources, such as Common Crawl. The total size …


UNVEILING THE COGNITIVE CAPACITY OF CHATGPT: ASSESSING ITS HUMAN-LIKE REASONING ABILITIES

… OpenAI utilizes a dataset termed Common Crawl, which is a collection of publicly available websites. This dataset includes billions of web pages and is one of the largest text datasets available. Meanwhile, this dataset is just a starting line. …


Using Large Language Models to Probe Cognitive Constructs, Augment Data, and Design Instructional Materials

F Kieser, P Wulff - … in Educational Sciences: Approaches, Applications and …, 2024

… It was shown that transformer-based large language models (LLMs could be pretrained on large language data sets such as the Common Crawl of the Internet, the Books Corpus, or Wikipedia, and then further fine-tuned in specific tasks [9, 10] …


Using Large Language Models to Understand Telecom Standards

A Karapantelakis, M Shakur, A Nikou, F Moradi… - arXiv preprint arXiv …, 2024

… They have billions of parameters and are trained on corpora such as Common Crawl [5]. For foundation models to adapt to application domains, such as telecommunications, prompt engineering and fine-tuning approaches are used …


Using rhetorical strategies to design prompts: a human-in-the-loop approach to make AI useful

N Ranade, M Saravia, A Johri - AI & SOCIETY, 2024

… It consumed the text content of the popular CommonCrawl dataset of web pages from 2016 to 2019. Although OpenAI curated it to eliminate redundancy and improve quality, it is nominally 45-terabytes worth of compressed text data. …


UzRoberta: A pre-trained language model for Uzbek

R Davronov, F Adilova - AIP Conference Proceedings, 2024

… Common Crawl datasets, was generated distributed word representations for 157 different languages [8]. Author’s Uzbek Wikipedia model has 110K words, compared to -830K words in the Common Crawl … It is pre-trained on 2.5TB of filtered …


UZROBERTA: AN UZBEK LANGUAGE PRE-TRAINED MODEL

F Adilova, R Davronov, R Safarov - Universum: технические науки, 2023

… Distributed word representations for 157 distinct languages were created using the Wikipedia and Common Crawl databases [8]. The author's Uzbek Wikipedia model has 110K words, but the Common Crawl model only has 830K words. They …


Validating and Exploring Large Geographic Corpora

J Dunn - arXiv preprint arXiv:2403.08198, 2024

… Beginning with a 427 billion word corpus derived from the Common Crawl, three methods are used to improve the quality of sub-corpora representing specific language-country pairs … This paper starts with an existing web corpus derived …


VERB: Visualizing and Interpreting Bias Mitigation Techniques Geometrically for Word Representations

YAN ZHENG, CCM YEH, J WANG, WEI ZHANG - 2024

Complicated and massive datasets are becoming more and more commonly represented as vectorized embeddings. They are part of a de facto model for words in word vector embeddings such as Word2Vec [51] and GloVe [54], and are …


Verifying Claims About Metaphors with Large-Scale Automatic Metaphor Identification

K Aono, R Sasano, K Takeda - arXiv preprint arXiv:2404.01029, 2024

… Third, in this study, the examples were collected from CommonCrawl, which is a corpus of texts on the Web, and thus contains mostly written language. It is unknown whether similar results can be obtained when analyzing examples collected from a …


ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

LT Nguyen - arXiv preprint arXiv:2405.14141, 2024

… Leveraging a Transformer architecture and trained on a massive dataset of 2.5TB filtered CommonCrawl text across 100 languages, it employs a masked language modeling (MLM) objective to learn crosslingual representations. This approach …


ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models

TH Nguyen, AC Le, VC Nguyen - arXiv preprint arXiv:2404.11086, 2024

The rapid advancement of large language models (LLMs) necessitates the development of new benchmarks to accurately assess their capabilities. To address this need for Vietnamese, this work aims to introduce ViLLM-Eval, the …


VISION2UI: A Real-World Dataset with Layout for Code Generation from UI Designs

Y Gui, Z Li, Y Wan, Y Shi, H Zhang, Y Su, S Dong… - arXiv preprint arXiv …, 2024

… we opt to create our dataset on top of the Common Crawl3 dataset. The Common Crawl dataset is a vast collection of Web page data from … This partial dataset from the original Common Crawl dataset includes 3.35 billion pages, amounting to a total …


Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey

J Huang, J Zhang, K Jiang, H Qiu, S Lu - arXiv preprint arXiv:2312.16602, 2023

Traditional computer vision generally solves each single task independently by a dedicated model with the task instruction implicitly designed in the model architecture, arising two limitations: (1) it leads to task-specific models, which require …


VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding

PNT Do, SQ Tran, PG Hoang, K Van Nguyen… - arXiv preprint arXiv …, 2024

… 250002 multilingual CommonCrawl XLM-Robertalarge … This model was trained on a Transformers-based masked language task using two terabytes of CommonCrawl data across more than a hundred languages. The model has two …


WaCadie: Towards an Acadian French Corpus

J Robichaud, P Cook - Proceedings of the 2024 Joint International Conference …, 2024

… We treat each post as one document, similar to one website’s HTML from the Common Crawl corpus. We refer to this corpus as r/acadie. … We used CommonCrawl’s extensive database and scraped New Brunswick (.nb.ca) websites …


WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

J Qiu, H Lv, Z Jin, R Wang, W Ning, J Yu, CB Zhang… - arXiv preprint arXiv …, 2024

… Moreover, Common Crawl contains a large amount of unsafe content, such as toxic and pornographic … Common Crawl data, which includes data extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality …


WCC-EC 2.0: Enhancing Neural Machine Translation with a 1.6 M+ Web-Crawled English-Chinese Parallel Corpus

J Zhang, K Su, Y Tian, T Matsumoto - Electronics, 2024

This research introduces WCC-EC 2.0 (Web-Crawled Corpus—English and Chinese), a comprehensive parallel corpus designed for enhancing Neural Machine Translation (NMT), featuring over 1.6 million English-Chinese sentence pairs …


We have no idea what we are walking into: AI and ethical considerations

KB Forrest - Annals of the New York Academy of Sciences, 2024

… called “Common Crawl” provide data, “C4” is a curated (de-duplicated and some nonsensical junk eliminated) set of the Common Crawl data, then there is a data set well named “The Pile”, and others that are compilations of books, music …


Weakly supervised learning for an effective focused web crawler

PRJ Dhanith, K Saeed, G Rohith, SP Raja - Engineering Applications of Artificial …, 2024

Focused crawler traverses the Web to only collect pages that are relevant to a particular topic, and is increasingly considered as a way to get around the scalability issues with current general-purpose search engines. But the data diversity in the …


Web Crawl Refusals: Insights From Common Crawl

… Abstract: This study investigates server-side blocks encountered by Common Crawl, a major … that approximately 1.68% of websites in a Common Crawl snapshot exhibit some form of explicit … of these blocks and the effectiveness of …


Web Data Commons-RDFa, Microdata, Embedded JSON-LD, and Microformats Data Sets-October 2023

The Web Data Commons RDFa, Microdata and Microformats data sets has been extracted from the September/October 2023 release of the Common Crawl. In summary, we found structured data within 1.7 billion HTML pages out of the 3.4 …


Web Data Commons-Schema. org Table Corpus 2023

The WDC schema.org Table Corpus consists of ~5 million relational tables which were generated by extracting schema.org data from the Common Crawl and grouping the data into seperate tables for each class/host combination, eg all …


Web scraping and Generative Models training in the Directive 790/19

… The companies that created GM never disclosed the exact corpus of training data they employed to train their models; however, it has become clear sources that many copyrighted materials were employed as part of larger datasets such as the …


WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations

H Deng, C Wang, X Li, D Yuan, J Zhan, T Zhou, J Ma… - arXiv preprint arXiv …, 2024

Enhancing the attribution in large language models (LLMs) is a crucial task. One feasible approach is to enable LLMs to cite external sources that support their generations. However, existing datasets and evaluation methods in this domain still …


WebGraph: The Next Generation (Is in Rust)

T Fontana, S Vigna, S Zacchiroli - ACM Web Conference 2024, 2024

We report the results of a yearlong effort to port the WebGraph framework from Java to Rust. For two decades WebGraph has been instrumental in the analysis and distribution of large graphs for the research community of TheWebConf, but the …


What are Large Language Models Made of?

C Stinson - Leading the Way: Envisioning the Future of Higher …, 2024

… CommonCrawl is the lowest quality but largest of these datasets, consisting of text scraped from all over the web. Their data from the years 2016 to 2019 were used to train GPT-3. OpenAI’s quality control measure for filtering CommonCrawl was to …


What's in a Name? Experimental Evidence of Gender Bias in Recommendation Letters Generated by ChatGPT

DM Kaplan, R Palitsky, SJ Arconada Alvarez… - Journal of Medical Internet …, 2024

… In the case of GPT-3.5 (the current version at the time of this study), the model contains 175 billion parameters and was trained with commonly available text training sets that represent a broad swath of data on the internet (Common Crawl) …


When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

J Wang, C Huang, S Yan, W Xie, D He - International Journal of Human–Computer …, 2024

As a novel AI-powered conversational system, large language models (LLMs) have the potential to be used in various applications. Recent advances in LLMs like ChatGPT have made LLM-based academic tools possible. However, most of the …


Where does In-context Translation Happen in Large Language Models

S Sia, D Mueller, K Duh - arXiv preprint arXiv:2403.04510, 2024

Self-supervised large language models have demonstrated the ability to perform Machine Translation (MT) via in-context learning, but little is known about where the model performs the task with respect to prompt instructions and demonstration …


Which Word Embeddings for Modeling Web Search Queries? Application to the Study of Search Strategies

In order to represent the global strategies deployed by a user during an information retrieval session on the Web, we compare different pretrained vector models capable of representing the queries submitted to a search engine. More precisely …


Who's Afraid of ChatGPT?: Large Language Models and Conspiracies

… GPT-3 added Common Crawl to the training datasets, 570GB of text obtained after filtering from 45TB of plaintext. The filtering was done by comparing new documents to documents in WebText, with the latter acting as a proxy for high-quality …


Willkommens-Merkel, Chaos-Johnson, and Tore-Klose: Modeling the Evaluative Meaning of German Personal Name Compounds

We present a comprehensive computational study of the under-investigated phenomenon of personal name compounds (PNCs) in German such as Willkommens-Merkel (‘Welcome-Merkel’). Prevalent in news, social media, and …


Worldwide Federated Training of Language Models

A Iacob, L Sani, B Marino, P Aleksandrov, ND Lane - arXiv preprint arXiv:2405.14446, 2024

… To simulate geographically distributed systems, we use a subset of Multilingual Colossal Common Crawl (mC4)[81], covering high and low-… For IID experiments, we use the standard Cleaned Colossal Common Crawl (C4) English dataset …


WRITTEN IN PERSIAN USING A TRANSFORMER-BASED MODEL

T Firoozi, MJ Gierl - The Routledge International Handbook of Automated …, 2024

… This model is created using contents from Common Crawl and Wikipedia. It is trained using CBOW with position weights, in dimension 300, with character n-grams of length 5, and a window of size 5 and 10. The model gives 300-dimensional vector …


X-Phishing-Writer: A Framework for Cross-Lingual Phishing Email Generation

SW Guo, YC Fan - ACM Transactions on Asian and Low-Resource …, 2024

… The mBART model employs various language corpora from the Common Crawl dataset and undergoes multilingual pretraining based on the BART architecture. The training data is a concatenation of data from � diferent languages, denoted as � = �1,�2...�� …


XLMR4MD: New Vietnamese dataset and framework for detecting the consistency of description and permission in Android applications using large language models

QN Nguyen, NT Cam, K Van Nguyen - Computers & Security, 2024

… The most significant update that XLMRoBERTa provides compared to the original is that the amount of training data has increased significantly, the cleaned CommonCrawl data it was trained on takes up a considerable memory block of up to …


XLORE 3: A Large-scale Multilingual Knowledge Graph from Heterogeneous Wiki Knowledge Resources

K Zeng, H Jin, X Lv, F Zhu, L Hou, Y Zhang, F Pang… - ACM Transactions on …, 2024

In recent years, Knowledge Graph (KG) has attracted significant attention from academia and industry, resulting in the development of numerous technologies for KG construction, completion, and application. XLORE is one of the largest …


Yi: Open Foundation Models by 01. AI

A Young, B Chen, C Li, C Huang, G Zhang, G Zhang… - arXiv preprint arXiv …, 2024

… We start with web documents from Common Crawl, use the CCNet pipeline [79] for language identification and perplexity scoring. Then … Notably, the Chinese content extracted from Common Crawl present unique challenges, particularly with a …


Yuan 2.0-M32: Mixture of Experts with Attention Router

S Wu, J Luo, X Chen, L Li, X Zhao, T Yu, C Wang… - arXiv preprint arXiv …, 2024

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention Router, is proposed and adopted for a more efficient selection of experts …


Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data

N Popp, JH Metzen, M Hein - arXiv preprint arXiv:2404.16637, 2024

… The first one involves relying on large-scale data such as common crawl datasets [7,38]. While this approach is feasible for large foundation models, it poses challenges for smaller models as these lack the same level of generalization capabilities due to …


Zero-shot learning for multilingual discourse relation classification

E Metheniti, P Muller, C Braud, M Hernández-Casas - 5th Workshop on …, 2024

Classifying discourse relations is a hard task: discourse-annotated data is scarce, especially for languages other than English, and there exist different theoretical frameworks that affect textual spans to be linked and the label set used. Thus, work …


Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon

F Koto, T Beck, Z Talat, I Gurevych, T Baldwin - arXiv preprint arXiv:2402.02113, 2024

Improving multilingual language models capabilities in low-resource languages is generally difficult due to the scarcity of large-scale data in those languages. In this paper, we relax the reliance on texts in low-resource languages by using …


ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using ChatGPT

A Kabir, S Wang, Y Tian, M Asaduzzaman, W Zhang - arXiv preprint arXiv …, 2024

Technical question and answering (Q&A) sites such as Stack Overflow have become an important source for software developers to seek knowledge. However, code snippets on Q&A sites are usually uncompilable and semantically incomplete for …


Zyda: A 1.3 T Dataset for Open Language Modeling

Y Tokpanov, B Millidge, P Glorioso, J Pilault, A Ibrahim… - arXiv preprint arXiv …, 2024

… Additionally, unlike these works which directly filter from Common Crawl, we aim to instead collect and collate existing open-source datasets under one banner, and to uniformly filter and deduplicate them against one another so as to create a dataset …