KazQAD: Kazakh Open-Domain Question Answering Dataset

R Yeshpanov, P Efimov, L Boytsov, A Shalkarbayuli… - arXiv preprint arXiv …, 2024

… Kaz-RoBERTa6 is a monolingual model trained on a collection of Kazakh texts from Common Crawl, the Leipzig Corpora Collection, the … XLM-R was pre-trained on a larger multilingual corpus derived from Common Crawl, in which Kazakh is …


KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes

R Yeshpanov, HA Varol - arXiv preprint arXiv:2403.19335, 2024

This paper presents KazSAnDRA, a dataset developed for Kazakh sentiment analysis that is the first and largest publicly available dataset of its kind. KazSAnDRA comprises an extensive collection of 180,064 reviews obtained from various sources …


KEPA-CRF: Knowledge expansion prototypical amortized conditional random field for few-shot event detection

R Wu, L Yu, S Tian, J Long, T Zhou, B Wang - Journal of Intelligent & Fuzzy Systems

Event Detection (ED) has long struggled with the ambiguous definition of event categories, making it challenging to accurately classify events. Previous endeavors aimed to tackle this problem by constructing prototypes for specific event categories …


Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks

N Chirkova, V Nikoulina - arXiv preprint arXiv:2402.12279, 2024

… For Intermediate tuning we finetune models for 100k steps on the CommonCrawl data uniformly sampled across all target languages and English, with the batch size of 5k tokens and the LR chosen to optimize fluency of model generations, inspected …


Knowledge Induced Transformer Network for Causality Prediction

T Dasgupta, M Sinha, A Naskar - Companion Proceedings of the ACM on Web …, 2024

… The CausalBank dataset3 contains 314 million pairs of cause-effect statements scraped from the Common Crawl corpus using causal lexical patterns. Baselines: Most of the earlier work on causality detection is based on identifying cause, effect …


Knowledge Sources

M Jiang, BY Lin, S Wang, Y Xu, W Yu, C Zhu - Knowledge-augmented Methods for …, 2024

… To be more specific, the dataset includes Pile-CommonCrawl (227.12G, web crawling data), PubMed Central (90.27G, biomedical articles), Books (100.96G, a mix of fiction and nonfiction books), OpenWebText (262.77G, content from Reddit) …


Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models

R Vemulapalli, H Pouransari, F Faghri, S Mehta… - Forty-first International Conference …

… 2023) which contains 1.28B images filtered from Common Crawl (web-crawled data). We take 10 randomly augmented crops from each image and create a gallery of 12.8B images. See Appendix B.2 for additional details about this gallery set and …


Knowledge-augmented Methods for Natural Language Processing

Ever since language was invented and used, humans have leveraged this powerful tool to pass down experience over generations. These kinds of knowledge summarize precious findings and inspiring ideas, which constitute the human …


Knowledge-Enhanced Neural Networks for Machine Reading Comprehension

Machine Reading Comprehension is a language understanding task where a system is expected to read a given passage of text and typically answer questions about it. When humans assess the task of reading comprehension, in addition to the …


KOMPYUTER LINGVISTIKASIDA TA'MINOT MASALASI VA SINONIM BIRLIKLARNING LINGVISTIK TA'MINOTI

N SOTVOLDIYEVA - News of UzMU journal, 2024

… Lingvistik ma'lumotlar konsorsiumi (LDC) va Common Crawl[8] loyihasi kabi ochiq hamkorlik tashabbuslari lingvistik resurslarning mavjudligiga sezilarli hissa qo'shdi. Bu hamkorlik resurslar almashinuvini targ‘ib qiladi, ortiqchalikni kamaytiradi va …


Krishiq-BERT: A Few-Shot Setting BERT Model to Answer Agricultural-Related Questions in the Kannada Language

P Ajawan, V Desai, S Kale, S Patil - Journal of The Institution of Engineers (India) …, 2024

… corpora for 11 languages from two language families obtained primarily by sourcing articles from news crawls [3], and OSCAR (Open Super-large Crawled ALMAnaCH coRpus), a huge multilingual corpus procured by language …


KVP10k: A Comprehensive Dataset for Key-Value Pair Extraction in Business Documents

O Naparstek, R Pony, I Shapira, FA Dahood, O Azulai… - arXiv preprint arXiv …, 2024

… : extensive web data from Common Crawl and a collection of images from publicfiles.fcc.gov. From Common Crawl, we employed a systematic … A schematic describing the data acquisition process for the common crawl data is shown in Fig.2 …


Labadain-30k+: A Monolingual Tetun Document-Level Audited Dataset

G de Jesus, S Nunes - Proceedings of the 3rd Annual Meeting of the Special …, 2024

This paper introduces Labadain-30k+, a monolingual dataset comprising 33.6 k documents in Tetun, a low-resource language spoken in Timor-Leste. The dataset was acquired through web crawling and augmented with Wikipedia documents …


Language Model Frame Filling for Low Resource Languages

Acceptability judgment tasks are key tools used in the understanding of the grammar of languages by linguists. A typical way to construct these tasks is to use semantic frames. We provide evidence that the fill-mask objective for language models (LMs) …


Language Models Fine-Tuning for Automatic Format Reconstruction of SEC Financial Filings

G Lombardo, G Trimigno, M Pellegrino, S Cagnoni - IEEE Access, 2024

The analysis of financial reports is a crucial task for investors and regulators, especially the mandatory annual reports (10-K) required by the SEC (Securities and Exchange Commission) that provide crucial information about a public company in …


Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining

N Ljubešić, V Suchomel, P Rupnik, T Kuzman… - arXiv preprint arXiv …, 2024

The world of language models is going through turbulent times, better and ever larger models are coming out at an unprecedented speed. However, we argue that, especially for the scientific community, encoder models of up to 1 billion parameters …


Language models scale reliably with over-training and on downstream tasks

SY Gadre, G Smyrnis, V Shankar, S Gururangan… - arXiv preprint arXiv …, 2024

Scaling laws are useful guides for developing language models, but there are still gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal …


Large Language Models (LLMs) on Tabular Data: Predic-tion, Generation, and Understanding-A Survey

X Fang, W Xu, FA Tan, J Zhang, Z Hu, Y Qi… - arXiv preprint arXiv …, 2024

Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table …


Large Language Models and the Disappearing Private Sphere

A Cappello, M Dada, V Grigoreva, R Khan, C Stinson… - 2024

… The higher quality datasets are sampled more often during training, but the Reddit approved contents of CommonCrawl still … of CommonCrawl contains fan fiction, video game chats, conspiracy theories, pornography, junk advertising, and …


Large Language Models aren't all that you need

KV Holla, C Kumar, A Singh - arXiv preprint arXiv:2401.00698, 2024

This paper describes the architecture and systems built towards solving the SemEval 2023 Task 2: MultiCoNER II (Multilingual Complex Named Entity Recognition) [1]. We evaluate two approaches (a) a traditional Conditional Random …


LARGE LANGUAGE MODELS AS AN AID FOR SOFTWARE DEVELOPMENT USING GRAPHICAL PROGRAMMING LANGUAGE.

… In pre-training, the model is trained with the Common Crawl dataset, internet books, and Python code, summing up to 400B tokens. After the model is pre-trained, the phase preference model pre-training comes, and after that, the preference model …


Large language models as oracles for instantiating ontologies with domain-specific knowledge

G Ciatto, A Agiollo, M Magnini, A Omicini - arXiv preprint arXiv:2404.04108, 2024

… Models following the RoBERTa [30] approach were trained on (i) the same datasets as BERT, plus (ii) a sample of the CommonCrawl4 News dataset (a dump of news … 4https://commoncrawl.org 5https://mistral.ai/news/mixtral-of-experts …


Large language models for biomedicine: foundations, opportunities, challenges, and best practices

SS Sahoo, JM Plasek, H Xu, Ö Uzuner, T Cohen… - Journal of the American …, 2024

… In the pre-training phase, generic training data, such as the Common Crawl dataset or social media posts, are used for unsupervised learning.With further training on prompt-response pairs (such as the Alpaca dataset developed at Stanford23) …


Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review

J Su, C Jiang, X Jin, Y Qiao, T Xiao, H Ma, R Wei… - arXiv preprint arXiv …, 2024

… The training data for GPT-3 includes the Common Crawl dataset for lower quality, as well as the WebText2 dataset for higher quality, as well as the Books1, Books2, and Wikipedia datasets for higher quality [148]. GPT-3 assigns different weights to …


Large Language Models for Human-Machine Collaborative Particle Accelerator Tuning through Natural Language

J Kaiser, A Eichler, A Lauscher - arXiv preprint arXiv:2405.08888, 2024

Autonomous tuning of particle accelerators is an active and challenging field of research with the goal of enabling novel accelerator technologies cutting-edge high-impact applications, such as physics discovery, cancer research and material sciences. A …


Large Language Models for identification of medical data in unstructured records

… Sources for the training data include web pages extracted by CommonCrawl, opensource GitHub repositories, Wikipedia in twenty languages, Project Gutenberg books in the public domain, LaTeX source code of scientific articles in ArXiv and …


Large Language Models for Reliable Information Extraction

This project addresses the ongoing challenge of achieving reliable Information Extraction (IE), particularly in domains which require near-perfect precision. ELICIT (Butcher et al., 2023) introduced a novel approach that combines the processing speed of …


Large Language Models on Tabular Data--A Survey


Large Language Models: A Survey

S Minaee, T Mikolov, N Nikzad, M Chenaghlu… - arXiv preprint arXiv …, 2024

… Common Crawlbased dataset consisting of texts in 101 languages. … CommonCrawl. They also released an extract of 600 billion tokens from our REFINEDWEB dataset, and 1.3/7.5B parameters language models trained on it. 27 …


Large Malaysian Language Model Based on Mistral for Enhanced Local Language Understanding

H Zolkepli, A Razak, K Adha, A Nazhan - arXiv preprint arXiv:2401.13565, 2024

… An analysis of the widely utilized Common Crawl dataset reveals a mere 0.0742% contribution from the Malay language based on CCMAIN-… We also replicated the same technique for Malaysia Hansard and MS CommonCrawl samples. All synthetic …


Large-language models facilitate discovery of the molecular signatures regulating sleep and activity

D Peng, L Zheng, D Liu, C Han, X Wang, Y Yang… - Nature Communications, 2024

Sleep, locomotor and social activities are essential animal behaviors, but their reciprocal relationships and underlying mechanisms remain poorly understood. Here, we elicit information from a cutting-edge large-language model (LLM), generative pre-trained …


Large-Language-Models (LLM)-Based AI Chatbots: Architecture, In-Depth Analysis and Their Performance Evaluation

V Kumar, P Srivastava, A Dwivedi, I Budhiraja… - International Conference on …, 2023

… LLaMA models span from 7 billion to 65 billion parameters and were trained on trillions of tokens derived from publicly accessible datasets, including Common Crawl and Wikipedia. The primary objective of LLaMA models is to serve as …


Latxa: An Open Language Model and Evaluation Suite for Basque

J Etxaniz, O Sainz, N Perez, I Aldabe, G Rigau, E Agirre… - arXiv preprint arXiv …, 2024

We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the …


Law and the Political Economy of AI Production

The governance of Artificial Intelligence (AI) is at a historical juncture. Legislative acts, global treaties, export controls, and technical standards are now dominating the discourse over what used to be a predominantly market-driven space. Amidst all this …


Law, Technology and Humans

… Around 60% of GPT-3 training data comes from a filtered version of Common Crawl—that is, a non-profit organisation that archives and provides open-access petabytes of data collected since 2011. [30] The remaining 40% comes from diverse …


Learning with Noisy Foundation Models

H Chen, J Wang, Z Wang, R Tao, H Wei, X Xie… - arXiv preprint arXiv …, 2024

… GPT2 is pre-trained on WebText [29], a scraped web dataset from Common Crawl that contains low-quality raw texts. We also leverage OpenAI’s API service “text-ada-002”. We cannot use larger and more recent language models such as LLaMA [30], since …


Leveraging AI to Deliver Culturally Responsive Mental Health Care at Scale

A Cerezo, D Cooper, V Palat, A Jolley, S Peregrine

… One of the most commonly used datasets to train LLMs is The Common Crawl dataset, a large collection of web pages scraped from the internet since 2014. These pages contain the gamut from high-quality news, science, and literature to the …


Leveraging distant supervision and deep learning for twitter sentiment and emotion classification

M Kastrati, Z Kastrati, A Shariq Imran, M Biba - Journal of Intelligent Information …, 2024

Nowadays, various applications across industries, healthcare, and security have begun adopting automatic sentiment analysis and emotion detection in short texts, such as posts from social media. Twitter stands out as one of the most popular …


Leveraging Open Large Language Models for Multilingual Policy Topic Classification: The Babel Machine Approach

M Sebők, Á Máté, O Ring, V Kovács, R Lehoczki - Social Science Computer Review, 2024

The article presents an open-source and freely available natural language processing system for comparative policy studies. The CAP Babel Machine allows for the automated classification of input files based on the 21 major policy topics of …


Leveraging Parameter Efficient Training Methods for Low Resource Text Classification: A case study in Marathi

P Deshmukh, N Kulkarni, S Kulkarni, K Manghani… - 2024 IEEE 9th International …, 2024

With the surge in digital content in low-resource languages, there is an escalating demand for advanced Natural Language Processing (NLP) techniques tailored to these languages. BERT (Bidirectional Encoder Representations from Transformers) …


Leveraging Zero-Shot Prompting for Efficient Language Model Distillation

L Vöge, V Gurgul, S Lessmann - arXiv preprint arXiv:2403.15886, 2024

… Specifically, we use the T5 v1.1 models which were only pretrained on the C4 Common Crawl dataset for general language understanding without additional task-specific finetuning. The use of text-to-text models allows for straightforward processing, as …


Lib2Life–Digital Library Services Empowered with Advanced Natural Language Processing Techniques

M Nitu, M Dascalu, MD Dascalu, LM Neagu…

Educational institutions are struggling to keep up with the accelerated technological advancements; hence, sustainable and supportive tools have become essential to reshape traditional models into intelligent learning systems. This paper introduces …


LiBERTa: Advancing Ukrainian Language Modeling through Pre-training from Scratch

M Haltiuk, A Smywiński-Pohl - Proceedings of the Third Ukrainian Natural Language …, 2024

… CC-100, a multilingual text corpus sourced from Wikipedia and CommonCrawl, was processed following the CCNet1 methodology. The Ukrainian segment of CC-100 encompasses 6.5 billion tokens, equivalent to 84 GiB of data2. This corpus primarily …


Limitations of Language Models in The Oil & Gas Upstream Operations

A Alsultan, FA Razak - International Petroleum Technology Conference, 2024

… Although the process is simple in principle, it is done on very large training corpra which can be as big as dataset provided by the Common Crawl project (428.81 TB as of the date of writing this paper). Thus, training these models using these large …


LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages

AM Bean, S Hellsten, H Mayne, J Magomere, EA Chi… - arXiv preprint arXiv …, 2024

… ], while the second and fourth rows are sorted by Common Crawl Share [16]. The upper two … of pages tagged with a given language in the Common Crawl dataset [16]. Figure 12 show the … LLMs are trained on, using the Common Crawl share and …


Linguistic Profiling of Deepfakes: An Open Database for Next-Generation Deepfake Detection

Y Wang, Z Huang, Z Ma, X Hong - arXiv preprint arXiv:2401.02335, 2024

The emergence of text-to-image generative models has revolutionized the field of deepfakes, enabling the creation of realistic and convincing visual content directly from textual descriptions. However, this advancement presents considerably greater …


Linguistic Structure from a Bottleneck on Sequential Information Processing

R Futrell, M Hahn - arXiv preprint arXiv:2405.12109, 2024

… Table 3: English expressions for the given meanings, along with frequencies from the English Common Crawl web corpus [64]. Example … n-gram counts and language models from the common crawl. In Language Resources and Evaluation …


Llama (LLM)

… Data was mostly from CommonCrawl and C4. Llama-2-70B saw 2T tokens but Llama-3-70B saw 15T tokens. Llama-3-8B and Llama-3-70B took 1.3M and 6.4M GPU hours for pretraining. …


LLaMA Beyond English: An Empirical Study on Language Capability Transfer

J Zhao, Z Zhang, Q Zhang, T Gui, X Huang - arXiv preprint arXiv:2401.01055, 2024

In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks. However, many mainstream LLMs (eg LLaMA) are …


LLM-Datasets: An Open Framework for Pretraining Datasets of Large Language Models

M Ostendorff, PO Suarez, LF Lage, G Rehm

… All of the numbers are given by Common Crawl dump. … 2 show the number of documents by both date of the CommonCrawl dump and language, the percentage of documents removed at each filtering step, and the final document and token …


LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

N Lee, T Wattanawong, S Kim, K Mangalam, S Shen… - arXiv preprint arXiv …, 2024

… We assume that we are given an LLM model M (eg, GPT-3.5 or LLaMA2-7B) that is pre-trained on some source dataset (eg, Common Crawl). The goal is to adapt M (hereon called the student model) to a new target domain by using a small seed dataset D …


LLMs achieve adult human performance on higher-order theory of mind tasks

W Street, JO Siy, G Keeling, A Baranes, B Barnett… - arXiv preprint arXiv …, 2024

This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (eg I think that you …


LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings

F Alam, SA Chowdhury, S Boughorbel, M Hasanain - … of the 18th Conference of the …, 2024

The recent breakthroughs in Artificial Intelligence (AI) can be attributed to the remarkable performance of Large Language Models (LLMs) across a spectrum of research areas (eg, machine translation, question-answering, automatic speech …


LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments

BP Cipriano, P Alves - arXiv preprint arXiv:2403.06254, 2024

… are high-level descriptions of the training datasets (eg a filtered version of Common Crawl 14, English Wikipedia, among others) [3], it was not possible to perform this analysis for those models. 13https:github.com/camilaabrantes/CursoJava …


LMaaS: Exploring Pricing Strategy of Large Model as a Service for Communication

P Wu, Q Liu, Y Dong, F Wang - arXiv preprint arXiv:2401.02675, 2024

The next generation of communication is envisioned to be intelligent communication, that can replace traditional symbolic communication, where highly condensed semantic information considering both source and channel will be extracted and …


Long-input summarization using Large Language Models

… the colossal and cleaned version of common crawl CNN … T5 was pre-trained on the C4 (The Colossal and Cleaned version of Common Crawl) dataset. C4 contains text from 350M web pages; the dataset is 750 GB. A maximum sequence length of …


Long-Term Ad Memorability: Understanding & Generating Memorable Ads

SI Harini, S Singh, Y Kumar, A Bhattacharyya, V Baths…

Marketers spend billions of dollars on advertisements, but to what end? At purchase time, if customers cannot recognize the brand for which they saw an ad, the money spent on the ad is essentially wasted. Despite its importance in marketing, until now …


LongWanjuan: Towards Systematic Measurement for Long Text Quality

K Lv, X Liu, Q Guo, H Yan, C He, X Qiu, D Lin - arXiv preprint arXiv:2402.13583, 2024

… In the English data, the CommonCrawl domain predominates, accounting for over 50% of the data. Apart from a significant amount of aggregated texts in the CommonCrawl domain, the majority of data in other domains consists of holistic texts …


Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

Q Zhang, T Du, H Huang, Y Wang, Y Wang - Forty-first International Conference on Machine …

In recent years, the rise of generative self-supervised learning (SSL) paradigms has exhibited impressive performance across visual, language, and multi-modal domains. While the varied designs of generative SSL objectives lead to distinct properties in …


Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training

Z Zhong, M Xia, D Chen, M Lewis - arXiv preprint arXiv:2405.03133, 2024

… We suspect that segment-level routing may be particularly effective at handling out-of-domain evaluation data (we assume that there is only a very small part of Python code in Commoncrawl). Our analysis in Section 5.4 shows that segment-level routing …


M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

G Thakkar, S Hakimov, M Tadić - arXiv preprint arXiv:2404.01753, 2024

… 2020b) is a large multilingual language model trained on 2.5 TB of filtered Common Crawl data containing 100 languages. The model was trained with the Masked Language Modelling (MLM) objective, with 15% of the input words masked …


Machine Learning and the Analysis of Culture

The focus of this chapter is on how machine learning (ML) impacts the analysis of culture in sociology. It shows how ML has greatly advanced the analysis of culture, with new tools enabling a massive and fine-grained extraction of information from …


Machine Translation that Peeks at the Reference

Abstract Machine translation with lexical constraints is a popular research topic, especially for terminology translation. Existing approaches for lexical control in MT are usually complex and not easily applicable to all existing MT toolkits. We propose …


Machines Do See Color: A Guideline to Classify Different Forms of Racist Discourse in Large Corpora

DD Gordillo, J Timoneda, SV Vera - arXiv preprint arXiv:2401.09333, 2024

Current methods to identify and classify racist language in text rely on small-n qualitative approaches or large-n approaches focusing exclusively on overt forms of racist discourse. This article provides a step-by-step generalizable guideline to …


MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

K Zhang, Y Luan, H Hu, K Lee, S Qiao, W Chen, Y Su… - arXiv preprint arXiv …, 2024

… We collect all images with the same URL from Common Crawl2 as a group of images from the same web page for potential pairing. Due to the inevitable noisy images introduced by simple grouping, we remove duplicated, low-resolution, and …


MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews

O Ignat, X Xu, R Mihalcea - arXiv preprint arXiv:2404.12938, 2024

Deceptive reviews are becoming increasingly common, especially given the increase in performance and the prevalence of LLMs. While work to date has addressed the development of models to differentiate between truthful and …


MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness

S Zhou, H Shan, B Plank, R Litschko - arXiv preprint arXiv:2404.02570, 2024

This paper presents our system developed for the SemEval-2024 Task 1: Semantic Textual Relatedness (STR), on Track C: Cross-lingual. The task aims to detect semantic relatedness of two sentences in a given target language without access to …


MambaByte: Token-free Selective State Space Model

J Wang, T Gangavarapu, JN Yan, AM Rush - arXiv preprint arXiv:2401.13660, 2024

Token-free language models learn directly from raw bytes and remove the bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences, and standard autoregressive Transformers scale poorly in such settings …


MAmmoTH2: Scaling Instructions from the Web

X Yue, T Zheng, G Zhang, W Chen - arXiv preprint arXiv:2405.03548, 2024

… We argue that the pre-training corpus (eg, Common Crawl) already contains a vast amount of high-quality instruction data for LLM reasoning. For example, the web corpus contains a large amount of educational materials in the form of instruction-following …


MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series

G Zhang, S Qu, J Liu, C Zhang, C Lin, CL Yu, D Pan… - arXiv preprint arXiv …, 2024

… Inspired by [89], which adopts an iterative pipeline to facilitate the acquisition of large-scale, high-quality data from Common Crawl, we propose to select high-quality data for mathematics, scientific exam synthetic data, and wiki-based content in our Matrix. …


Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking

Y Fung, R Zhao, J Doo, C Sun, H Ji - arXiv preprint arXiv:2402.09369, 2024

Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions …


MASTERARBEIT/MASTER'S THESIS

Today, a huge amount of unstructured data from various sources is available. Natural language processing (NLP) techniques using such data are applied to several tasks such as named entity recognition (NER). This study is designed to …


Mastering Transformers: The Journey from BERT to Large Language Models and Stable Diffusion

Explore transformer-based language models from BERT to GPT, delving into NLP and computer vision tasks, while tackling challenges effectively Key Features Understand the complexity of deep learning architecture and transformers …


Materials science in the era of large language models: a perspective

G Lei, R Docherty, SJ Cooper - arXiv preprint arXiv:2403.06949, 2024

Large Language Models (LLMs) have garnered considerable interest due to their impressive natural language capabilities, which in conjunction with various emergent properties make them versatile tools in workflows ranging from complex …


Matthew Coscia and

… URL: https://github.com/commoncrawl/ia-web-commons. [9] Image Details - Container Registry - Google Cloud Platform. 2022. …


MDSAA

In the field of digital marketing, efficient content gap analysis is crucial for developing effective SEO strategies. Traditional approaches to this task are time-consuming, representing a challenge for Organic Performance teams, who must process large …


Measuring and Improving the Energy Efficiency of Large Language Models Inference

MF Argerich, M Patiño-Martínez - IEEE Access, 2024

… 4) RedPajama [42]: another family of models based on Pythia with 7B and 3B parameters, fine-tuned on a dataset with the same name as the model, with over 100B text documents coming from 84 CommonCrawl snapshots. Similar to Dolly …


Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

I García-Ferrero, R Agerri, AA Salazar, E Cabrio… - arXiv preprint arXiv …, 2024

… mT5 was trained using mC4, a 1 Trillion token Common Crawl-based dataset covering 101 languages. The pre-training is based on a masked language modeling “spancorruption” objective, where consecutive spans of input tokens are replaced …


Meeting the challenge: A benchmark corpus for automated Urdu meeting summarization

B Sadia, F Adeeba, S Shams, K Javed - Information Processing & Management, 2024

Meeting summarization has become crucial as the world is gradually shifting towards remote work. Nowadays, automation of meeting summary generation is really needed in order to minimize the time and effort. The surge in online meetings …


MELTing point: Mobile Evaluation of Language Transformers

S Laskaridis, K Kateveas, L Minto, H Haddadi - arXiv preprint arXiv:2403.12844, 2024

Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with ``sparks of intelligence''. However, their runtime requirements have prevented them from being …


Memorization and Privacy Risks in Domain-Specific Large Language Models

X Yang, Z Wen, W Qu, Z Chen, Z Xiang, B Chen, H Yao - ICLR 2024 Workshop on Reliable …

… Common Crawl3 is a nonprofit organization that provides a large and open web crawl data repository for public use. It collects web pages from the internet every month and stores them on Amazon Web Services. Common Crawl’s … C4 cleanses …


Mentions of prejudice in news media–an international comparison

D Rozado - Journal of Computational Social Science, 2024

Previous research has identified a post-2010 sharp increase of terms used to denounce prejudice (ie racism, sexism, homophobia, Islamophobia, anti-Semitism, etc.) in US and UK news media content. Here, we extend previous analysis to an …


Methods Futures Report–Sea, Sky, and Land: Engaging in Solidarity in Endangered Ecologies–4S Conference Nov 2023

This is a report of the 4S 2023 conference. The 4S (Society for the Social Studies of Science) conference 2023 was a large, disciplinarily diverse gathering with over 400 sessions spread over four days. Navigating a particular route through the conference …


Milestones in Bengali Sentiment Analysis leveraging Transformer-models: Fundamentals, Challenges and Future Directions

S Sengupta, S Ghosh, P Mitra, TI Tamiti - arXiv preprint arXiv:2401.07847, 2024

Sentiment Analysis (SA) refers to the task of associating a view polarity (usually, positive, negative, or neutral; or even fine-grained such as slightly angry, sad, etc.) to a given text, essentially breaking it down to a supervised (since we have the view …


MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

S Hu, Y Tu, X Han, C He, G Cui, X Long, Z Zheng… - arXiv preprint arXiv …, 2024

The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This …


Misinformation Resilient Search Rankings with Webgraph-based Interventions

P Carragher, EM Williams, KM Carley - arXiv preprint arXiv:2404.08869, 2024

… To this end, we investigate the distribution of changes in CommonCrawl PageRank as a result of our interventions for all domains in the webgraph, not just news domains. We then take the domains whose PR score changes significantly ( 50%) …


Mission Critical--Satellite Data is a Distinct Modality in Machine Learning

E Rolf, K Klemmer, C Robinson, H Kerner - arXiv preprint arXiv:2402.01444, 2024

Satellite data has the potential to inspire a seismic shift for machine learning -- one in which we rethink existing practices designed for traditional data modalities. As machine learning for satellite data (SatML) gains traction for its real-world impact …


MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

J Ni, F Xue, X Yue, Y Deng, M Shah, K Jain, G Neubig… - arXiv preprint arXiv …, 2024

… In the detection phase, we train open-source LLMs on self-collected data to detect queries in Common Crawl splits. During filtering, we utilize GPT-4 Turbo to exclude non-query sentences. In classification, we categorize the filtered queries by input …


MobileVLM: A Fast, Reproducible and Strong Vision Language Assistant for Mobile Devices

X Chu, L Qiao, X Lin, S Xu, Y Yang, Y Hu, F Wei… - arXiv preprint arXiv …, 2023

We present MobileVLM, a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It is an amalgamation of a myriad of architectural designs and techniques that are mobile-oriented, which comprises a set of language …


Modeling Caption Diversity in Contrastive Vision-Language Pretraining

S Lavoie, P Kirichenko, M Ibrahim, M Assran… - arXiv preprint arXiv …, 2024

There are a thousand ways to caption an image. Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an …


Modeling the Sacred: Considerations when Using Considerations when Using Religious Texts in Natural Language Processing

B Hutchinson - arXiv preprint arXiv:2404.14740, 2024

This position paper concerns the use of religious texts in Natural Language Processing (NLP), which is of special interest to the Ethics of NLP. Religious texts are expressions of culturally important values, and machine learned models have a …


mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

M Futeral, A Zebaze, PO Suarez, J Abadji, R Lacroix… - arXiv preprint arXiv …, 2024

… This motivated us to collect and release the first large-scale multilingual and multimodal document dataset derived from Common Crawl.Our dataset, multimodal OSCAR (mOSCAR), follows the OSCAR initiative [Ortiz Suárez et al., 2019, Abadji et …


MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Z Zong, B Ma, D Shen, G Song, H Shao, D Jiang, H Li… - arXiv preprint arXiv …, 2024

As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in …


mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models

P Lin, C Hu, Z Zhang, AFT Martins, H Schütze - Findings of the Association for …, 2024

Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible to employ mPLMs to …


mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

A Hu, H Xu, J Ye, M Yan, L Zhang, B Zhang, C Li… - arXiv preprint arXiv …, 2024

Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition …


MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels

Q Chen, X Geng, C Rosset, C Buractaon, J Lu, T Shen… - Companion Proceedings of …, 2024

Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked …


Multi Class Depression Detection Through Tweets using Artificial Intelligence

MO Nusrat, W Shahzad, SA Jamal - arXiv preprint arXiv:2404.13104, 2024

… Typically, GloVe embeddings were pretrained on extensive corpora like Wikipedia or Common Crawl and fine-tuned for specific tasks. 300 LSTM units were used with a dropout rate of 0.4. For depression classification, the dense layer was …


Multi-class hate speech detection in the Norwegian language using FAST-RNN and multilingual fine-tuned transformers

E Hashmi, SY Yayilgan - Complex & Intelligent Systems, 2024

… FastText, a word representation tool developed by Facebook’s research division, offers both unsupervised and supervised modes and features a comprehensive database of 2 million words from Common Crawl, each represented by 300 …


Multi-Head Mixture-of-Experts

X Wu, S Huang, W Wang, F Wei - arXiv preprint arXiv:2404.15045, 2024

Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for …


Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation

J Choi, J Yun, K Jin, YB Kim - arXiv preprint arXiv:2404.09682, 2024

… an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Multi-Reference Benchmarks for Russian Grammatical Error Correction

FP Gomez, A Rozovskaya - Proceedings of the 18th Conference of the European …, 2024

This paper presents multi-reference benchmarks for the Grammatical Error Correction (GEC) of Russian, based on two existing single-reference datasets, for a total of 7,444 learner sentences from a variety of first language backgrounds. Each …


Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation

M Guan, Y Wang, G Ma, J Liu, M Sun - arXiv preprint arXiv:2405.05672, 2024

Sign language serves as a non-vocal means of communication, transmitting information and significance through gestures, facial expressions, and bodily movements. The majority of current approaches for sign language recognition (SLR) …


Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings

I Mohr, M Krimmel, S Sturua, MK Akram, A Koukounas… - arXiv preprint arXiv …, 2024

… Seeking further diversity and volume, we utilize Common Crawl data to create two types of pairs: one from web page titles and their contents, and another by mining questionanswer pairs from FAQ and support-related pages. Additionally, we pair …


Multilingual Diversity Improves Vision-Language Representations

T Nguyen, M Wallingford, S Santy, WC Ma, S Oh… - arXiv preprint arXiv …, 2024

… Data We experiment with the medium pool of the DataComp benchmark [14], which consists of 128M image-text pairs randomly sampled from Common Crawl dumps between 2014 and 2022, and deduplicated. Unlike other heavily filtered …


Multilingual E5 Text Embeddings: A Technical Report

L Wang, N Yang, X Huang, L Yang, R Majumder, F Wei - arXiv preprint arXiv …, 2024

This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a …


Multilingual Hate Speech Detection: Comparison of Transfer Learning Methods to Classify German, Italian, and Spanish Posts

J Fillies, MP Hoffmann, A Paschke - 2023 IEEE International Conference on Big Data …, 2023

… This datasetincludes Wikipedia dumps for all supported languages anda substantial 2.5 terabytes of Common Crawl Data collectedfrom the Internet. The ”RoBERTa” component in the model’sname indicates the adherence to the training procedure …


Multilingual Model Fine-tuning for Sentiment Analysis

ML Elrefai, MI Khalil, HM Abbas - … on Intelligent Computing and Information Systems …, 2023

… A sizable multilingual language model was created using 2.5 TB of filtered CommonCrawl data. One of the outstanding XLMRoBERTa in low-resource languages, Low-resource languages are languages with limited available data for …


Multilingual Models in Neural Machine Translation

… XGLM [19] uses the decoder-only model architecture similar to that of GPT-3 [6], and was pre-trained on CC100-XL that covers 68 Common Crawl (CC) snapshots and 134 languages. Experiment results show that XGLM (7.5B) outperforms GPT-3 (6.7B) …


MultiLS: A Multi-task Lexical Simplification Framework

K North, T Ranasinghe, M Shardlow, M Zampieri - arXiv preprint arXiv:2402.14972, 2024

Lexical Simplification (LS) automatically replaces difficult to read words for easier alternatives while preserving a sentence's original meaning. LS is a precursor to Text Simplification with the aim of improving text accessibility to various target …


Multimodal Learning for Visual Question Answering using World Knowledge

Navigating the frontier of the Visual Turing Test, this research delves into multimodal learning to bridge the gap between visual perception and linguistic interpretation, a foundational challenge in artificial intelligence. It scrutinizes the integration of visual …


Multiscale cascaded domain-based approach for Arabic fake reviews detection in e-commerce platforms

N Qandos, G Hamad, M Alharbi, S Alturki, W Alharbi… - Journal of King Saud …, 2024

Fake reviews in e-commerce can lead to customer deception and financial losses. Despite the importance of fake reviews detection, studies for Arabic language are scarce due to the lack of comprehensive datasets. This study addresses this gap by …


Multivariate Log-based Anomaly Detection for Distributed Database

L Zhang, T Jia, M Jia, Y Li, Y Yang, Z Wu - arXiv preprint arXiv:2406.07976, 2024

… Following this, MultiLog utilizes pre-trained word vectors based on the Common Crawl Corpus with the FastText algorithm[45], which is adept at capturing the inherent relationships among words in natural language. This means each word in a …


Musical Word Embedding for Music Tagging and Retrieval

SH Doh, J Lee, D Jeong, J Nam - arXiv preprint arXiv:2404.13569, 2024

… We used two publicly available embeddings trained on Common Crawl, a large-scale general word corpus, with the GloVe and skipgram … It demonstrates that word embeddings trained with a general word corpus, such as Common Crawl and Wiki …


Myanmar XNLI: Building a Dataset and Exploring Low-resource Approaches to Natural Language Inference with Myanmar

Despite dramatic recent progress in NLP, it is still a major challenge to apply Large Language Models (LLM) to low-resource languages. This is made visible in benchmarks such as Cross-Lingual Natural Language Inference (XNLI), a key task …


Named Entity Recognition in Italian Lung Cancer Clinical Reports using Transformers

D Paolo, A Bria, C Greco, M Russano, S Ramella… - 2023 IEEE International …, 2023

The widespread adoption of electronic health records (EHRs) offers a valuable opportunity to support clinical research by containing crucial patient information, including diagnoses, symptoms, medications, lab tests, and more. Despite the …


NATALITY AT RISK? RAISING DOUBTS ON THE EDUCATIONAL IMPORTANCE OF ChatGPT

Digital tools are part of our daily life. Thus, they also enter educational contexts. With reference to Hannah Arendt’s writings this paper explores if, and to what extent, digital tools can be considered to be helpful and desirable in education. For this …


Natural Language Processing Advancements: Breaking Barriers in Human-Computer Interaction

JGC Ramírez - Journal of Artificial Intelligence General science (JAIGS …, 2024

Natural Language Processing (NLP) advancements have revolutionized human-computer interaction, breaking barriers and opening new frontiers in technology. NLP techniques enable machines to understand, interpret, and generate human …


Natural Language Processing for Dialects of a Language: A Survey

A Joshi, R Dabre, D Kanojia, Z Li, H Zhan, G Haffari… - arXiv preprint arXiv …, 2024

State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language …


Natural Language Processing Journal

Natural language processing and vision tasks have recently seen large improvements through the rise of Transformer architectures. The high-performing large language models (LLMs) benefit from large textual datasets that are …


Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey

DVT Le, L Bigo, M Keller, D Herremans - arXiv preprint arXiv:2402.17467, 2024

Several adaptations of Transformers models have been developed in various domains since its breakthrough in Natural Language Processing (NLP). This trend has spread into the field of Music Information Retrieval (MIR), including studies …


Natural Language Processing Tools for Romanian–Going Beyond a Low-Resource Language

… Oscar (Open Super-large Crawled Aggregated coRpus) [64] is a multilingual dataset obtained via filtering the common crawl corpus. It encompasses 151 languages. The Romanian sub-corpus contains over 4,5 million documents …


Natural Language Processing

CH Focke, RM Sylvester - Statistical Methods in Epilepsy, 2024

In this chapter, we introduce modern natural language processing (NLP) libraries, techniques and their applications. This chapter focuses on deep learning methods and less on computational linguistics that require nuanced knowledge of linguistics …


Navigating Challenges and Technical Debt in Large Language Models Deployment

A Menshawy, Z Nawaz, M Fahmy - Proceedings of the 4th Workshop on Machine …, 2024

… These models, trained on an internet scale data including common crawl, books, and articles, have acquired an extensive understanding of language patterns, nuances, and contexts. This profound comprehension allows LLMs to perform a …


Navigating the ethical landscape behind ChatGPT

L Peng, B Zhao - Big Data & Society, 2024

… While ChatGPT and similar text-generating AIs claim to be trained on mostly public data such as the Common Crawl archives, rendering … content in public datasets used for AI training, such as the Common Crawl, which is beyond the …


Navigating the Legal Landscape: Technical Implementation of Copyright Reservations for Text and Data Mining in the Era of AI Language Models

L Löbling, C Handschigl, K Hofmann, J Schwedhelm - JIPITEC–Journal of Intellectual …, 2023

… However, limitations arose from the potential omission of specific pages of a website in individual monthly crawls by Common Crawl. Through manual investigation, a section containing the terms of use could be identified for 40 …


Near to Mid-term Risks and Opportunities of Open Source Generative AI

F Eiras, A Petrov, B Vidgen, CS de Witt, F Pizzati… - arXiv preprint arXiv …, 2024

In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential …


Need a Programming Exercise Generated in Your Native Language? ChatGPT's Got Your Back: Automatic Generation of Non-English Programming Exercises Using …

M Jordan, K Ly, AG Soosai Raj - Proceedings of the 55th ACM Technical Symposium …, 2024

Large language models (LLMs) like ChatGPT are changing computing education and may create additional barriers to those already faced by non-native English speakers (NNES) learning computing. We investigate an opportunity for a positive …


Nemotron-4 15B Technical Report

J Parmar, S Prabhumoye, J Jennings, M Patwary… - arXiv preprint arXiv …, 2024

We introduce Nemotron-4 15B, a 15-billion-parameter large multilingual language model trained on 8 trillion text tokens. Nemotron-4 15B demonstrates strong performance when assessed on English, multilingual, and coding tasks: it …


Neural Approaches to Decode Semantic Similarities in Spanish Song Lyrics for Enhanced Recommendation Systems

… In this model, it was found that traditional methods of pre-training, which rely on large amounts of data from sources such as Common Crawl… The mC4 dataset is a multilingual variant of C4, containing natural text in 101 languages from the public …


NEURAL MODELS FOR ONTOLOGY ANNOTATIONS-NEMO

Words cannot express my gratitude to my advisors Dr. Somya D. Mohanty and Dr. Prashanti Manda for their invaluable supervision and guidance. I am extremely grateful to them for their continuous support, constructive suggestions, and …


Neural Network-based Language Modelling for Code-Switching in South African Languages

Language models, specifically those based on neural network architectures, have become an attractive alternative to classical statistical language models for natural language processing and speech recognition. Naturally, therefore, the field of neural …


Neural Networks with Model Compression

B Zhang, T Wang, S Xu, D Doermann - 2024

… Firstly, the availability of large-scale datasets, such as ImageNet for computer vision or the Common Crawl dataset for natural language processing, has enabled the training of deep neural networks with millions or even billions of parameters …


No Filter: Cultural and Socioeconomic Diversityin Contrastive Vision-Language Models

A Pouget, L Beyer, E Bugliarello, X Wang, AP Steiner… - arXiv preprint arXiv …, 2024

We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training …


Non-governmental Governance of Trust on the Internet: WebPKI as Public Good

… The population of sites we sampled from was indexed by the non-profit foundation Common Crawl. We then employed a script to pull certificate information from our list of URLs. Slightly more than half the sample did not return certificate …


Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models

This paper identifies a cultural dominance issue within large language models (LLMs) due to the predominant use of English data in model training (eg, ChatGPT). LLMs often provide inappropriate English-culture-related answers that are not relevant to …


Notes towards infrastructure governance for large language models

This paper draws on information infrastructures (IIs) in science and technology studies (STS), as well as on feminist STS scholarship and contemporary critical accounts of digital technologies, to build an initial mapping of the infrastructural …


NSINA: A News Corpus for Sinhala

H Hettiarachchi, D Premasiri, L Uyangodage… - arXiv preprint arXiv …, 2024

The introduction of large language models (LLMs) has advanced natural language processing (NLP), but their effectiveness is largely dependent on pre-training resources. This is especially evident in low-resource languages, such as Sinhala …


Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis

A Shah, A Hiray, P Shah, A Banerjee, A Singh… - arXiv preprint arXiv …, 2024

In this paper, we investigate the influence of claims in analyst reports and earnings calls on financial market returns, considering them as significant quarterly events for publicly traded companies. To facilitate a comprehensive analysis, we construct a …


NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural

W Wongso, DS Setiawan, S Limcorn, A Joyoadikusumo - arXiv preprint arXiv …, 2024

… We also note that NusaBERT was pre-trained on Wikipedia and common crawl corpora, which explains its effectiveness on and closeness to NusaX and NusaTranslation source domains, but not so for NusaParagraph. Thus, due to their high linguistic and …


Nyonic Technical Report

J Tian, R Wang, C Li, Y Zhou, J Liu, J Wang - arXiv preprint arXiv:2404.15702, 2024

… Our curated dataset is specifically designed to meet these criteria, incorporating sources such as the Common Crawl, books, Wikipedia, code, and … Our data sources encompass Common Crawl, Wiki, and various coding repositories. The final …


Ocean Decade Vision 2030 White Papers–Challenge 8: Create a digital representation of the ocean.

JB Calewaert, PC Sierra-Correa, O McMeel… - 2024

… The Common crawl and WebImageText are examples, with communities such as Hugging Face and KAGGLE creating new communities of practice16F … 17 https://commoncrawl.org, https://github.com/google-research-datasets/wit, https://huggingface.co, https://www.kaggle.com …


OLMo: Accelerating the Science of Language Models

D Groeneveld, I Beltagy, P Walsh, A Bhagia, R Kinney… - arXiv preprint arXiv …, 2024

… well on evaluations predominated by Common Crawl, such as C4, though different ways of postprocessing Common Crawl are best fit by … The RedPajama evaluation shows a similar pattern, perhaps as only 2 of its 7 domains are from …


Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models

S Ruan, Y Dong, H Liu, Y Huang, H Su, X Wei - arXiv preprint arXiv:2404.12139, 2024

Vision-Language Pre-training (VLP) models like CLIP have achieved remarkable success in computer vision and particularly demonstrated superior robustness to distribution shifts of 2D images. However, their robustness under 3D viewpoint …


On Labs and Fabs: Mapping How Alliances, Acquisitions, and Antitrust are Shaping the Frontier AI Industry

T Aguirre - arXiv preprint arXiv:2406.01722, 2024

As frontier AI models progress, policy proposals for safe AI development are gaining increasing attention from researchers and policymakers. This paper evaluates the present landscape of integration within the AI supply chain, emphasizing vertical …


On Protecting the Data Privacy of Large Language Models (LLMs): A Survey

B Yan, K Li, M Xu, Y Dong, Y Zhang, Z Ren, X Cheng - arXiv preprint arXiv …, 2024

… For example, GPT-3, developed by OpenAI, was pre-trained using CommonCrawl, constituting 45TB of compressed plaintext before filtering [37]. Regarding multimodal LLMs, CLIP’s training dataset encompasses 400 million pairs of images and text …


On the data scarcity problem of neural-based named entity recognition

… The initial release trained on Wikipedia includes word representations for nine languages, namely Arabic, Czech, German, English, Spanish, French, Italian, Romanian and Russian, and has quickly expanded to 157 languages by training on larger …


On the evaluation and application of neural language models for grammatical error detection

Neural language models (NLM) have become a core component in many downstream applications within the field of natural language processing, including the task of data-driven automatic grammatical error detection (GED). This thesis …


On the importance of Data Scale in Pretraining Arabic Language Models

A Ghaddar, P Langlais, M Rezagholizadeh, B Chen - arXiv preprint arXiv:2401.07760, 2024

Pretraining monolingual language models have been proven to be vital for performance in Arabic Natural Language Processing (NLP) tasks. In this paper, we conduct a comprehensive study on the role of data in Arabic Pretrained Language …


On the Question of Authorship in Large Language Models

C Soos, L Haroutunian - KO KNOWLEDGE ORGANIZATION, 2024

… CommonCrawl exemplifies the problem of documentation debt. The dataset is an effort by The Common Crawl Foundation to “[democratize] access to web information by producing and maintaining an open repository of web crawl data” (Common …


On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe

N Xu, Q Zhang, M Zhang, P Qian, X Huang - arXiv preprint arXiv:2402.14404, 2024

Probing and enhancing large language models' reasoning capacity remains a crucial open question. Here we re-purpose the reverse dictionary task as a case study to probe LLMs' capacity for conceptual inference. We use in-context learning …


One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support

V Rasiah, R Stern, V Matoshi, M Stürmer, I Chalkidis…

Recent strides in Large Language Models (LLMs) have saturated many NLP benchmarks (even professional domain-specific ones), emphasizing the need for more challenging ones to properly assess LLM capabilities. Domain-specific and …


Online health search via multi-dimensional information quality assessment based on deep language models

B Zhang, N Naderi, R Mishra, D Teodoro - 2024

Background Widespread misinformation in Web resources can lead to serious implications for individuals seeking health advice. Despite that, information retrieval models are often focused only on the query-document relevance dimension to rank …


Online Health Search Via Multidimensional Information Quality Assessment Based on Deep Language Models: Algorithm Development and Validation

B Zhang, N Naderi, R Mishra, D Teodoro - JMIR AI, 2024

… In this study, we simulated online health information search scenarios with a topic set of 32 different health-related inquiries and a corpus containing 1 billion web documents from the April 2019 snapshot of Common Crawl. Using state-of-the-art …


Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis

N Li, T Bailleux, Z Bouraoui, S Schockaert - arXiv preprint arXiv:2403.17216, 2024

We consider the problem of finding plausible knowledge that is missing from a given ontology, as a generalisation of the well-studied taxonomy expansion task. One line of work treats this task as a Natural Language Inference (NLI) problem, thus relying …


Open Assistant Toolkit--version 2

S Fischer, F Rossetto, C Gemmell, A Ramsay, I Mackie… - arXiv preprint arXiv …, 2024

… To source knowledge and tasks, we release a new offline pipeline to parse and augment task data from CommonCrawl. The offline pipeline supports using LLMs and additional open-source multimodal data sources to enhance tasks and make …


Open Science at the Generative AI Turn: An Exploratory Analysis of Challenges and Opportunities

Technology influences Open Science (OS) practices, because conducting science in transparent, accessible, and participatory ways requires tools/platforms for collaborative research and sharing results. Due to this direct relationship …


OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

F Xue, Z Zheng, Y Fu, J Ni, Z Zheng, W Zhou, Y You - arXiv preprint arXiv:2402.01739, 2024

To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from …


OptiComm-GPT: a GPT-based versatile research assistant for optical fiber communication systems

X Jiang, M Zhang, Y Song, Y Zhang, Y Wang, C Ju… - Optics Express, 2024

… Generally, LLMs are pre-trained on publicly available datasets such as BooksCorpus, Common Crawl, and Wikipedia, thereby ensuring comprehensive coverage of knowledge. Meanwhile, with the support of powerful analysis, reasoning …


Optimization and Evaluation in Machine Learning Challenges

To develop new machine learning methods, it is necessary to evaluate them reliably. This doctoral thesis discusses some aspects of preparing machine learning challenges and techniques for developing their solutions. The work consists of …


Optimization of news dissemination push mode by intelligent edge computing technology for deep learning

JL DeGe, S Sang - Scientific Reports, 2024

… Realnew (https://paperswithcode.com/dataset/realnews) is a large corpus of news articles from CommonCrawl. The data is crawled from CommonCrawl, which is limited to 5000 news fields of Google News Index. The author uses the …


Optimization of Unsupervised Neural Machine Translation Based on Syntactic Knowledge Improvement

… Common Crawl is a multilingual aligned dataset based on web crawling. Para Crawl is a multilingual parallel corpus for large-scale web crawling. As shown in Table III, the detection accuracies of LSTM in Newstest2020, Newstest2021 …


Optimizing Data I/O for LLM Datasets on Remote Storage

Training large language models (LLMs) demands increasingly larger datasets for optimal performance [13]. In practice, these datasets may include hundreds of terabytes (TB) or even petabytes of data. For example, Common Crawl [3] needs …


Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean

CS Choi, Y Jeong, S Park, IH Won, HS Lim, SM Kim… - arXiv preprint arXiv …, 2024

… Llama2 is a multilingual language model trained using largescale publicly available data (including CommonCrawl, Github, Wikipedia, and ArXiv) for more than 28 languages. Thus, it possesses cross-language understanding capabilities …


ORCID

How are Asian and Black men and women stereotyped? Research from the gendered race and stereotype content perspectives has produced mixed empirical findings. Using BERT models pre-trained on English language books, news articles …


Organic Data-Driven Approach for Turkish Grammatical Error Correction and LLMs

A Ersoy, OT Yıldız - arXiv preprint arXiv:2405.15320, 2024

… 2020) which contains random paragraphs sampled from the CommonCrawl dataset3. Some studies put some effort into filling the gap and … 2020), a multilingual pre-trained encoder-decoder text-to-text transformer trained on a Common Crawl-based …


Outlier Detection in Serbian CommonCrawl Data

V Kalušev, D Ćulibrk - 2024 23rd International Symposium INFOTEH …, 2024

The surge in large language models (LLMs) has greatly advanced natural language processing. However, their development is often hampered by the limited availability and quality of training datasets, particularly for underrepresented languages. Our …


Outsmarting Artificial Intelligence in the Classroom—Incorporating Large Language Model-Based Chatbots into Teaching

J Wutzler - Issues in Accounting Education, 2024

Since the release of ChatGPT in November 2022, large language model-based chatbots have attracted much attention. Although businesses value their potential for efficiency gains, academics are concerned about their effects on learning and …


Overview of Existing LLM Families

… data that are considered as redundant—namely the Common Crawl and Colossal Cleaned Common Crawl (C4). Researchers who created the … to Common Crawl when it came to training LLMs [30] and should be used instead of the whole …


Overview of the TREC 2023 NeuCLIR Track (Notebook Paper)

D Lawrie, S MacAvaney, J Mayfield, P McNamee…

… For more information about extracting the text for the documents from CommonCrawl News and how the collection can be obtained, see … of the track including HC4 — a CLIR collection built over three years of Common Crawl data in …


Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization

C Mavromatis, P Karypis, G Karypis - arXiv preprint arXiv:2404.11531, 2024

… 2019) is sa publicly available distribution of a Common Crawl snapshot. Following cBTM, we use 168B tokens of the no blocklist version (en.noblocklist) that is out of distribution to the OPT model’s pretraining corpus. S2ORC (Lo et al.…


PAL: Proxy-Guided Black-Box Attack on Large Language Models

C Sitawarin, N Mu, D Wagner, A Araujo - arXiv preprint arXiv:2402.09674, 2024

Large Language Models (LLMs) have surged in popularity in recent months, but they have demonstrated concerning capabilities to generate harmful content when manipulated. While techniques like safety fine-tuning aim to minimize harmful use …


Parameter-Efficient Tuning of Large Convolutional Models

W Chen, Z Miao, Q Qiu - arXiv preprint arXiv:2403.00269, 2024

To address the high computational and parameter complexity associated with fine-tuning large pre-trained models, researchers have developed parameter-efficient methods, where only partial parameters are updated for downstream tasks. However, these …


Paraphrase Generation and Supervised Learning for Improved Automatic Short Answer Grading

L Ouahrani, D Bennouar - International Journal of Artificial Intelligence in …, 2024

We consider the reference-based approach for Automatic Short Answer Grading (ASAG) that involves scoring a textual constructed student answer comparing to a teacher-provided reference answer. The reference answer does not cover the variety of student …


Parsability revisited and reassessed

S Monakhov - Journal of Linguistics, 2023

This paper provides evidence that the inveterate way of assessing linguistic items’ degrees of analysability by calculating their derivation to base frequency ratios may obfuscate the difference between two meaning processing models: one based on …


Part-of-Speech Tagger for Bodo Language using Deep Learning approach

D Pathak, S Narzary, S Nandi, B Som - arXiv preprint arXiv:2401.03175, 2024

Language Processing systems such as Part-of-speech tagging, Named entity recognition, Machine translation, Speech recognition, and Language modeling (LM) are well-studied in high-resource languages. Nevertheless, research on these …


Pathological Liars: Algorithmic Knowing in the Rhetorical Ecosystem of Wallstreetbets

MH Yang, ZP Majdik - Rhetoric Society Quarterly, 2024

This essay demonstrates the value of using artificial intelligence (AI) technologies to address specific kinds of research questions in rhetoric. The essay builds on a study of a novel rhetorical object first observed by Yang on the Reddit subreddit r/wallstreetbets …


Pathological-Llama: an Explainable Medical Visual Question An

This thesis introduces Pathological-Llama, an explainable medical visual question answering system that integrates computer vision and natural language processing to accurately interpret medical images through a generative task approach. By …


Pedro L. Rodriguez Arthur Spirling Brandon M. Stewart Elisa M. Wirsching

… from Common Crawl and Wikipedia. A strength of the fastText model is that it uses subword … On inspection, we saw that Common Crawl includes many typos and rare terms (plus many … Beyond this potential for noise, Common Crawl is not …


Performance Comparison of Large Language Models, GPT and Gemini on Turkish News Classification Task

… This was achieved by training a Transformer-based masked language model on 100 languages using more than two terabytes of filtered CommonCrawl data. The balance between high and low-resource languages and the impact of language …


Performance Evaluation of Word Embedding Algorithms

S Kulshretha, L Lodha - Performance Evaluation, 2023

This study intends to explore the field of word embedding and thoroughly examine and contrast various word embedding algorithms. Words retain their semantic relationships and meaning when they are transformed into vectors using word …


Performance of ChatGPT on American Board of Surgery In-Training Examination Preparation Questions

CG Tran, J Chang, SK Sherman, JP De Andrade - Journal of Surgical Research, 2024

… The training dataset of ChatGPT 3.5, where the artificial intelligence (AI) “knowledge” is derived, is comprised of Common Crawl (a monthly updated web archive of data), books, journals, and Wikipedia and totals over 500 billion encoded “tokens.” Many …


Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models

Z Ankner, C Blakeney, K Sreenivasan, M Marion… - arXiv preprint arXiv …, 2024

… Secondly, while previous work only investigates pruning on datasets composed of just one domain (CommonCrawl1), we consider two … dataset, with 81.31% of its data being derived from the CommonCrawl. We find that successful pruning …


Personal Science Guide to AI

Imagine you have access to a zillion documents, preferably curated in some way reassures you about their quality and consistency. Wikipedia, for example, or maybe Reddit and other posts that have been sufficiently up-voted. Maybe you also have a …


Personality trait detection via transfer learning

B Alshouha, J Serrano-Guerrero, F Chiclana… - Comput Mater Continua …, 2024

Personality recognition plays a pivotal role when developing user-centric solutions such as recommender systems or decision support systems across various domains, including education, e-commerce, or human resources. Traditional machine …


Perspectives of Global and Hong Kong's Media on China's Belt and Road Initiative

LC Khoo, A Datta - arXiv preprint arXiv:2312.17013, 2023

This study delves into the media analysis of China's ambitious Belt and Road Initiative (BRI), which, in a polarized world, and furthermore, owing to the very polarizing nature of the initiative itself, has received both strong criticisms and …


PHISHING DETECTION SYSTEM THROUGH MACHINE LEARNING BASED ON URL

PU Chandra, TV Teja, VN Reddy, S Agarwalla

… Legitimate site data is often gathered from Alexa's top-ranking website database or through common-crawl. Additionally, publicly available datasets like those from the UCI machine learning repository, containing over 11,000 records with 31 …


Phishing Detection Using Hybrid Algorithm Based on Clustering and Machine Learning

L Al-Shalabi, Y Hasan Jazyah - International Journal of Computing and Digital …, 2024

Phishing is a prevalent and evolving cyber threat that continues to exploit human vulnerability to deceive individuals and organizations into revealing sensitive information. Phishing attacks encompass a range of tactics, from deceptive emails …


Physics-informed Machine Learning with Uncertainty Quantification

Physics Informed Machine Learning (PIML) has emerged as the forefront of research in scientific machine learning with the key motivation of systematically coupling machine learning (ML) methods with prior domain knowledge often available in the …


PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods

S Dadas, M Perełkiewicz, R Poświata - arXiv preprint arXiv:2402.13350, 2024

We present Polish Information Retrieval Benchmark (PIRB), a comprehensive evaluation framework encompassing 41 text information retrieval tasks for Polish. The benchmark incorporates existing datasets as well as 10 new, previously …


Pitfalls of Conversational LLMs on News Debiasing

IB Schlicht, D Altiok, M Taouk, L Flek - arXiv preprint arXiv:2404.06488, 2024

This paper addresses debiasing in news editing and evaluates the effectiveness of conversational Large Language Models in this task. We designed an evaluation checklist tailored to news editors' perspectives, obtained generated texts from three …


PLAID SHIRTTT for Large-Scale Streaming Dense Retrieval

D Lawrie, E Kayi, E Yang, J Mayfield, DW Oard - arXiv preprint arXiv:2405.00975, 2024

… With ten million documents from the news subset of CommonCrawl, NeuCLIR is presently the largest available test collection for MLIR; It is tiny though relative to ClueWeb, and thus has relatively small shards. The NeuCLIR MLIR task is to …


Please be polite to your peers: a multi-task model for assessing the tone and objectivity of critiques of peer review comments

PK Bharti, M Agarwal, A Ekbal - Scientometrics, 2024

The peer-review process plays a pivotal role in maintaining the quality and credibility of scientific publications. However, in recent times, there has been an increase in unhelpful and overly critical reviews, which can be detrimental to the …


Politehnica University of Bucharest, Splaiul Independentei Nr. 313, Cector 6, Bucharest, Romania radu. stefan@ live. de, radu. stefan3@ student. upt. ro

R Stefan, G Carutasu, M Mocan - The 17th International Conference Interdisciplinarity in …

This paper aims to investigate the ethical considerations surrounding the implementation and usage of large language models (LLMs). LLMs have shown tremendous advance in natural language processing and have the potential to …


Position: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

S Longpre, R Mahari, N Obeng-Marnu, W Brannon…

… Common Crawl. For pretraining text data, models routinely rely on the Common Crawl.For instance, sixty percent of GPT-3 training data is … Common Crawl is accurate with wide coverage, but not detailed. Hugging Face can be inaccurate with …


Position: Mission Critical–Satellite Data is a Distinct Modality in Machine Learning

E Rolf, K Klemmer, C Robinson, H Kerner - Forty-first International Conference on Machine …

Satellite data has the potential to inspire a seismic shift for machine learning---one in which we rethink existing practices designed for traditional data modalities. As machine learning for satellite data (SatML) gains traction for its real-world impact …


Position: Near to Mid-term Risks and Opportunities of Open-Source Generative AI

F Eiras, A Petrov, B Vidgen, CS de Witt, F Pizzati… - Forty-first International Conference …

In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential …


Position: Will we run out of data? Limits of LLM scaling based on human-generated data

P Villalobos, A Ho, J Sevilla, T Besiroglu, L Heim… - Forty-first International Conference …

… Assuming that Common Crawl is a representative sample of the indexed web,11 we can use it to estimate the average amount of plain text bytes per web page. This number has increased over time, from around 6100 bytes in 2013 to about 8200 …


Post-processing Techniques for Word Embedding

Word embedding has been a significant breakthrough in natural language processing (NLP). Although word representation has improved remarkably and resulted in better performance in downstream NLP applications, interpretability of …


Posthumanist AI

… In their now canonical paper on “Stochastic Parrots”, Bender, Gebru, McMillan-Major, and Mitchell note that "it is easy to imagine that very large datasets, such as Common Crawl “…petabytes of data collected over 8 years of web crawling” a …


Pre-Trained Acoustic-and-Textual Modeling for End-To-End Speech-To-Text Translation

W Zhang, H Zhang, C Liu, Z Ye, X Zhou, C Lin, L Dai - ICASSP 2024-2024 IEEE …, 2024

… The monolingual data are mainly from News Commentary, News crawl and Common Crawl. We apply language identification [23] to retain sentences predicted as desired language, remove sentences longer than 250 tokens and with a source/target …


Pre-trained language models in medicine: A survey

X Luo, Z Deng, B Yang, MY Luo - Artificial Intelligence in Medicine, 2024

With the rapid progress in Natural Language Processing (NLP), Pre-trained Language Models (PLM) such as BERT, BioBERT, and ChatGPT have shown great potential in various medical NLP tasks. This paper surveys the cutting-edge …


Pre-trained Large Language Models for Financial Sentiment Analysis

W Luo, D Gong - arXiv preprint arXiv:2401.05215, 2024

Financial sentiment analysis refers to classifying financial text contents into sentiment categories (eg positive, negative, and neutral). In this paper, we focus on the classification of financial news title, which is a challenging task due to a lack of …


Pre-training Small Base LMs with Fewer Tokens

S Sanyal, S Sanghavi, AG Dimakis - arXiv preprint arXiv:2404.08634, 2024

We study the effectiveness of a simple approach to develop a small base language model (LM) starting from an existing large base LM: first inherit a few transformer blocks from the larger LM, and then train this smaller model on a very small subset (0.1\%) …


Predicting fraud in MD&A sections using deep learning

S Velloor Sivasubramanian, D Skillicorn - Journal of Business Analytics, 2024

Conventional data analytic techniques have been successfully applied to detecting fraud in the Management’s Discussion and Analysis sections of company filings mandated by the SEC. Here, we investigate whether fraud detection can be …


Predicting semantic category of answers for question answering systems using transformers: a transfer learning approach

S CM, J Prakash, VS Alaparthi - Multimedia Tools and Applications, 2024

… The model uses a data set of web extracted text that has been created with the help of the Common Crawl and preprocessed to remove short, duplicate, and non-English content, resulting in a corpus called "Colossal Clean Crawled Corpus" (C4). An …


Prediction of Author's Profile basing on Fine-Tuning BERT model

B Bsir, N Khoufi, M Zrigui - Informatica, 2024

… Indeed, XLMRoBERTa is the multilingual variant of RoBERTa trained with a multilingual MLM on one hundred languages, with more than two terabytes of filtered Common Crawl data.XLM-RoBERTa showed its superiority over BERT by its …


Preemptive Answer" Attacks" on Chain-of-Thought Reasoning

R Xu, Z Qi, W Xu - arXiv preprint arXiv:2405.20902, 2024

Large language models (LLMs) showcase impressive reasoning capabilities when coupled with Chain-of-Thought (CoT) prompting. However, the robustness of this approach warrants further investigation. In this paper, we introduce a novel scenario …


Pretraining and Updating Language-and Domain-specific Large Language Model: A Case Study in Japanese Business Domain

K Takahashi, T Omi, K Arima, T Ishigaki - arXiv preprint arXiv:2404.08262, 2024

… , mC4, and Common Crawl. These datasets contribute to general knowledge, which is essential for diverse natural language processing tasks. … the proportion of clean domain-specific data, we aim to mitigate the impact of noisy data sources such …


Probability-Aware Word-Confusion-Network-To-Text Alignment Approach for Intent Classification

E Villatoro-Tello, S Madikeri, B Sharma, D Khalil… - ICASSP 2024-2024 IEEE …, 2024

… The Language Model (LM) was trained with 34M utterances from publicly available English datasets including People’s Speech, Fisher, Switchboard, AMI, Wikitext103, and subsets of Common Crawl and Reddit datasets. The model was …


Probing and enhancing the reliance of Transformer models on poetic information

Transformer models have achieved remarkable success in the widest variety of domains, spanning not just a multitude of tasks within natural language processing, but also those in computer vision, speech, and reinforcement learning. The key to …


Probing Critical Learning Dynamics of PLMs for Hate Speech Detection

S Masud, MA Khan, V Goyal, MS Akhtar, T Chakraborty - arXiv preprint arXiv …, 2024

… Each variant is trained on a training corpus from Wikipedia, and Common-Crawl is curated and updated before the date associated with … that are pretrained on regular and latest Internet snapshots obtained via Common Crawl and Wikipedia …


Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics

F Lin, D Altshuler, JB Pierrehumbert - arXiv preprint arXiv:2404.03301, 2024

… 2018) trained on 600B Common Crawl data as a baseline. For the baseline, we add the representations of strongest and weakest adjectives on s as ⃗s, then also rank cos(⃗a, ⃗s) for each a to calculate MRR. Here, we only report the results for the …


PRODIS-a speech database and a phoneme-based language model for the study of predictability effects in Polish

Z Malisz, J Foremski, M Kul - arXiv preprint arXiv:2404.10112, 2024

We present a speech database and a phoneme-level language model of Polish. The database and model are designed for the analysis of prosodic and discourse factors and their impact on acoustic parameters in interaction with predictability effects. The …


Prometheus' Digital Fire: The Civic Responsibilities of Artificial Intelligence

[Prometheus], the noble son of Iapetus outwitted [Zeus] and stole the far-seen gleam of unwearying fire in a hollow fennel stalk. And Zeus who thunders on high was stung in spirit, and his dear heart was angered when he saw amongst men the far-seen …


Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars

Z Wu, X Lin, Z Dai, W Hu, Y Shu, SK Ng, P Jaillet… - arXiv preprint arXiv …, 2024

… [2], it is highly likely that GPT-3 has been trained on common benchmark datasets potentially contained in Common Crawl due to data contamination. Hence, further providing high-quality in-context exemplars could hardly improve the performance of …


Prompting Change: ChatGPT's Impact on Digital Humanities Pedagogy–A Case Study in Art History

Q Guo - International Journal of Humanities and Arts …, 2024

This article explores the transformative impact of ChatGPT, a Generative Pre-trained Transformers model, on humanities methodologies and its integration into a digital humanities (DH) course. The advent of ChatGPT enhances the capacity of people …


PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models

H Tan, Z Guo, Z Shi, L Xu, Z Liu, X Li, Y Wang, L Shang… - arXiv preprint arXiv …, 2024

… Nevertheless, scaling the context window during the pretraining phase remains a daunting task as the computational demands surge quadratically with the length of the attention span, and a majority of texts within standard corpora, like Common …


PSILENCE: A Pseudonymization Tool for International Law

LA Cabrera-Diego, A Gheewala - Proceedings of the Workshop on Computational …, 2024

Since the announcement of the GDPR, the pseudonymization of legal documents has become a high-priority task in many legal organizations. This means that for making public a document, it is necessary to redact the identity of certain entities …


PSQE: Personalized Semantic Query Expansion for user-centric query disambiguation

… Furthermore, they enable explainable intelligent systems, in that the results of querying the embedding for similar terms relate directly to the training set, rather than webscale collections such as Common Crawl. To our knowledge, our work is …


PSQS: Parallel Semantic Querying Service for Self-describing File Formats

C Niu, W Zhang, S Byna, Y Chen - 2023 IEEE International Conference on Big Data …, 2023

… Typically, theseword representations are learned from a training dataset suchas the Common Crawl web corpus (840 billion tokens, 2.2million words) or Wikipedia 2014 (6 billion tokens, 400,000words). However, this assumption does not always …


Public perception of the Chinese president's visit to Saudi Arabia and the China–Arab Summit: sentiment analysis of Arabic tweets

AAM Hassan - Social Network Analysis and Mining, 2024

In recent years, China and Saudi Arabia have had frequent exchanges in the political and economic fields, and public opinion evaluation is an essential aspect of evaluating the two countries’ current relations. This paper analyzes tweets in Arabic …


PUBLIC-PRIVATE PARTNERSHIPS IN HEALTH SECTOR INNOVATION: LESSONS FROM AROUND THE WORLD

CC Ebulue, OR Ebulue, CS Ekesiobi - International Medical Science Research …, 2024

Public-Private Partnerships (PPPs) have emerged as a crucial mechanism for fostering innovation in the health sector globally. This review encapsulates the lessons learned from diverse PPP models worldwide, highlighting their significance …


PyLoomer-The Automated Weaving of Python Text

A Sahani, RB Trivedi, T Singh, SR Goyal - 2024 2nd International Conference on …, 2024

The introduction of attention mechanisms has evolved NLP methodologies, giving rise to Large Language Models (LLMs) that excel in various sequence-to-sequence tasks like translation, question-answering, summarization, intent, and entity …


Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora

S Ranathunga, N de Silva, M Velayuthan, A Fernando… - arXiv preprint arXiv …, 2024

… 2020) is a dataset created using 68 snapshots of CommonCrawl10. Document … of CommonCrawl. It contains around 4.5 billion parallel sentences across 576 language pairs. In building CCMatrix, it was assumed the aligned sentence could …


Quantifying Gender Bias in Arabic Pre-trained Language Models

W Alrajhi, H Al-Khalifa, AM Al-Salman - IEEE Access, 2024

The current renaissance in the development of Arabic Pre-trained Language models (APLMs) has yielded significant advancement across many fields. Nevertheless, no study has explored the dimensions of gender bias in these models. It is argued that …


Quantifying Geospatial in the Common Crawl Corpus

I Ilyankou, M Wang, J Haworth, S Cavazzi - arXiv e-prints, 2024

… Common Crawl corpus. However, the geospatial content within CC remains largely unexplored, impacting our understanding of LLMs' spatial reasoning. This paper investigates the prevalence of geospatial data in recent Common Crawl …


Quantifying task-related gaze

K Walter, M Freeman, P Bex - Attention, Perception, & Psychophysics, 2024

… For this study, we used the pretrained word vector Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors), which can be obtained at https://nlp.stanford.edu/projects/glove/. To utilize GloVe with the LabelMe images, we used it in conjunction with LASS …


Quantifying the Importance of Data Alignment in Downstream Model Performance

K Chawla, A Sahai, M DePavia, S Sundar, B Miranda - 2024

Contrary to the conventional emphasis on dataset size, we explore the role of data alignment---an often overlooked aspect of data quality---in training capable Large Language Models (LLMs). To do so, we use the Task2Vec-based alignment …


Query Obfuscation for Information Retrieval through Differential Privacy

Protecting the privacy of a user querying an Information Retrieval (IR) system is of utmost importance. The problem is exacerbated when the IR system is not cooperative in satisfying the user’s privacy requirements. To address this …


Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora

Z Fei, Y Shao, L Li, Z Zeng, H Yan, X Qiu, D Lin - arXiv preprint arXiv:2401.14624, 2024

… become increasingly rich, including the Pile, RedPajama, and Common Crawl. To ensure the diversity of data sources for retrieval, we constructed our retrieval corpus based on Common CrawlCommon Crawl is an open-source web crawler project …


Query Performance Prediction using Relevance Judgments Generated by Large Language Models

C Meng, N Arabzadeh, A Askari, M Aliannejadi… - arXiv preprint arXiv …, 2024

Query performance prediction (QPP) aims to estimate the retrieval quality of a search system for a query without human relevance judgments. Previous QPP methods typically return a single scalar value and do not require the predicted …


QuRating: Selecting High-Quality Data for Training Language Models

A Wettig, A Gupta, S Malik, D Chen - arXiv preprint arXiv:2402.09739, 2024

… We visualize the distributions for the cluster in CommonCrawl and C4 in Figure 7 in the appendix. They are similarly encouraging, eg, the clusters associated with cells, protein, gene and energy, climate, species are rated highly on required …


RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors

L Dugan, A Hwang, F Trhlik, JM Ludan, A Zhu, H Xu… - arXiv preprint arXiv …, 2024

… The model was allegedly trained with a variety of data including the Common Crawl (fil- … Released on July 18th 2023, LLaMA 2 is the successor to the original LLaMA model which was trained on webpages from CommonCrawl, multilingual …


Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese

Y Sun, Z Wan, N Ueda, S Yahata, F Cheng, C Chu… - arXiv preprint arXiv …, 2024

The creation of instruction data and evaluation benchmarks for serving Large language models often involves enormous human annotation. This issue becomes particularly pronounced when rapidly developing such resources for a non-English …


Real-time phishing detection using deep learning methods by extensions

DM Linh, HD Hung, HM Chau, QS Vu, TN Tran - International Journal of Electrical …, 2024

… [2] used two algorithms, random forest (RF) and long short-term memory (LSTM), to evaluate the PhishTank and Common Crawl datasets, achieving accuracies of 93.5% and 98.7%, respectively. Similarly, Kumar et al. [3] evaluated using the …


Real-World Robot Applications of Foundation Models: A Review

K Kawaharazuka, T Matsushima, A Gambardella… - arXiv preprint arXiv …, 2024

… In terms of the data, the datasets for training LLMs are Internet-scale; for example, the pre-training of GPT-3 [47] uses CommonCrawl covering Internet data from years 2016 to 2019, which amounts to 45TB of compressed plain text and 410 billion …


Reasoning for fact verification using language models

In response to the proliferation of misinformation on social media platforms, this thesis introduces the Triple-R framework (Retriever, Ranker, Reasoner) to enhance fact-checking by leveraging the Web for evidence retrieval and generating …


Recent Advances in Large Language Models for Healthcare

K Nassiri, MA Akhloufi - BioMedInformatics, 2024

Recent advances in the field of large language models (LLMs) underline their high potential for applications in a variety of sectors. Their use in healthcare, in particular, holds out promising prospects for improving medical practices. As we highlight in …


Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark

H Cao - arXiv preprint arXiv:2406.01607, 2024

… data: (post, comment) pairs from Reddit3, (question, upvoted answer) pairs from Stackexchange4, (entity name + section title, passage) pairs from English Wikipedia, (title, abstract) and citation pairs from Scientific papers [53], and (title, passage) pairs …


Recognizing Textual Inference in Mongolian Bar Exam Questions

G Khaltarkhuu, B Batjargal, A Maeda - Applied Sciences, 2024

This paper examines how to apply deep learning techniques to Mongolian bar exam questions. Several approaches that utilize eight different fine-tuned transformer models were demonstrated for recognizing textual inference in Mongolian bar exam …


Referring expression segmentation: from conventional to generalized

In recent years, many remarkable achievements have been made in the field of deep machine learning in various data modalities, such as image processing and natural language comprehension. Based on the good performance of deep neural …


Reliability Estimation of News Media Sources: Birds of a Feather Flock Together

S Burdisso, D Sánchez-Cortés, E Villatoro-Tello…

Evaluating the reliability of news sources is a routine task for journalists and organizations committed to acquiring and disseminating accurate information. Recent research has shown that predicting sources’ reliability represents an …


Reliability of large language models in managing odontogenic sinusitis clinical scenarios: a preliminary multidisciplinary evaluation

AM Saibene, F Allevi, C Calvo-Henriquez, A Maniaci… - European Archives of Oto …, 2024

Purpose This study aimed to evaluate the utility of large language model (LLM) artificial intelligence tools, Chat Generative Pre-Trained Transformer (ChatGPT) versions 3.5 and 4, in managing complex otolaryngological clinical scenarios …


ReLU $^ 2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs

Z Zhang, Y Song, G Yu, X Han, Y Lin, C Xiao, C Song… - arXiv preprint arXiv …, 2024

… For the corpus used for calculating the sparsity statistics, we a mixture of texts from Wikipedia, Pile, and Common Crawl. The total number of tokens is about 217. Since we need to store the activation values of all neurons, we only use a part of the …


Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

P Maini, S Seto, H Bai, D Grangier, Y Zhang, N Jaitly - arXiv preprint arXiv …, 2024

… There are different versions of CommonCrawl data and our selection of C4 for pretraining is driven by driven by its size and quality. … The dataset is also derived from the CommonCrawl, however has a more stringent filtering process. Our …


Representation Learning for Hierarchical Classification of Entity Titles

E Chistova - Proceedings of the 6th International Conference on …, 2023

We present a method for effective title encoding for hierarchical classification in a large taxonomy. The method enables taxonomy-aware encoding in pre-trained text encoders, such as fastText and BERT, which are additionally fine-tuned for the …


Representing Online Handwriting for Recognition in Large Vision-Language Models

A Fadeeva, P Schlattner, A Maksai, M Collier… - arXiv preprint arXiv …, 2024

The adoption of tablets with touchscreens and styluses is increasing, and a key feature is converting handwriting to text, enabling search, indexing, and AI assistance. Meanwhile, vision-language models (VLMs) are now the go-to solution …


Representing relational knowledge with language models

Relational knowledge is the ability to recognize the relationship between instances, and it has an important role in human understanding a concept or commonsense reasoning. We, humans, structure our knowledge by understanding individual …


Research and Application of Large Model-Based Intelligent Customer Service System

Y Xi - International Journal of Emerging Technologies and …, 2024

With the rapid development of artificial intelligence technology, intelligent customer service systems have been widely used. This paper addresses the limitations of traditional intelligent customer service systems, such as limited language …


Research on cross-lingual multi-label patent classification based on pre-trained model

Y Lu, L Chen, X Tong, Y Peng, H Zhu - Scientometrics, 2024

Patent classification is an important part of the patent examination and management process. Using efficient and accurate automatic patent classification can significantly improve patent retrieval performance. Current monolingual patent classification …


ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models

J Baek, SK Jauhar, S Cucerzan, SJ Hwang - arXiv preprint arXiv:2404.07738, 2024

Scientific Research, vital for improving human life, is hindered by its inherent complexity, slow pace, and the need for specialized experts. To enhance its productivity, we propose a ResearchAgent, a large language model-powered …


Retrieval-Based Grammatical Error Correction

… a cleaned version of the common crawl to produce nearly 200 million ungrammatical sentences with a realistic distribution of errors. … CWEB consists of randomly selected texts from the first 18 dumps of the Common Crawl, which include …


Revisión de modelos de lenguaje y aplicación a la automatización de tareas de oficina en empresas industriales

This project focuses on the evaluation and application of Large Language Models (LLMs) in natural language processing (NLP). The technical properties, advantages, and disadvantages of LLMs are analysed, highlighting their wide use in machine …


Revisiting Adversarial Training at Scale

Z Wang, X Li, H Zhu, C Xie - arXiv preprint arXiv:2401.04727, 2024

… DataComp-1B is a more recent dataset with about 1.3B samples filtered from a candidate pool of 12.8B image-text pairs from Common Crawl, which has been recorded to yield superior performance for contrastive training. To summarize, our …


Revisiting Joke Comprehension with Surprisal and Contextual Similarity: Implication from N400 and P600 Components

H Xu, M Nakanishi, S Coulson - Proceedings of the Annual Meeting of the Cognitive …, 2024

… Specifically, we used the version with a 2.2 million word vocabulary and 300-dimensional vectors trained on 840 billion tokens from the Common Crawl corpus. The contextual vector of all the words preceding the final word in each sentence was computed by …


Revisiting Pre-training of Embedding Layers in Transformer-based Neural Machine Translation

M Neishi, N Yoshinaga - Journal of Natural Language Processing, 2024

Recent trends in the pre-training and fine-tuning paradigm have made significant advances in several natural language processing tasks, including machine translation (MT), particularly for low-resource situations. However, it is reported that …


Rho-1: Not All Tokens Are What You Need

Z Lin, Z Gou, Y Gong, X Liu, Y Shen, R Xu, C Lin… - arXiv preprint arXiv …, 2024

Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that "Not all tokens in a corpus are equally important for language model training". Our initial …


RoBERTaNET: Enhanced RoBERTa Transformer Based Model for Cyberbullying Detection with GloVe Features

F Alrowais, AA Jamjoom, H Karamti, M Umer, S Alsubai… - IEEE Access, 2024

… 3) FastText FastText is a word illustration library, that encompasses 300dimensional 2 million common crawl words, resulting in 600 billion word vectors [31]. Particularly, hand-crafted ngrams are used as features alongside solo words. Its straightforward …


Robust Neural Machine Translation for Abugidas by Glyph Perturbation

H Kaing, C Ding, H Tanaka, M Utiyama - Proceedings of the 18th Conference of the …, 2024

Neural machine translation (NMT) systems are vulnerable when trained on limited data. This is a common scenario in low-resource tasks in the real world. To increase robustness, a solution is to intently add realistic noise in the training phase. Noise …


RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian

A Cosma, B Iordache, P Rosso - arXiv preprint arXiv:2402.13222, 2024

… Even if code is present in the dataset (from Common Crawl), it is described in English and comments and documentation are in English. The same argument can be made for mathematics and other scientific disciplines. A large, highly curated and …


ROME: Memorization Insights from Text, Probability and Hidden State in Large Language Models

B Li, Q Zhao, L Wen - arXiv preprint arXiv:2403.00510, 2024

Probing the memorization of large language models holds significant importance. Previous works have established metrics for quantifying memorization, explored various influencing factors, such as data duplication, model size, and prompt length …


RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict

Y Zeng, X Ding, Y Zhao, X Li, J Zhang, C Yao, T Liu… - arXiv preprint arXiv …, 2024

Fact-checking is the task of verifying the factuality of a given claim by examining the available evidence. High-quality evidence plays a vital role in enhancing fact-checking systems and facilitating the generation of explanations that are understandable to …