1-Diffractor: Efficient and Utility-Preserving Text Obfuscation Leveraging Word-Level Metric Differential Privacy

S Meisenbacher, M Chevli, F Matthes - arXiv preprint arXiv:2405.01678, 2024

The study of privacy-preserving Natural Language Processing (NLP) has gained rising attention in recent years. One promising avenue studies the integration of Differential Privacy in NLP, which has brought about innovative methods in a variety …


101 Billion Arabic Tokens Dataset

M Aloui, H Chouikhi, G Chaabane, H Kchaou…

… , we heavily depend on Common Crawl (1) as our primary source of web-scraped content. Common Crawl provides an accessible web archive … a Dataset Pipeline expressed in the following figure (1) aimed at cleaning up the web-extracted text …


101 Billion Arabic Words Dataset

M Aloui, H Chouikhi, G Chaabane, H Kchaou… - arXiv preprint arXiv …, 2024

… We undertook a large-scale data mining project, extracting a substantial volume of text from the Common Crawl WET files, specifically targeting Arabic content. The extracted data underwent a rigorous cleaning and deduplication process, using …


2 Text Preprocessing Method

This paper uses easy data augmentation (EDA) and back translation for text enhancement. After text enhancement, the number of samples in the dataset can be significantly increased, which is helpful for model parameters through training and …


4 CHATGPT IN ACADEMIA: AN IN-DEPTH EXPLORATION OF STUDENT VIEWS–PROS AND CONS

D Rajkumar, P Murugeswari, M Karthigaieswari - TRANSDISCIPLINARY THREADS

… GPT-3’s training data came from five existing datasets: • Common Crawl: A collection of text pulled from billions of web pages containing trillions of words. OpenAI filtered it for high-quality reference material only. • WebText2: OpenAI …


6 Performance Analysis

A Safi, S Singh, G Kaur, T Kaur - Intelligent Security Solutions for Cyber-Physical …, 2024

… The dataset used in this research was gathered from Alexa and the Common Crawl archive. … from a variety of sources, including Common Crawl, Alexa, and PhishTank for the phishing … They used three datasets, including real URLs …


" Previously on..." From Recaps to Story Summarization

AK Singh, D Srivastava, M Tapaswi - arXiv preprint arXiv:2405.11487, 2024

We introduce multimodal story summarization by leveraging TV episode recaps - short video sequences interweaving key story moments from previous episodes to bring viewers up to speed. We propose PlotSnap, a dataset featuring two crime …


“More than Words”: A Legal Approach to the Risks of Commercial Chatbots Powered by Generative Artificial Intelligence

S Migliorini - European Journal of Risk Regulation, 2024

… In the training of ChatGPT, this “clean”19 version of the Common Crawl has been complemented and filtered with the addition of a few … also that the training gave more prominence to the “clean” Common Crawl.Although this explanation may be …


A Balancing Act: Data Protection Compliance of Artificial Intelligence

Neural network based generative artificial intelligence (GenAI) systems, in particular large language models (LLMs), hold huge potential for economic, scientific, and artistic progress. However, they are also the subject of ongoing fierce data protection …


A bilingual benchmark for evaluating large language models

M Alkaoud - PeerJ Computer Science, 2024

This work introduces a new benchmark for the bilingual evaluation of large language models (LLMs) in English and Arabic. While LLMs have transformed various fields, their evaluation in Arabic remains limited. This work addresses this …


A Canary in the AI Coal Mine: American Jews May Be Disproportionately Harmed by Intellectual Property Dispossession in Large Language Model Training

H Precel, A McDonald, B Hecht, N Vincent - arXiv preprint arXiv:2403.13073, 2024

Systemic property dispossession from minority groups has often been carried out in the name of technological progress. In this paper, we identify evidence that the current paradigm of large language models (LLMs) likely continues this long history …


A Comparative Analysis of Text Embedding Models for Bug Report Semantic Similarity

A Patil, K Han, A Jadon - 2024 11th International Conference on Signal …, 2024

… For Fasttext, the ”crawl-300d-2M-subword” model was employed, which consisted of 2 million word vectors trained with subword information on the Common Crawl dataset, encompassing 600 billion tokens. In the case of Doc2Vec, the ”GoogleNews-vectors-negative300” …


A Comparative Analysis of Word Embeddings Techniques for Italian News Categorization

F Rollo, G Bonisoli, L Po - IEEE Access, 2024

Text categorization remains a formidable challenge in information retrieval, requiring effective strategies, especially when applied to low-resource languages such as Italian. This paper delves into the intricacies of categorizing Italian news articles …


A Comparative Study of ChatGPT and Seq2Seq Chatbot for Effective Students Advising

SK Assayed, M Alkhatib, K Shaalan - ChatGPT and Global Higher Education: Using …, 2024

Artificial Intelligence (AI) chatbots are increasingly becoming a part of students’ lives, from school learning to university admissions and orientations. AI chatbots are finding ways to facilitate and improve learning experiences. Moreover, AI chatbots …


A comparative study of cross-lingual sentiment analysis

P Přibáň, J Šmíd, J Steinberger, A Mištera - Expert Systems with Applications, 2024

This paper presents a detailed comparative study of the zero-shot cross-lingual sentiment analysis. Namely, we use modern multilingual Transformer-based models and linear transformations combined with CNN and LSTM neural networks. We …


A Comparison of Pretrained Models for Classifying Issue Reports

J Heo, G Kwon, C Kwak, S Lee - IEEE Access, 2024

Issues are evolving requirements which are the main factor that increase the cost of software evolution. To help developers manage issues, GitHub provides issue labeling mechanisms in issue management systems. However, manually labeling …


A Comprehensive Review on Large Language Models

A Yadav - … Software Engineering Through AI, Federated Learning …, 2024

In the realm of computer science and language, large language models (LLMs) stand out as remarkable tools of artificial intelligence (AI). Proficient in deciphering intricate language nuances, LLMs offer sensible responses and find applications in …


A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions

A Casheekar, A Lahiri, K Rath, KS Prabhakar… - Computer Science Review, 2024

This review paper offers an in-depth analysis of AI-powered virtual conversational agents, specifically focusing on OpenAI’s ChatGPT. The main contributions of this paper are threefold: (i) an exhaustive review of prior literature on chatbots, (ii) a …


A Copious Void: Rhetoric as Artificial Intelligence 1.0

A Hallsby - Rhetoric Society Quarterly, 2024

Rhetoric is a trace retained in and by artificial intelligence (AI) technologies. This concept illuminates how rhetoric and AI have faced issues related to information abundance, entrenched social inequalities, discriminatory biases, and the …


A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl

S Baack - The 2024 ACM Conference on Fairness, Accountability …, 2024

… among LLM builders about the implications of using Common Crawl’s data. This paper discusses what Common Crawl’s popularity for LLM development means … Our qualitative analysis is based on indepth interviews with Common Crawl staffers …


A critical examination of document-level machine translation systems

The need for accurate and effective translation cannot be overstated in an increasingly globalised world where communication is paramount. Bridging language barriers is important for promoting understanding and cooperation among …


A CURATEd CATalog: Rethinking the Extraction of Pretraining Corpora for Mid-Resourced Languages

J Palomar-Giner, JJ Saiz, F Espuña, M Mina, S Da Dalt… - Proceedings of the 2024 …, 2024

We present and describe two language resources in this paper: CATalog 1.0, the largest text corpus in Catalan to date, and CURATE (Corpus Utility for RAting TExt), a modular, parallelizable pipeline used for processing and scoring documents …


A Dataset for Pharmacovigilance in German, French, and Japanese: Annotating Adverse Drug Reactions across Languages

L Raithel, HS Yeh, S Yada, C Grouin, T Lavergne… - arXiv preprint arXiv …, 2024

User-generated data sources have gained significance in uncovering Adverse Drug Reactions (ADRs), with an increasing number of discussions occurring in the digital world. However, the existing clinical corpora predominantly revolve around scientific …


A Dataset for the Detection of Dehumanizing Language

P Engelmann, PB Trolle, C Hardmeier - arXiv preprint arXiv:2402.08764, 2024

… The Common Crawl is an open repository of web crawled data, which features crawls from all … As the Common Crawl includes several petabytes of data in total, we have selectively ex… As we limit ourselves to English, the Common Crawl data …


A Decade's Battle on Dataset Bias: Are We There Yet?

Z Liu, K He - arXiv preprint arXiv:2403.08632, 2024

We revisit the "dataset classification" experiment suggested by Torralba and Efros a decade ago, in the new era with large-scale, diverse, and hopefully less biased datasets as well as more capable neural network architectures. Surprisingly, we …


A field guide to using LLMs for online conflict analysis

E Borra, P Bollini, L Caroleo, B Cerigo, R Dubèl, F Gaw…

… Two ways in which we studied LLMs was by looking at some of their open-source datasets, which one can sift through in WebText2, CommonCrawl and Wikipedia. Another is by studying autocompletions as perpetuations of standardized knowledge …


A Framework for Embedding Entities in a Textual Narrative: a Case Study on Les Misérables

In this article, we propose a general and flexible framework in order to study narrative entities found in a literary work. This framework is exposed starting from a broad perspective, consisting in how to segment the work into textual units and …


A Framework for Preparing a Balanced and Comprehensive Phishing Dataset

… • Choon Lin Tan (2018) [19] - dataset contains 5000 legitimate records sourced from Alexa and Common Crawl and 5000 phishing records from PhishTank and … Common Crawl (commoncrawl.org)- is a humongous web archive collected by …


A Framework to Construct Financial Causality Knowledge Graph from Text

Z Xu, H Takamura, R Ichise - 2024 IEEE 18th International Conference on Semantic …, 2024

Causality analysis holds a prominent role in finance, and the presentation of causality could offer valuable insights for risk mitigation, investment decisions, and portfolio optimization. Recent research has extensively investigated the identification …


A Galician Corpus for Misogyny Detection Online

LM Álvarez-Crespo, LM Castro - Proceedings of the 16th International Conference on …, 2024

Social networks are virtual spaces where millions of people share ideas, opinions, and experiences. However, this broad social interaction also exposes negative and harmful behaviors, such as harassment and misogyny. Misogyny, particularly, is a …


A Generalize Hardware Debugging Approach for Large Language Models Semi-Syntectic Datasets

W Fu, S Li, Y Zhao, K Yang, X Zhang, Y Jin, X Guo

… C4 specifically employs the Common Crawl [44] dataset, focusing on deduplication, removing non-English content, and filtering out offensive language. Redpajama combines sources from Wikipedia and StackExchange, similar to C4 …


A Holistic Review on Detection of Malicious Browser Extensions and Links using Deep Learning

T Zonta, M Sathiyanarayanan - 2024 IEEE 3rd International Conference on AI in …, 2024

… Common Crawl [36] is a website which crawls over the web and suggest many datasets of legitimate website and their metadata namely … Available: https://commoncrawl.or g/2021/10/september-2021-crawlarchive-now-available/ [37] AC Bahnsen, EC …


A Hybrid Deep BiLSTM-CNN for Hate Speech Detection in Multi-social media

A Kumar, S Kumar, K Passi, A Mahanti - ACM Transactions on Asian and Low …, 2024

Nowadays, ways of communication among people have changed due to advancements in information technology and the rise of online multi-social media. Many people express their feelings, ideas, and emotions on social media sites such …


A Japanese-Chinese Parallel Corpus Using Crowdsourcing for Web Mining

M Nagata, M Morishita, K Chousa, N Yasuda - arXiv preprint arXiv:2405.09017, 2024

… In ParaCrawl, they determine which websites to crawl by analyzing the Common Crawl … In this study, we analyzed 12 sets of Common Crawl archives (104TB in total) published from … and bilingual website URLs obtained from Common Crawl …


A Language Model Trained on Uruguayan Spanish News Text

JP Filevich, G Marco, S Castro, L Chiruzzo, A Rosá - LREC-COLING 2024, 2024

This paper presents a language model trained from scratch exclusively on a brand new corpus consisting of about 6 GiB of Uruguayan newspaper text. We trained the model for 30 days on a single Nvidia P100 using the RoBERTa-base architecture …


A Large-Scale Exploration of $\mu $-Transfer

L Lingle - arXiv preprint arXiv:2404.05728, 2024

Large neural network models have become a mainstay of natural language processing and computer vision, yet their initialization and learning rates are set in a largely heuristic fashion, potentially varying from paper to paper and one model size …


A Legal Framework for Natural Language Model Training in Portugal

R Almeida, E Amorim - Proceedings of the Workshop on Legal and Ethical …, 2024

Recent advances in deep learning have promoted the advent of many computational systems capable of performing intelligent actions that, until then, were restricted to the human intellect. In the particular case of human languages, these …


A Legal Framework for Natural Language Processing Model Training in Portugal

R Almeida, E Amorim - arXiv preprint arXiv:2405.00536, 2024


A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish

M Kaneko, T Baldwin - arXiv preprint arXiv:2403.16139, 2024

… The most common sources included in all LLMs are web page sources such as C4, CommonCrawl, and the Pile. Because they are collected from various web pages, there is a risk that they may contain personal information, copyrighted texts …


A Longitudinal Study of Content Control Mechanisms

M Dinzinger, M Granitzer - Companion Proceedings of the ACM on Web …, 2024

… Both resources, robots.txt and regular pages, were collected from the Common Crawl web archive 7 and processed through a customized … As Common Crawl is furhtermore not focused on particular web domains, but randomly samples the …


A methodology for using players' chat content for dynamic difficulty adjustment in metaverse multiplayer games

MM Rezapour, A Fatemi, MA Nematbakhsh - Applied Soft Computing, 2024

… RoBERTa, on the other hand, was trained on a much larger corpus of text that includes not only Wikipedia and BookCorpus but also Common Crawl, a dataset that contains billions of web pages. This larger and more diverse corpus of text allows …


A Multilingual Dataset for Investigating Stereotypes and Negative Attitudes Towards Migrant Groups in Large Language Models

D Sorato, CC Ventura, D Zavala-Rojas - … of the 16th International Conference on …, 2024

… 2020), which is a cleaned version of the CommonCrawl corpus. … where stereotypes are presented in more subtle and/or strategic ways (when compared to social media/CommonCrawl texts) and the explicit discrimination of migrant groups …


A Multilingual Perspective on Probing Gender Bias

K Stańczak - arXiv preprint arXiv:2403.10699, 2024

Gender bias represents a form of systematic negative treatment that targets individuals based on their gender. This discrimination can range from subtle sexist remarks and gendered stereotypes to outright hate speech. Prior research has …


A New Massive Multilingual Dataset for High-Performance Language Technologies

O de Gibert, G Nail, N Arefyev, M Bañón… - arXiv preprint arXiv …, 2024

We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls …


A Permutaion Importance Based feature selection method and Deep Learning Model to Detect Phishing Websites

Phishing attacks pose a significant and escalating threat to cybersecurity in recent times. This deceptive scam aims to trick naive users, luring them into visiting harmful websites and sharing sensitive information, including credentials, credit card …


A phenomenology and epistemology of large language models: Transparency, trust, and trustworthiness

R Heersmink, B de Rooij, MJC Vázquez, M Colombo

… We know that its dataset includes the Common Crawl, which is a publicly available corpus of Webpages, including billions of Webpages … whether sources in the Common Crawl or Wikipedia are prioritised. This is again a serious issue to do …


A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)

… For low-resource languages, XLM models trained on CommonCrawl-100 did better than the ones trained using Wikipedia. Another example is mT5 [32], which uses multilingual corpus mC4 to pretrain the model. When dealing with multilingual …


A Review of Multi-Modal Large Language and Vision Models

K Carolan, L Fennelly, AF Smeaton - arXiv preprint arXiv:2404.01322, 2024

… The Falcon LLM team created the RefinedWed dataset, which is a massive web dataset based on CommonCrawl. TII focused on scaling and improving the quality of web data by employing large-scale de-duplication and strict filtering, resulting in …


A Robust Approach to E-Banking Phishing Detection using Ensemble Methods and LSTM

NVS Reddy, ARR Saai, TV Ramanujan, NRK Reddy… - … International Conference on …, 2024

… range of datasets from reputable community platforms like VirusTotal, Common Crawl, and PhishTank. This extensive dataset, spanning … Integration of Common Crawl further enriched the dataset, ensuring a diverse set of websites for effective …


A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism

B Thompson, MP Dhaliwal, P Frisch, T Domhan… - arXiv preprint arXiv …, 2024

… 2021), which is in turn based on Common CrawlCommon Crawl is a long running web-scraping project which maintains a free, open source repository of web-scraped data. ccMatrix is created by embedding Common Crawl sentences into a …


A Short Commentary on Trinh & Le (2018)

… corpus (trained on the language model LM-1-Billion, CommonCrawl, and SQuAD), the probabilities of s1 and s2 appearing in a large corpus. The substitution that turns out to be more probable, is considered to be the more probable referent of “it”. There …


A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization

T Kumarage, G Agrawal, P Sheth, R Moraffah… - arXiv preprint arXiv …, 2024

We have witnessed lately a rapid proliferation of advanced Large Language Models (LLMs) capable of generating high-quality text. While these LLMs have revolutionized text generation across various domains, they also pose significant …


A Survey of Multimodal Large Language Model from A Data-centric Perspective

T Bai, H Liang, B Wan, L Yang, B Li, Y Wang, B Cui… - arXiv preprint arXiv …, 2024

… CommonCrawl project serves as the most commonly … , CommonCrawl’s vast archive of web pages, which includes numerous image-text pairs, has also become a vital resource for constructing multimodal pre-training datasets such as LAION-5B [236] …


A Survey of Neural Machine Translation based on Knowledge Distillation

MA Chang, T Yonghong, Z Xiaoli, SUN Kangkang - Journal of Frontiers of Computer Science …

Abstract: Machine Translation (MT) is the process of using a computer to convert a language into another language with the same semantics. With the introduction of neural network, neural machine translation (NMT), as a powerful machine …


A Survey of Resource-efficient LLM and Multimodal Foundation Models

M Xu, W Yin, D Cai, R Yi, D Xu, Q Wang, B Wu, Y Zhao… - arXiv preprint arXiv …, 2024

Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment …


A Survey of Web Content Control for Generative AI

M Dinzinger, F Heß, M Granitzer - arXiv preprint arXiv:2404.02309, 2024

… The documents were sourced from the web archive of Common Crawl,16 offering a broad and randomly selected cross-section of the internet. These files were all collected in November and December 2023. Further details on the experimental …


A Survey on Data Selection for Language Models

A Albalak, Y Elazar, SM Xie, S Longpre, N Lambert… - arXiv preprint arXiv …, 2024

… They train a classifier using the “high-quality” reference corpora as the positive class and unfiltered Common Crawl documents as the negative class. This classifier then defines the utility metric for selecting data that follows the desired distribution …


A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

K Huang, F Mo, H Li, Y Li, Y Zhang, W Yi, Y Mao, J Liu… - arXiv preprint arXiv …, 2024

The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing, attracting global attention in both academia and industry. To mitigate potential discrimination …


A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias

Y Xu, L Hu, J Zhao, Z Qiu, Y Ye, H Gu - arXiv preprint arXiv:2404.00929, 2024

… A significant portion of these training data originates from multilingual repositories like Common Crawl, WikiPedia and Web documents, encompassing a broad range of languages. These multilingual repositories are crucial for enhancing the cross-lingual …


A survey on recent advances in named entity recognition

I Keraghel, S Morbieu, M Nadif - arXiv preprint arXiv:2401.10825, 2024

Named Entity Recognition seeks to extract substrings within a text that name real-world objects and to determine their type (for example, whether they refer to persons or organizations). In this survey, we first present an overview of recent popular …


A systematic review of multidimensional relevance estimation in information retrieval

G Peikos, G Pasi - Wiley Interdisciplinary Reviews: Data Mining and …

In information retrieval, relevance is perceived as a multidimensional and dynamic concept influenced by user, task, and domain factors. Relying on this perspective, researchers have introduced multidimensional relevance models addressing …


A Systematic Study of the Role of Data Quality and Alignment for Fine-tuning LLMs for Enhanced Autoformalization

K Chawla, A Sahai, M DePavia, B Miranda - The Second Tiny Papers Track at ICLR 2024

This study explores the role of data quality, particularly alignment, in fine-tuning Large Language Models (LLMs) for the task of autoformalization. Contrary to the conventional emphasis on dataset size, our research highlights the importance of …


A Trustworthy Automated Short-Answer Scoring System Using a New Dataset and Hybrid Transfer Learning Method

M Maslim, HC Wang, CD Putra, YD Prabowo - 2024

To measure the quality of student learning, teachers must conduct evaluations. One of the most efficient modes of evaluation is the short answer question. However, there can be inconsistencies in teacher-performed manual evaluations due to an …


AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

L Lucy, S Gururangan, L Soldaini, E Strubell… - arXiv preprint arXiv …, 2024

… From this Common Crawl data, we identify website hostnames that include an ABOUT page, which we define as URL paths containing about, aboutme, about-us, or bio (Appendix A). We then pair each ABOUT page with a random page on the …


Abusive Language Detection in Khasi Social Media Comments

A Baruah, L Wahlang, F Jyrwa, F Shadap, F Barbhuiya… - ACM Transactions on Asian …, 2024

This paper describes the work performed for automated abusive language detection in the Khasi language, a low-resource language spoken primarily in the state of Meghalaya, India. A dataset named Khasi Abusive Language Dataset (KALD) was …


Accelerating Spiking Neural Networks with Parallelizable Leaky Integrate-and-Fire Neurons

… To achieve this, these models undergo training on massive datasets, such as the Common Crawl corpus1, which contains petabytes of textual data. Training on such extensive datasets necessitates the use of fast machine learning models like the …


ACCEPTANCE OF GENERATIVE AI IN KNOWLEDGE WORK

… The GPT-3 was trained with Corpus, a large and structured collection of texts or language data, called Common Crawl including nearly a trillion words. (Brown et al., 2020) In addition, the training set was also expanded with known high-quality …


ACCSAMS: Automatic Conversion of Exam Documents to Accessible Learning Material for Blind and Visually Impaired

D Wilkening, O Moured, T Schwarz, K Muller… - arXiv preprint arXiv …, 2024

… We collected exam materials from the Common Crawl project4, a public archive of web data. We analyzed eight datasets, released between mid 2022 and the end of 2023, extracting PDFs, which include “exam” or “klausur” as a keyword in the URL …


Accuracy of ChatGPT in Neurolocalization

WF Dabbas, YM Odeibat, M Alhazaimeh, MY Hiasat… - Cureus, 2024

… The datasets used for pre-training included Common Crawl dataset, an expanded version of the WebText dataset, two internet-based books corpora (Books1 and Books2), and English-language Wikipedia [13]. …


Adapting LLMs to Downstream Applications

… The model was fed with text that generally precedes private information in a dataset used to train GPT-2—specifically the “Common Crawl” dataset to achieve a better recall. 2 With this approach, authors were not only able to extract extensive …


Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection

Z Lai, X Zhang, S Chen - arXiv preprint arXiv:2403.13335, 2024

… 5) XLMRoberta: XLMRoberta is a large-scale multilingual transformer-based LM pretrained on a diverse set of one hundred languages using extensive CommonCrawl data [68]. We use the “xlm roberta base multi” as pretrained backbone weight. …


Addressing Both Statistical and Causal Gender Fairness in NLP Models

H Chen, Y Ji, D Evans - arXiv preprint arXiv:2404.00463, 2024

… 2019) comprising nearly 400,000 online biographies of 28 unique occupations scraped from the CommonCrawl. The task is to predict the occupation given in the biography with the occupation title removed. Each biography includes the name and …


Addressing the Challenge of Online Health Misinformation: Detection, Retrieval, and Explainability

Nell’odierna epoca digitale, le piattaforme online costituiscono uno dei mezzi principali utilizzati dalle persone per cercare informazioni relative alla propria salute. Nonostante il web sia un vasto repository di conoscenze in tale ambito, è affetto dal …


Adjusting Interpretable Dimensions in Embedding Space with Human Judgments

K Erk, M Apidianaki - arXiv preprint arXiv:2404.02619, 2024

… The GLoVe embeddings used were trained on Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors), downloaded from https: //nlp.stanford.edu/projects/glove/. In order to generate embeddings for contextualized instances of words in our …


Adult Content Detection on Indonesian Tweets by Fine-tuning Transformer-based Models

AF Hidayatullah, RA Apong, DTC Lai, A Qazi - 2023 6th International Conference on …, 2023

The prevalence of adult content on social media has harmful effects on the moral values of young individuals. Therefore, effectively filtering inappropriate content on social media like Twitter is essential. Researchers have utilized machine learning …


Advancing Fake News Detection: Hybrid Deep Learning with FastText and Explainable AI

E Hashmi, SY Yayilgan, MM Yamin, S Ali, M Abomhara - IEEE Access, 2024

… FastText, a word representation tool developed by Facebook’s research division, provides both unsupervised and supervised modes, featuring an extensive lexicon of 2 million words sourced from Common Crawl. Each word is represented in a 300-dimensional …


Advancing Internet Viewpoint Diversity: A Novel Algorithm and a Corpus Creation Tool

… To achieve this, we first develop a big data processing architecture for creating indexed corpora from the Common Crawl web archives. The … Utilizing this tool, we processed approximately 1.2 billion web pages from the Common Crawl dataset …


Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural …

F Stöhr - Journal of Computational Social Science, 2024

This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are:(1) to train the embeddings …


Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

K Uehara, N Goswami, H Wang, T Baba, K Tanaka… - arXiv preprint arXiv …, 2024

The increasing demand for intelligent systems capable of interpreting and reasoning about visual content requires the development of Large Multi-Modal Models (LMMs) that are not only accurate but also have explicit reasoning capabilities. This paper …


Advancing medical imaging with language models: featuring a spotlight on ChatGPT

M Hu, JY Qian, S Pan, Y Li, RLJ Qiu, X Yang - Physics in Medicine and Biology, 2024

This review paper aims to serve as a comprehensive guide and instructional resource for researchers seeking to effectively implement language models in medical imaging research. First, we presented the fundamental principles and …


Advancing Real-time Pandemic Forecasting Using Large Language Models: A COVID-19 Case Study

H Du, J Zhao, Y Zhao, S Xu, X Lin, Y Chen, LM Gardner - arXiv preprint arXiv …, 2024

HF Yang, H Du, J Zhao, Y Zhao, S Xu, X Lin, Y Chen… - 2024

Forecasting the short-term spread of an ongoing disease outbreak is a formidable challenge due to the complexity of contributing factors, some of which can be characterized through interlinked, multi-modality variables such as epidemiological …


Afrikaans Literary Genre Recognition using Embeddings and Pre-Trained Multilingual Language Models

E Kotzé, BA Senekal - 2024 International Conference on Artificial Intelligence …, 2024

… XLM-R (or XLM-RoBERTa) is a multilingual version of RoBERTa with 279M parameters and was trained on a 2.5TB of CommonCrawl data containing 100 languages. Afrikaans is included in all three of these multilingual models. …


Aggression and Misogyny in Hindi and Bangla: A Study of YouTube Comments

R Kumar, B Lahiri - Studies in Pragmatics, 2024

… The pre-trained models are typically trained on web-scale data, ie text collected from all over the web such as the whole of Wikipedia and a common crawl of the websites currently on the internet amounting to billions (or trillions) of tokens and …


AI unveiled personalities: Profiling optimistic and pessimistic attitudes in Hindi dataset using transformer‐based models

Both optimism and pessimism are intricately intertwined with an individual's inherent personality traits and people of all personality types can exhibit a wide range of attitudes and behaviours, including levels of optimism and pessimism. This paper …


AI's Secret Weapon in Education. ChatGPT–The Future of Personalized Learning

A Popescu - Bulletin of the Transilvania University of Brasov. Series …, 2023

This article delves into the fascinating intersection of AI and education, with a focus on ChatGPT, a conversational AI developed by OpenAI. Noted for its human-like responses, ChatGPT is positioned as a game-changer in personalized learning. The …


AI-Powered Code Review Assistant for Streamlining Pull Request Merging

C Adapa, SS Avulamanda, ARK Anjana, A Victor - 2024 IEEE International …, 2024

… The model's reliance on Refined Web, primarily sourced from Common Crawl, underscores its commitment to exceptional quality through advanced data augmentation techniques, notably largescale deduplication and rigorous filtering. Unlike conventional …


AI-Proof Your Assignments

ChatGPT and other AI tools are powerful, free, and seemingly everywhere. Faculty love it, fear it, and everything in between. Students also love it, fear it, and everything in between; All the while, they are using it. In this session, we’ll cover a bit about how …


AI-Replicas as Ethical Practice: Introducing an Alternative to Traditional Anonymization Techniques in Image-Based Research

This paper introduces the use of AI-replicas as an alternative to traditional anonymization methods in image-based qualitative research. It emphasizes the ethical and practical dilemmas posed by current anonymization methods, such as …


Algorithmic amplification of biases on Google Search

H Habib, R Stoldt, A High, B Ekdale, A Peterson… - arXiv preprint arXiv …, 2024

The evolution of information-seeking processes, driven by search engines like Google, has transformed the access to information people have. This paper investigates how individuals' preexisting attitudes influence the modern information-seeking …


Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs

AM Kassem, O Mahmoud, N Mireshghallah, H Kim… - arXiv preprint arXiv …, 2024

In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data …


Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages

M Andersland - arXiv preprint arXiv:2403.06354, 2024

… Less than 0.1% of CommonCrawl2 is Amharic, and even when combining open source datasets without deduplication, we find that less than 500 million tokens of Amharic are available. In addition, the content of this data tends to be biased …


An Ambiguous Technique for Nonvisual Text Entry

Text entry is a common daily task for many people, but it can be a challenge for people with visual impairments when using virtual touchscreen keyboards that lack physical key boundaries. In this thesis, we investigate using a small number of …


An Analysis of Langauge Frequency and Error Correction for Esperanto

J Liang - arXiv preprint arXiv:2402.09696, 2024

… Goyal builds Commoncrawl Corpus in 100 languages and then uses a language identification model to select Esperanto contexts[8], which … OSCAR [11] provides a collection of unannotated raw data and devises a method to improve the quality of …


An Assessment on Comprehending Mental Health through Large Language Models

M Arcan, PD Niland, F Delahunty - arXiv preprint arXiv:2401.04592, 2024

… The training data for GPT-3 is primarily sourced from a filtered version of Common Crawl, contributing to 60% of the weighted pre-training dataset, comprising 410 billion bytepair-encoded tokens. Other data sources include 19 billion tokens from …


An Extensive Survey on Investigation Methodologies for Text Summarization

A Saklecha, P Uplavdiya, MPS Chawla - Indian Journal of Signal Processing (IJSP), 2023

Natural language processing (NLP) is a fastexpanding field, and text summarization has recently gained a lot of research interest. The necessity for automatic summarizing approaches to effectively digest massive amounts of textual data has …


An Integrated Data Processing Framework for Pretraining Foundation Models

Y Sun, F Wang, Y Zhu, WX Zhao, J Mao - arXiv preprint arXiv:2402.16358, 2024

… In the end-to-end evaluation, we train two GPT-2 models using CommonCrawl before and after processing, respectively. The model trained on the refined data demonstrates remarkable performance enhancement across downstream tasks of …


An Introduction to Vision-Language Modeling

F Bordes, RY Pang, A Ajay, AC Li, A Bardes, S Petryk… - arXiv preprint arXiv …, 2024

… Multiple curation steps are used to curate this dataset where English data is collected from common crawl and deduplicated followed by pre-processing HTML document where useful DOM nodes are identified and retained, then for each DOM …


Analysing Slow Thinking Capabilities in Large Language Model Agent-Agent Dialogue

… LLMs can be given data from books, CommonCrawl, Reddit links and Wikipedia, where CommonCrawl consists of petabytes of web archives [113]. Having large datasets makes it more difficult to curate the data. Documentation of the data can …


Analysing The Impact of Sequence Composition on Language Model Pre-Training

Y Zhao, Y Qu, K Staniszewski, S Tworkowski, W Liu… - arXiv preprint arXiv …, 2024

Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and …


Analysing the landscape of Deep Fake Detection: A Survey

K Vyas, P Pareek, R Jayaswal, S Patil - … of Intelligent Systems and Applications in …, 2024

… Grover utilizes a test set created by the free and open-source web crawler and archive Common Crawl [35]. As an example, a team of academics from Harvard and the MIT-IBM Watson laboratory developed the Giant language model test room, a …


AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets

P Lesci, A Vlachos - arXiv preprint arXiv:2404.05623, 2024

Active learning for imbalanced classification tasks is challenging as the minority classes naturally occur rarely. Gathering a large pool of unlabelled data is thus essential to capture minority instances. Standard pool-based active learning is …


and S. Abirami

RS Gulecha, SKNR Subramanian - Speech and Language Technologies for Low …

In recent years, the Internet and social media have sparked a revolution in the way information is exchanged. The growth of social media and micro-blogging sites not only provides platforms for empowering freedom of expression and individual voices …


Answer Agnostic Question Generation in Bangla Language

AR Fahad, N Al Nahian, MA Islam, RM Rahman - International Journal of Networked …, 2024

… mT5 is a massively multilingual model pre-trained using fresh Common Crawl-based data that includes 101 languages [14]. Another model, BanglaT5, a Transformer model for the Bangla language, was trained using a 27.5 GB clean corpus of Bangla …


AntiPhishStack: LSTM-based Stacked Generalization Model for Optimized Phishing URLs Detection

S Aslam, H Aslam, A Manzoor, C Hui, A Rasool - 2024

The escalating reliance on revolutionary online web services has introduced heightened security risks, with persistent challenges posed by phishing despite extensive security measures. Traditional phishing systems, reliant on machine …


AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

J Zhan, J Dai, J Ye, Y Zhou, D Zhang, Z Liu, X Zhang… - arXiv preprint arXiv …, 2024

We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any …


Application of an Improved Convolutional Neural Network Algorithm in Text Classification

J Peng, S Huo - Journal of Web Engineering, 2024

This paper proposes a text classification model based on a combination of a convolutional neural network (CNN) and a support vector machine (SVM) using Amazon review polarity, TREC, and Kaggle as experimental data. By adding an …


Application of Open-source Large Language Model (LLM) for Simulation of a Vulnerable IoT System and Cybersecurity Best Practices Assistance

This paper explores the role of open-source large language models in IoT cybersecurity world. The threats of malicious activity on the Internet and the loss of private information are very real and lead to serious consequences. The purpose of …


Applications of AI in Computer Vision and NLP

SB Rajasekaran - Journal of Artificial Intelligence & Cloud Computing …, 2023

… In language AI, there are several commonly used datasets such as Common Crawl, Wikipedia, and OpenWebText. However, creating and maintaining these datasets can also be a challenging task, as it needs a big amount of work and it can …


Applying German word vectors to assess flexibility performance in the associative fluency task.

M Camenzind, M Single, SM Gerber, T Nef, CL Bassetti… - Psychology of Aesthetics …, 2024

… One model was trained on common crawl, a nonprofit organization crawling the web and making resulting data publicly available, and the other was trained on Wikipedia. The vectors were trained using the CBOW algorithm with a special focus …


Applying Named Entity Recognition and Graph Networks to Extract Common Interests from Thematic Subfora on Reddit

J Sawicki, M Ganzha, M Paprzycki, Y Watanobe - Applied Sciences, 2024

… , accessed on 13 February 2024) (pre-trained BERT large models fine-tuned on the CoNLL-2003 dataset) and flair/ner-english-large (https://huggingface.co/flair/ner-english-large, accessed on 13 February 2024) (an XLM-RoBERTa [89] pre-trained on a cleaned …


Applying Transfer Learning to German Metaphor Prediction

M Berger, SM Reimann, NM Kiwitt - Proceedings of the 2024 Joint International …, 2024

… -scores using CommonCrawl embeddings (F1 60%) and News Commentary embeddings (F1 58%) while CommonCrawl contains over 2 m… 4), we can see that the bilingual embeddings approach, especially using Europarl and CommonCrawl …


Arabic sarcasm detection: An enhanced fine-tuned language model approach

MA Galal, AH Yousef, HH Zayed, W Medhat - Ain Shams Engineering Journal, 2024

Sarcasm is a complex linguistic phenomenon involving humor, criticism, or phrases that convey the opposite meaning, mask true feelings, and play pivotal roles in various aspects of communication. Therefore, identifying sarcasm is essential for …


Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models

L Merrick, D Xu, G Nuti, D Campos - arXiv preprint arXiv:2405.05374, 2024

This report describes the training dataset creation and recipe behind the family of \texttt{arctic-embed} text embedding models (a set of five models ranging from 22 to 334 million parameters with weights open-sourced under an Apache-2 license). At the time of …


Are Human Conversations Special? A Large Language Model Perspective

T Jawale, C Animesh, S Vallath, K Talamadupula… - arXiv preprint arXiv …, 2024

… We analyze the web data from CommonCrawl[21] dumps for human conversation data and the types of conversations in Table 1. We randomly sample a subset of the dump and deduplicate it so that it can be used to approximate the data distribution …


Are LLMs Robust for Spoken Dialogues?

SM Mousavi, G Roccabruna, S Alghisi, M Rizzoli… - arXiv preprint arXiv …, 2024

… The LLM finetuned for this task is T5 Small [21], (12 layers, 60M parameters), a transformer-based encoder-decoder model, pre-trained on the Common Crawl dataset with 750GB of web page text. The fine-tuning was performed following the …


Artificial Intelligence for Anesthesiology Board-style Examination Questions: Role of Large Language Models

AA Khan, R Yunus, M Sohail, TA Rehman, S Saeed… - Journal of Cardiothoracic …, 2024

Background : New artificial intelligence tools have been developed that have implications for medical usage. Large language models such as the widely used ChatGPT, developed by OpenAI, have not been explored in the context of …


Springer MRW: [AU:, IDX:] Page 1 Artificial Intelligence in the Media Economy: A Systematic Review of Use Cases, Application Potentials, and Challenges of Generative Language Models Tim Prien and Klaus Goldhammer Contents 1 …


Artificial intelligence in the news: how AI retools, rationalizes, and reshapes journalism and the public arena

Despite growing interest, the effects of AI on the news industry and our information environment — the public arena — remain poorly understood. Insufficient attention has also been paid to the implications of the news industry’s dependence on …


Artificial Intelligence Supporting Independent Student Learning: An Evaluative Case Study of ChatGPT and Learning to Code

K Hartley, M Hayak, UH Ko - Education Sciences, 2024

Artificial intelligence (AI) tools like ChatGPT demonstrate the potential to support personalized and adaptive learning experiences. This study explores how ChatGPT can facilitate self-regulated learning processes and learning computer programming …


Artificial Intelligence, Large Language Models, and Design Thinking in TPC Classrooms

C Masters-Wheeler, J Bay, P Sullivan - Programmatic Perspectives, 2023

… Common Crawl, for instance, is an open repository of web data that is accessible to anyone and can be used to train AI models. Large language models (LLMs) use the textual data from sources like Common Crawl and other freely accessible data …


Asking GPT for the Ordinary Meaning of Statutory Terms

C Engel, RH McAdams - MPI Collective Goods Discussion Paper, 2024

We report on our test of the Large Language Model (LLM) ChatGPT (GPT) as a tool for generating evidence of the ordinary meaning of statutory terms. We explain why the most useful evidence for interpretation involves a distribution of replies rather …


Asking Questions Framework for Oral History Archives

J Švec, M Bulín, A Frémund, F Polák - European Conference on Information Retrieval, 2024

… The resulting Wav2Vec 2.0 end-to-end model was combined with a lower-case four-gram language model estimated from CommonCrawl data. To decrease the model’s … The model was pre-trained on a self-supervised text-restoration task …


Assessing and Optimizing Large Language Models on Spondyloarthritis Multi-Choice Question Answering: Protocol for Enhancement and Assessment

A Wang, Y Wu, X Ji, X Wang, J Hu, F Zhang, Z Zhang… - JMIR Research Protocols, 2024

Background Spondyloarthritis (SpA), a chronic inflammatory disorder, predominantly impacts the sacroiliac joints and spine, significantly escalating the risk of disability. SpA’s complexity, as evidenced by its diverse clinical presentations and symptoms …


Assessing the research landscape and clinical utility of large language models: a scoping

YJ Park, A Pillai, J Deng, E Guo, M Gupta, M Paget… - 2024

… Introduced in November 2022, ChatGPT was trained using a large corpora of unlabelled text, including CommonCrawl, WebText, and Wikipedia, as well as internet-based book corpora spanning multiple languages [3]. GPT, along with other …


Asymmetric Polysemous Reasoning for Image-Text Matching

H Zhang, M Yang - 2023 IEEE International Conference on Data Mining …, 2023

Image-text matching has received growing interest since it bridges vision and language. The key challenge lies in how to learn correspondence between image and text. Upon observation, we find existing works suffer from two limitations. Firstly …


Asynchronous Local-SGD Training for Language Modeling

B Liu, R Chhaparia, A Douillard, S Kale, AA Rusu… - arXiv preprint arXiv …, 2024

Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD update per communication. This work presents an empirical …


Atmospheric Limitations for High-frequency Ground-based Very Long Baseline Interferometry

DW Pesce, L Blackburn, R Chaves, SS Doeleman… - The Astrophysical Journal, 2024

Very long baseline interferometry (VLBI) provides the highest-resolution images in astronomy. The sharpest resolution is nominally achieved at the highest frequencies, but as the observing frequency increases, so too does the atmospheric contribution …


AtomGPT: Atomistic Generative Pre-trained Transformer for Forward and Inverse Materials Design

K Choudhary - arXiv preprint arXiv:2405.03680, 2024

Large language models (LLMs) such as generative pretrained transformers (GPTs) have shown potential for various commercial applications, but their applicability for materials design remains underexplored. In this article, we introduce AtomGPT, a …


Auditing Large Language Models for Enhanced Text-Based Stereotype Detection and Probing-Based Bias Evaluation

Z Wu, S Bulathwela, M Perez-Ortiz, AS Koshiyama - arXiv preprint arXiv:2404.01768, 2024

Recent advancements in Large Language Models (LLMs) have significantly increased their presence in human-facing Artificial Intelligence (AI) applications. However, LLMs could reproduce and even exacerbate stereotypical outputs from …


Augmenting a Spanish clinical dataset for transformer-based linking of negations and their out-of-scope references

AJ Tamayo-Herrera, DA Burgos, A Gelbukh - Natural Language Processing

A negated statement consists of three main components: the negation cue, the negation scope, and the negation reference. The negation cue is the indicator of negation, while the negation scope defines the extent of the negation. The negation …


Augmenting Knowledge-Based Conversational Search Systems With Large Language Models

Conversational interfaces are increasingly used by websites and apps, including office software as well as creativity tools. This trend reflects a shift towards natural language in human-computer interaction, which is also facilitated by utilizing Large …


AutoMathText: Autonomous Data Selection with Language Models for Mathematical Texts

Y Zhang, Y Luo, Y Yuan, ACC Yao - arXiv preprint arXiv:2402.07625, 2024

To improve language models' proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy that leverages base language models for autonomous data selection. Departing from conventional supervised fine-tuning or …


Automatic Coding of Contingency in Child-Caregiver Conversations

A Agrawal, M Nikolaus, B Favre, A Fourtassi - 2024

One of the most important communicative skills children have to learn is to engage in meaningful conversations with people around them. At the heart of this learning lies the mastery of contingency, ie, the ability to contribute to an ongoing exchange …


Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach

HV Vo, V Khalidov, T Darcet, T Moutakanni… - arXiv preprint arXiv …, 2024

… By filtering Common Crawl data, the authors managed to obtain large datasets to train reliable … To this end, we apply our curation pipeline to two text pools based on Common Crawl. The first … Doing so keeps the original data distribution closer to …


Automatic detection of hate speech in code-mixed Indian languages in twitter social media interaction using DConvBLSTM-MuRIL ensemble method

P Kakati, D Dandotiya - Social Network Analysis and Mining, 2024

… During the pre-training process, 2.5 terabytes of Common Crawl data in 100 languages are used. This model gives a much better performance than multilingual BERT when code-switching, especially for languages with little resources, because …


Automatic Extraction of User-Centric Aspects for Tourist Spot Recommender Systems Using Reviews in Japanese

F Uwano, R Kobayashi, M Ohta - International Conference on Human-Computer …, 2024

In tourist reviews, various pieces of information are described to confirm the characteristics of tourist spots. This paper proposes a method to extract tourist spot aspects using Japanese tourist reviews automatically. The aspects of tourist spots …


Automating Code Adaptation for MLOps--A Benchmarking Study on LLMs

H Patel, BA Ramanan, MA Khan, T Williams… - arXiv preprint arXiv …, 2024

This paper explores the possibilities of the current generation of Large Language Models for incorporating Machine Learning Operations (MLOps) functionalities into ML training code bases. We evaluate the performance of OpenAI (gpt-3.5-turbo) and …


Automating String Encoding in AutoML

Automated Machine Learning (AutoML) systems often struggle with data sets with string data. Due to their unstructured nature and dependence on domain knowledge, string data are a frequent cause of errors and coming up with generalized ways of …


Autonomous Data Selection with Language Models for Mathematical Texts

Y Zhang, Y Luo, Y Yuan, AC Yao - ICLR 2024 Workshop on Navigating and …, 2024

To improve language models’ proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy that leverages base language models for autonomous data selection. Departing from conventional supervised fine-tuning or …


BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

J Xue, M Zheng, Y Hu, F Liu, X Chen, Q Lou - arXiv preprint arXiv:2406.00083, 2024

Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations." Retrieval-Augmented Generation (RAG) addresses these limitations by combining …


Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding

LC Chen, ZR Li - arXiv preprint arXiv:2404.00862, 2024

… Owing to computational constraints, we further conducted additional sampling of the OSCAR dataset and the Common Crawl dataset to mitigate the issue related to dataset size. In the long run, there are around 13 billion tokens left in the corpus. The …


Balanced Data Sampling for Language Model Training with Clustering

Y Shao, L Li, Z Fei, H Yan, D Lin, X Qiu - arXiv preprint arXiv:2402.14526, 2024

Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sampling strategy in training remains an open question. Most …


Balancing Multilingual Model Training Data Using Exponential Smoothing

… The XLM-R [9] model is pre-trained over a larger common crawl corpus similarly to MBERT, using language sampling according to exponentially-smoothed language probabilities. Again, performance on low-resources tasks seems to benefit …


Bangla Emergency Post Classification on Social Media using Transformer Based BERT Models

AA Nabil, D Das, MS Salim, S Arifeen, HMA Fattah - 2023 6th International …, 2023

… It is a large multi-lingual language model trained on 2.5TB of filtered Common Crawl data. XLMRoBERTa is a multilingual model trained in 100 different languages. Unlike some XLM multilingual models, it does not require lang tensors to understand …


Benchmarking PathCLIP for Pathology Image Analysis

S Zheng, X Cui, Y Sun, J Li, H Li, Y Zhang, P Chen… - arXiv preprint arXiv …, 2024

… All these versions are trained using masked language modeling and next-word prediction techniques with training data sourced from publicly available repositories like Wikipedia, Common Crawl. The results of LLaMA demonstrate that cuttingedge …


Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

R Aiyappa, S Senthilmani, J An, H Kwak, YY Ahn - arXiv preprint arXiv:2403.00236, 2024

We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations …


BengaliLCP: A Dataset for Lexical Complexity Prediction in the Bengali Texts

N Ayman, MA Hossain, A Aziz, RU Faruqui, AN Chy - Proceedings of the 2024 Joint …, 2024

Encountering intricate or ambiguous terms within a sentence produces distress for the reader during comprehension. Lexical Complexity Prediction (LCP) deals with predicting the complexity score of a word or a phrase considering its context. This …


BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformer Models in Political Science Text

Transformer models such as BERT, RoBERTa, and DeBERTa have revolutionized the field of Natural Language Processing in recent years with substantial improvements in the contextual understanding of text. While political scientists have …


BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformers Models in Political Science Text


BERTicelli at HaSpeeDe 3: Fine-tuning and Cross-validating Large Language Models for Hate Speech Detection Leonardo Grotti¹², Patrick Quick¹ ¹Universiteit …

L Grotti¹ - EVALITA Proceedings of the Eighth Evaluation …, 2024

The present paper describes the results from the experiments carried out for the HaSpeeDe 3 shared task, an Italian-language Hate Speech (HS) detection task, at EVALITA 2023. Two BERT-based language models were selected: UmBERTO (cased) …


BERTs are Generative In-Context Learners

D Samuel - arXiv preprint arXiv:2406.04823, 2024

This paper explores the in-context learning capabilities of masked language models, challenging the common view that this ability does not 'emerge' in them. We present an embarrassingly simple inference technique that enables DeBERTa to operate as …


Beyond Lexical Boundaries: LLM-Generated Text Detection for Romanian Digital Libraries

M Nitu, M Dascalu - Future Internet, 2024

Machine-generated content reshapes the landscape of digital information; hence, ensuring the authenticity of texts within digital libraries has become a paramount concern. This work introduces a corpus of approximately 60 k Romanian documents …


Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

M Ochieng, V Gumma, S Sitaram, J Wang, K Ronen… - arXiv preprint arXiv …, 2024

The deployment of Large Language Models (LLMs) in real-world applications presents both opportunities and challenges, particularly in multilingual and code-mixed communication settings. This research evaluates the performance of seven leading …


Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data

A Lee, B Miranda, S Sundar, A Casasola, S Koyeyo

… Since many datasets are publicly available (eg Common Crawl, Wikipedia), data used to train new models may be curated from such … We evaluate on Pile-CC (Pile Common Crawl) and OpenWebText2 since those datasets align with intuitively …


BharatBhasaNet-A unified framework to identify Indian code mix Languages

S Dey, S Thakur, A Kandwal, R Kumar, S Dasgupta… - IEEE Access, 2024

In the rapidly globalizing digital communication sphere, the imperative for advanced multilingual text recognition and identification is increasingly evident. Contrasting the previous works, which were predominantly constrained to 2-3 languages, this …


Bi-directional GRU-Based Approach for Multi-Class Text Error Identification System

S Reeha, BV Nikith, GMP Reddy, PB Pati - 2024 IEEE 9th International Conference …, 2024

… The C4 dataset, utilized in this study, is an extensively cleaned and curated version of Common Crawl's web crawl corpus. Common Crawl is a publicly available dataset accessible at commoncrawl.org. and boasting over 700 million …


Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models

C Campanella, R van der Goot - arXiv preprint arXiv:2403.08046, 2024

Large language models (LLMs) have emerged as a useful technology for job matching, for both candidates and employers. Job matching is often based on a particular geographic location, such as a city or region. However, LLMs have known …


Bioeconomy firms and where to find them

… This dataset uses the open-source web repository CommonCrawl to identify German company websites and has proven to be a valuable database for spatial research. From this data, we identify bioeconomy firms using a combination of …


BioMedLM: A 2.7 B Parameter Language Model Trained On Biomedical Text

E Bolton, A Venigalla, M Yasunaga, D Hall, B Xiong… - arXiv preprint arXiv …, 2024

… The Pile contains multiple sub-corpora, including general content from Common Crawl and specialized content like PubMed, Github, and FreeLaw. This model serves as an important baseline for comparison with our domain-specific models; …


BIS Papers

I Aldasoro, S Doerr, L Gambacorta, S Notra, T Oliviero… - 2024

Generative artificial intelligence (gen AI) introduces novel opportunities to strengthen central banks’ cyber security but also presents new risks. We use data from a unique survey among cyber security experts at major central banks to shed …


Botlitica: A generative AI-based tool to assist journalists in navigating political propaganda campaigns

E Musi, EEG Aguilar, L Federico - Studies in Communication Sciences, 2024

The hype on generative AI has raised concerns about the spread of disinformation but also opened up new opportunities for hybrid journalism. The proliferation of political propaganda campaigns spread across digital media during election periods …


Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

S Sukhbaatar, O Golovneva, V Sharma, H Xu, XV Lin… - arXiv preprint arXiv …, 2024

We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts …


Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models

T Blevins, T Limisiewicz, S Gururangan, M Li, H Gonen… - arXiv preprint arXiv …, 2024

Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which …


Bridging large language model disparities: Skill tagging of multilingual educational content

Y Kwak, ZA Pardos - British Journal of Educational Technology, 2024

The adoption of large language models (LLMs) in education holds much promise. However, like many technological innovations before them, adoption and access can often be inequitable from the outset, creating more divides than they bridge. In …


Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking

EC Acikgoz, M Erdogan, D Yuret - arXiv preprint arXiv:2405.04685, 2024

… This dataset was generated by extracting content from 71 monthly snapshots of the internet via Common Crawl (CC). CulturaX contains version … However, the diversity of these datasets should be sufficient, including common crawl, book …


Building a Large Japanese Web Corpus for Large Language Models

N Okazaki, K Hattori, H Shota, H Iida, M Ohi, K Fujii… - arXiv preprint arXiv …, 2024

… The corpus developed in this study is made from 21 snapshots of Common Crawl (from CC-MAIN-2020-40 to CC-MAIN-2023-23), and the size of the corpus after cleaning is 173,350,375 pages and 312,093,428,689 characters. To confirm the …


Building Question-Answer Data Using Web Register Identification

A Eskelinen, A Myntti, E Henriksson, S Pyysalo… - Proceedings of the 2024 …, 2024

… (2015), is a corpus of Finnish collected between 2015 and 2016 from Common Crawl and by crawling .fi domains. As all of these resources are Common Crawl based, in the post-processing stage we make sure that duplicate QA pairs are …


By Senior Judge Stephanie Domitrovich and Judge Herbert B. Dixon Jr.

CJJ Federal Judiciary - Judges Journal, 2024

… For example, the datasets used to train ChatGPT 3 included text from the following sources: (1) Common Crawl, containing petabytes of data collected from the internet since 2008; (2) WebText2, text from webpages with highly ranked Reddit …


Bye-bye Bias: What to Consider When Training Generative AI Models on Subjective Marketing Metrics

… However, these models are trained on vast data sets scraped from the Internet, such as Common Crawl or LAION, that contain little information on the types of perceptual measures marketing is often interested in, such as perceived brand …


Cache Me If You Can: The Case For Retrieval Augmentation in Federated Learning

A Muhamed, P Thaker, MT Diab, V Smith - Privacy Regulation and Protection in Machine …

We propose retrieval augmentation (RA) as an enhancement to federated learning (FL) that can improve privacy protection and ensure regulatory compliance. FL, primarily designed for data privacy preservation, faces challenges with conventional …


CAM 2.0: End-to-End Open Domain Comparative Question Answering System

A Shallouf, H Herasimchyk, M Salnikov, RG Veliz…

… After extracting objects and aspects, we look for their matches in the Common Crawl text. It is also important to mention, that for the final … (lemmatized) corpus containing 14.3 billion English sentences from the Common Crawl. To retrieve …


Can ChatGPT learn Chinese or Swahili? Considering how large language models might act differently if trained in different languages.

… Thien Nguyen, a professor of computer science at the University of Oregon, looked at the CommonCrawl corpus, a selection of text scraped … Then, because only a small percentage of the CommonCrawl corpus is in Korean, the company …


Can ChatGPT Plan Your Retirement?: Generative AI andFinancial Advice

We identify some of the most pressing issues facing the adoption of large language models (LLMs) in practical settings, and propose a research agenda to reach the next technological inflection point in generative AI. We focus on three challenges …


Can cross-domain term extraction benefit from cross-lingual transfer and nested term labeling?

HTH Tran, M Martinc, A Repar, N Ljubešić, A Doucet… - Machine Learning, 2024

Automatic term extraction (ATE) is a natural language processing task that eases the effort of manually identifying terms from domain-specific corpora by providing a list of candidate terms. In this paper, we treat ATE as a sequence-labeling task and …


Cascaded transformer-based networks for wikipedia large-scale image-caption matching

N Messina, DA Coccomini, A Esuli, F Falchi - Multimedia Tools and Applications, 2024

… It uses an XLM-RoBERTa masked language model pre-trained on the CommonCrawl dataset and fine-tuned on the image URL-caption match classification task. The CLS token in output from XLM-RoBERTa is attached to a feed-forward …


Causality-driven multivariate stock movement forecasting

A Díaz Berenguer, Y Da, MN Bossa, MC Oveneke… - PloS one, 2024

Our study aims to investigate the interdependence between international stock markets and sentiments from financial news in stock forecasting. We adopt the Temporal Fusion Transformers (TFT) to incorporate intra and inter-market …


CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl

I Ilyankou, J Haworth, S Cavazzi - arXiv preprint arXiv:2405.11039, 2024

The Common Crawl (CC) corpus is the largest open web crawl dataset containing 9.5+ petabytes of data captured since 2008. The dataset is instrumental in training large language models, and as such it has been studied for (un)desirable content …


CCSUM: A large-scale and high-quality dataset for abstractive news summarization

… Accordingly, among 35 million CommonCrawl News articles, we identify pairs of articles about the same news story and use one article’s first sentence as the summary for the other article. To ensure high quality, we apply strict filters whose …


CERM: Context-aware Literature-based Discovery via Sentiment Analysis

JC Young, U Akujuobi - arXiv preprint arXiv:2402.01724, 2024

Driven by the abundance of biomedical publications, we introduce a sentiment analysis task to understand food-health relationship. Prior attempts to incorporate health into recipe recommendation and analysis systems have primarily focused on …


Challenge design roadmap

HJE Balderas, I Guyon, A Howard, W Reade, S Treguer - AI Competitions and …, 2023

Challenges can be seen as a type of game that motivates participants to solve serious tasks. As a result, competition organizers must develop effective game rules. However, these rules have multiple objectives beyond making the game enjoyable …


Chatbots in Airport Customer Service—Exploring Use Cases and Technology Acceptance

I Auer, S Schlögl, G Glowka - Future Internet, 2024

… Those pre-trained transformer systems like BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) have been trained on vast language corpora such as Wikipedia and Common Crawl …


ChatGPT Alternative Solutions: Large Language Models Survey

H Alipour, N Pendar, K Roy - arXiv preprint arXiv:2403.14469, 2024

In recent times, the grandeur of Large Language Models (LLMs) has not only shone in the realm of natural language processing but has also cast its brilliance across a vast array of applications. This remarkable display of LLM capabilities has ignited a …


ChatGPT and GPT-4: utilities in the legal sector

Artificial intelligence systems such as ChatGPT, the OpenAI chatbot, based on the language model family GPT (generative pre-trained transformers), as well as other solutions built on this technology and fine-tuned for specific tasks, have generated …


ChatGPT and Language Translation: A Small Case Study Evaluating English–Mandarin Translation

C Woodrum - International Conference on Human-Computer …, 2024

… has been informed speculation that it relies heavily on common crawl [13], a compendium of publicly available … common crawl. The \(15^{th}\) most common language, Indonesian, makes up less than 1% of the data set [14, 15]. It is important …


ChatGPT as the Marketplace of Ideas: Should Truth-Seeking Be the Goal of AI Content Governance?

J Zhang - arXiv preprint arXiv:2405.18636, 2024

As one of the most enduring metaphors within legal discourse, the marketplace of ideas has wielded considerable influence over the jurisprudential landscape for decades. A century after the inception of this theory, ChatGPT emerged as a …


ChatGPT in Cyber Onslaught and Fortification: Past, Present, and Future

M Gunda, V Manda, P Naradasu, S Mekala… - 2024 IEEE 9th International …, 2024

… GPT-1 was unveiled in 2018 and trained on a pair of data sets: over 11,000 books from Book Corpus and the enormous Common Crawl … With 1.5 billion features and a much more extensive and diversified training dataset comprising Common Crawl …


ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis

W Ahmed, M Saturno, R Rajjoub, AH Duey, B Zaidat… - European Spine Journal, 2024

… Furthermore, the largest dataset used to train both ChatGPT 3.5 and 4.0 is CommonCrawl, a platform that does not include PubMed and consists predominantly of open access articles, some of which may be of less rigorous if not …


ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis

A Buscemi, D Proverbio - arXiv preprint arXiv:2402.01715, 2024

… It relied on a dataset referred to as the Common Crawl [30], a publicly accessible repository that comprises billions of web pages, making it one of the most extensive text databases available. It is important to note that the choice of dataset significantly …


ChatGPT's applications in marketing: a topic modeling approach

W Tafesse, A Wien - Marketing Intelligence & Planning, 2024

Purpose ChatGPT is a versatile technology with practical use cases spanning many professional disciplines including marketing. Being a recent innovation, however, there is a lack of academic insight into its tangible applications in the marketing …


ChatGPT-3.5 can create and justify short Afrikaans poems: Implications for practice

CJ van Staden - Suid-Afrikaanse Tydskrif vir Natuurwetenskap en …, 2024

ChatGPT-3.5 can create poems but the implications thereof has not yet been sufficiently investigated. The purpose of this explorative case study was to determine to which extent ChatGPT-3.5 can create and justify Afrikaans poems. Because …


ChatQA: Building GPT-4 Level Conversational QA Models

Z Liu, W Ping, R Roy, P Xu, M Shoeybi, B Catanzaro - arXiv preprint arXiv …, 2024

… We collect 7k documents (average ∼1k words per document) from common crawl, which cover a wide range of domains. Each document is used for generation of a single conversational QA sample, which leads to a total of 7k multi-turn QA dialogues …


Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data

L Castro-Gonzalez, YL Chung, HR Kirk, J Francis… - arXiv preprint arXiv …, 2024

The field of machine learning has recently made significant progress in reducing the requirements for labelled training data when building new models. These `cheaper' learning techniques hold significant potential for the social sciences, where …


Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

X Du, Z Yu, S Gao, D Pan, Y Cheng, Z Ma, R Yuan… - arXiv preprint arXiv …, 2024

In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional …


CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems

S Rosenthal, A Sil, R Florian, S Roukos - arXiv preprint arXiv:2404.02103, 2024

… However, they use sentencelevel matching (by encoding sentences for semantic similarity comparisons) to retrieve up to top 7 documents from Common Crawl while avoiding exact matches as the abstractive dataset. In the extractive version, the …


CLASSLA-web: Comparable Web Corpora of South Slavic Languages Enriched with Linguistic and Genre Annotation

N Ljubešić, T Kuzman - arXiv preprint arXiv:2403.12721, 2024

This paper presents a collection of highly comparable web corpora of Slovenian, Croatian, Bosnian, Montenegrin, Serbian, Macedonian, and Bulgarian, covering thereby the whole spectrum of official languages in the South Slavic language space …


Cleaner Pretraining Corpus Curation with Neural Web Scraping

Z Xu, Z Liu, Y Yan, Z Liu, C Xiong, G Yu - arXiv preprint arXiv:2402.14652, 2024

… The web-crawled datasets, such as Common Crawl, have been widely used for pretraining, facilitating the development of language … htmlparser serves as the text pre-extraction tool for CommonCrawl6 WET file (containing pre-extracted text), while …


CLEARNESS: Coreference Resolution for Generating and Ranking Arguments Extracted from Debate Portals for Queries

… Refining Arg-CTRL with data from Common Crawl leads to a higher quality of generated arguments compared to using user discussions from Reddit comments. In the mentioned studies, new arguments are generated through the use of knowledge …


CLIP and the City: Addressing the Artificial Encoding of Cities in Multimodal Foundation Deep Learning Models

In this project, we propose and explore a computational pipeline to examine urban cultural landscapes through the lens of artificial intelligence, and for questioning modes of embedding culture in machine learning models. By employing machine …


Co-constructing AI Authoring Using Ethical Theories

B Jones - 88th Annual International Conference

We must show ethical humility, historically contextualizing ethical standards, as we implement AI tools in courses. The pace of change requires agile approaches to defining standards for the ethical use of AI. We invited students to co-construct a …


CoastTerm: a Corpus for Multidisciplinary Term Extraction in Coastal Scientific Literature

J Delaunay, HTH Tran, CE González-Gallardo… - arXiv preprint arXiv …, 2024

… – Multilingual pre-trained model: We opt for XLMR [9], a transformer-based model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. This multilingual version of RoBERTa, achieves benchmark performance in ATE for rich-resourced …


Comparative Study on Synthetic and Natural Error Analysis with BART & MarianMT

R Rohit, SA Gandheesh, GS Sannala, PB Pati - 2024 IEEE 9th International …, 2024

Text is essential for communication, information sharing, knowledge acquisition, and analysis. It shapes public opinion, supports education, and drives online content, making it crucial in various domains. While there are various language models …


Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

H Bansal, A Suvarna, G Bhatt, N Peng, KW Chang… - arXiv preprint arXiv …, 2024

A common technique for aligning large language models (LLMs) relies on acquiring human preferences by comparing multiple generations conditioned on a fixed context. This only leverages the pairwise comparisons when the generations are …


Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models

Z Zhao, N Aletras - arXiv preprint arXiv:2403.12809, 2024

In many real natural language processing application scenarios, practitioners not only aim to maximize predictive performance but also seek faithful explanations for the model predictions. Rationales and importance distribution given by feature …


Comparing Foundations: Insights into the Construction of Financial Causal Knowledge Graphs with and without Ontology

Z Xu, R Ichise - 人工知能学会全国大会論文集 第 38 回 (2024), 2024

Simplicity in information navigation, interpretation, and reasoning has positioned mono-relation knowledge graphs (KGs) as a focus of attention, particularly in targeted scenarios. However, their deficiency in hierarchical interactions between …


Comparing generative and retrieval-based chatbots in answering patient questions regarding age-related macular degeneration and diabetic retinopathy

KX Cheong, C Zhang, TE Tan, BJ Fenner, WM Wong… - British Journal of …, 2024

… ChatGPT was trained on multiple extremely large datasets, of which Common Crawl (open repository of web crawl data) constituted the bulk of the datasets.30 31 Google Bard was trained on Infiniset (data comprising public forum dialogue …


Compass: Large Multilingual Language Model for South-east Asia

S Maria - arXiv preprint arXiv:2404.09220, 2024

… We categorized these sources into seven types: CommonCrawl, C4, Wikipedia, WebText, Academic, Books, and Code. We upsampled and … We expect that the introduction of biases may originate from CommonCrawl, as this substantial dataset …


Compilation of a Synthetic Judeo-French Corpus

I Nikolova-Stoupak, G Lejeune, E Schaeffer-Lacroix - Proceedings of the 8th Joint …, 2024

This is a short paper describing the process of derivation of synthetic Judeo-French text. Judeo-French is one of a number of rare languages used in speaking and writing by Jewish communities as confined to a particular temporal and …


Compositional Text-to-Image Generation with Dense Blob Representations

W Nie, S Liu, M Mardani, C Liu, B Eckart, A Vahdat - arXiv preprint arXiv:2405.08246, 2024

Existing text-to-image models struggle to follow complex text prompts, raising the need for extra grounding inputs for better controllability. In this work, we propose to decompose a scene into visual primitives - denoted as dense blob representations …


Comprehensive analysis of natural language processing

RK Yadav, A Madaan - Global Journal of Engineering and Technology …, 2024

… Huge datasets like the Penn Treebank and the Common Crawl drove the development of more sophisticated NLP models. Transfer learning techniques like BERT enabled finetuning of pre-trained models for specific tasks. TensorFlow and …


COMPREHENSIVE STUDY OF CLINICAL ENTITY EXTRACTION AND CLASSIFICATION USING LARGE LANGUAGE MODELS

M Faedi, P Torroni, DA Galassi, DG Grundler…

This project aims to evaluate various techniques for extraction and categorization of clinical terms in unstructured documents. The purpose of this study is to assist automated systems operating in the biomedical domain by highlighting relevant …


Compressing Lengthy Context With UltraGist

P Zhang, Z Liu, S Xiao, N Shao, Q Ye, Z Dou - arXiv preprint arXiv:2405.16635, 2024

Compressing lengthy context is a critical but technically challenging problem. In this paper, we propose a new method called UltraGist, which is distinguished for its high-quality compression of lengthy context due to the innovative design of the compression and …


Compression Represents Intelligence Linearly

Y Huang, J Zhang, Z Shan, J He - arXiv preprint arXiv:2404.09937, 2024

… Concretely, for assessing knowledge and commonsense, we have compiled texts from the latest Common Crawl dataset. To evaluate coding ability, we have sourced data from GitHub repositories mainly on the Python language since the downstream …


Computational criminology: at-scale quantitative analysis of the evolution of cybercrime forums

Cybercrime forums and marketplaces are used by members to share hacking techniques, general community-building discussions, and trade hacking tools. While there is a large corpus of literature studying these platforms, from a cross-forum …


Computerized diagnostic decision support systems–a comparative performance study of Isabel Pro vs. ChatGPT4

… ChatGPT4 is trained on Common Crawl, a publicly available dataset and one of the most extensive text datasets. While not trained explicitly for medical diagnosis, ChatGPT4 accesses medical textbooks, medical websites, medical papers, and …


Comquest: an Adaptive Crawler for User Comments on the Web

This thesis introduces Comquest, an adaptive framework designed for the large-scale collection and integration of user comments from the Web. User comments are featured on many websites and there is growing interest in mining and studying user …


Comquest: Large Scale User Comment Crawling and Integration

Z Chen, L He, A Mukherjee, E Dragut - 2024

… To train the model, we collect an extensive dataset from the Common Crawl Project. We filter the webpages that show apparent code signals (eg, loading the commenting system libraries) of the supported commenting systems. …


Considering the Role of Fairness in Copyright Fair Use

Moral intuitions regarding when it is "fair" to engage in unauthorized copying or derivation from a copyrighted work may often take into account factors that the fair use statute, 17 USC § 107, does not enumerate among fair use considerations. This …


Construction of Text Summarization Corpus in Economics Domain and Baseline Models

S Jumpathong, A Takhom, P Boonkwan… - 2024

… It was pre-trained with a data collection of 101 languages from Common Crawl Due to time and resource constraints, we chose mT5-small… It was pre-trained with a data collection of 101 languages from Common Crawl. Due to time and resource …


Context-aware Transliteration of Romanized South Asian Languages

C Kirov, C Johny, A Katanova, A Gutkin, B Roark - Computational Linguistics, 2024

While most transliteration research is focused on single tokens such as named entities – eg, transliteration of “અમદાવાદ” from the Gujarati script to the Latin script “Ahmedabad” – the informal romanization prevalent in South Asia and elsewhere …


Contextual Chart Generation for Cyber Deception

DD Nguyen, D Liebowitz, S Nepal, SS Kanhere… - arXiv preprint arXiv …, 2024

… (1) HPN-T5 is based on Text-to-Text Transfer Transformer (T5) [42], a highly scalable sequence-to-sequence Transformer pre-trained on a large corpus harvested from Common Crawl. T5 introduced a novel pre-training objective that …


Contextual Word Embedding for Biomedical Knowledge Extraction: a Rapid Review and Case Study

D Vithanage, P Yu, L Wang, C Deng - Journal of Healthcare Informatics Research, 2024

Recent advancements in natural language processing (NLP), particularly contextual word embedding models, have improved knowledge extraction from biomedical and healthcare texts. However, limited comprehensive research compares these models …


Contextually Enriched Meta-Learning Ensemble Model for Urdu Sentiment Analysis Symmetry 2023, 15, 645

K Ahmed, MI Nadeem, D Li, Z Zheng, N Al-Kahtani… - 2023

… On the other hand, this model makes use of pre-trained word vectors that were trained on “common crawl” and “Wikipedia” through the use of the fastText model [82]. The CBOW algorithm with position-weighting is used to train this word vector. After …


Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

K Fujii, T Nakamura, M Loem, H Iida, M Ohi, K Hattori… - arXiv preprint arXiv …, 2024

Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an …


Controllable Sentence Simplification in Dutch

T Seidl, V Vandeghinste - Computational Linguistics in the Netherlands Journal, 2024

Text simplification aims to reduce complexity in vocabulary and syntax, enhancing the readability and comprehension of text. This paper presents a supervised sentence simplification approach for Dutch using a pre-trained large language …


Controllable Sentence Simplification in Swedish Using Control Prefixes and Mined Paraphrases

J Monsen, A Jönsson - Proceedings of the 2024 Joint International Conference …, 2024

… 2020), an opensource repository with tools to download and clean Common Crawl snapshots. Each file in the CC-100 corpus contains … The Common Crawl snapshots contain a large portion of low-quality data, including offensive and …


Copilot for Microsoft 365: A Comprehensive End-user Training Plan for Organizations

This thesis presents a comprehensive end user training plan for Copilot for Microsoft 365, a generative AI tool that integrates with Microsoft’s productivity suite of applications. The research is conducted within the context of Sulava Ltd., a Finnish …


Copyleft for Alleviating AIGC Copyright Dilemma: What-if Analysis, Public Perception and Implications

X Guo, Y Li, Y Peng, X Wei - arXiv preprint arXiv:2402.12216, 2024

As AIGC has impacted our society profoundly in the past years, ethical issues have received tremendous attention. The most urgent one is the AIGC copyright dilemma, which can immensely stifle the development of AIGC and greatly cost the entire …


CoRePooL—Corpus for Resource‐Poor Languages: Badaga Speech Corpus

HB Barathi Ganesh, G Jyothish Lal, R Jairam… - … Speech Recognition and …, 2024

This chapter presents a corpus named CoRePooL that stands for Corpus for Resource‐Poor Languages. As voice‐specific human‐machine interaction applications are accelerated by deep learning algorithms, the lack of resources …


Corpus Considerations for Annotator Modeling and Scaling

OO Sarumi, B Neuendorf, J Plepi, L Flek, J Schlötterer… - arXiv preprint arXiv …, 2024

Recent trends in natural language processing research and annotation tasks affirm a paradigm shift from the traditional reliance on a single ground truth to a focus on individual perspectives, particularly in subjective tasks. In scenarios where …


CorpusNÓS: A massive Galician corpus for training large language models

I de-Dios-Flores, SP Suárez, CC Pérez, DB Outeiriño… - Proceedings of the 16th …, 2024

CorpusNÓS is a massive Galician corpus made up of 2.1 B words primarily devised for training large language models. The corpus sources are varied and represent a relatively wide range of genres. CorpusNÓS is, to the best of our knowledge, the …


Counter (media) Intelligence and Visioning: An Interview with Adam Harvey

… Among the largest and most popular datasets used in AI research projects are Common Crawl (for NLP), and ImageNet and COCO (for computer vision), which are all derived from user-generated content. In my research project Exposing.ai, I’ve …


Creating Ad Campaigns Using Generative AI Check for updates

A Bulut, B Arslan - Applications of Generative AI

Search campaigns consist of ad groups. An ad group contains a related set of keywords and ads. During an online campaign, search advertisers experiment with different marketing messages such as subtle vs. strong being used in ad copies, with …


Creating Ad Campaigns Using Generative AI

A Bulut, B Arslan - Applications of Generative AI, 2024

… In particular, T5 and PEGASUS models were trained on 750 GB of English-language text from the public Common Crawl web scrape while the BART model was trained on the CNN/Daily Mail dataset, which contains roughly 300K unique news articles …


Creating Parallel Corpora for Ukrainian: a German-Ukrainian Parallel Corpus (ParaRook|| DE-UK)

M Shvedova, A Lukashevskyi - Proceedings of the Third Ukrainian Natural Language …, 2024

Parallel corpora are currently a popular and vibrantly developing category of linguistic resources, used both in literature and translation studies, as well as in the field of NLP. For Ukrainian, though, there are still not enough significant parallel …


CroissantLLM: A Truly Bilingual French-English Language Model

M Faysse, P Fernandes, N Guerreiro, A Loison, D Alves… - arXiv preprint arXiv …, 2024

We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local …


Cross-cultural Inspiration Detection and Analysis in Real and LLM-generated Social Media Data

O Ignat, GG Lakshmy, R Mihalcea - arXiv preprint arXiv:2404.12933, 2024

Inspiration is linked to various positive outcomes, such as increased creativity, productivity, and happiness. Although inspiration has great potential, there has been limited effort toward identifying content that is inspiring, as opposed to just engaging …


Cross-Language Harmonization of Linguistic Resources

The presented work consists of two parts. In the first part I summarize the main directions of my research since the defense of my PhD thesis in 2005. I start with cross-language transfer of parsing models to languages that have little or no …


Cross-Lingual Cross-Modal Retrieval with Noise-Robust Fine-Tuning

R Cai, J Dong, T Liang, Y Liang, Y Wang, X Yang… - IEEE Transactions on …, 2024

… from snapshots of the CommonCrawl public dataset2. Experiments in this paper concern parallel data between English and other four languages, namely Chinese (ZH), French (FR), German (DE) and Czech (CZ). Evaluation metrics. … https://commoncrawl.org/ …


Cross-lingual Editing in Multilingual Language Models

H Beniwal, M Singh - arXiv preprint arXiv:2401.10521, 2024

The training of large language models (LLMs) necessitates substantial data and computational resources, and updating outdated LLMs entails significant efforts and resources. While numerous model editing techniques (METs) have emerged to …


Cyber Risks of Machine Translation Critical Errors: Arabic Mental Health Tweets as a Case Study

H Saadany, A Tantawy, C Orasan - arXiv preprint arXiv:2405.11668, 2024

With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like …


Data Augmentation and Large Language Model for Legal Case Retrieval and Entailment

MQ Bui, DT Do, NK Le, DH Nguyen, KVH Nguyen… - The Review of Socionetwork …, 2024

The Competition on Legal Information Extraction and Entailment (COLIEE) is a well-known international competition organized each year with the goal of applying machine learning algorithms and techniques in the analysis and understanding of legal …


Data augmentation for language generation inspired by machine translation

The field of natural language processing has witnessed a surge in the adoption of deep learning, which faces notable hurdles when the training data is scarce. This thesis aims to study automatic data augmentation for language generation tasks …


Data Augmentation with Semantic Enrichment for Deep Learning Invoice Text Classification

WW Chi, TY Tang, NM Salleh, M Mukred, H AlSalman… - IEEE Access, 2024

Natural language processing (NLP) is a research field that provides huge potential to automate accounting tasks dealing with text data. This research studies the application of NLP in automatically categorizing invoices based on the invoice text …


Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

S Longpre, R Mahari, N Obeng-Marnu, W Brannon… - arXiv preprint arXiv …, 2024

… Common Crawl is accurate with wide coverage, but not detailed. Hugging Face can be inaccurate with varying levels of detail, but it has extensive coverage. The Data Provenance Initiative is highly accurate and detailed but is currently limited in scope. …


Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus

G de Jesus, SS Nunes - Proceedings of the 2024 Joint International Conference …, 2024

This paper proposes Labadain Crawler, a data collection pipeline tailored to automate and optimize the process of constructing textual corpora from the web, with a specific target to low-resource languages. The system is built on top of Nutch, an …


Data Engineering for Scaling Language Models to 128K Context

Y Fu, R Panda, X Niu, X Yue, H Hajishirzi, Y Kim… - arXiv preprint arXiv …, 2024

We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular \textit{the ability to utilize information at arbitrary input …


Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining

C Ge, Z Ma, D Chen, Y Li, B Ding - arXiv preprint arXiv:2405.14908, 2024

Large language models exhibit exceptional generalization capabilities, primarily attributed to the utilization of diversely sourced data. However, conventional practices in integrating this diverse data heavily rely on heuristic schemes, lacking …


Data-Centric Methods for Decentralizing Large Language Models

… from large web crawl services like Common Crawl. Web crawl is perceived to be wholly representative of activity on the Internet [Brügger… Indeed, web dumps like Common Crawl offer the promise of more diverse text than what is available in …


DataComp Challenge

… billion sample image-text dataset filtered from a comparable pool of samples to Common Crawl. For the creation of the LAION-2B dataset, first language filtering is applied to the original Common Crawl pool so that only samples with English texts …


Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

H Pouransari, CL Li, JHR Chang, PKA Vasu, C Koc… - arXiv preprint arXiv …, 2024

… Although the benefits of ICLM with large-scale Common Crawl data (used in our experiments) are marginal in regular evaluation, it significantly boosts long-context evaluation metrics from 27.5 to 28.7 by constructing long data sequences through …


Dataset Growth

Z Qin, Z Xu, Y Zhou, Z Zheng, Z Cheng, H Tang… - arXiv preprint arXiv …, 2024

Deep learning benefits from the growing abundance of available data. Meanwhile, efficiently dealing with the growing data scale has become a challenge. Data publicly available are from different sources with various qualities, and it is …


Datasets for Large Language Models: A Comprehensive Survey

Y Liu, J Cao, C Liu, K Ding, L Jin - arXiv preprint arXiv:2402.18041, 2024

… The first method involves building upon Common Crawl1. Common Crawl is a massive, unstructured, multilingual web corpus that provides public access to web archives by regularly crawling and storing webpage data from the Internet. However …


Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

H Park, S Lee, G Gim, Y Kim, D Kim, C Park - arXiv preprint arXiv:2403.19340, 2024

To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of …


Dated Data: Tracing Knowledge Cutoffs in Large Language Models

J Cheng, M Marone, O Weller, D Lawrie, D Khashabi… - arXiv preprint arXiv …, 2024

… Our analysis reveals two reasons for these inconsistencies: (1) temporal biases of CommonCrawl data due to non-trivial amounts of old data in new dumps and (2) complications in LLM deduplication schemes involving semantic duplicates and …


DE-COP: Detecting Copyrighted Content in Language Models Training Data

AV Duarte, X Zhao, AL Oliveira, L Li - arXiv preprint arXiv:2402.09910, 2024

How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim …


Dear GPT-3: Collaborative Writing with Neural Networks

J Becker - Artificial Intelligence-Intelligent Art?: Human-Machine …, 2024

GPT-3 and I engaged in this dialogue about four months ago. I wanted to develop a collaborative writing project, not only to collectively realise a novel, but also to negotiate poetological questions which might arise during the process. These …


DeBERTa-BiLSTM: A multi-label classification model of arabic medical questions using pre-trained models and deep learning

BS Al-Smadi - Computers in Biology and Medicine, 2024

… Using data from CommonCrawl in 100 different languages, the authors in [38] introduce XLM-RoBERTa, an advanced masked multilingual language model. XLM-RoBERTa outperformed previous multilingual models such as XLM and mBERT. [39] …


Deconstructing In-Context Learning: Understanding Prompts via Corruption

N Shivagunde, V Lialin, S Muckatira, A Rumshisky - arXiv preprint arXiv:2404.02054, 2024

The ability of large language models (LLMs) to "learn in context" based on the provided prompt has led to an explosive growth in their use, culminating in the proliferation of AI assistants such as ChatGPT, Claude, and Bard. These AI …


Deep learning aided clinical decision support

Medical professionals create vast amounts of clinical texts during patient care. Often, these documents describe medical cases from anamnesis to the final clinical outcome. Automated understanding and selection of relevant medical records pose …


Deep Learning Based Multi-document Summarization

In this era of rapidly advancing technology, the exponential increase of data availability makes analyzing and understanding text files a tedious, labor-intensive, and time-consuming task. Multi-document summarization (MDS) is an effective tool …