Data augmentation and generation for natural language processing

Natural Language Processing (NLP) is a rapidly growing field that deals with the interaction between computers and human language. One of the major challenges in NLP is dealing with low-resource scenarios, where limited amounts of labeled …


Data Cartography for Low-Resource Neural Machine Translation

A Richburg, M Carpuat - Findings of the Association for Computational …, 2022

While collecting or generating more parallel data is necessary to improve machine translation (MT) in low-resource settings, we lack an understanding of how the limited amounts of existing data are actually used to help guide the collection of …


Data Dialogue with ChatGPT: Using Code Interpreter to Simulate and Analyse Experimental Data

A Low, ZY Kalender - arXiv preprint arXiv:2311.12415, 2023

Artificial intelligence (AI) has the potential to revolutionise many aspects of physics education, including introductory laboratory courses. In this study, we explored the capability of OpenAI's ChatGPT-4 to interpret and complete an introductory …


Data Distillation: A Survey

N Sachdeva, J McAuley - arXiv preprint arXiv:2301.04272, 2023

… Textual data is available in large amounts from sources like websites, news articles, academic manuscripts, etc., and is also readily accessible with datasets like the common crawl1 which sizes up to almost 541TB. Furthermore, with the advent of …


Data Filtering Networks

A Fang, AM Jose, A Jain, L Schmidt, A Toshev… - arXiv preprint arXiv …, 2023

… ), and gradually replace it with unfiltered data from Common Crawl until this pool only contains Common Crawl. We train DFNs on these data mixes, and use these DFNs to CLIP filter a separate pool of 128 million Common Crawl samples from …


Data Management For Large Language Models: A Survey

Z Wang, W Zhong, Y Wang, Q Zhu, F Mi, B Wang… - arXiv preprint arXiv …, 2023

Data plays a fundamental role in the training of Large Language Models (LLMs). Effective data management, particularly in the formulation of a well-suited training dataset, holds significance for enhancing model performance and improving training …


Data Portraits: Recording Foundation Model Training Data

M Marone, B Van Durme - arXiv preprint arXiv:2303.03919, 2023

Foundation models are trained on increasingly immense and opaque datasets. Even while these models are now key in AI system building, it can be difficult to answer the straightforward question: has the model already encountered a given …


Data Pruning for Efficient Model Pruning in Neural Machine Translation

A Azeemi, I Qazi, A Raza - Findings of the Association for Computational …, 2023

… We use the T5-small variant pre-trained on the 750 GB C4 dataset containing text from the public web scrape of the common crawl. This variant has 60 million parameters, 6 layers in the encoder and decoder each, and 8-headed attention. …


Data Selection for Language Models via Importance Resampling

SM Xie, S Santurkar, T Ma, P Liang - arXiv preprint arXiv:2302.03169, 2023

Selecting a suitable training dataset is crucial for both general-domain (eg, GPT-3) and domain-specific (eg, Codex) language models (LMs). We formalize this data selection problem as selecting a subset of a large raw unlabeled dataset to match a …


Data-Centric Debugging: mitigating model failures via targeted image retrieval

S Singla, AM Chegini, M Moayeri, S Feizi - Proceedings of the IEEE/CVF Winter …, 2024

Deep neural networks can be unreliable in the real world when the training set does not adequately cover all the settings where they are deployed. Focusing on image classification, we consider the setting where we have an error distribution E …


Data-Driven Approach to Automated Hypernym Hierarchy Construction for the Ukrainian WordNet

… The authors used publicly available data sources such as web pages scraped by CommonCrawl (67%), open source code repositories from GitHub, Wikipedia in 20 different languages, including Ukrainian, public domain books from Project …


DATA-EFFICIENT LEARNING FOR HEALTHCARE QUERIES IN LOW-RESOURCE AND CODE MIXED LANGUAGE SETTINGS

S Mwongela, J Patel, S Rajasekharan, L Wotton…

… MT5 is a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages (Xue et al., 2020). … It is pre-trained on 2.5TB of filtered Common Crawl data containing 100 languages. It was introduced in the …


Data-Juicer: A One-Stop Data Processing System for Large Language Models

D Chen, Y Huang, Z Ma, H Chen, X Pan, C Ge, D Gao… - arXiv preprint arXiv …, 2023

… As an illustrative example, we describe our text quality classifier for culling high-quality text from heterogeneous data sources like CommonCrawl. This tool is a reproduced model based on the closed-source GPT-3 quality scorer [10]. Moreover, we have …


Data: A Critical Element for AI

T Taulli - ChatGPT and Bard for Business Automation: Achieving …, 2023

… CommonCrawl News: This is a dataset of news stories from websites across the globe. …


Data: The Fuel for Generative AI

T Taulli - Generative AI: How ChatGPT and Other AI Tools Will …, 2023

A large language model (LLM) processes huge amounts of data for its generative AI systems. They are on the scale of petabytes. Consider that a petabyte is 1000 terabytes. This would hold about 500 billion pages of standard text. No doubt, the …


Databases, Search Engines

BVV Martens - Keywords In and Out of Context, 2023

… (2019) at OpenAI and pre-trained with the Common Crawl dataset, forms the current backbone of the Bing search engine. …


DataComp: In search of the next generation of multimodal datasets

S Yitzhak Gadre, G Ilharco, A Fang, J Hayase… - arXiv e-prints, 2023

SY Gadre, G Ilharco, A Fang, J Hayase, G Smyrnis… - arXiv preprint arXiv …, 2023

… We provide a testbed for dataset experiments centered around a new candidate pool of 12.8B image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their …


Dataset Factory: A Toolchain For Generative Computer Vision Datasets

D Kharitonov, R Turner - arXiv preprint arXiv:2309.11608, 2023

… As an illustrative example, the prestigious DataComp competition built around the Common Crawl data features four scale tracks (12.8M, 128M, 1.28B, and 12.8B samples [3]), but in the first few months, the leaderboard for the top two tracks have …


Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data

Y Yang, AK Singh, M Elhoushi, A Mahmoud… - arXiv preprint arXiv …, 2023

… In this race for higher performance, some training datasets have swelled to petabyte size, sourced from extensive repositories like the Common Crawl. While significant effort has gone into optimizing the computational aspects of training LLMs …


Decomposing geographical judgments into spatial, temporal and linguistic components

D Gatti, G Anceresi, M Marelli, T Vecchi, L Rinaldi - 2023

When mentally exploring large-scale maps, humans are assumed to mainly rely on spatial information derived from direct perceptual experience (eg, prior visual experience with the geographical map itself). In the present study, we rather tested …


Deep Learning Algorithms for Cybersecurity Applications

S Takale, S Pawar, V Khot, VS Gutte, A Acharya

… Around one million legitimate and phishing URLs were used on the dataset collected from PhishTank and Common Crawl. To build the IPDS, the LSTM and CNN classifier used over 10,000 images and one million URLs for training. The …


Deep learning and low-resource languages: How much data is enough? A case study of three linguistically distinct South African languages

… data like Common Crawl or Wikipedia, which have some inherent limitations (as discussed later in this paper), but also have limited availability for South African languages. For instance, isiNdebele has no Wikipedia data and is therefore not …


Deep Learning Benchmark Studies on an Advanced AI Engineering Testbed from the Open Compass Project

MY Wang, J Uran, P Buitrago - Practice and Experience in Advanced Research …, 2023

We present the Open Compass project’s pilot deep learning benchmark results with various AI accelerators. Those accelerators are NVIDIA V-100 and A-100, AMD MI100, as well as emerging novel accelerators such as Cerebras CS-2 and …


Deep learning for clinical texts in low-data regimes

… noticed that FastText embeddings trained on Wikipedia performed worse than GloVe embeddings trained on the Common Crawl corpus. … Third, RoBERTa is trained on an about ten times larger dataset, which includes the Common Crawl-News …


Deep Learning Model for Automated Trainee Assessment During High-Fidelity Simulation

Problem: Implementation of competency-based medical education has necessitated more frequent trainee assessments. Use of simulation as an assessment tool is limited by access to trained examiners, cost, and concerns with interrater reliability …


Deep learning model for Mongolian Citizens Feedback Analysis using Word Vector Embeddings

Z Dashdorj, T Munkhbayar, S Grigorev - arXiv preprint arXiv:2302.12069, 2023

… The vector model of words taught in Mongolian text on Wikipedia and Common Crawl, is 300-dimensional and contains 600,000 Mongolian words. The model was trained using CBOW with position weights with character n-grams of length 5, a …


Deep Learning Transformer Models for Building a Comprehensive and Real-time Trauma Observatory: Development and Validation Study

G Chenais, C Gil-Jardiné, H Touchais, MA Fernandez… - JMIR AI, 2023

… BelGPT2 was the first GPT-2 model fine-tuned on a French heterogeneous corpus (CommonCrawl, French Wikipedia, and EuroParl) released on the Hugging Face platform. The self-supervised training of transformers in a specific domain can …


Deep Learning-based POS Tagger and Chunker for Odia Language Using Pre-trained Transformers

T Dalai, TK Mishra, PK Sa - ACM Transactions on Asian and Low-Resource …, 2023

Developing effective natural language processing (NLP) tools for low-resourced languages poses significant challenges. This article centers its attention on the task of Part-of-speech (POS) tagging and chunking, which pertains to the identification …


Deepfake Detection on Social Media: Leveraging Deep Learning and FastText Embeddings for Identifying Machine-Generated Tweets

S Sadiq, T Aljrees, S Ullah - IEEE Access, 2023

… 2) FastText Subword FastText Subword has 2 million word vectors that were learned using Common Crawl’s subword information (600B tokens). By breaking down each word into its component words, subword embedding gives us more …


Defeasible Reasoning with Large Language Models–Initial Experiments and Future Directions

As Large Language Models gain prominence in the AI landscape, it is essential to understand their capabilities and limitations, among others in terms of reasoning. This paper is a first step towards understanding the capabilities in terms of …


Demystifying CLIP Data

H Xu, S Xie, XE Tan, PY Huang, R Howes, V Sharma… - arXiv preprint arXiv …, 2023

Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP …


Dense Information Retrieval on a Latin digital library via LaBSE and LatinBERT embeddings

FA Galatolo, G Martino, MGCA Cimino, CO Tommasi

… The authors also demonstrated the effectiveness of the LaBSE model by mining parallel data from CommonCrawl repository and using it to train competitive Neural Machine Translation (NMT) models for English-Chinese and English-German. One …


Densest Subhypergraph: Negative Supermodular Functions and Strongly Localized Methods

Y Huang, DF Gleich, N Veldt - arXiv preprint arXiv:2310.13792, 2023

Dense subgraph discovery is a fundamental primitive in graph and hypergraph analysis which among other applications has been used for real-time story detection on social media and improving access to data stores of social networking systems …


Deobfuscating Leetspeak With Deep Learning to Improve Spam Filtering

I Vélez de Mendizabal, X Vidriales, V Basto-Fernandes… - 2023

The evolution of anti-spam filters has forced spammers to make greater efforts to bypass filters in order to distribute content over networks. The distribution of content encoded in images or the use of Leetspeak are concrete and clear examples of …


DEPARTMENT OF COMPUTER ENGINEERING

Introduction to Web. Information Retrieval and Web Search.: Basic Concepts of Information Retrieval, Information Retrieval Methods–Boolean Model, Vector Space Model and Statistical Language Model, Relevance Feedback, Evaluation Measures …


Depression clinical detection model based on social media: a federated deep learning approach

… Specifically, a pre-processed Common Crawl dataset of more than 2TB based on 100 languages is used to train cross-language representations in a self-supervised manner. This includes generating new unlabeled corpora for low-resource …


Design and Data Mining Techniques for Large-Scale Scholarly Digital Libraries and Search Engines

The exponential growth of digital libraries and the proliferation of scholarly content in electronic formats have made data mining and information retrieval essential tools for effectively managing, organizing, and disseminating knowledge. This thesis …


Designing a Multilingual Fact-Checking Dataset from Existing Question-Answering Data

… To create a dataset, Common Crawl was used as a source of text. Due to the scraped text data a significant portion of it is not natural language. It consists of gibberish, duplicate text, menus, error messages, and other non-useful content. To …


Designing with Language: Wireframing UI Design Intent with Generative Large Language Models

S Feng, M Yuan, J Chen, Z Xing, C Chen - arXiv preprint arXiv:2312.07755, 2023

… Since the training samples of LLMs are typically scraped from the raw web page data, eg, GPT was trained on 410 billion tokens from the Common Crawl web corpus, we use HTML syntax as the domain-specific language to convert the UI screens into …


Detecting Human Emotion by Text Classification.

U Brunda, P Akhilesh, K Kalaiselvi - Grenze International Journal of Engineering & …, 2023

… Using more than two terabytes of filtered CommonCrawl data, we train a Transformer-based masked language model on 100 different languages. On a number of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13 …


Detecting Human Rights Violations on Social Media during Russia-Ukraine War

P Nemkova, S Ubani, SO Polat, N Kim, RD Nielsen - arXiv preprint arXiv:2306.05370, 2023

… We also evaluated XLM-RoBERTa20, a transformer model pretrained on a large corpus in a self-supervised fashion on 2.5TB of filtered CommonCrawl data spanning 100 languages [12]. Finally, we also tried RuBERT, a BERT-based model …


Detecting Kids Cyberbullying Using Transfer Learning Approach: Transformer Fine-Tuning Models

WMS Yafooz, A Al-Dhaqm, A Alsaeedi - Kids Cybersecurity Using Computational …, 2023

… XLMRoBERTa: This is a pertained model trained on 2.5 TB of text collected from Wikipedia and Common Crawl data of 100 languages including the Arabic language [38]. It is developed by Facebook and it contains a vocabulary size of 250 k. It has …


Detecting Modern Maldocs: An Analytical Study of PDF Feature Importance Using KDD Cup 99 as the Process Framework

… For instance, the Govdocs1 PDF corpus (Govdocs1 – Digital Corpora, nd) was the seminal corpus that laid the groundwork for future web crawls, and Common Crawl (Common Crawl - Open Repository of Web Crawl Data, nd) and SafeDocs (Allison et …


Detecting Natural Language Biases with Prompt-based Learning

MA Aowal, MT Islam, PM Mammen, S Shetty - arXiv preprint arXiv:2309.05227, 2023

In this project, we want to explore the newly emerging field of prompt engineering and apply it to the downstream task of detecting LM biases. More concretely, we explore how to design prompts that can indicate 4 different types of biases: (1) …


Detecting Personal Information in Training Corpora: an Analysis

N Subramani, AS Luccioni, J Dodge, M Mitchell

… This corpus consists of text taken from Common Crawl then passed through a number of filters with the intention of retaining high-quality English text The C4-en validation set of the C4 dataset that we analyzed was created by taking the April …


Detecting Phishing URLs through Deep Learning Models

S Noor, SU Bazai, S Tareen, S Ullah - … Applications: Volume One: Image Security and …, 2024

… Kaggle Common Crawl, Common Crawl 2 Phish Tank, Alexa … PhishTank, ALEXA, and Common Crawl data sets are mostly used by authors for training models. For evaluating the model, the authors mostly used accuracy, precision, recall, and F1-score …


Detecting Salient Antimetaboles in English Texts Using Deep and Transfer Learning

… In our case, we opted for GloVe24 in its Common Crawl (48B) version as our embedding software. We decided to use it because of its high result in other NLP tasks while being easy to access and use directly thanks to the existence of several …


Detection and Mitigation of Gender Bias in Natural Language Processing

This thesis contributes to our collective understanding of how gender bias arises in natural language processing systems, provides new detection and measurement tools, and develops mitigation methods. More specifically, we quantify and reduce …


Detection of Privacy-Harming Social Media Posts in Italian

F Peiretti, RG Pensa - International Symposium on Security and Privacy in …, 2023

As many psychological and sociological study reveal, many people disclose too much privacy-harming information in social media in the form of text and multimedia posts, thus exposing themselves and other persons to several security risks …


Detection of Propagandistic Techniques in Text using Pretrained Language Models

… mC4 consists of natural text in 101 languages extracted from the public Common Crawl web scrape. However, pre-trained generative multilingual language models can exhibit problematic behavior in the zero-shot setting. They may translate a …


Detection of Truthful, Semi-Truthful, False and Other News with Arbitrary Topics Using BERT-Based Models

E Shushkevich, J Cardiff, A Boldyreva - 2023 33rd Conference of Open Innovations …, 2023

Easy and uncontrolled access to the Internet provokes the wide propagation of false information, which freely circulates in the Internet. Researchers usually solve the problem of fake news detection (FND) in the framework of a known topic and binary …


Determining sentiment views of verbal multiword expressions using linguistic features

M Wiegand, M Schulder, J Ruppenhofer - Natural Language Engineering, 2023

We examine the binary classification of sentiment views for verbal multiword expressions (MWEs). Sentiment views denote the perspective of the holder of some opinion. We distinguish between MWEs conveying the view of the speaker of the …


DeTexD: A Benchmark Dataset for Delicate Text Detection

A Chernodub, S Yavnyi, O Sliusarenko, J Razzaghi… - The 7th Workshop on Online …, 2023

… For our DeTexD Benchmark dataset (Table 3), we extracted data from various websites in CommonCrawl5, where we specifically targeted news websites, forums discussing sensitive topics (eg, Mental Health Forum6, and able2know7 which …


Developing a Flexible System for a Friendly Robot to Ease Dementia (FRED) Using Cloud Technologies and Software Design Patterns

In this work, we designed two prototypes for a friendly robot to ease dementia (FRED). This affordable social robot is designed to provide company to older adults with cognitive decline, create reminders for important events and tasks, like taking …


Developing a Named Entity Recognition Dataset for Tagalog

LJV Miranda - arXiv preprint arXiv:2311.07161, 2023

We present the development of a Named Entity Recognition (NER) dataset for Tagalog. This corpus helps fill the resource gap present in Philippine languages today, where NER resources are scarce. The texts were obtained from a pretraining …


Developing an Incivility Dictionary for German Online Discussions–a Semi-Automated Approach Combining Human and Artificial Knowledge

A Stoll, L Wilms, M Ziegele - Communication Methods and Measures, 2023

… of Common Crawl and Wikipedia, consisting of over 67 billion tokens (Bojanowski et al., 2017; Grave et al., 2018). The common crawl project … In addition to common crawl, the German Wikipedia dump 15 was considered to train the embedding …


Developing Hybrid Model for Analyzing Sentiment of Textual Data for Amazon Product Reviews Using Deep Learning

MAM Ali, DRSN Lokhande, SAS Alshaibani - Harbin Gongcheng Daxue Xuebao …, 2023

… Common Crawl and Wikipedia, has also contributed to the progress of NLP [8]. These resources enable researchers and practitioners to leverage transfer learning techniques, where models pre-trained on vast amounts of unlabeled text can be fine-tuned …


Development of a web service for creating tests based on text analysis using natural language processing technologies

TA Vakaliuk, OV Chyzhmotria, SO Didkivska, I Linevych - International Journal of …, 2023

… small in size, while the Common Crawl dataset (Common crawl) is huge and very diverse but has a relatively low quality. So, to satisfy all requirements, Google developed the Colossal Clean Crawled Corpus (C4), a cleaned version of Common …


Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints …

T Hirosawa, Y Harada, M Yokose, T Sakamoto… - International Journal of …, 2023

The diagnostic accuracy of differential diagnoses generated by artificial intelligence (AI) chatbots, including the generative pretrained transformer 3 (GPT-3) chatbot (ChatGPT-3) is unknown. This study evaluated the accuracy of differential-diagnosis lists …


Diagnostic and Management Performance of ChatGPT in Obstetrics and Gynecology

… ChatGPT is one of the largest AI models, with over 175 billion model parameters and 570 gigabytes of training data from the “common crawl” dataset, which contains almost all unstructured text on the internet. The Common Crawl corpus constitutes …


DIATOPIT: A Corpus of Social Media Posts for the Study of Diatopic Language Variation in Italy

A Ramponi, C Casula - Tenth Workshop on NLP for Similar Languages …, 2023

We introduce DiatopIt, the first corpus specifically focused on diatopic language variation in Italy for language varieties other than Standard Italian. DiatopIt comprises over 15K geolocated social media posts from Twitter over a period of two …


Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models

M Meeus, S Jain, M Rei, YA de Montjoye - arXiv preprint arXiv:2310.15007, 2023

… Finding members is typically easy as most LLMs today have seen sources such as Common Crawl [15] or Project Gutenberg [24]. Given … Data retrieved from the frequently updated Common Crawl [15] has for instance been the majority of the …


Digital humanities and web archives: Possible new paths for

This article discusses the importance of web archives making their collections available as data and not only as sources seen through the Wayback Machine’s interface where only individual web pages are displayed. This will help unlock the …


DiLoCo: Distributed Low-Communication Training of Language Models

A Douillard, Q Feng, AA Rusu, R Chhaparia… - arXiv preprint arXiv …, 2023

Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightly interconnected accelerators, with devices …


Disambiguating Italian homographic heterophones with SoundChoice and testing ChatGPT as a data-generating tool

Text-To-Speech systems are challenged by the presence of homographs, words that have more than one possible pronunciation. Rule-based approaches are often still the preferred solution to this issue in the industry. However, there have been …


Disaster Tweets: Analysis from the Metaphor Perspective and Classification Using LLM's

T Alcántara, O García-Vázquez, H Calvo… - … International Conference on …, 2023

Nowadays, social networks, specially Twitter (now X), allow the spread of information about all topics; since this platform is completely open, there is little to none restriction on what a user can post, hence, creating a lack of confidence and …


Discerning mental illnesses from social media posts using machine and deep learning algorithms

R Harish, A Vaid, SS Byakod, A Kumar, A Arya - AIP Conference Proceedings, 2023

Deep learning has played a pivotal role in solving a wide array of problems. These powerful models can also be used for detecting mental illness. Knowing this in advance shall help individuals to utilize appropriate prophylactic measures and …


DISCO-10M: A Large-Scale Music Dataset

LA Lanzendörfer, F Grötschla, E Funke, R Wattenhofer - arXiv preprint arXiv …, 2023

… In the field of text processing, datasets such as CommonCrawl [1] and the Pile [13] have made substantial amounts of written content available for training large language models. For instance, the Pile contains an extensive dataset of 825 …


Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling

L Wang, Z Du, D Liu, C Deng, D Yu, H Jiang, Y Wang… - arXiv preprint arXiv …, 2023

Modeling discourse -- the linguistic phenomena that go beyond individual sentences, is a fundamental yet challenging aspect of natural language processing (NLP). However, existing evaluation benchmarks primarily focus on the evaluation of inter-sentence …


Disentangling Transformer Language Models as Superposed Topic Models

J Lim, H Lauw - Proceedings of the 2023 Conference on Empirical …, 2023

Topic Modelling is an established research area where the quality of a given topic is measured using coherence metrics. Often, we infer topics from Neural Topic Models (NTM) by interpreting their decoder weights, consisting of top-activated words projected …


DistilGREEK-BERT: A distilled version of the GREEK-BERT model

EA Karavangeli, DA Pantazi, M Iliakis - 2023

ABSTRACT Language Models for Natural Language Processing (NLP) have constituted subject of engagement and endless research for over two decades. All models, having been introduced through the years, have the tendency to increase in …


Distilling semantic concept embeddings from contrastively fine-tuned language models

N Li, H Kteich, Z Bouraoui, S Schockaert - 2023

Learning vectors that capture the meaning of concepts remains a fundamental challenge. Somewhat surprisingly, perhaps, pretrained language models have thus far only enabled modest improvements to the quality of such concept embeddings …


Distinguishing Romanized Hindi from Romanized Urdu

E Nielsen, C Kirov, B Roark - Proceedings of the Workshop on Computation and …, 2023

We examine the task of distinguishing between Hindi and Urdu when those languages are romanized, ie, written in the Latin script. Both languages are widely informally romanized, and to the extent that they are identified in the Latin script by …


Distributed Marker Representation for Ambiguous Discourse Markers and Entangled Relations

D Ru, L Qiu, X Qiu, Y Zhang, Z Zhang - arXiv preprint arXiv:2306.10658, 2023

Discourse analysis is an important task because it models intrinsic semantic structures between sentences in a document. Discourse markers are natural representations of discourse in our daily language. One challenge is that the …


Diverse Paraphrase Generation for Evaluating Model Robustness

… The BoW model consisted of GloVe (300 dimension embeddings trained on 840B CommonCrawl tokens) [49] vectors as the embedding layer. The average of all word vectors for the input sequence is treated as its final representation. The …


DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection

Y Chen, Z Ding, X Chen, D Wagner - arXiv preprint arXiv:2304.00409, 2023

We propose and release a new vulnerable source code dataset. We curate the dataset by crawling security issue websites, extracting vulnerability-fixing commits and source codes from the corresponding projects. Our new dataset contains 150 …


Diving into Deep Learning: On Reading and Interpreting Black Boxes

The deep neural networks used in computer vision and in recent large language models are widely recognized as black boxes, a term that describes their complicated architectures and opaque decision-making mechanisms. This essay …


DNS Dependencies as an Expression of the Digital Divide: the Example of Australia

N Nazemi, O Tavallaie, AY Zomaya, R Holz

This paper investigates the relationship between the digital divide, Internet transparency, and DNS dependencies. The term “digital divide” refers to a gap between how different population groups can access and use digital technology …


Do AIs dream of electric comics? Generative AI models, digital memory, and creativity

G Busi Rizzi - La memoria digitale. Forme del testo e organizzazione …, 2023

… called LAION, that processes through a CLIP model image-text pairs obtained from another non-profit organization called Common CrawlCommon Crawl, in turn, gets its images by scraping billions of web pages monthly and releasing them as …


Do large language models solve verbal analogies like children do?

CE Stevenson, M ter Veen, R Choenni… - arXiv preprint arXiv …, 2023

Analogy-making lies at the heart of human cognition. Adults solve analogies such as \textit{Horse belongs to stable like chicken belongs to ...?} by mapping relations (\textit{kept in}) and answering \textit{chicken coop}. In contrast, children often use association …


Do Localization Methods Actually Localize Memorized Data in LLMs?

TY Chang, J Thomason, R Jia - arXiv preprint arXiv:2311.09060, 2023

Large language models (LLMs) can memorize many pretrained sequences verbatim. This paper studies if we can locate a small set of neurons in LLMs responsible for memorizing a given sequence. While the concept of localization is often mentioned …


Do Not Discard–Extracting Useful Fragments from Low-Quality Parallel Data to Improve Machine Translation

S Steingrímsson, P Lohar, H Loftsson, A Way - CoCo4MT 2023, 2023

When parallel corpora are preprocessed for machine translation (MT) training, a part of the parallel data is commonly discarded and deemed non-parallel due to odd-length ratio, overlapping text in source and target sentences or failing some other form of a …


DocEdit: Language-Guided Document Editing

P Mathur, R Jain, J Gu, F Dernoncourt, D Manocha… - Proceedings of the AAAI …, 2023

Professional document editing tools require a certain level of expertise to perform complex edit operations. To make editing tools accessible to increasingly novice users, we investigate intelligent document assistant systems that can make or …


Document Entity Retrieval with Massive and Noisy Pre-training

L Yu, J Miao, X Sun, J Chen, AG Hauptmann, H Dai… - arXiv preprint arXiv …, 2023

Visually-Rich Document Entity Retrieval (VDER) is a type of machine learning task that aims at recovering text spans in the documents for each of the entities in question. VDER has gained significant attention in recent years thanks to its broad …


DOES CLIP'S GENERALIZATION PERFORMANCE MAINLY STEM FROM HIGH TRAIN-TEST SIMILARITY?

P Mayilvahanan, T Wiedemer, E Rusak, M Bethge… - arXiv preprint arXiv …, 2023

… In comparison, LAION-400M comprises roughly 400 million samples scraped from the Common Crawl dataset*. The Common Crawl dataset comprises petabytes of data collected over more than a decade of web crawling; LAION-400M, therefore …


Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?

AY Sun, E Zemour, A Saxena, U Vaidyanathan, E Lin… - arXiv preprint arXiv …, 2023

… For these half of these generations, we use prompts from randomly select strings with character length 100 from an English-only subset of Common Crawl to serve as naive prompts to probe our model, inspired by the naive prompting strategy used by (Diera …


Does GPT-3 qualify as a co-author of a scientific paper publishable in peer-review journals according to the ICMJE criteria? A case study

A Osmanovic-Thunström, S Steingrimsson - Discover Artificial Intelligence, 2023

This paper explores the potential for a system to be a co-author on an academic paper based on the criteria proposed by the International Committee of Medical Journal Editors (ICMJE). We used a third generation generative pretrained …


Does GPT-3 qualify as a co-author of a scientific paper publishable in peer-review journals according to the ICMJE criteria?-A Case Study.


Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation?

H Tamura, T Hirasawa, H Kim, M Komachi - Findings of the Association for …, 2023

Pre-training masked language models (MLMs) with artificial data has been proven beneficial for several natural language processing tasks such as natural language understanding and summarization; however, it has been less explored for neural …


Does the first letter of one's name affect life decisions? A natural language processing examination of nominative determinism.

P Chatterjee, H Mishra, A Mishra - Journal of Personality and Social Psychology, 2023

… That is, when running the analysis using the Common Crawl corpus, we obtained the embedding of person name, profession name, and city name from the Common Crawl corpus. This helped us test for nominative determinism using the millions of co-occurrences …


DoGE: Domain Reweighting with Generalization Estimation

S Fan, M Pagliardini, M Jaggi - arXiv preprint arXiv:2310.15393, 2023

… We see how DOGE seem to upweights significantly the common-crawl (cc) domain. In (b) we show DOGE domain weights for out-of-domain generalization. Those domain weights obtained by training on various training mixtures consisting …


Dolphin: A Challenging and Diverse Benchmark for Arabic NLG

EMB Nagoudi, A El-Shangiti, AR Elmadany… - arXiv preprint arXiv …, 2023

We present Dolphin, a novel benchmark that addresses the need for an evaluation framework for the wide collection of Arabic languages and varieties. The proposed benchmark encompasses a broad range of 13 different NLG tasks, including text …


Don't Just Clean It, Proxy Clean It: Mitigating Bias by Proxy in Pre-Trained Models

S Panda, A Kobren, M Wick, Q Shen - Findings of the Association for Computational …, 2022

Transformer-based pre-trained models are known to encode societal biases not only in their contextual representations, but also in downstream predictions when fine-tuned on task-specific data. We present D-Bias, an approach that selectively eliminates …


Don't Worry Accountants, ChatGPT Won't Be Taking Your Job... Yet

ChatGPT has demonstrated the ability to generate plausible human-like text and research is underway to evaluate and benchmark its current performance in various domains. The research we present here provides a preliminary benchmark on …


DONOTTRAIN: A Metadata Standard for Indicating Consent for Machine Learning

… take the Common Crawl, a publicly available crawl of the internet, as their starting point, and filter and process it into a training dataset. The Common Crawl currently respects robots.txt. For the learners.txt protocol to be useful, we would work with the Common …


DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining

SM Xie, H Pham, X Dong, N Du, H Liu, Y Lu, P Liang… - arXiv preprint arXiv …, 2023

The mixture proportions of pretraining data domains (eg, Wikipedia, books, web text) greatly affect language model (LM) performance. In this paper, we propose Domain Reweighting with Minimax Optimization (DoReMi), which first trains a small proxy …


Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness

This paper investigates the significant impact different prompts have on the behaviour of ChatGPT when used for health information seeking. As people more and more depend on generative large language models (LLMs) like ChatGPT, it is …


Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctness

G Zuccon, B Koopman - arXiv preprint arXiv:2302.13793, 2023

… The actual documents are web pages from the noclean version of the C4 dataset: ≈1B English text extracts from the April 2019 snapshot of Common Crawl. Some passages are long and exceed the maximum token limit of ChatGPT. We trimmed …


Dream the Impossible: Outlier Imagination with Diffusion Models

X Du, Y Sun, X Zhu, Y Li - arXiv preprint arXiv:2309.13415, 2023

… The model was trained on 5 billion pairs of images and captions taken from LAION-5B [87], a publicly available dataset derived from Common Crawl data scraped from the web. Given a class name y, the generation process can be mathematically denoted …


DreamShot: Teaching Cinema Shots to Latent Diffusion Models

T Cerquitelli, B Vacchetti, T Massaglia - 2023

This thesis work presents a comprehensive overview of recent advancements in image synthesis models, exploring the recent developments of Diffusion Models [1] and their finetuning. The primary contribution consists in a novel approach that …


Dynamic Masking Rate Schedules for MLM Pretraining

Z Ankner, N Saphra, D Blalock, J Frankle, ML Leavitt - arXiv preprint arXiv …, 2023

Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. Our work instead dynamically schedules the masking ratio throughout training. We found that linearly …


Dzongkha to English translation using the neural machine translation approach

K Wangchuk, SC Navaneethakrishnan, Y Jamtsho… - Indonesian Journal of …, 2023

… The word embeddings were trained using the continuous bag of words (CBOW) model on common crawl and Wikipedia. Word vectors were trained using CBOW with position-weights, embedding dimension of 300, ngram characters of length five …


EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data

S Ma, S Huang, S Huang, X Wang, Y Li, HT Zheng… - arXiv preprint arXiv …, 2023

… Generally, commonly used data for LLM pretraining is obtained from organized long text on web pages, such as CommonCrawl and Wikipedia. However, in some specific domains including ecommerce domain, a substantial amount of text data is …


Ecosystem Graphs: The Social Footprint of Foundation Models

R Bommasani, D Soylu, TI Liao, KA Creel, P Liang - arXiv preprint arXiv:2303.15772, 2023

Foundation models (eg ChatGPT, StableDiffusion) pervasively influence society, warranting immediate social attention. While the models themselves garner much attention, to accurately characterize their impact, we must consider the broader …


EDSA-Ensemble: an Event Detection Sentiment Analysis Ensemble Architecture

A Petrescu, CO Truică, ES Apostol, A Paschke - arXiv preprint arXiv:2301.12805, 2023

… The additional data included CommonCrawl News dataset (63 million articles, 76 GB), Web text corpus (38 GB), and Stories from Common Crawl (31 GB). This coupled with a whopping 1024 V100 Tesla GPUs running for a day, led to pre-training …


Educational data augmentation in physics education research using ChatGPT

F Kieser, P Wulff, J Kuhn, S Küchemann - arXiv preprint arXiv:2307.14475, 2023

… [22] Early LLM were trained based on large text corpora such as the Common Crawl (ie, dump of the Internet, 60%[23]), WebText2 (22%), or Wikipedia (3%) with the training objective to predict next words in a sequence (a sort of Cloze test, [18]) …


Effective Long-Context Scaling of Foundation Models

W Xiong, J Liu, I Molybog, H Zhang, P Bhargava… - arXiv preprint arXiv …, 2023

We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are …


Effects of paraphrasing and demographic metadata on NLI classification performance

… It is a neural network that has been pre-trained on a large dataset of texts made up from a filtered version of CommonCrawl, WebText2, Books1, Books2 (two internet-based books corpora) and Wikipedia. GPT-3 can generate human-like text in a variety of …


Efficient Approximate Nearest Neighbor Search in Multi-dimensional Databases

YUN PENG, B CHOI, T CHAN, J YANG, J XU - 2023

Approximate nearest neighbor (ANN) search in multi-dimensional databases is a fundamental search and has many applications, such as image retrieval [28, 44], recommendation [12], entity resolution [21], and sequence matching [10]. Many ANN …


Efficient Continual Pre-training for Building Domain Specific Large Language Models

Y Xie, K Aggarwal, A Ahmad - arXiv preprint arXiv:2311.08545, 2023

… We use two sources of data for the financial corpus: the financial news common crawl and SEC filings. Financial News CommonCrawl is curated by filtering out financial news from the public CommonCrawl data. We follow the de-duplication …


Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning

M Ostendorff, G Rehm - arXiv preprint arXiv:2301.09626, 2023

Most Transformer language models are primarily pretrained on English text, limiting their use for other languages. As the model sizes grow, the performance gap between English and other languages with fewer compute and data resources …


Efficient Phishing Detection and Prevention Using Support Vector Machine (SVM) Algorithm

M Arivukarasi, A Manju, R Kaladevi, S Hariharan… - 2023 IEEE 12th …, 2023

Phishing issues influence the electronic trade in light of the fact that web-based clients trust the Internet climate less. Phishers use procedures that advance to bait online clients, making new phishing sites and spreading messages that attempt to …


Efficient Pre-training for Localized Instruction Generation of Videos

A Batra, D Moltisanti, L Sevilla-Lara, M Rohrbach… - arXiv preprint arXiv …, 2023

Procedural videos show step-by-step demonstrations of tasks like recipe preparation. Understanding such videos is challenging, involving the precise localization of steps and the generation of textual instructions. Manually annotating steps and writing …


Efficient, Adaptable and Interpretable NLP

In natural language processing (NLP), a central concern is how to develop and evaluate language model pretraining that better transfers and adapts to downstream tasks. Due to their black box character, it is hard to understand how models transfers …


Efficiently Adapting Pretrained Language Models To New Languages

Z Csaki, P Pawakapan, U Thakker, Q Xu - arXiv preprint arXiv:2311.05741, 2023

… For the Thai pre-training corpus, we combine the Thai subsets of OSCAR [41], MC4 [42], and CCNet [43], which are all derived from Common Crawl. The entire combined corpus was processed with MinHash deduplication [44, 45] with 1-grams …


Einsatzbereiche semantisch strukturierter Daten bei der Unternehmensanalyse

This dissertation consists of 6 papers. The papers investigate the areas of application of semantically structured data in company analysis. The first paper examines the extent to which alternative data can support tax audits. It is also …


Email Spam Detection Using Multi-head CNN-BiGRU Network

A Gupta, J Patil, S Soni, A Rajan - … , ANTIC 2022, Varanasi, India, December 22–24 …, 2023

Spam emails refer to unsolicited email messages, usually sent in bulk to a large list of recipients with the purpose of marketing, or luring individuals to download malware or click on phishing links. Although much research has been done on this …


Embedded Lexica

Researchers oftentimes find themselves in positions where they need to extract information, such as events or target topics, from large corpora. One common solution involves applying semantically-related keywords to identify tweets, news …


EMit at EVALITA 2023: Overview of the Categorical Emotion Detection in Italian Social Media Task

O Araque, S Frenda, R Sprugnoli, D Nozza, V Patti - Proceedings of the Eighth …, 2023

The Emotions in Italian (EMit) task is the first edition of a shared task on emotion analysis and opinion mining in Italian messages at EVALITA 2023. EMit presents two subtasks:(i) Subtask A, that consists in an emotion detection challenge, and (ii) …


EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning

P Guo, X Wei, Y Hu, B Yang, D Liu, F Huang, J Xie - arXiv preprint arXiv:2310.17233, 2023

Expressing universal semantics common to all languages is helpful in understanding the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal …


Emotion Hunters at EMit: Categorical Emotion Detection combining BERT and ChatGPT models

G Calò, F Massafra, B De Carolis, C Loglisci - 2023

Emotion detection in text plays a crucial role in various applications, such as customer feedback analysis, social media monitoring, or for the analysis of the verbal part of human communication. Deep learning techniques have shown …


Emotion recognition in italian political language for prefiguring crisis in the balance of the parties' coalitions

A Forciniti, E Zavarrone, M Paolillo - Quality & Quantity, 2023

The purpose of this study is to describe the emotions that dominated the political vocabulary used on Twitter during the Covid-19 epidemic by the leaders of the major Italian political parties, including Forza Italia, Movimento 5 Stelle, Fratelli d’Italia …


Empirical Analysis of Beam Search Curse and Search Errors with Model Errors in Neural Machine Translation

J He, S Sun, X Jia, W Li - Proceedings of the 24th Annual Conference of the …, 2023

Beam search is the most popular decoding method for Neural Machine Translation (NMT) and is still a strong baseline compared with the newly proposed sampling-based methods. To better understand beam search, we investigate its two well-recognized …


Empirical evaluation of Uncertainty Quantification in Retrieval-Augmented Language Models for Science

S Wagle, S Munikoti, A Acharya, S Smith… - arXiv preprint arXiv …, 2023

Large language models (LLMs) have shown remarkable achievements in natural language processing tasks, producing high-quality outputs. However, LLMs still exhibit limitations, including the generation of factually incorrect information. In safety-critical …


Empirical study of pretrained multilingual language models for zero-shot cross-lingual generation

N Chirkova, S Liang, V Nikoulina - arXiv preprint arXiv:2310.09917, 2023

Zero-shot cross-lingual generation assumes finetuning the multilingual pretrained language model (mPLM) on a generation task in one language and then using it to make predictions for this task in other languages. Previous works notice a frequent …


Empowering Short Answer Grading: Integrating Transformer-Based Embeddings and BI-LSTM Network

WH Gomaa, AE Nagib, MM Saeed, A Algarni, E Nabil - Big Data and Cognitive …, 2023

… The T5 model was trained on the 700 GB Colossal Clean Crawled Corpus dataset, a cleaned version of the Common Crawl dataset that included only English text. It achieved state-of-the-art performance on several NLP benchmarks and could be …


Enabling Cross-lingual Information Retrieval for African Languages

… Another common approach is to exploit existing large multilingual corpora, eg, the Common Crawl5 and Wikipedia. For example, the HC4 corpus for cross-lingual information retrieval was created from Common Crawl data [30]. Examples of …


Encoding the Enemy: The Politics Within and Around Ethical Algorithmic War

This article develops a critique of the politics of algorithmic war and autonomous weapons systems. While much of the existing debate is focused on whether algorithmic weapons technologies can satisfy the ethics and laws of war, we argue …


End-to-End Speech-to-Text Translation: A Survey

N Sethiya, CK Maurya - arXiv preprint arXiv:2312.01053, 2023

… Finally, a language model (LM) is trained on CommonCrawl data and combined with the ST model to generate text via beam-search decoding. Following along, for training E2E model, (Wang et al., 2021b) produces pseudo-labels by cascading ASR …


Energy Estimates Across Layers of Computing: From Devices to Large-Scale Applications in Machine Learning for Natural Language Processing, Scientific …

S Shankar - arXiv preprint arXiv:2310.07516, 2023

… These AI/ML methods depend on training on a large corpus, namely significant amounts of data using words, phrases, part-of speech requirements, existing collections of text from academic journals, books, social network websites, Wikipedia …


Engineering a Distributed-Memory Triangle Counting Algorithm

P Sanders, TN Uhl - arXiv preprint arXiv:2302.11443, 2023

Counting triangles in a graph and incident to each vertex is a fundamental and frequently considered task of graph analysis. We consider how to efficiently do this for huge graphs using massively parallel distributed-memory machines …


Engineering the Best In-Context Input for GPT-3 in the OpenQA Task

GPT-3, since its release, has garnered the attention of the NLP community due to its versatility across a wide range of NLP tasks. In this work, we use GPT-3 to approach the OpenQA task, where the model needs to answer input questions without being …


Enhanced Emotion and Sentiment Recognition for Empathetic Dialogue System Using Big Data and Deep Learning Methods

M Kozłowski, K Gabor-Siatkowska, I Stefaniak… - International Conference on …, 2023

… The process of using the Common Crawl web archive to create an enlarged corpus, named CORTEX+ pCC, is presented. An empathetic dialogue system named Terabot, incorporating the elaborated method, is also described. The system …


Enhanced Phishing URL Detection Using Leveraging BERT with Additional URL Feature Extraction

KS Jishnu, B Arthi - 2023 5th International Conference on Inventive …, 2023

… Their heuristic-based deep learning technique made use of RNN models and datasets including PhishTank, Alexa, and Common Crawl. … Their research used the PhishTank and Common Crawl databases, which contain legal and phishing …


Enhancing Customer Support with Knowledge Graph-Based Question Answering

Many companies don’t utilize the huge amount of unstructured data they possess. Old issue tickets are one example. A company that possesses a lot of old issue tickets are Stibo Systems. Meanwhile, customer support staff receive issues that has …


Enhancing EFL reading and writing through AI-powered tools: design, implementation, and evaluation of an online course

JC Hsiao, JS Chang - Interactive Learning Environments, 2023

During the Covid-19 pandemic, global teachers gained extensive experiences with teaching online courses. To design quality online courses in the post-pandemic era, the impact of the latest technology, such as artificial intelligence (AI), must be …


Enhancing Neural Text Detector Robustness with µAttacking and RR-Training. Electronics 2023, 12, 1948

G Liang, J Guerrero, F Zheng, I Alsmadi - 2023

With advanced neural network techniques, language models can generate content that looks genuinely created by humans. Such advanced progress benefits society in numerous ways. However, it may also bring us threats that we have not seen …


Enhancing RDF Verbalization with Descriptive and Relational Knowledge

F Zhang, M Zhang, S Liu, Y Sun, N Duan - ACM Transactions on Asian and Low …, 2023

… T5 is pre-trained on a large and clean unlabeled text dataset named C4 from Common Crawl (about 750 GB) and achieves state-of-the-art performance on various benchmarks such as CNN/Daily Mail [4] and SuperGLUE [33]. T5 has ive …


Enhancing Search Engine Results: A Comparative Study of Graph and Timeline Visualizations for Semantic and Temporal Relationship Discovery

… Examples of open source data sets for search engine data are Common Crawl (CC) 10 which provides a large-scale data set of the crawled web, providing easy access to such data which is typically not made available for commercial search engines …


Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approach

TJ Park, K Dhawan, N Koluguri, J Balam - arXiv preprint arXiv:2309.05248, 2023

Large language models (LLMs) have shown great promise for capturing contextual information in natural language processing tasks. We propose a novel approach to speaker diarization that incorporates the prowess of LLMs to exploit contextual cues …


Enhancing Text Summarization for Indian Languages: Mono, Multi and Cross-lingual Approaches

The internet serves as a vast repository of information covering a diverse array of topics, ranging from blogs and articles to websites. However, it is important to note that not all of this information is valuable or relevant. Navigating through the plethora …


Enhancing Traceability Link Recovery with Fine-Grained Query Expansion Analysis

T Peng, K She, Y Shen, X Xu, Y Yu - Information, 2023

… The training datasets were Wikipedia and CommonCrawl datasets [16], which contain texts from various domains. Using common domain datasets for training enabled the model to better represent different software terminologies and vocabularies. …


EnML: Multi-label Ensemble Learning for Urdu Text Classification

F Mehmood, R Shahzadi, H Ghafoor, MN Asim… - ACM Transactions on Asian …, 2023

… Crawl News Dataset (63 million articles, 76 GB), Web text corpus (38 GB), and Stories from Common Crawl were among the supplementary data (31 GB). We have utilized RoBERTa-Large-Multilingual with 12 attention heads and 12 hidden layers …


ENTITY LINKING IN LOW-ANNOTATION DATA SETTINGS

… While these LMs were originally trained on standard NLP domain texts such as Wikipedia and CommonCrawl, the unsupervised nature of their training means that models tailored to specific domains can be produced. This is especially useful in the …


Entity Matching using Large Language Models

R Peeters, C Bizer - arXiv preprint arXiv:2310.11244, 2023

… The benchmark was built by extracting schema.org data from the Common Crawl We use the most difficult version of the benchmark including 80% corner-cases (hard positives and hard negatives) and use the following attributes: brand, title, currency …


ERATE: Efficient Retrieval Augmented Text Embeddings

V Raina, N Kassner, K Popat, P Lewis, N Cancedda… - The Fourth Workshop on …, 2023

… An additional 100 million sentences sampled from common crawl (CC)5 are included in an expanded datastore to investigate the impact of increasing the datastore size. Table 1 details the statistics for each of these subsets. Sentences …


Escaping the sentence-level paradigm in machine translation

M Post, M Junczys-Dowmunt - arXiv preprint arXiv:2304.12959, 2023

It is well-known that document context is vital for resolving a range of translation ambiguities, and in fact the document setting is the most natural setting for nearly all translation. It is therefore unfortunate that machine translation -- both research and …


Essays in Environmental Economics

This dissertation consists of three essays in the field of environmental economics. The first chapter provides the first causal evidence that hostile activities online lead to physical violence. Given the recently documented relationship between pollution …


Establishing An Optimal Online Phishing Detection Method: Evaluating Topological NLP Transformers on Text Message Data

H Milner, M Baron - Journal of Data Science and Intelligent Systems, 2023

This research establishes an optimal classification model for online SMS spam detection by utilizing topological sentence transformer methodologies. The study is a response to the increasing sophisticated and disruptive activities of malicious actors …


Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation

Y Li - arXiv preprint arXiv:2309.10677, 2023

… For example, if the model M was trained on a Common Crawl dataset spanning 2016-2019, we would gather web pages specifically from this period. Similarly, if the model was trained on a 2019 dump of Wikipedia, we would collect articles from …


Ethical challenges of large language models-a systematic literature review

… Current LLMs are by and large trained on massive datasets collected from the internet often the Common Crawl -material, a collection of vast amounts of internet content [S111]. However, just because private data is available online does not …


Evading Text Based Emotion Detection Mechanism via Adversarial Attacks

A Bajaj, DK Vishwakarma - Neurocomputing, 2023

Textual Emotion Analysis (TEA) seeks to extract and assess the emotional states of users from the text. Various Deep Learning (DL) algorithms have emerged rapidly and demonstrated success in numerous disciplines, including audio, image, and …


Evaluating Biased Attitude Associations of Language Models in an Intersectional Context

SO Sabbaghi, R Wolfe, A Caliskan - arXiv preprint arXiv:2307.03360, 2023

… Models trained on the Pile have been shown to outperform models trained on both raw and filtered versions of the Common Crawl on many benchmarks and downstream evaluations [23]. Prior work finds that GPT-Neo most strongly encodes …


Evaluating Cross Lingual Transfer for Morphological Analysis: a Case Study of Indian Languages

S Pawar, P Bhattacharyya, P Talukdar - … of the 20th SIGMORPHON workshop on …, 2023

… It is pre-trained on the Common Crawl-based dataset and covers 101 languages. It is an encoder-decoder sequence generation model, unlike mBERT, which is an encoder-only multilingual model. Our task of root word extraction requires the …


Evaluating Event Embeddings on Commodity Future Prices

… We developed a library for pulling and filtering websites from the Common Crawl Index, accessible here: github.com/networkdynamics/seldonite. We pulled Reuters news articles from Common Crawl from 2006 to 2021. We only keep articles that are …


Evaluating Gender Bias in Pre-trained Filipino FastText Embeddings

LC Gamboa, MRJ Estuar - 2023 International Conference on IT Innovation and …, 2023

… This embedding was trained on Common Crawl and Wikipedia data written in Filipino, the Philippine national language based primarily on Tagalog (a regional language in the country) but also containing words from foreign (eg, …


Evaluating Knowledge Graphs with Hybrid Intelligence

… WebIsALOD is a large KG containing 400M hypernymy relations, automatically extracted from the CommonCrawl web corpus, describing generic knowledge [6]. The current verification of the resource relies on machine learning models trained …


Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics

J Holmes, Z Liu, L Zhang, Y Ding, TT Sio, LA McGee… - arXiv preprint arXiv …, 2023

We present the first study to investigate Large Language Models (LLMs) in answering radiation oncology physics questions. Because popular exams like AP Physics, LSAT, and GRE have large test-taker populations and ample test …


Evaluating Paraphrastic Robustness in Textual Entailment Models

D Verma, YK Lal, S Sinha, B Van Durme, A Poliak

We presentˆP aRT E, a collection of 1,126 pairs of Recognizing Textual Entailment (RTE) examples to evaluate whether models are robust to paraphrasing. We posit that if RTE models understand language, their predictions should be consistent across …


Evaluating Prompt-based Question Answering for Object Prediction in the Open Research Knowledge Graph

J D'Souza, M Hrou, S Auer - arXiv preprint arXiv:2305.12900, 2023

There have been many recent investigations into prompt-based training of transformer language models for new text genres in low-resource settings. The prompt-based training approach has been found to be effective in generalizing pre-trained …


Evaluating Rule-Based and Neural Merging Strategies of Entity Clusters for Coreference Resolution

Coreference resolution is an essential pre-processing step for many natural language processing tasks. In the past, there has been a shift from rule-based approaches to machinelearning-based approaches. A recent approach that focuses …


Evaluating Supervised Machine Learning Models for Zero-Day Phishing Attack Detection: A Comprehensive Study

Z Lotfi, S Valipourebrahimi, T Tran - 2023

To have highly secure e-commerce websites, detecting and preventing cyber-attacks is of high importance. Among diverse types of cyber-attacks, identifying zero-day attacks is problematic since they are unknown to the security system. It is because …


Evaluating synthesized speech: the cognitive approach

G Navickas, GA Melnik-Leroy - DAMSS: 13th conference on data analysis methods …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Evaluating Temporal Persistence Using Replicability Measures

J Keller, T Breuer, P Schaer - arXiv preprint arXiv:2308.10549, 2023

In real-world Information Retrieval (IR) experiments, the Evaluation Environment (EE) is exposed to constant change. Documents are added, removed, or updated, and the information need and the search behavior of users is evolving. Simultaneously …


Evaluating the Capabilities of Large Language Models for Spatial and Situational Understanding

… With the advent of new generation compute platforms1, curation of large scale datasets like the CommonCrawl2, and the formal verification of the power scaling, development of larger and larger language models boomed. In Table 2.1 we give a …


Evaluating the Effectiveness of Pre-trained Language Models in Predicting the Helpfulness of Online Product Reviews

A Boluki, JPR Sharami, D Shterionov - arXiv preprint arXiv:2302.10199, 2023

Businesses and customers can gain valuable information from product reviews. The sheer number of reviews often necessitates ranking them based on their potential helpfulness. However, only a few reviews ever receive any helpfulness votes on …


Evaluating the Effectiveness of Retrieval-Augmented Large Language Models in Scientific Document Reasoning

S Munikoti, A Acharya, S Wagle, S Horawalavithana - arXiv preprint arXiv:2311.04348, 2023

… In addition to the ATLAS model pretrained with common crawl (CC) and Wikipedia, we also train ATLAS-Science (220M) model from scratch with the S2ORC scientific text datasets. For a fair comparison with ATLAS, we initialize the ATLAS-Science …


Evaluating the Efficiency of Quantum Simulators Using Practical Application Benchmarks

Quantum computing holds the potential to revolutionize various industries by solving problems that classical computers cannot solve efficiently. However, building quantum computers is still in its infancy, and simulators are currently the best …


Evaluating the Moral Beliefs Encoded in LLMs

N Scherrer, C Shi, A Feder, DM Blei - arXiv preprint arXiv:2307.14324, 2023

This paper presents a case study on the design, administration, post-processing, and evaluation of surveys on large language models (LLMs). It comprises two components: (1) A statistical method for eliciting beliefs encoded in LLMs. We …


Evaluation of African American Language Bias in Natural Language Generation

N Deas, J Grieser, S Kleiner, D Patton, E Turcan… - arXiv preprint arXiv …, 2023

We evaluate how well LLMs understand African American Language (AAL) in comparison to their performance on White Mainstream English (WME), the encouraged "standard" form of English taught in American classrooms. We measure …


Evaluation of ChatGPT Applicability to Learning Quantum Physics

A Stefńnska, TP Stefański, M Czubenko - 2023 16th International Conference on …, 2023

ChatGPT is an application that uses a large language model. Its purpose is to generate answers to various questions as well as provide information, help solve problems and participate in conversations on a wide range of topics. This …


Evaluation of ChatGPT as a Question Answering System for Answering Complex Questions

Y Tan, D Min, Y Li, W Li, N Hu, Y Chen, G Qi - arXiv preprint arXiv:2303.07992, 2023

… FLAN-T5 FLAN-T5 (Text-to-Text Transfer Transformer, 11B [8])is an encoderdecoder Transformer language model that is trained on a filtered variant of CommonCrawl (C4) [34]. The release date and model size for this model are also based on [34]. …


Evaluation of Different Machine Learning, Deep Learning and Text Processing Techniques for Hate Speech Detection

Social media has become a domain that involves a lot of hate speech. Some users feel entitled to engage in abusive conversations by sending abusive messages, tweets, or photos to other users. It is critical to detect hate speech and prevent …


Evaluation of Estimating Preschool Children's Verbal Communication Skills in Group Discussions

H Yamaguchi, K Horio - 2022 Joint 12th International Conference on Soft …, 2022

In recent years, “First grade problem”, in which first graders who have just entered elementary school are unable to adjust to school life, has been occurring. One of the causes of this problem is a lack of communication skills. Since children have more …


Evaluation of GPT and BERT-based models on identifying protein-protein interactions in biomedical text

H Rehana, NB Çam, M Basmaci, Y He, A Özgür, J Hur - arXiv preprint arXiv …, 2023

Detecting protein-protein interactions (PPIs) is crucial for understanding genetic mechanisms, disease pathogenesis, and drug design. However, with the fast-paced growth of biomedical literature, there is a growing need for automated and accurate …


Evaluation of Language Models on Romanian XQuAD and RoITD datasets

CD Nicolae, RK Yadav, D Tufiş - INTERNATIONAL JOURNAL OF COMPUTERS …, 2023

… This multi-language corpus was extracted via language classification, filtered, and cleaned from Common Crawl. Additionally, the latest version of the Wikipedia Dumps for Romanian was obtained, parliamentary debates from 1996-2017 were …


Evaluation of strategies for the adaptation of large neural models to the task of machine translation in constrained scenarios

[EN] Historically, machine translation (MT) has been one of the most active areas within artificial intelligence and more precisely, within the field of machine learning. Thanks to the significant progress in training large neural networks on massive …


Evidence retrieval for causal questions using query expansion and reranking

A Gaden, B Reinhold, L Zeit-Altpeter, N Rausch - Working Notes of CLEF, 2023

We present our runs to the Touché Lab on Argument and Causal Retrieval at CLEF 2023, aiming to retrieve relevant documents for given causal questions. Our approaches rely heavily on query expansions. Both a transformer-based and a …


Examining the effect of whitening on static and contextualized word embeddings

S Sasaki, B Heinzerling, J Suzuki, K Inui - Information Processing & Management, 2023

… GloVe840B is trained on the Common Crawl dataset containing 840 billion tokens, while GloVe6B is trained on the Wikipedia and Gigaward dataset containing 6 billion tokens. GNews is trained by using a CBOW algorithm (Mikolov, Chen, et al., 2013) …


Examining the Text-to-Image Community of Practice: Why and How do People Prompt Generative AIs?

T Sanchez - Proceedings of the 15th Conference on Creativity and …, 2023

Image generation gained popularity with machine learning (ML) models generating images from text, fuelling new online communities of practices. This work explores the sociology, motivations, and usages of AI art hobbyists. We analyzed an online …


Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models

K Gao, S He, Z He, J Lin, QZ Pei, J Shao, W Zhang - arXiv preprint arXiv:2308.14149, 2023

… GPT-3 [2] devised an automated filtering method to effectively eliminate low-quality documents from the Common Crawl dataset. It used a … Moreover, GPT-3 adopted fuzzy deduplication of documents within each dataset and the removal of WebText …


ExpertQA: Expert-Curated Questions and Attributed Answers

C Malaviya, S Lee, S Chen, E Sieber, M Yatskar… - arXiv preprint arXiv …, 2023

SL ChaitanyaMalaviya, ES SihaoChen, M Yatskar…

As language models are adapted by a more sophisticated and diverse set of users, the importance of guaranteeing that they provide factually correct information supported by verifiable sources is critical across fields of study & professions. This is …


Explainable clinical coding with in-domain adapted transformers

G López-García, JM Jerez, N Ribelles, E Alba… - Journal of Biomedical …, 2023

Background and Objective Automatic clinical coding is a crucial task in the process of extracting relevant information from unstructured medical documents contained in Electronic Health Records (EHR). However, most of the existing computer-based …


Explainable Crowd Decision Making methodology guided by expert natural language opinions based on Sentiment Analysis with Attention-based Deep Learning and …

C Zuheros, E Martínez-Cámara, E Herrera-Viedma… - Information Fusion, 2023

There exist a high demand to provide explainability to artificial intelligence systems, where decision making models are included. This paper focuses on crowd decision making using natural language evaluations from social media with the aim to …


Explainable multi-task convolutional neural network framework for electronic petition tag recommendation

Z Yang, J Feng - Electronic Commerce Research and Applications, 2023

Electronic petition (e-petition) is an electronic government (e-government) service that allows citizens to file petitions to governments via the internet. The complexity of the e-petition filing process and the unexplainable e-government tools would reduce …


Explaining Sarcasm of Tweets using Attention Mechanism

RL Keerthana, AK Singh, P Saini, D Malhotra - Scalable Computing: Practice and …, 2023

Emotion identification from text can help boost the effectiveness of sentiment analysis models. Sarcasm is one of the more difficult emotions to detect, particularly in textual data. Even though several models for detecting sarcasm have been …


Explanatory Argument Extraction of Correct Answers in Resident Medical Exams

I Goenaga, A Atutxa, K Gojenola, M Oronoz, R Agerri - arXiv preprint arXiv …, 2023

Developing the required technology to assist medical experts in their everyday activities is currently a hot topic in the Artificial Intelligence research field. Thus, a number of large language models (LLMs) and automated benchmarks have recently …


Exploration of transfer learning capability of multilingual models for text classification

M Bhargava, K Vijayan, O Anand, G Raina - Proceedings of the 2023 5th International …, 2023

The use of multilingual models for natural language processing is becoming increasingly popular in industrial and business applications, particularly in multilingual societies. In this study, we investigate the transfer learning capabilities …


Exploring acceptance of autonomous vehicle policies using KeyBERT and SNA: Targeting engineering students

J Ha, D Kim - arXiv preprint arXiv:2307.09014, 2023

This study aims to explore user acceptance of Autonomous Vehicle (AV) policies with improved text-mining methods. Recently, South Korean policymakers have viewed Autonomous Driving Car (ADC) and Autonomous Driving Robot (ADR) as …


Exploring and Repairing Gender Fairness Violations in Word Embedding-based Sentiment Analysis Model through Adversarial Patches

LS Khoo, JQ Bay, MLK Yap, MK Lim, CY Chong…

With the advancement of sentiment analysis (SA) models and their incorporation into our daily lives, fairness testing on these models is crucial, since unfair decisions can cause discrimination to a large population. Nevertheless, some challenges in …


Exploring Applications of NLP to Create Deceptive Sandbox Environments with Honeyfiles

Various techniques are in practice to create a sandboxed environment to study malicious actors and/or divert them from gaining access to critical assets. Honeypots and honeynets have long been used to create decoy systems that lure adversaries …


Exploring ChatGPT's accuracy and confidence in high-resource languages

… web content and of the Common Crawl dataset in different European languages. Although, for example, 1.6% might seem like a very small percentage, all the languages listed are still commonly considered high-resource, which is usually …


Exploring Divergent Thinking for Cyber Soldier Selection

The Swedish Defence Research Agency is developing a test battery (CyberTest Future Soldiers—CTFS) to select conscript cyber soldier candidates for the Swedish Armed Forces since 2019. Comprising knowledge and ability components, CTFS …


Exploring Instagram Messages on the Right to Abortion: User Profiling and Natural Language Processing Tasks

… [25] presents XLM-R, a Transformer based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data, and it provides substantial gains over previous multilingual models like mBERT [26] …


Exploring natural language processing in mechanical engineering education: Implications for academic integrity

J Lesage, R Brennan, SE Eaton, B Moya, B McDermott… - International Journal of …, 2023

In this paper, the authors review extant natural language processing models in the context of undergraduate mechanical engineering education. These models have advanced to a stage where it has become increasingly more difficult to discern …


Exploring Register Variation in Turkish Web Corpus

S Erten - 14–15 September 2023, University of Mannheim …

… collected by Common Crawl (commoncrawl. org) and cleaned and pre-processed within Massively Multilingual Modelling of Registers in Web-scale Corpora project run by TurkuNLP team at the University of Turku, Finland. Altogether, the corpus …


Exploring Semantic Word Representations for Recognition-Free NLP on Handwritten Document Images

O Tüselmann, GA Fink - International Conference on Document Analysis and …, 2023

A semantic analysis of documents offers a wide range of practical application scenarios. Thereby, the combination of handwriting recognizer and textual NLP models constitutes an intuitive solution. However, due to the difficulty of recognizing …


Exploring the Constructicon: Linguistic Analysis of a Computational CxG

J Dunn - arXiv preprint arXiv:2301.12642, 2023

Recent work has formulated the task for computational construction grammar as producing a constructicon given a corpus of usage. Previous work has evaluated these unsupervised grammars using both internal metrics (for example, Minimum …


Exploring the Conversations of a Motivational Interviewing Chatbot: A Topic Modeling Approach

… GPT-3, alongside the mentioned datasets, was also trained on CommonCrawl, a collection of webpages containing 410 billion words [31]. These models gained state-of-the-art performance on many language task benchmarks, including machine translation …


Exploring the Cookieverse: A Multi-Perspective Analysis of Web Cookies

A Rasaii, S Singh, D Gosain, O Gasser - arXiv preprint arXiv:2302.05353, 2023

Web cookies have been the subject of many research studies over the last few years. However, most existing research does not consider multiple crucial perspectives that can influence the cookie landscape, such as the client's location, the impact of …


Exploring the impact of intermediate languages on machine translation

The field of translation predates the computer, and as technology has advanced, it has evolved and adapted to new discoveries, constantly striving to become more efficient and precise. Machine translation, an integral part of Natural Language …


Exploring the limits of early predictive maintenance applying anomaly detection technique

A Serackis, M Jankauskas - DAMSS 2022: 13th conference on data analysis …, 2022

DAMSS-2022 is the 13th International Conference on Data Analysis Methods for Software Systems, held in Druskininkai, Lithuania. Every year at the same place and time. The exception was in 2020, when the world was gripped by the Covid-19 …


Exploring the Potential of Large Language models in Traditional Korean Medicine: A Foundation Model Approach to Culturally-Adapted Healthcare

D Jang, CE Kim - arXiv preprint arXiv:2303.17807, 2023

… In fact, the pre-training dataset for GPT-3, a model released in 2020 that ChatGPT is based on, is mainly derived from a modified version of Common Crawl (19) in which English-based data accounts for about 50%, while Korean data accounts for …


Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Task

Z Wu, L Zhang, C Cao, X Yu, H Dai, C Ma, Z Liu… - arXiv preprint arXiv …, 2023

… GPT-3 has a massive 175 billion parameters and was trained on a diverse collection of internet data, including Common Crawl3 and Wikipedia4. GPT-4 is the most powerful LLM to date, a successor of GPT-3. Although the technical details of …


Exploring the Why in AI: Investigating how Visual Question Answering models can be interpreted by post-hoc linguistic and visual explanations

With the increase in accuracy and usability of Artificial Intelligence (AI), especially deep neural networks, there has been a big demand for these networks. These methods are implemented in various domains to increase productivity, create new …


Exploring zero-shot and joint training cross-lingual strategies for aspect-based sentiment analysis based on contextualized multilingual language models

D Van Thin, H Quoc Ngo, D Ngoc Hao… - Journal of Information and …, 2023

Aspect-based sentiment analysis (ABSA) has attracted many researchers' attention in recent years. However, the lack of benchmark datasets for specific languages is a common challenge because of the prohibitive cost of manual annotation. The zero-shot …


Extended context for InstructGPT with LlamaIndex

… To train the extensive amount of parameters, a refined version of the Common Crawl dataset1 was used, consisting of almost a trillion … 1https://commoncrawl.org/the-data/ 2The modifications include a “modified initialization, pre-normalization, and …


Extended Hell (o): A Comprehensive Large-Scale Study on Email Confidentiality and Integrity Mechanisms in the Wild

The core specifications of electronic mail as used today date back as early as the 1970s. At that time, security did not play a significant role in developing communication protocols. These shortcomings still manifest themselves today in the …


Extracting Geographic Knowledge from Large Language Models: An Experiment

K Salmas, DA Pantazi, M Koubarakis - … on Knowledge Base Construction from Pre …, 2023

We perform a thorough analysis of how the inner architecture of large language models behaves whilst extracting geographic knowledge. Our aim is to conclude on whether models actually incorporate geospatial information or simply follow …


Extracting Intersectional Stereotypes from Static and Contextualized Embeddings

TES Charlesworth, K Ghate, A Caliskan, MR Banaji - 2023

… text corpora, ranging from static GloVe embeddings trained on 840 billion words from the Common Crawl to contextualized BERT embeddings trained on a combination of Wikipedia and Common Crawl text. For the simplest case of static …


Extracting Key-Value Pairs in Business Documents

E Thomas, DS Kafle, IS Mahamoud, A Joseph… - International Conference on …, 2023

Key-value extraction is a challenging task in document AI, particularly in business documents such as invoices. Accurately extracting key-value pairs from such documents is crucial for downstream tasks like accounting, analytics, and decision-making …


Extracting Mental Health Indicators From English and Spanish Social Media: A Machine Learning Approach

ME Villa-Pérez, LA Trejo, MB Moin, E Stroulia - IEEE Access, 2023

… For the English Twitter dataset, we used word vectors trained on Common Crawl from the … The Common Crawl corpus contains petabytes of data collected over 12 years of web … Available: https://commoncrawl.org/the-data/ [56] O. Loyola-González …


Extracting relations from texts using vector language models and a neural network classifier

M Shishaev, V Dikovitsky, V Pimeshkov, N Kuprikov… - PeerJ Computer Science, 2023

The article investigates the possibility of identifying the presence of SKOS (Simple Knowledge Organization System) relations between concepts represented by terms on the base of their vector representation in general natural language models …


Extracting representative subset from extensive text data for training pre-trained language models

J Suzuki, H Zen, H Kazawa - Information Processing & Management, 2023

This paper investigates the existence of a representative subset obtained from a large original dataset that can achieve the same performance level obtained using the entire dataset in the context of training neural language models. We employ the …


Extracting Training Data from GPT-2

… We collect tokens from the Common Crawl dataset and use them (5-10 tokens) as prefixes to prompt GPT-2 to generate samples. … Common Crawl dataset has all the data that has been crawled from the public internet. So if we prompt GPT-2 using …


Extraction of microservices from monolithic software based on the database model

D Mažeika, EK Kazlauskas - DAMSS 2022: 13th conference on data analysis …, 2022


Extraction of Parallel Sentences

S Sharoff, R Rapp, P Zweigenbaum - Building and Using Comparable Corpora for …, 2023

… billion parallel sentences in 90 languages from CommonCrawl snapshots, among which less than … This system, trained from parallel sentences collected from CommonCrawl as explained … sentence embeddings to extract parallel sentences …


Extractive Summarization of Telugu Text Using Modified Text Rank and Maximum Marginal Relevance

A Babu Gl, S Badugu - ACM Transactions on Asian and Low-Resource …, 2023

With the rapid growth of digital content, the need for the automatic text summarizer is arising to provide short text from the long text document. Many research works have been presented for extractive text summarization (ETS). This paper mainly focuses …


Förenkla nyhetssammanfattning med hjälp av AI: En analys av GPT-3 modellers förmåga och begränsningar

Everyday we are flooded with news from all around the world and this information can be overwhelming. In our study we analyze the possibilities to implement GPT-3 models in the work of news summarization in swedish and automize this process. In …


Face Recognition in the age of CLIP & Billion image datasets

A Bhat, S Jain - arXiv preprint arXiv:2301.07315, 2023

… LAION-5B was collected by parsing files in the Common Crawl dataset to find image tags with alt-text values. The corresponding images were downloaded and filtered using CLIP to keep only those images whose content resembled their alt-text …


Factors Affecting the Reliability of Information: The Case of ChatGPT

J Morato, JM Diaz-Nafria, S Sanchez-Cuadrado - International Conference on …, 2023

The abundance of current information makes it necessary to select the highest quality documents. For this purpose, it is necessary to deepen the knowledge of information quality systems. The different dimensions of quality are analyzed, and …


Fairness and Bias in Algorithmic Hiring

A Fabris, N Baranowska, MJ Dennis, P Hacker… - arXiv preprint arXiv …, 2023

… Bias in Bios, [60] is composed of textual biographies written in English and extracted from the Common Crawl dataset. It was initially proposed to study fairness in occupation classification. Gender is automatically extracted based on the use of …


Fairness and Fair Use in Generative AI

Although we are still a long way from the science fiction version of artificial general intelligence that thinks, feels, and refuses to “open the pod bay doors”, 1 recent advances in machine learning and artificial intelligence (“AI”) have captured the …


Fairness in Language Models Beyond English: Gaps and Challenges

K Ramesh, S Sitaram, M Choudhury - arXiv preprint arXiv:2302.12578, 2023

With language models becoming increasingly ubiquitous, it has become essential to address their inequitable treatment of diverse demographic groups and factors. Most research on evaluating and mitigating fairness harms has been concentrated on …


Fake News Detection via Deep Learning Approaches

M Li - 2023 4th International Symposium on Computer …, 2023

… RealNews: RealNews is a corpus of news articles whose data is taken from Common Crawl. The body and metadata in each news article is extracted by the Newspaper Python library. The training data uses news data from December 2016 to March 2019. …


Fake news detection: Taxonomy and comparative study

F Farhangian, RMO Cruz, GDC Cavalcanti - Information Fusion, 2023

The proliferation of social networks has presented a significant challenge in combating the pervasive issue of fake news within modern societies. Due to the large amount of information and news produced daily in text, audio, and video, the …


Faking It: Artificial Intelligence in a Human World


Fast and Energy-Efficient Inference for Attention-Based Natural Language Processing Models

Creating machines that can ``understand’’ our language and ``interact’’ with us as we interact with each other has been a dream that motivated many and captured the imaginations of even more. Attention-Based Transformer models have demonstrated …


Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

G Bao, Y Zhao, Z Teng, L Yang, Y Zhang - arXiv preprint arXiv:2310.05130, 2023

Large language models (LLMs) have shown the ability to produce fluent and cogent content, presenting both productivity opportunities and societal risks. To build trustworthy AI systems, it is imperative to distinguish between machine-generated …


Feature Learning in Infinite-Depth Neural Networks

G Yang, D Yu, C Zhu, S Hayou - NeurIPS 2023 Workshop on Mathematics of Modern …, 2023

… block is deeper (such as modern transformers), then we find fundamental limitations in all possible infinite-depth limits of such parametrizations, which we illustrate both theoretically and empirically on simple networks as well as Megatron …


Feature-Level Ensemble Learning for Robust Synthetic Text Detection with DeBERTaV3 and XLM-RoBERTa

SS Joy, TD Aishi - Proceedings of ALTA, 2023

As large language models, or LLMs, continue to advance in recent years, they require the development of a potent system to detect whether a text was created by a human or an LLM in order to prevent the unethical use of LLMs. To address this …


Federated Learning with Client-Exclusive Classes

J Zhang, X Zhang, X Zhang, D Hong, RK Gupta… - arXiv preprint arXiv …, 2023

Existing federated classification algorithms typically assume the local annotations at every client cover the same set of classes. In this paper, we aim to lift such an assumption and focus on a more general yet practical non-IID setting where every …


Few-Shot Learning for Identification of COVID-19 Symptoms Using Generative Pre-trained Transformer Language Models

K Jiang, M Zhu, GR Bernard - Machine Learning and Principles and Practice of …, 2023

Since the onset of the COVID-19 pandemic, social media users have shared their personal experiences related to the viral infection. Their posts contain rich information of symptoms that may provide useful hints to advancing the knowledge …


Few-shot Learning with Multilingual Generative Language Models

XV Lin, T Mihaylov, M Artetxe, T Wang, S Chen… - Proceedings of the 2022 …, 2022

… As such we also note the potential difference in genres between CommonCrawl and the genres used in GPT-3 comprising in addition to CommonCrawl, corpora such as BookCorpus and Wikipedia. Moreover, GPT-3 is trained on 118 languages …


Filtering and Extended Vocabulary based Translation for Low-resource Language pair of Sanskrit-Hindi

P Jha, R Kumar, V Sahula - ACM Transactions on Asian and Low-Resource …, 2023

… We used the reported Model 3 (Hindi to Sanskrit) and fed it with a small fraction of monolingual Hindi data (about 10k small sentences from CommonCrawl6 as used by [29]) to generate Sanskrit translations. We used a small fraction of the data to …


Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

F Radenovic, A Dubey, A Kadian, T Mihaylov… - arXiv preprint arXiv …, 2023

Vision-language models trained with contrastive learning on large-scale noisy data are becoming increasingly popular for zero-shot recognition problems. In this paper we improve the following three aspects of the contrastive pre-training pipeline …


FIMO: A Challenge Formal Dataset for Automated Theorem Proving

C Liu, J Shen, H Xin, Z Liu, Y Yuan, H Wang, W Ju… - arXiv preprint arXiv …, 2023

We present FIMO, an innovative dataset comprising formal mathematical problem statements sourced from the International Mathematical Olympiad (IMO) Shortlisted Problems. Designed to facilitate advanced automated theorem proving at the IMO …


Financial Machine Learning with Alternative Data

In the first chapter, we reconsider the idea of trend-based predictability using methods that flexibly learn price patterns that are most predictive of future returns, rather than testing hypothesized or pre-specified patterns (eg, momentum and …


Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here But Not Quite There Yet

K Tom, E Avramidis, R Bawden, O Bojar, A Dvorkovich… - WMT23-Eighth Conference …, 2023

This paper presents the results of the General Machine Translation Task organised as part of the 2023 Conference on Machine Translation (WMT). In the general MT task, participants were asked to build machine translation systems for any of 8 …


Findings of the CoCo4MT 2023 Shared Task on Corpus Construction for Machine Translation

A Ganesh, M Carpuat, W Chen, K Kann, C Lignos… - CoCo4MT 2023, 2023

This paper provides an overview of the first shared task on choosing beneficial instances for machine translation, conducted as part of the CoCo4MT 2023 Workshop at MTSummit. This shared task was motivated by the need to make the …


Findings of the Second WMT Shared Task on Sign Language Translation

M Müller, M Alikhani, E Avramidis, R Bowden, A Braffort…

This paper presents the results of the Second WMT Shared Task on Sign Language Translation (WMT-SLT23) 1. This shared task is concerned with automatic translation between signed and spoken 2 languages. The task is unusual in the …


Findings of the WMT 2023 shared task on parallel data curation

S Sloto, B Thompson, H Khayrallah, T Domhan… - Proceedings of the Eighth …, 2023

… We release processed Common Crawl data, along with various intermediate states from a strong baseline system, which we believe will … All of our inputs were derived from the 2023-06 snapshot of Common Crawl. We extracted the plain text …


Fine Tuning Transformer Models for Domain Specific Feature Extraction

The nature of Natural Language Processing has drastically changed in the past years. The implementation of Large Language Models pre-trained on thousands of unlabelled data has opened the door to a new layer of comprehension of text …


Fine-Grained Language Relatedness for Zero-Shot Silesian-English Translation

E Signoroni - RASLAN 2023 Recent Advances in Slavonic Natural …, 2023

… -cleaned version of the Common Crawl web dump containing about 750GB of English text. T5 uses a unified” text-to-text” format for all text-based NLP problems. Xue et al.(2021)[29] present mT5, a multilingual variant of T5 trained on a Common …


Fine-tuning MBART-50 with French and Farsi data to improve the translation of Farsi dislocations into English and French

B Namdarzadeh, S Mohseni, L Zhu, G Wisniewski… - Proceedings of Machine …, 2023

In this paper, we discuss the improvements brought by the fine-tuning of mBART50 for the translation of a specific Farsi dataset of dislocations. Given our BLEU scores, our evaluation is mostly qualitative: we assess the improvements of our fine-tuning …


FinGPT: Large Generative Models for a Small Language

R Luukkonen, V Komulainen, J Luoma, A Eskelinen… - arXiv preprint arXiv …, 2023

… Common Crawl and a targeted Internet crawl seeded by the .fi domain registry content and all URLs of Finnish material in Common Crawl. … CC-Fi To maximize coverage of Finnish text in Common Crawl resources, we applied a custom …


Finnish Internet Parsebank

J Luotolahti, J Kanerva, J Luoma, V Skantsi, S Pyysalo… - 2023

… The Common Crawl dataset includes both plain text and raw HTML files, at the time without … plain text from the Amazon cloud service that hosts Common Crawl As shown in Table 2, this … of all Finnish text pages we gathered from Common …


Focused Crawling for Automated IsiXhosa Corpus Building

C Marquard, H Suleman - Annual Conference of South African Institute of …, 2023

IsiXhosa is a low-resource language, which means that it does not have many large, high-quality corpora. This makes it difficult to perform many kinds of research with the language. This paper examines the use of focused Web crawling for automatic …


Form to meaning mapping and the impact of explicit morpheme combination in novel word processing

R Bonandrini, S Amenta, S Sulpizio, M Tettamanti… - Cognitive Psychology, 2023

In the present study, we leveraged computational methods to explore the extent to which, relative to direct access to semantics from orthographic cues, the additional appreciation of morphological cues is advantageous while inducing the meaning of …


Foundation Models and Fair Use

P Henderson, X Li, D Jurafsky, T Hashimoto… - arXiv preprint arXiv …, 2023

… 2021), rely on CommonCrawl data which is crawled only if users explicitly allow it through their robots.txt file. CommonCrawl is able to host a snapshot of the internet largely because of fair use arguments. As the organization’s director argues, there is …


Foundation models and the privatization of public knowledge

F Ferrari, J van Dijck, A van den Bosch - Nature Machine Intelligence, 2023

… This translates to approximately 300 billion words extracted from sources such as Wikipedia, CommonCrawl and GitHub. Overall, the training procedure involved the learning of 175 billion parameters. ChatGPT relies on GPT-4 as well as on a specific …


Foundation Models in Robotics: Applications, Challenges, and the Future

R Firoozi, J Tucker, S Tian, A Majumdar, J Sun, W Liu… - arXiv preprint arXiv …, 2023

… GPT-3 is trained on the Common Crawl dataset. Common Crawl contains petabytes of publicly available data over 12 years of web crawling and includes raw web page data, metadata, and text extracts. LLMs can also be multilingual. For …


Foundational Models Defining a New Era in Vision: A Survey and Outlook

M Awais, M Naseer, S Khan, RM Anwer, H Cholakkal… - arXiv preprint arXiv …, 2023

… common crawl. Schuhmann et al. [227] further scaled it up and released a multilingual, multi-modal dataset called LAION-5B which contains 5.8 billion data points curated from Common Crawl after filtering through existing CLIP model. Utilizing largescale …


FP8-LM: Training FP8 Large Language Models

H Peng, K Wu, Y Wei, G Zhao, Y Yang, Z Liu, Y Xiong… - arXiv preprint arXiv …, 2023

… 2022) across CommonCrawl snapshots to enhance data quality. Tab. 10 in Appendix A.3 provides details of our pre-training data, including … 2023) to distinguish documents similar to Wikipedia pages from randomly sampled …


From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models

M Suzuki, M Hirano, H Sakaji - arXiv preprint arXiv:2309.03412, 2023

Instruction tuning is essential for large language models (LLMs) to become interactive. While many instruction tuning datasets exist in English, there is a noticeable lack in other languages. Also, their effectiveness has not been well …


From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding

L Sun, F Luisier, K Batmanghelich, D Florencio… - arXiv preprint arXiv …, 2023

… Most generic language models are pre-trained on web-crawled text corpora including Wikipedia and Common Crawl. But in real world deployments, models are often used in a different domain, an issue referred to as domain shift. In order to …


From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy

M Gupta, CK Akiri, K Aryal, E Parker, L Praharaj - arXiv preprint arXiv:2307.00691, 2023

… Initially, GPT-1 was trained with the Common Crawl dataset, made up of web pages, and the BookCorpus dataset, which contained over … GPT-2: GPT-2 was trained on Common Crawl just like GPT-1 but combined that with WebText, which …


From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

J Coil, V Shwartz - arXiv preprint arXiv:2305.10568, 2023

Noun compound interpretation is the task of expressing a noun compound (eg chocolate bunny) in a free-text paraphrase that makes the relationship between the constituent nouns explicit (eg bunny-shaped chocolate). We propose modifications …


From GPT to AutoGPT: a Brief Attention in NLP Processing using DL

in this paper, we intend to present the evolution of the so called “Generative Pre-Training (GPT) models”. Starting from GPT-1 to the new one GPt-4 and AutoGPT, Generative Pre-Training (GPT) models are trained on unlabeled dataset (which are available in …


From Pre-training to Meta-Learning: A journey in Low-Resource-Language Representation Learning

D Zaikis, I Vlahavas - IEEE Access, 2023

Language representation learning is a vital field in Natural Language Processing (NLP) that aims to capture the intricate semantics and contextual information of text. With the advent of deep learning and neural network architectures, representation …


From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

S Feng, CY Park, Y Liu, Y Tsvetkov - arXiv preprint arXiv:2305.08283, 2023

Large language models (LMs) are pretrained on diverse data sources: news, discussion forums, books, online encyclopedias. A significant portion of this data includes facts and opinions which, on one hand, celebrate democracy and diversity …


From task to evaluation: an automatic text summarization review

L Lu, Y Liu, W Xu, H Li, G Sun - Artificial Intelligence Review, 2023

Automatic summarization is attracting increasing attention as one of the most promising research areas. This technology has been tried in various real-world applications in recent years and achieved a good response. However, the …